WO2023237105A1

WO2023237105A1 - Method for displaying virtual surgical instrument on surgeon console, and surgeon console

Info

Publication number: WO2023237105A1
Application number: PCT/CN2023/099487
Authority: WO
Inventors: 仇高延; 常新朝
Original assignee: 上海微创医疗机器人（集团）股份有限公司
Priority date: 2022-06-10
Filing date: 2023-06-09
Publication date: 2023-12-14
Also published as: CN115068114A

Abstract

The present specification relates to the technical field of endoscopic surgeries. Particularly disclosed is a method for displaying a virtual surgical instrument on a surgeon console, and the surgeon console. The method comprises: acquiring an image of a main control arm of the surgeon console; determining, according to the image of the main control arm, attitude data of the main control arm; and generating, on the basis of the attitude data of the main control arm and a three-dimensional model of a surgical instrument, virtual attitude data of the surgical instrument so as to display a virtual instruction instrument image in a three-dimensional monitor of the surgeon console. According to the scheme above, the virtual instruction instrument image is displayed in the three-dimensional monitor, so that a surgeon can conveniently discover misoperation in time, find instrument failures in time, and is assisted in the operation, thereby improving the surgical efficiency, improving the surgical quality and success rate, and improving the surgical experience of surgeons and patients.

Description

Method for displaying virtual surgical instruments on doctor's console and doctor's console

This application requires the priority of a Chinese patent application with a filing date of June 10, 2022, an application number of 202210652288.5, and an application name of "Method for displaying virtual surgical instruments on a doctor's console and a doctor's console", and the entire content thereof incorporated herein by reference.

Technical field

This specification relates to the technical field of endoscopic surgery, and in particular to a method for displaying virtual surgical instruments on a doctor's console and a doctor's console.

Background technique

In the existing endoscopic medical robot surgery, the doctor uses the main control arm and foot pedal in the doctor's console to control the surgical instruments in the endoscopic surgery environment, which requires the doctor to switch between hand movements and foot pedal movements. Cooperate. Images of surgical instruments in an endoscopic surgical environment are displayed on a stereoscopic monitor in the doctor's console. Regarding the doctor's footwork and hand movements, doctors can only rely on training or muscle memory from past experience when operating. It is easy to misoperate, affecting the quality and success rate of the operation.

In response to the above problems, no effective solution has been proposed yet.

Contents of the invention

Embodiments of this specification provide a method for displaying virtual surgical instruments on a doctor's console and a doctor's console to solve the problem in the prior art of easy misoperations when performing surgery only by relying on endoscopic images on a stereoscopic monitor. .

Embodiments of this specification provide a method for displaying virtual surgical instruments on a doctor's console, including: acquiring an image of a main control arm of the doctor's console; and determining, based on the image of the main control arm, the main control arm. Posture data; based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, generate virtual posture data of the surgical instrument to display a virtual command instrument image in the stereoscopic monitor of the doctor's console.

Embodiments of this specification also provide a doctor's console, including: an image acquisition device, used to collect images of the main control arm of the doctor's console; and an image processing device, used to determine the image of the main control arm based on the image of the main control arm. posture data of the main control arm; also used to generate virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument; and a stereoscopic monitor for displaying virtual posture data according to the virtual posture data. Command device image.

Embodiments of this specification also provide a medical device, including a processor and a memory used to store instructions executable by the processor. When the processor executes the instructions, it implements the method described in any of the above embodiments for performing the steps on the doctor console. Steps of a method of displaying virtual surgical instruments.

Embodiments of this specification also provide a computer-readable storage medium on which computer instructions are stored. When the instructions are executed, the steps of the method for displaying virtual surgical instruments on a doctor's console described in any of the above embodiments are implemented. .

In the embodiment of this specification, a method for displaying virtual surgical instruments on a doctor's console is provided. An image of the main control arm of the doctor's console can be obtained, and the main control arm can be determined based on the obtained image of the main control arm. Arm posture data, based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, generate virtual posture data of the surgical instrument to display a virtual instruction instrument image on the stereoscopic monitor of the doctor's console. In the above scheme, the attitude data of the main control arm is first determined based on the image of the main control arm. Since there is a master-slave relationship between the main control arm and the surgical instruments, the attitude data of the main control arm and the three-dimensional model of the surgical instruments can be used. Generate virtual posture data of the surgical instrument, and then display a virtual instruction instrument image on the stereoscopic monitor based on the virtual posture data of the surgical instrument. By displaying virtual command instrument images on a stereoscopic monitor, it is possible for doctors to detect misoperations and instrument failures in a timely manner, assist doctors in operation, improve surgical efficiency, improve surgical quality and success rate, and improve the surgical experience of doctors and patients.

Description of the drawings

The drawings described here are used to provide a further understanding of this specification, constitute a part of this specification, and do not constitute a limitation of this specification. In the attached picture:

Figure 1 shows a schematic structural diagram of an endoscopic surgery system according to an embodiment of this specification;

Figure 2 shows a schematic diagram of a doctor's console in an embodiment of this specification;

Figure 3 shows a schematic diagram of the image acquisition device in the embodiment of this specification;

Figure 4 shows a schematic diagram of a patient surgical platform in an embodiment of the present specification;

Figure 5 shows a schematic structural diagram of the image platform in the embodiment of this specification;

Figure 6 shows a flow chart of a method for displaying virtual surgical instruments on a doctor's console in an embodiment of the present specification;

Figure 7 shows a structural block diagram of the doctor's console in the embodiment of this specification;

Figure 8 shows the enhanced display control logic block diagram in the embodiment of this specification;

Figure 9 shows a flow chart of virtual command instrument image display in the embodiment of this specification;

Figure 10 shows a flow chart of the association between the main hand posture and the virtual instrument posture in the embodiment of this specification;

Figure 11 shows a schematic diagram of the camera coordinate system and the world coordinate system in the embodiment of this specification;

Figure 12 shows a schematic diagram of the transformation of the world coordinate system and the camera coordinate system in the embodiment of this specification;

Figure 13 shows a schematic diagram of the virtual foot position and foot action image display in the embodiment of this specification;

Figure 14 shows a schematic diagram of the prompt for excessive position following deviation of the surgical instrument in the embodiment of this specification;

Figure 15 shows a schematic diagram of the position display of the foot pedal panel in the embodiment of this specification;

Figure 16 shows a schematic diagram of foot action display and effective pedaling display in the embodiment of this specification;

Figure 17 shows a schematic diagram of the operation flow of the endoscopic surgical robot in the embodiment of this specification;

Figure 18 shows a schematic flow chart of virtual instruction device acquisition and display in the embodiment of this specification;

Figure 19 shows a schematic flow chart of virtual foot position and foot movement display in the embodiment of this specification;

Figure 20 shows a schematic structural diagram of the medical equipment in the embodiment of this specification.

Detailed ways

The principles and spirit of the present specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present specification, and are not intended to limit the scope of the present specification in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art know that the embodiments of this specification can be implemented as a system, device, method or computer program product. Therefore, the disclosure in this specification can be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

Embodiments of this specification provide a method for displaying virtual surgical instruments on a doctor's console and a doctor's console, which can be applied to endoscopic surgery systems. Figure 1 shows a schematic structural diagram of the endoscopic surgery system. As shown in Figure 1, the endoscopic surgery system can be It consists of a doctor's console, a patient surgical platform (including instruments and endoscopic instruments), and an image platform. Among them, the doctor can operate at the doctor's console and provide motion control instructions to the instruments on the patient's surgical platform. The endoscopic images are processed by the image platform and displayed on the image platform and the three-dimensional monitor on the doctor's console respectively.

Please refer to FIG. 2 , which shows a schematic diagram of a doctor's console in an embodiment of this specification. The doctor's console is the control center of the endoscopic surgery system, which can provide doctors with clear images and provide control signal input necessary for surgical operations. The main parts of the doctor's console may include: a stereoscopic monitor 201, a main operating hand (or main control arm) 202, a foot panel 203, and an image acquisition device (not shown in Figure 2).

The stereoscopic monitor can present the same image or video information through the left and right screens respectively. The left eye observes the left screen and the right eye observes the right screen. The brain automatically synthesizes images into stereoscopic vision. The main functions of the stereoscopic monitor may include: displaying endoscopic visual images, displaying virtual command instrument images, and displaying virtual foot pedal positions and foot movements.

The foot pedal panel can be an auxiliary operation function, operating in conjunction with the main operating hand/main control arm control to provide control signals to surgical instruments. The main functional parts of the foot panel may include: operating surgical instruments, operating endoscopes, and other auxiliary functions.

The doctor's console can be equipped with two main operating hands/main control arms, which receive operating signals from the operator's left and right hands respectively. The main control arm can monitor the operator's hand movement information and is the main motion control input of the entire system. The operator can control the movement of the tool arm by operating the control handle at the end of the main control arm, thereby controlling the operation of the endoscope and surgical instruments.

Figure 3 shows a schematic diagram of the image acquisition device in the embodiment of this specification. The image acquisition device may include one or more binocular cameras. Binocular cameras can use the difference in pixel positions of feature points in images taken by two lenses to obtain object depth information. Using VR's 3D technology, the position and posture of objects can be obtained. The image acquisition device mainly monitors the position of the operator's hand, foot panel and foot movements.

In some embodiments of this specification, the image collection device may include a binocular camera and an infrared rangefinder. By setting up an additional infrared rangefinder, it is easier to obtain the depth information of the object and then obtain the attitude data of the main control arm.

In some embodiments of the present specification, the image capture device may include an RGB-D (RGB-Depth) camera. The depth map collected by the RGB-D camera contains an image channel with information related to the surface distance of the viewpoint scene object. The channel itself is similar to a grayscale image. Each pixel value is the actual distance from the object measured by the sensor, and the object depth information and position information, and then obtain the attitude data of the main control arm.

In some embodiments of this specification, the image collection device may include multiple monocular cameras. By setting up multiple monocular cameras. The multiple monocular cameras may be two monocular cameras arranged vertically. The depth information of the object can be obtained by processing the collected images, and then the attitude data of the main control arm can be obtained.

Referring to Figure 4, a schematic diagram of the patient surgical platform is shown. The patient surgical platform is the operating platform of the endoscopic surgery system located next to the patient's operating bed. It consists of three single machines: a surgical trolley, an adjustment arm, and a tool arm (including an image arm). The instruments and endoscopes installed on the tool arm The mirror serves as a slave mechanism and accepts motion control commands from the doctor's console.

Please refer to Figure 5, which shows a schematic structural diagram of the image platform. The image platform is the visual feedback subsystem of the endoscopic surgery system, which can include three stand-alone machines: a three-dimensional electronic endoscope, an endoscope image processing host, and an imaging car. It can provide energy, endoscopic vision processing, image display and other functions for the instrument. The image of the synchronous doctor's stereo monitor is mainly for viewing by personnel other than the operating doctor.

Figure 6 shows a flowchart of a method for displaying virtual surgical instruments on a doctor's console in an embodiment of this specification. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or less operation steps or module units may be included in the method or device based on routine or without creative effort. . In the steps or structures where there is no necessary logical causal relationship, the execution order of these steps or the module structure of the device is not limited to the execution described in the embodiments of this specification and shown in the drawings. Line order or module structure. When the described methods or module structures are applied to actual devices or terminal products, they can be executed sequentially or in parallel (such as parallel processors or multi-thread processing) according to the methods or module structures shown in the embodiments or drawings. environment, even a distributed processing environment).

Specifically, as shown in Figure 6, a method for displaying virtual surgical instruments on a doctor's console provided by an embodiment of this specification may include the following steps:

Step S601: Obtain an image of the main control arm of the doctor's console.

The methods in the embodiments of this specification can be applied to image processing devices. The image processing device may acquire an image of the main control arm of the doctor's console. Wherein, the main control arm may include a left main control arm and a right main control arm.

In one embodiment, images of the main control arm during endoscopic surgery may be acquired by an image acquisition device. The image processing device may acquire the image of the main control arm from the image acquisition device. The image of the main control arm may include an image of the left main control arm and an image of the right main control arm at at least one moment in the endoscopic surgical environment.

In one embodiment, the image acquisition device can acquire images of the main control arm during endoscopic surgery in real time. The image processing device may acquire images of the main control arm from the image acquisition device every preset time period.

Step S602: Determine the posture data of the main control arm based on the image of the main control arm.

After obtaining the image of the main control arm, the image processing device may determine the attitude data of the main control arm based on the image of the main control arm. The attitude data of the main control arm may include attitude data of multiple feature points on the main control arm. The attitude of the main control arm can be characterized by the attitude data of multiple feature points. The posture data of each feature point may include at least one of the following: position coordinates, displacement direction, displacement amount, rotation angle, rotation direction and other various data. The image processing device can determine the posture data of multiple feature points on the main control arm based on the images of the main control arm at two adjacent times, and then determine the posture data of the main control arm.

Step S603: Generate virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument to display a virtual instruction instrument image on the stereoscopic monitor of the doctor's console.

Since there is a master-slave relationship between the main control arm and the surgical instrument, the posture of the surgical instrument can be adjusted through the operation of the main control arm. Therefore, the posture data of the surgical instrument can be generated based on the posture data of the main control arm and the three-dimensional model of the surgical instrument. Virtual posture data. The virtual posture data of the surgical instrument may include posture data of the surgical instrument corresponding to the posture data of the main control arm.

In one embodiment, the correspondence between the attitude data of the main control arm and the attitude data of the surgical instrument may be stored in the image processing device. The posture data of the surgical instrument may also include posture data of multiple instrument feature points. The posture data of each instrument feature point may include at least one of the following: position coordinates, displacement direction, displacement amount, rotation angle, rotation direction and other various data.

After determining the posture data of the main control arm, the image processing device can generate virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument. The three-dimensional model of the surgical instrument may be pre-established model data of the surgical instrument in a three-dimensional coordinate system. The virtual posture data of the surgical instrument refers to the posture data of the surgical instrument under the operation of the main control arm.

After the virtual posture data of the surgical instrument is generated, the virtual instruction instrument image can be displayed on the stereoscopic monitor of the doctor's console according to the virtual posture data of the surgical instrument. In one embodiment, the virtual instruction instrument image and the actual instrument image collected by the endoscope can be displayed simultaneously on the stereoscopic monitor.

In the above embodiment, the attitude data of the main control arm can be determined first based on the image of the main control arm. Since there is a master-slave relationship between the main control arm and the surgical instrument, the attitude data of the main control arm and the three-dimensional model of the surgical instrument can be used. Generate virtual posture data of the surgical instrument, and then display a virtual instruction instrument image on the stereoscopic monitor based on the virtual posture data of the surgical instrument. By displaying virtual command instrument images on a stereoscopic monitor, it is easier for doctors to detect misoperations and instrument failures in a timely manner, thereby reducing misoperations, improving surgical quality and success rate, and improving the surgical experience of doctors and patients.

In some embodiments of this specification, the method may further include: acquiring a foot image of the doctor's console and/or an image of the operator's foot; generating foot position data based on the foot image, and/or generating foot position data based on the foot image. The foot image generates foot position data to display the footrest area and/or foot area in the stereoscopic monitor.

Considering that the operator cannot see the foot pedals and foot movements during the operation and can only perform foot pedal operations based on experience, the image acquisition device can collect the foot pedal images of the doctor's console and the operator's foot pedal operations in the pedal area. Foot image inside. The image processing device can acquire the foot images and foot images collected by the image acquisition device.

Afterwards, the image processing device may generate foot pedal position data based on the foot pedal image. The footrest position data may include position data and size data for each of a plurality of footrests in the physician console. The image processing device can display the footrest area on the stereoscopic monitor based on the footrest position data. The footrest area may include multiple footrest sub-areas, and each footrest sub-area corresponds to one footrest.

The image processing device may also generate foot position data based on the foot image. The foot position data may include position data and size data of the operator's left and/or right foot. Image processing allows the foot area to be displayed on a stereoscopic monitor based on foot position data. The foot area may include a left foot area corresponding to the left foot and a right foot area corresponding to the right foot.

In some embodiments of the present description, the method may further include: when one or more foot pedals of the doctor's console are effectively stepped on, on the stereoscopic monitor, the steps on the foot pedal area that are effectively stepped on are displayed. One or more pedals for highlighting.

In order to facilitate the operator to know in time whether he or she is stepping on a certain pedal, when one or more pedals of the doctor's console are effectively stepped on, one or more of the pedals in the pedal area that are effectively stepped on can be displayed on the stereo monitor. pedal to highlight.

In one embodiment, a pressure sensor is provided below each of the plurality of foot pedals of the doctor's console. When the pressure detected by the pressure sensor is greater than a preset value, the corresponding foot pedal is effectively stepped on. The pressure sensor corresponding to each pedal can be connected to an image processing device. The image processing device can determine whether the pressure value detected by each pressure sensor is greater than a preset pressure threshold, and assign the pedal corresponding to the pressure sensor with a pressure value greater than the preset pressure threshold. It is judged to be effectively stepped on. Based on the determination result, the image processing device highlights one or more pedals that are effectively stepped on in the pedal area on the stereoscopic monitor. Through the above method, the operator can be informed in time whether the foot pedal has been stepped on effectively or mistakenly, which can improve the quality and success rate of the operation and improve the operator's operating experience.

In some embodiments of this specification, the image of the main control arm may include a left-eye image and a right-eye image of the main control arm collected by a binocular camera.

In this embodiment, the image acquisition device may include a binocular camera. Correspondingly, the captured image of the main control arm may include the left eye image and the right eye image of the main control arm captured by the binocular camera. In this embodiment, by setting the image acquisition device as a binocular camera, the image processing device can obtain depth information based on the left eye image and the right eye image, so that the instrument can be displayed stereoscopically.

In some embodiments of this specification, determining the posture data of the main control arm based on the image of the main control arm may include: obtaining the spatial position data of the left-eye camera and the right-eye camera in the binocular camera; based on the The spatial position data and the position parameters of each of the plurality of characteristic points of the main control arm in the left-eye image and the right-eye image are used to determine the spatial position data corresponding to each feature point; according to the plurality of The spatial position data corresponding to each feature point at time determines the attitude data of the main control arm.

In this embodiment, the main control arm can be regarded as a rigid body. The position or attitude of three points that are not on a straight line can determine the position or attitude of the rigid body. Therefore, the attitude of the main control arm can be determined through the attitude of at least three characteristic points on the main control arm. The spatial position data of the orbital left-eye camera and the right-eye camera in the binocular camera can be obtained. Afterwards, based on the spatial position data of the left-eye camera and the right-eye camera and the position parameters of at least three feature points on the main control arm, the spatial position data corresponding to each feature point can be determined, that is, the spatial position of each feature point in the geographical coordinate system. Location data. After obtaining the spatial position data corresponding to each feature point, the attitude data of the main control arm can be determined based on the spatial position data corresponding to each feature point. Through the above method, it can be based on The acquired image of the main control arm determines the attitude data of the main control arm.

In some embodiments of this specification, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument may include: generating the virtual posture data of the surgical instrument based on the posture data of the main control arm. The control instruction information of the main control arm is applied to the three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

Specifically, the image processing device may generate the control instruction information of the main control arm based on the posture data of the main control arm. The control instruction information may include an operation instruction for the robotic arm, and the operation instruction may include but is not limited to at least one of the following: a specified translation distance, a specified rotation angle, a rotation direction, a rotation axis, a rotation position, a specified opening angle, etc. In one embodiment, the image processing device may store a correspondence between the attitude data of the control arm and the control instruction or a conversion formula between the two. Based on the stored correspondence or conversion formula, the main control arm can be Attitude data generates control command information for the main control arm. After obtaining the control instruction information of the main control arm, the control instruction corresponding to the control instruction information is applied to the three-dimensional model of the surgical instrument, thereby obtaining the virtual posture data of the surgical instrument. For example, the three-dimensional model of the surgical instrument can be operated according to the operating instructions in the control instruction to obtain the virtual posture data of the surgical instrument. In the above manner, virtual posture data of the surgical instrument can be generated based on the posture data of the control arm and the three-dimensional model of the surgical instrument.

In some embodiments of this specification, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument may include: obtaining a target machine learning model; the target machine learning model is Pre-established based on the three-dimensional model of the surgical instrument; input the posture data of the main control arm into the target machine learning model to obtain the virtual posture data of the surgical instrument.

Specifically, after obtaining the posture data of the control arm and the three-dimensional model of the surgical instrument, a preset control algorithm can be used to generate virtual posture data of the surgical instrument. In one embodiment, the preset control algorithm may include multiple operators in the machine learning model. After inputting the attitude data of the main control arm into the machine learning model, the virtual attitude data of the surgical instrument can be output. The machine learning model can be pre-built based on a three-dimensional model of the surgical instrument.

In some embodiments of this specification, before obtaining the target machine learning model, it may also include: constructing a machine learning model based on the three-dimensional model of the surgical instrument; obtaining the posture data of the control arm during endoscopic surgery and the corresponding The posture data of the surgical instruments, and construct a training sample set based on the posture data of the control arm during the endoscopic surgery and the posture data of the corresponding surgical instruments; use the constructed training sample set to train the machine The learning model is trained to obtain the target machine learning model.

A machine learning model can be built based on the three-dimensional model of the surgical instrument. During endoscopic surgery, the posture data of the control arm and the posture data of the corresponding surgical instruments can be obtained. Based on the obtained posture data of the control arm during endoscopic surgery and the posture data of the corresponding surgical instruments, a training sample set is constructed. Among them, the attitude data of the control arm is the input data, and the attitude data of the surgical instrument is the label. The pre-established machine learning model is trained using the constructed training sample set to obtain a target machine learning model. The target machine learning model can determine the posture data of the surgical instrument based on the posture data of the control arm for a specific surgical instrument. Through the above method, the virtual posture data of the surgical instrument can be generated directly based on the posture data of the control arm and the three-dimensional model of the surgical instrument.

In some embodiments of this specification, the actual instrument image of the surgical instrument collected by the endoscope is also displayed on the stereoscopic monitor; after the virtual instruction instrument image is displayed on the stereoscopic monitor of the doctor's console, it may also include: : Comparing the actual instrument image with the virtual instruction instrument image; determining, based on the comparison result, the target portion of the virtual instruction instrument image that has a difference between the virtual instruction instrument image and the actual instrument image exceeding a preset range; The stereoscopic monitor is also used to annotate and display the target portion.

In this embodiment, the actual instrument image of the surgical instrument collected by the endoscope and the virtual instruction instrument image determined based on the posture data of the main control arm can be displayed simultaneously on the stereoscopic monitor. The image processing device can compare the actual instrument image with the virtual command instrument image. In one embodiment, the image processing device can divide the surgical instrument into multiple parts, and divide the parts in the virtual instruction instrument image into Each of the multiple parts is compared with the corresponding part in the actual device image. According to the comparison results, the target portion of the virtual instruction instrument image whose difference between the actual instrument image and the actual instrument image exceeds a preset range can be determined. The determined target part can be marked and displayed on the stereo monitor. For example, if a certain part in the virtual instruction instrument image is translated by a distance greater than a preset distance relative to the corresponding part in the actual instrument image, then the part is determined to be the target part. For another example, if the rotation angle of a certain part in the virtual instruction instrument image about a fixed axis relative to the corresponding part in the actual instrument image is greater than the preset angle, then the part is determined as the target part. For another example, if the rotation angle of a certain part in the virtual instruction instrument image around a fixed point relative to the corresponding part in the actual instrument image is greater than the preset angle, then the part is determined as the target part.

After the target part is determined, the image processing device may control the label display of the target part in the stereoscopic monitor. The annotation display may display the target part in different colors, for example, it may be highlighted in red, highlighted in yellow, etc. For another example, the annotation display may be a flashing display of the target part. In one embodiment, the actual instrument image and the virtual instruction instrument image can be displayed in the same coordinate system and at the same scale, which facilitates the operator to observe the target part in time, reduces misoperation or detects instrument failure in time, and improves the quality of surgery and the success rate of surgery. .

In some embodiments of this specification, after determining the target portion of the virtual instruction instrument image that differs from the actual instrument image by exceeding a preset range, the method may further include: determining that the target portion is in the surgical operation. Position data in the instrument; generate early warning information based on the position data, and provide early warning to the operator in a preset manner based on the early warning information.

Specifically, the image processing device divides the surgical instrument into multiple parts and determines the target part. The position of the target part in the surgical instrument can be obtained according to the division rules. The position data can include the position of the target part in the surgical instrument and its location. The specific component names and other data described above. After the location data of the surgical instrument is determined, early warning information can be generated based on the location data. An early warning can be issued to the operator in a preset manner based on the early warning information. For example, it can be displayed in a stereoscopic monitor in the form of voice broadcast or text.

Each embodiment in this specification is described in a progressive manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. For details, reference may be made to the foregoing descriptions of related processing embodiments, which will not be described in detail here.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain implementations.

Based on the same inventive concept, the embodiments of this specification also provide a doctor console, as described in the following embodiments. Since the problem-solving principle of the doctor's console is similar to the method for displaying virtual surgical instruments on the doctor's console, the implementation of the doctor's console can be referred to the implementation of the method for displaying virtual surgical instruments on the doctor's console, and there will be no duplication. Again. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements predetermined functions. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated. Figure 7 is a structural block diagram of the doctor's console in the embodiment of this specification. As shown in Figure 7, it includes: an image acquisition device 701, an image processing device 702 and a stereoscopic monitor 703. This structure will be described below.

The image acquisition device 701 is used to acquire images of the main control arm of the doctor's console.

The image processing device 702 is configured to determine the attitude data of the main control arm according to the image of the main control arm; and is also used to generate the posture data of the main control arm and the three-dimensional model of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument. Virtual posture data.

The stereoscopic monitor 703 is used to display a virtual instruction instrument image according to the virtual posture data.

In some embodiments of this specification, the image acquisition device is also used to collect the foot image of the doctor's console and the operator's foot image; the image processing device is also used to generate the foot position based on the foot image. data, and/or generate foot position data according to the foot image; the stereoscopic monitor is also used to display the foot area according to the foot position data, and/or display the foot area according to the foot position data. area.

In some embodiments of this specification, the stereoscopic monitor is also used to highlight one or more pedals that are effectively stepped on in the pedal area.

In some embodiments of this specification, the image acquisition device includes a binocular camera, and the binocular camera is used to collect left-eye images and right-eye images of the main control arm.

In some embodiments of this specification, the image processing device is specifically configured to: obtain the spatial position data of the left-eye camera and the right-eye camera in the binocular camera; and obtain multiple spatial position data based on the spatial position data and the main control arm. The position parameters of each feature point in the left-eye image and the right-eye image determine the spatial position data corresponding to each feature point; based on the spatial position data corresponding to each feature point at multiple times, Determine the attitude data of the main control arm.

In some embodiments of this specification, the image processing device is specifically configured to: generate control instruction information of the main control arm based on the posture data of the main control arm; apply the control instruction information of the main control arm to The three-dimensional model of the surgical instrument generates virtual posture data of the surgical instrument.

In some embodiments of this specification, the image processing device is specifically configured to: use a preset control algorithm to generate virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument. .

In some embodiments of this specification, the stereoscopic monitor may also display actual instrument images of surgical instruments collected by the endoscope; the image processing device may also be used to: compare the actual instrument images with the virtual Compare the command instrument images; according to the comparison results, determine the target portion of the virtual command instrument image that differs from the actual instrument image by exceeding a preset range; the stereoscopic monitor is also used to monitor the target portion Display labels.

The above method and the doctor's console will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating this specification and does not constitute an improper limitation of this specification.

In this specific embodiment, an augmented reality doctor console is provided. As shown in Figure 1, the doctor's console may include a stereoscopic monitor, a main control arm and a foot panel, and an image acquisition device.

Enhanced display mainly realizes the display of endoscopic visual images, virtual command instrument images, foot position images, and foot movement images in a stereoscopic monitor together. This real-time fusion display can provide prompts to the operator and reduce misoperations. Please refer to Figure 8, which shows an enhanced display control logic block diagram. As shown in Figure 8, augmented reality can include: virtual instruction device image display, virtual foot position and foot action image display.

Please refer to FIG. 9 , which shows a flow chart of virtual command instrument image display. As shown in Figure 9, through the image acquisition device, preferably binocular vision, the depth information is used to obtain the main hand operating posture, and then fused with the main hand control instructions to obtain the virtual command instrument posture. The display of the virtual instruction device image has a switch on the operating platform that can control its opening or closing. When an excessive deviation fault occurs, a red prompt will be displayed on the part where the instrument deviation is excessive.

Please refer to Figure 10, which shows a flow chart of the association between the main hand posture and the virtual instrument posture. As shown in Figure 10, after the start, binocular vision is initiated. The positions of multiple target feature points on the main control arm within the field of view are obtained through the left eye camera (camera 1). The positions of multiple target feature points on the main control arm within the field of view are obtained through the right eye camera (camera 2). Based on the spatial positions of the two cameras and the positions of each feature point in the two fields of view, the spatial position of each target feature point is calculated. Through continuous image processing and position Analyze and eliminate the continuous motion trajectories of each target feature point. The continuous motion trajectory of each feature point is associated with the posture of the 3D model of the virtual instrument to obtain the posture data of the virtual instrument.

Please refer to Figure 11, which shows a schematic diagram of the camera coordinate system and the world coordinate system. As shown in Figure 11: XcYcZc is the camera coordinate system; XwYwZw is the world coordinate system; XY is the imaging plane coordinate system. According to the coordinate information of the target feature point on the main control arm imaged in two directions, the position of the feature point on the three-dimensional coordinate system is deduced. Continuous image and position calculation can obtain the motion displacement/trajectory of feature points. Then it is associated with the three-dimensional model of the virtual equipment to obtain the animation effect of the virtual equipment model following the movement.

Please refer to Figure 12, which shows a schematic diagram of the transformation of the world coordinate system and the camera coordinate system. The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation, that is, the object will not deform and only needs to be rotated and translated. R represents the rotation matrix. S represents the translation vector. The transformation formula is as follows:

Among them, (x, y, z) are the coordinates in the camera coordinate system, (x _w , y _w , z _w ) are the coordinates in the world coordinate system, r _ij (i, j = 1, 2, 3) is the rotation The elements in the matrix R, s _k (k=x, y, z) are the elements in the translation vector S, and T is the matrix transpose.

Please refer to Figure 13, which shows a schematic diagram of virtual foot position and foot action image display. As shown in Figure 13, through the image acquisition device, preferably binocular vision, the depth information is used to obtain foot position and foot movement information. The virtual footrest position is displayed on the stereo monitor to guide the doctor when switching footrests. The virtual foot movements are displayed on the stereo monitor. When the doctor moves the foot and switches the pedal, there is real-time visual feedback to prevent misstepping. The calculation method for collection and display: take the center of the lower left pedal as the origin of the X, Y, and Z coordinate systems, and obtain the real-time displacement of the foot in the XYZ direction through the depth camera. The displacement is displayed in a specified area in the stereoscopic monitor according to a certain proportion.

Please refer to Figure 14, which shows a schematic diagram of a prompt for excessive position following deviation of a surgical instrument. During the operation, when there is an excessive deviation between the posture of the commanded given instrument and the posture of the actual instrument, a fault will be reported, and there will be corresponding fault location indication and text prompts in the stereo monitor. As shown in Figure 14, parts with excessive deviations can be displayed in red (marked in dark gray in the figure). This enhanced display can intuitively and effectively point out the fault point and provide prompts such as fault solutions.

Please refer to Figure 15, which shows a schematic diagram of the foot panel position display. As shown in Figure 15, during the operation process, the foot pedal position and foot movements are displayed in real time. The purpose is to give the operator visual feedback during the switching operation and reduce the probability of operation errors. There are both effective pedal position prompts and a real-time display of the current foot position, which can correct each other to prevent errors. During the operation, the foot movements are displayed in real time. When the operator wants to switch the pedal, the real-time display image of the foot can give the doctor feedback on whether the movement direction of the pedal reached is correct. When the foot pedal is depressed, the color highlights to determine whether the pedal function is effectively depressed. The above two points can assist the doctor in the smooth operation of switching pedals. Please refer to Figure 16, which shows a schematic diagram of foot action display and effective pedaling display. As shown in Figure 16, the pedal where the foot is is effectively stepped on.

Please refer to Figure 17, which shows a schematic diagram of the operation flow of the endoscopic surgical robot. As shown in Figure 17, the main process includes: Operation: On the doctor's platform, the main hand and foot pedal operations are coordinated to give control instructions to the instruments on the surgical platform; Monitoring: On the doctor's platform, the main hand movements and foot pedal movements are obtained through the image acquisition device Information, virtual command equipment posture, this posture and the actual control instructions are mutually corrected. Alarm when the deviation is too large; display: the endoscope image is displayed on the stereo monitor, and the virtual command instrument, pedal position and foot movement are also displayed on the monitor in real time as real-time feedback information for the doctor's operation.

Please refer to Figure 18, which shows a schematic flow chart of virtual instruction device acquisition and display. As shown in Figure 18, the main process includes: displaying the virtual instruction device based on image acquisition on the stereo monitor; when an excessive deviation fault occurs, the fault location is displayed with color highlighting and flashing prompts.

Please refer to Figure 19, which shows a schematic flow chart of virtual pedal position and foot movement display. As shown in Figure 19, the main process includes: based on the image acquisition device, obtain the foot position image and display it in the stereo monitor, and highlight the effective pedal to confirm; based on the image acquisition device, obtain the real-time action image of the foot, and display it on the stereo monitor In-monitor display gives visual feedback on target position movement.

In the above solution, an image acquisition module is installed on the doctor's console to monitor the main hand and foot pedal parts of the doctor's console. Through the above image collection information, the action posture of the main hand and the action posture of the feet are recognized. Then, based on the control command information of the device from the control algorithm, VR display technology is used to virtualize and display the posture of the command device in the doctor's stereoscopic monitor. Based on the above image acquisition device, VR technology is used to virtually display the foot position and foot movements on the stereoscopic monitor to assist the doctor's operation. By adding an image acquisition device to monitor the operator's posture and foot movement position, VR technology is used to display and operate prompts on the doctor's stereoscopic monitor in real time to facilitate the doctor's operation, such as: placing the foot on the doctor's stereoscopic monitor. The pedal position and foot movement are displayed in real time. The doctor can clearly see the foot movement process and related prompts such as which foot needs to move to which pedal position next. It can avoid misoperation and improve the efficiency of surgery, and the prompts during the operation can assist the doctor in the operation.

From the above description, it can be seen that the embodiments of this specification achieve the following technical effects: first determine the attitude data of the main control arm according to the image of the main control arm. Since there is a master-slave relationship between the main control arm and the surgical instrument, it is possible to Based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, virtual posture data of the surgical instrument is generated, and then a virtual instruction instrument image is displayed on the stereoscopic monitor based on the virtual posture data of the surgical instrument. By displaying virtual command instrument images on a stereoscopic monitor, it is possible for doctors to detect misoperations and instrument failures in a timely manner, assist doctors in operation, improve surgical efficiency, improve surgical quality and success rate, and improve the surgical experience of doctors and patients.

The embodiment of this specification also provides a medical device. For details, please refer to the schematic structural diagram of the medical device based on the method for displaying virtual surgical instruments on a doctor's console provided by the embodiment of this specification as shown in Figure 20. The medical device Specifically, it may include an input device 21, a processor 22, and a memory 23. The memory 23 is used to store instructions executable by the processor. When the processor 22 executes the instructions, the steps of the method for displaying virtual surgical instruments on the doctor's console described in any of the above embodiments are implemented.

An embodiment of this specification also provides an endoscopic medical robot, including the doctor console described in any of the above embodiments.

In this embodiment, the input device may be one of the main devices for exchanging information between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input pad, voice input device, etc.; the input device is used to input raw data and programs for processing these numbers into the computer. The input device can also obtain and receive data transmitted from other modules, units, and devices. The processor may be implemented in any suitable manner. For example, a processor may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit ( Application Specific Integrated Circuit (ASIC), programmable logic controller and embedded microcontroller form, etc. The memory may specifically be a memory device used to store information in modern information technology. The memory can include multiple levels. In a digital system, anything that can save binary data can be a memory; in an integrated circuit, a circuit with a storage function that does not have a physical form is also called a memory, such as RAM, FIFO, etc.; In the system, storage devices in physical form are also called memories, such as memory sticks, TF cards, etc.

In this embodiment, the specific functions and effects achieved by the medical device can be explained in comparison with other embodiments, and will not be described again here.

The embodiment of this specification also provides a computer storage medium based on a method for displaying virtual surgical instruments on a doctor's console. The computer storage medium stores computer program instructions. When the computer program instructions are executed, the above-mentioned steps are implemented. arbitrary The steps of the method for displaying virtual surgical instruments on a doctor's console are described in the embodiments.

In this embodiment, the above-mentioned storage media includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), cache (Cache), and hard disk (Hard Disk Drive, HDD). Or Memory Card. The memory may be used to store computer program instructions. The network communication unit may be an interface configured in accordance with the standards specified by the communication protocol and used for network connection communication.

In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments, and will not be described again here.

Obviously, those skilled in the art should understand that each module or each step of the above embodiments of this specification can be implemented by a general computing device, and they can be concentrated on a single computing device, or distributed among multiple computing devices. on a network, optionally, they may be implemented in program code executable by a computing device, such that they may be stored in a storage device for execution by the computing device, and in some cases, may be implemented in a manner different from that described herein The steps shown or described are performed in sequence, or they are separately made into individual integrated circuit modules, or multiple modules or steps among them are made into a single integrated circuit module. As such, embodiments of the present description are not limited to any specific combination of hardware and software.

It should be understood that the above description is for purposes of illustration rather than limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art from reading the above description. The scope of the specification, therefore, should be determined, not with reference to the above description, but rather with reference to the foregoing claims, along with the full scope of equivalents to which such claims are entitled.

The above are only preferred embodiments of this specification and are not intended to limit this specification. For those skilled in the art, various modifications and changes may be made to the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this manual shall be included in the protection scope of this manual.

Claims

A method for displaying virtual surgical instruments on a doctor's console, characterized by including:

Acquire an image of the master control arm of the physician's console;

Determine the posture data of the main control arm according to the image of the main control arm;

Based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, virtual posture data of the surgical instrument is generated to display a virtual instruction instrument image in the stereoscopic monitor of the doctor's console.
The method for displaying virtual surgical instruments on a doctor's console according to claim 1, further comprising:

Acquire an image of the physician's console's feet and/or the operator's feet;

Generate foot position data based on the foot image, and/or generate foot position data based on the foot image to display a foot area and/or a foot area in the stereoscopic monitor.
The method for displaying virtual surgical instruments on a doctor's console according to claim 2, further comprising:

When one or more footrests of the doctor's console are effectively stepped on, the one or more footrests in the footrest area that are effectively stepped on are highlighted on the stereoscopic monitor.
The method for displaying virtual surgical instruments on a doctor's console according to claim 1, wherein the image of the main control arm includes a left eye image and a right eye image of the main control arm collected by a binocular camera; The image of the main control arm determines the posture data of the main control arm, including:

Obtain the spatial position data of the left-eye camera and the right-eye camera in the binocular camera;

Based on the spatial position data and the position parameters of each feature point in the left-eye image and the right-eye image among the plurality of feature points of the main control arm, determine the spatial position data corresponding to each feature point;

According to the spatial position data corresponding to each feature point at multiple times, the attitude data of the main control arm is determined.
The method for displaying virtual surgical instruments on a doctor's console according to claim 1, wherein the virtual posture data of the surgical instrument is generated based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, include:

Based on the posture data of the main control arm, generate control instruction information of the main control arm;

The control instruction information of the main control arm is applied to the three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.
The method for displaying virtual surgical instruments on a doctor's console according to claim 1, wherein the virtual posture data of the surgical instrument is generated based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, include:

Obtain a target machine learning model; the target machine learning model is pre-established based on the three-dimensional model of the surgical instrument;

Input the posture data of the main control arm into the target machine learning model to obtain the virtual posture data of the surgical instrument; before obtaining the target machine learning model, it also includes:

Construct a machine learning model based on the three-dimensional model of the surgical instrument;

Obtaining the posture data of the control arm and the corresponding posture data of the surgical instrument during the endoscopic surgery, and based on the posture data of the control arm and the corresponding posture of the surgical instrument during the endoscopic surgery Data construction training sample set;

The machine learning model is trained using the constructed training sample set to obtain a target machine learning model.
The method for displaying virtual surgical instruments on a doctor's console according to claim 1, wherein the stereoscopic monitor also displays actual instrument images of the surgical instruments collected by the endoscope;

After the virtual instruction instrument image is displayed on the stereoscopic monitor of the doctor's console, it also includes:

Compare the actual instrument image with the virtual command instrument image;

According to the comparison results, it is determined that the difference between the virtual instruction instrument image and the actual instrument image exceeds a preset range The target part is marked and displayed on the stereoscopic monitor.
The method for displaying virtual surgical instruments on a doctor's console according to claim 7, wherein when it is determined that the difference between the virtual instruction instrument image and the actual instrument image exceeds a preset range, After section, also includes:

determining position data of the target portion within the surgical instrument;

Early warning information is generated based on the location data, and an early warning is provided to the operator in a preset manner based on the early warning information.
A doctor's console is characterized by including:

An image acquisition device used to acquire images of the main control arm of the doctor's console;

An image processing device, configured to determine the attitude data of the main control arm based on the image of the main control arm; and also used to generate the posture data of the main control arm and the three-dimensional model of the surgical instrument based on the attitude data of the main control arm and the three-dimensional model of the surgical instrument. Virtual posture data;

A stereoscopic monitor configured to display a virtual instruction instrument image according to the virtual posture data.
A computer-readable storage medium having computer instructions stored thereon, characterized in that when the instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.