WO2023078290A1

WO2023078290A1 - Mark sharing method and apparatus for surgical robot, and system, device and medium

Info

Publication number: WO2023078290A1
Application number: PCT/CN2022/129234
Authority: WO
Inventors: 凌刚; 蒋梦倩; 孙洪军
Original assignee: 上海微创医疗机器人（集团）股份有限公司
Priority date: 2021-11-05
Filing date: 2022-11-02
Publication date: 2023-05-11
Also published as: CN114022587A

Abstract

The present application relates to a mark sharing method and apparatus for a surgical robot, and a system, a device and a medium. The mark sharing method comprises: acquiring the currently generated initial medical image; after a first mark instruction, which is input by a user, is acquired, sending first mark information to a second surgical robot end in real time, and fusing the first mark information into the initial medical image, wherein the first mark instruction comprises the first mark information; and after second mark information, which is sent by the second surgical robot end in real time, is received, fusing the second mark information into the initial medical image. Therefore, the real-time sharing of data marks of a master end and a slave end during surgery is realized, thereby effectively solving the problem of poor communication between a doctor at the master end and an assistant doctor at the slave end during a surgical operation process of a remote robot.

Description

Marker sharing method, device, system, equipment and medium for surgical robot

technical field

The present application relates to the field of medical technology, in particular to a marker sharing method for a surgical robot, a marker sharing device, a surgical robot system, a readable storage medium and electronic equipment.

Background technique

With the advancement of science and technology, the development of 5G technology, the rise of AI artificial intelligence. Telemedicine surgery has an opportunity for rapid development in the general environment. Telemedicine refers to relying on computer technology, remote sensing, telemetry, and remote control technology, giving full play to the advantages of medical technology and medical equipment in large hospitals or specialized medical centers, and treating injuries and diseases in remote areas, islands, or ships with poor medical conditions. It is a brand-new medical service that aims to improve the level of diagnosis and medical treatment, reduce medical expenses, and meet the health needs of the general public through remote diagnosis, treatment and consultation.

At present, telemedicine technology has developed from the initial TV monitoring and telephone remote diagnosis to the comprehensive transmission of numbers, images, and voices using high-speed networks, and has realized real-time voice and high-definition image communication, which provides a solid foundation for the application of modern medicine. Broader development space. The development of foreign countries in this field has a history of more than 40 years, while our country has only received attention and development in recent years. The current technologies used in telemedicine include: (1) drawing board marking technology, which provides a remote real-time sub-consultation system, which includes image browsing, image marking, text input, and real-time synchronization of audio and video; (2) medical image stereo Visual display, the computer obtains medical images through input devices, then processes the images through the software system, and finally displays and performs visual linkage through the stereo vision module, 3D module or 2D module; (3) remote real-time consultation of medical images, providing a real-time case Sub-consultation system, including consultation creation management module, electronic whiteboard, audio and video collection-playback module, network data transmission module and central processing module, etc.

The above technologies are basically independent systems, most of which are for remote consultation scenarios, for remote surgery drawing board sharing systems, etc., focusing on teaching and sharing, and do not have the ability to perform multi-terminal device data fusion and synchronization in real-time telemedicine operations, and to guide physicians in surgery role.

Contents of the invention

In order to solve the technical problems in the prior art, the purpose of this application is to provide a marker sharing method for surgical robots, a marker sharing device, a surgical robot system, a readable storage medium and electronic equipment, which can realize master-slave dual Real-time sharing of data markers on the end, effectively solving the problem of poor communication between the master doctor and the slave assistant doctor during remote robot surgery, thereby effectively improving the surgical accuracy in this scenario application and ensuring the safe and smooth operation.

In order to achieve the above purpose, according to the first aspect of the present application, a marker sharing method for surgical robots is provided, which is applicable to the first surgical robot end, and the marker sharing method includes:

Obtain the currently generated initial medical image;

After obtaining the first marking instruction input by the user, the first marking information is sent to the second surgical robot in real time, and the first marking information is fused in the initial medical image, wherein the first The marking instruction includes the first marking information;

After receiving the second marker information sent by the second surgical robot in real time, the second marker information is fused in the initial medical image.

Optionally, after receiving the second marker information sent by the second surgical robot in real time, the second marker information is fused in the initial medical image fused with the first marker information.

Optionally, the initial medical image fused with the first marker information and/or the second marker information is sent to the second surgical robot in real time.

Optionally, before acquiring the first marking instruction input by the user, the method further includes: starting a marking mode, where the marking mode allows the user to input the first marking instruction.

Optionally, fusing the first marker information in the initial medical image includes:

Reconstructing a first marker corresponding to the first marker information on the initial medical image according to the position information in the first marker information;

Fusing the second marker information in the initial medical image includes:

Reconstructing a second marker corresponding to the second marker information on the initial medical image according to the position information in the second marker information.

Optionally, after fusing the first marker information in the initial medical image and/or fusing the second marker information in the initial medical image, the marker sharing method further includes:

Acquiring image change information of the currently generated initial medical image, and updating the first marker information and/or the second marker information according to the image change information of the initial medical image, so that the first marker information and/or the second marker information follows changes in the initial medical image in real time.

Optionally, the tag sharing method also includes:

Obtain the flag state of the first surgical robot end and/or the second surgical robot end, and output master-slave control instructions or enable A marking command; the master-slave control command is used to control the movement of the first surgical robot end or the second surgical robot end; the enabling marking command is used to start the marking mode.

Optionally, the marking sharing method further includes: obtaining an exit marking instruction input by a user, and exiting the marking mode according to the exit marking instruction.

Optionally, also include:

Entering into an operation lock state according to the activation flag instruction, so as to prohibit the movement of the first surgical robot end or the second surgical robot end;

An operation unlock state is commanded according to the exit flag to allow movement of the first surgical robot end or the second surgical robot end.

Optionally, the tag sharing method also includes:

Acquiring the marking state of the second surgical robot, and starting the marking mode when it is determined that the second surgical robot is currently in a non-marking state, and sending the local current marking state to the second surgical robot in real time.

Optionally, the first marking information and the second marking information include graphics and/or text information.

Optionally, the graphics include custom graphics or preset graphics.

Optionally, the tag sharing method also includes:

Acquiring the operating state of the first foot pedal at the first surgical robot end, and starting the marking mode according to the operating state of the first foot pedal;

Obtaining the operation state of the second foot pedal on the first surgical robot end, and exiting the marking mode according to the operation state of the second foot pedal.

Optionally, the tag sharing method also includes:

Starting the marking mode according to an instruction output by a marking button on the interactive interface of the first surgical robot;

Exit the marking mode according to an instruction output by the exit marking button on the interaction interface of the first surgical robot.

Optionally, the marker sharing method further includes: wirelessly transmitting audio data and/or video data to the second surgical robot in real time.

Optionally, the acquiring the currently generated initial medical image includes: using 3D modeling and artificial intelligence algorithms to process the image data collected in real time, so as to generate the 3D initial medical image.

In order to achieve the above purpose, according to the second aspect of the present application, a marker sharing device for surgical robots is provided, which is suitable for the first surgical robot end, including:

An image data acquisition module, configured to acquire the currently generated initial medical image;

A marking instruction acquisition module, configured to acquire a first marking instruction input by a user, the first marking instruction including first marking information, and also used to acquire second marking information sent in real time by the second surgical robot end;

A data fusion module, configured to fuse the first marker information and/or the second marker information in the initial medical image.

Optionally, the marker sharing device further includes an information sharing module, configured to send the initial medical image fused with the first marker information and/or the second marker information to the second surgical robot in real time .

Optionally, the marker sharing device further includes a display module, configured to display the initial medical image fused with the first marker information and/or the second marker information.

In order to achieve the above object, according to the third aspect of the present application, a surgical robot system is provided, including a first surgical robot end and a second surgical robot end, the first surgical robot end includes the above-mentioned surgical robot end Mark the shared device.

Optionally, the surgical robot system further includes an image acquisition device, configured to acquire an initial medical image of a predetermined object.

Optionally, the first surgical robot end further includes a first foot pedal and a second foot pedal, and the marking instruction acquisition module includes a first operating arm and a second operating arm;

The first foot pedal is used to output an enable flag command to start the flag mode;

The second foot pedal is used to output an exit marking command to exit the marking mode;

The first operating arm is used to select a first marker corresponding to the first marker information according to an operation instruction input by a user;

The second operating arm is configured to use the selected first marker to create the first marker on the initial medical image according to an operation instruction input by a user.

Optionally, the first surgical robot terminal also includes an interactive interface, the interactive interface includes a marking button and an exit marking button; the marking instruction acquisition module includes a keyboard and a mouse;

The interactive interface displays the initial medical image;

The mark button is used to output the enable mark instruction to start the mark mode;

The exit marking button is used to output an exit marking instruction to exit the marking mode;

The keyboard and the mouse are used to generate a first mark corresponding to the first mark information according to an instruction input by a user.

Optionally, the surgical robot system further includes an audio and/or video collection module for collecting audio and/or video data.

In order to achieve the above object, according to a fourth aspect of the present application, a readable storage medium is provided, on which a program is stored, and when the program is executed, the above-mentioned marker sharing method for a surgical robot is executed.

In order to achieve the above object, according to a fifth aspect of the present application, an electronic device is provided for performing a marker sharing method for a surgical robot, including a processor and the readable storage medium, and the processor is controlled by configured to execute the program stored on the readable storage medium.

Compared with the prior art, the marker sharing method for surgical robots, marker sharing device, surgical robot system, readable storage medium and electronic equipment of the present application have the following advantages:

This application uses data fusion technology and marker sharing technology to realize real-time sharing of data markers between the master end and slave end during the operation process, effectively solving the problem of operation information between the master end doctor and the slave end assistant doctor during the remote surgical robot operation, such as determining the location of the lesion , Surgical plans and other unclear communication issues, so as to effectively improve the surgical accuracy rate in telemedicine and ensure the safe and smooth operation. In this way, this application realizes effective communication and communication between the master doctor and the slave assistant doctor during the operation process through the real-time sharing of data tags, so that the master doctor can better guide the slave assistant doctor to perform the operation and improve the operation efficiency. efficiency and surgical success.

This application uses the existing equipment in the surgical robot system, such as foot pedals, operating arms, keyboards, and mice, to perform marking operations, which facilitates the operation of doctors and effectively reduces the interference of marking operations on surgery.

This application uses 3D modeling and artificial intelligence algorithms to process the image data collected in real time to generate a 3D initial medical image, and to mark and communicate between the master and slave on the 3D initial medical image to improve more accurate The reference value greatly improves the application accuracy of remote surgery scenarios.

Description of drawings

The features, properties and advantages of the methods of implementing the application and related embodiments will be described with reference to the following drawings, in which:

Fig. 1 is a schematic diagram of a surgical scene where the master end and the slave end of a surgical robot system are in different places according to a preferred embodiment of the present application;

Fig. 2 is a structural block diagram of a surgical robot system according to a preferred embodiment of the present application;

Fig. 3 is a schematic structural diagram of a doctor's main console according to a preferred embodiment of the present application;

FIG. 4 is a schematic structural diagram of a slave control device according to a preferred embodiment of the present application;

Fig. 5 is the working flow chart that the main end mark creates according to a preferred embodiment of the present application;

Fig. 6 is a schematic diagram of the operation principle of master-side tag creation according to a preferred embodiment of the present application;

Fig. 7 is a schematic diagram showing a main terminal mark, a main terminal cancel button, and a graphic selection window on the main terminal interactive interface of the main terminal display module according to a preferred embodiment of the present application;

FIG. 8 is a flow chart of a master end sharing a master end mark to a slave end according to a preferred embodiment of the present application;

Fig. 9 is a workflow flow diagram of creation of slave-end tags according to a preferred embodiment of the present application;

Fig. 10 is a schematic diagram of the operation principle of creating a slave tag according to a preferred embodiment of the present application;

Fig. 11 is a schematic diagram showing a slave mark, a mark button, a mark cancel button, and a graphic selection window on the slave end interactive interface of the slave display module according to a preferred embodiment of the present application;

Fig. 12 is a flow chart of sharing a slave mark from a slave to a master according to a preferred embodiment of the present application;

Fig. 13 is a schematic diagram of data fusion by the surgical robot system according to a preferred embodiment of the present application;

Fig. 14 is a schematic diagram of a state of fusion and display of data by the surgical robot system according to a preferred embodiment of the present application.

In the figure: 100-master end;

101-doctor main console; 1011-master image data acquisition module; 1012-master marking command acquisition module; 1013-master data fusion module; 1014-master information sharing module; 1015-master display module; 1016- Main-end audio and/or video acquisition module; 1017-master-end data transmission module;

1041-left pedal; 1042-right pedal; 1043-left operating arm; 1044-right operating arm; 1045-left control handle; 1046-right control handle;

G1-master interface; G2-master drawing tool; G3-rectangular preset master mark; G4-circular preset master mark; G5-circular master mark; G6-custom master End mark; G7-rectangular main end mark; G8-master end undo button;

200-slave;

201 - surgical robot;

202-slave control device; 2021-slave image data acquisition module; 2022-slave label instruction acquisition module; 2023-slave data fusion module; 2024-slave information sharing module; 2025-slave display module; 2026- Slave audio and/or video acquisition module; 2027-slave data transmission module;

2071-keyboard; 2072-keyboard;

H1-slave interactive interface; H2-slave drawing tool; H3-rectangular preset slave mark; H4-round preset slave mark; H5-circular slave mark; H6-custom slave End mark; H7-Rectangular slave end mark; H8-Slave end undo button; H9-Mark button;

203-image trolley; 204-sterile table; 211-partition.

Detailed ways

Embodiments of the present application are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in this embodiment are only schematically illustrating the basic idea of the application, and only the components related to the application are shown in the diagrams rather than the number, shape and shape of the components in actual implementation. Dimensional drawing, the type, quantity and proportion of each component can be changed freely during its actual implementation, and its component layout type may also be more complicated.

In addition, each embodiment of the content described below has one or more technical features, but this does not mean that the applicant must implement all the technical features in any embodiment at the same time, or can only implement different embodiments separately. Some or all of the technical features. In other words, on the premise that the implementation is possible, those skilled in the art can selectively implement some or all of the technical features in any embodiment according to the disclosure of the application and depending on the design specifications or implementation requirements, or Selectively implement a combination of some or all of the technical features in multiple embodiments, thereby increasing the flexibility of implementing the present application.

As used in this specification, the singular forms "a", "an" and "the" include plural objects, and the plural form "a plurality" includes two or more objects, unless the content clearly states otherwise. As used in this specification, the term "or" is generally used in the sense including "and/or", unless the content clearly indicates otherwise, and the terms "install", "connect" and "connect" should be To understand it in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediary, and it can be the internal communication of two elements or the interaction relationship between two elements. Those of ordinary skill in the art can understand the specific meanings of the above terms in this application according to specific situations.

In order to make the purpose, advantages and features of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the drawings are all in very simplified form and use imprecise scales, and are only used to facilitate and clearly assist the purpose of illustrating the embodiments of the present application. In the case of no conflict, the following embodiments and the features in the embodiments may complement each other or be combined with each other.

Fig. 1 shows an application scenario of a surgical robot system according to a preferred embodiment of the present application. As shown in FIG. 1 , a preferred embodiment of the present application provides a surgical robot system, including a first surgical robot end and a second surgical robot end connected in communication. One of the first surgical robot end and the second surgical robot end is the master end 100 , and the other is the slave end 200 . For the sake of brevity, in the following description, it is assumed that the first surgical robot end is the main end 100, and the second surgical robot end is the slave end 200. Those skilled in the art should be able to modify the following description, and describe the description after making appropriate modifications in details. It is used when the first surgical robot end is the slave end 200 and the second surgical robot end is the main end 100 .

The master terminal 100 and the slave terminal 200 may be located in different rooms, different hospitals or different cities, so as to realize telemedicine. The main terminal 100 includes a doctor's main console 101 for the doctor to use his hands and feet to realize telemedicine. The slave end 200 includes a surgical robot 201, the surgical robot 201 includes a mechanical arm, and the end of the mechanical arm is used to detachably connect to a surgical instrument or an image acquisition device, and the image acquisition device includes but is not limited to an endoscope , to control the operation of surgical instruments or image acquisition devices. During the operation, using the master-slave control relationship formed by the doctor's main console 101 and the robotic arm and surgical instruments, the doctor remotely operates the surgical robot 201 through the main operation unit of the doctor's main console 101, so that the robotic arm and surgical instruments operate according to the doctor's The movement of the main console 101 moves, for example, according to the operation of the doctor's hands, so as to perform minimally invasive surgical treatment on the patient on the hospital bed.

As shown in FIG. 1 and FIG. 2 , the slave end 200 also includes a slave end control device 202 , which is usually integrated with an image trolley 203 . The image trolley 203 can also place an endoscope and other related equipment (such as a slave display module and some cables). The endoscope is used to obtain images of surgical environments such as human tissues and organs, surgical instruments, blood vessels, and body fluids, and the endoscope and surgical instruments enter the patient's position through wounds on the patient's body. The image trolley 203 is set beside the hospital bed and is set separately from the surgical robot 201 .

The slave control device 202 communicates with the surgical robot 201 by wire or wirelessly. The doctor master console 101 is used to generate and output master-slave control instructions according to external instructions, and send them to the slave-end control device 202 . The slave control device 202 is used to control the movement of the surgical robot 201 according to the received master-slave control instructions. In more detail, the slave control device 202 is used to output master-slave control instructions according to the motion information sent by the doctor's master console 101 and the preset master-slave mapping relationship, so as to control the surgical robot 201 to execute the master-slave control instructions to Drives the movement of surgical instruments or endoscopes. For example, the slave control device 202 controls the surgical robot 201 to drive the surgical instrument or endoscope to move according to the acquired moving speed of the main operating unit, and controls the surgical robot according to the acquired rotational angle or rotational speed of the main operating unit. 201 to drive the surgical instrument or endoscope to rotate, and also control the surgical robot 201 to drive the surgical instrument or endoscope to bend according to the obtained bending angle or bending direction of the main operation unit.

Further, the slave end 200 may also include auxiliary equipment in the operation, such as a sterile table, a ventilator, a detection device, and the like. In this embodiment, the slave end 200 also includes a sterile table 204, which is set next to the hospital bed for placing surgical instruments to establish a sterile barrier to prevent recontamination of sterile surgical instruments and auxiliary materials, and also to prevent surgical Omission or loss of equipment and auxiliary materials.

In addition, the surgical robot system of this embodiment aims to realize telemedicine, and can exchange information between the master end and the slave end during the telemedicine operation. The positioning of tissues, blood vessels, and lesion boundaries can also include surgical paths, surgical plans, and communication of unclear positions displayed in medical images, so as to effectively solve the problem between the master doctor and the slave assistant doctor during remote surgery. The problem of poor information exchange between them can effectively improve the accuracy rate of surgery in this telemedicine scene, ensure the safe and smooth operation of the operation, and improve the efficiency of the operation.

More specifically, this embodiment provides a marker sharing method for a surgical robot, which is applicable to the master terminal 100 or the slave terminal 200, taking the master terminal 100 as an example, including the following steps:

Step 1: Obtain the currently generated initial medical image; the initial medical image here can be a 3D medical image, and of course it can also be a 2D medical image in other cases; in addition, the initial medical image data includes but is not limited to The image can also be image data collected by other image acquisition devices, such as CT or MRI data collected by imaging equipment; in addition, the doctor's main console 101 can create images of predetermined objects in the body such as target tissues, organs or blood vessels based on the initial medical image data. an initial medical image, and share the created initial medical image with the slave end 200;

Step 2: After obtaining the first marking instruction input by the user, sending the first marking information to the slave terminal 200 in real time, and fusing the first marking information in the initial medical image, the first marking information The instruction includes first marking information; the first marking information mainly includes coordinate information corresponding to the first marking;

Step 3: After receiving the second marker information sent in real time from the terminal 200, fuse the second marker information into the initial medical image; the second marker information mainly includes coordinate information corresponding to the second marker.

Preferably, the above marker sharing method further includes: after receiving the second marker information sent in real time from the terminal 200, fusing the second marker information in the initial medical image fused with the first marker information; thus obtaining The initial medical image fused with the first marker information and the second marker information.

Preferably, the marker sharing method further includes: sending the initial medical image fused with the first marker information and/or the second marker information to the slave terminal 200 in real time. At this time, the slave terminal 200 can omit the step of data fusion, and can directly obtain the initial medical image fused with the first marker and/or the second marker from the master terminal 100 . The first mark corresponds to the first mark information, such as graphic and/or text information; the second mark corresponds to the second mark information, which can also be graphic and/or text information.

Of course, the above tag sharing method can also be applied to the slave end 200. At this time, the above steps are mostly the same, the difference is that in step 2, the slave end 200 sends the first tag information to the master end 100 in real time, and In step 3, after receiving the second marker information sent by the master terminal 100 in real time, the slave terminal 200 fuses the second marker information in the initial medical image. Another difference is that, in the preferred step, the slave terminal 200 sends the initial medical image fused with the first marker information and/or the second marker information to the master terminal 100 in real time, and further in the preferred step, from the The terminal 200 sends the initial medical image fused with the first marker information and/or the second marker information to the master terminal 100 in real time. Then, the master terminal 100 can also omit the step of data fusion, and can directly obtain the initial medical image containing the first marker and/or the second marker from the slave terminal 200 .

In the embodiment of the present application, the above marker sharing method is applicable to both the master terminal 100 and the slave terminal 200, so that the master terminal 100 and the slave terminal 200 can each perform data fusion to obtain the initial medical image containing the marker (defined as marker medical image), and finally display the initial medical image fused with the first marker and/or the second marker on the respective display modules, so that the markers made by the master doctor and the slave doctor on the initial medical image can be viewed.

Correspondingly, an embodiment of the present application also provides a label sharing device, which is applicable to the master terminal 100 or the slave terminal 200, and is used to implement the above label sharing method.

Taking the main terminal 100 as an example, the label sharing device includes an image data acquisition module, a label instruction acquisition module and a data fusion module; the image data acquisition module is used to acquire the currently generated initial medical image; the label instruction acquisition module The module is used to obtain the first marking instruction input by the user, and is also used to obtain the second marking information sent in real time from the terminal 200; the data fusion module is used to fuse the first marking information and/or in the initial medical image or the second tag information. Preferably, the marker sharing device further includes an information sharing module, configured to send the initial medical image fused with the first marker information and/or the second marker information to the slave end 200 in real time. In this case, it should be understood that the first marking instruction is the marking instruction input by the doctor at the master end 100, and the first marking instruction is defined as the marking instruction at the main end, while the auxiliary doctor at the slave end The input marking command at 200 is defined as a second marking command, the second marking command includes second marking information, and the second marking command is defined as a slave marking command.

When applicable to the slave end 200, the difference from the tag sharing device applicable to the master end is that the tag instruction acquisition module is used to obtain the second tag instruction sent by the master end 100 in real time, and in a preferred solution, the information sharing The module is configured to send the initial medical image fused with the first marker information and/or the second marker information to the main terminal 100 in real time. In this case, the first marking instruction is the marking instruction input by the assistant doctor at the slave end 200, and the first marking instruction at this time is defined as the marking instruction from the slave end, and the second marking instruction at this time is The instruction is a marking instruction input by the client doctor at the active 100 , and the second marking instruction at this time is defined as the client marking instruction.

In the embodiment of the present application, both the master end 100 and the slave end 200 include a tag sharing device. For more clarity, the tag sharing device of the master 100 is defined as a tag sharing device of the host, and the tag sharing device of the slave 200 is defined as a tag sharing device of the slave. Therefore, both the master terminal 100 and the slave terminal 200 can use data fusion technology to obtain the initial medical image containing the first marker and/or the second marker, so that both the master terminal 100 and the slave terminal 200 can display the initial medical image containing the first marker and/or the second marker. /or the initial medical image of the second mark, to realize timely and effective communication of information between the master doctor and the slave doctor.

As shown in FIG. 2 , the main-end label sharing device includes a main-end image data acquisition module 1011 , a main-end label instruction acquisition module 1012 , a main-end data fusion module 1013 and a main-end information sharing module 1014 . The main-end image data acquisition module 1011 is used to acquire the currently generated initial medical image; the main-end marking instruction acquisition module 1012 is used to acquire the main-end marking instruction input by the user, and the main-end marking instruction includes main-end marking information , and is also used to obtain the slave-end tag information sent by the slave-end 200 in real time; the master-end data fusion module 1013 is used to fuse the master-end tag information and/or the slave-end tag information in the initial medical image, An initial medical image fused with the marker information of the master end and/or the marker information of the slave end is obtained. Further, the master-end information sharing module 1014 is configured to send the initial medical image fused with the master-end tag information and/or the slave-end tag information to the slave end 200 in real time. Further, the master-end mark sharing device further includes a master-end display module 1015, configured to display the initial medical image fused with the master-end mark information and/or the slave-end mark information.

In the embodiment of the present application, the master-end mark sharing device is integrated in the doctor's main console 101, and the master-end doctor can mark the initial medical image on the doctor's main console 101, and An initial medical image without a marker and an initial medical image with a marker are displayed on the display at 101 . The initial medical image without a mark and the initial medical image with a mark can be displayed in different windows, or after the initial medical image is marked, only the initial medical image with a mark is displayed.

Continuing to refer to FIG. 2 , the device for sharing a slave tag includes a slave image data acquisition module 2021 , a slave tag command acquisition module 2022 , a slave data fusion module 2023 and a slave information sharing module 2024 . The slave image data acquisition module 2021 is used to acquire the currently generated initial medical image; the slave label instruction acquisition module 2022 is used to acquire a user input slave label instruction, and the slave label instruction includes slave label information , is also used to obtain the master-end tag information sent by the master-end 100; the slave-end data fusion module 2023 is used to fuse the master-end tag information and/or the slave-end tag information in the initial medical image to obtain The primary medical image is fused with the primary-side marker information and/or the slave-side marker information. Further, the slave-end information sharing module 2024 is configured to send the initial medical image fused with the master-end tag information and/or the slave-end tag information to the master end 100 in real time. Further, the device for sharing markers of the slave end further includes a slave display module 2025, configured to display the initial medical image fused with the marker information of the master end and/or the marker information of the slave end.

In the embodiment of the present application, the slave-end control device 200, such as the image trolley 203, is integrated with the slave-end marking sharing device, and the slave-end assistant doctor can mark the initial medical image on the slave-end control device 200, And the initial medical image without marker and the initial medical image with marker are displayed on the display of the slave-end control device 200 . In the device for sharing markers from the end, the initial medical images without markers and the initial medical images with markers can be displayed in different windows, or after the initial medical images are marked, only the initial medical images with markers are displayed.

It needs to be understood that the master-end data fusion module 1013 and the slave-end data fusion module 2023 respectively adopt data fusion algorithms to realize data fusion. The data fusion algorithm is mainly realized through the marked coordinates, and how to realize the data fusion is a well-known technology in the field, so this application will not describe it in detail.

In an example, the marker sharing device reconstructs the master-end marker corresponding to the master-end marker information on the initial medical image according to the position information in the master-end marker information, and based on the slave-end marker The location information in the information, reconstruct the secondary marker corresponding to the secondary marker information on the initial medical image.

More specifically, the marker sharing device obtains the image corresponding to the location of the master-end marker information in the initial medical image according to the coordinate information in the master-end marker information, and The main-end marker is generated on the image of the position, thereby fusing the main-end marker with the original medical image. The marker sharing device also obtains the image corresponding to the position of the slave marker information in the initial medical image according to the coordinate information in the slave marker information, and places the image at the position of the slave marker information generate the slave-end markers, so as to fuse the slave-end markers with the initial medical image; finally, the master-end data fusion module 1013 obtains the initial medical image fused with the master-end markers and the slave-end markers.

In a specific embodiment, the tag sharing device can also obtain an enable or disable tag instruction, and output a master-slave control instruction according to the disable tag instruction, and the master-slave control instruction is used to control the master terminal 100 or the slave terminal 200 , and activate the marking mode according to the enabling marking instruction, so as to allow the user to mark the initial medical image.

For the main terminal 100, the main terminal mark sharing device enables or prohibits the doctor main console 101 from operating to create the main end mark according to the obtained enable or disable mark instruction. And when it is allowed to create the main-end mark, the user can mark the initial medical image through the doctor's main console 101 and save the position information (including coordinates) of the main-end mark. For the slave 200, the slave tag sharing device enables or prohibits the slave control device 200 from creating a slave tag according to the acquired enabling or disabling tag instruction. And when the slave-end mark is allowed to be created, the user can mark the initial medical image through the slave-end control device 200 and save the position information (including coordinates) of the slave-end mark.

Further, the master information sharing module 1014 can receive the data sent by the slave 200, such as the slave marker information, and send the tag information and the initial medical image containing the tag information to the slave 200, and send the The received slave tag information is transmitted to the master tag instruction acquiring module 1012 . Further, the master-end mark instruction acquisition module 1012 can parse the slave-end mark information to obtain the position information in the slave-end mark information. Further, the master-end marking instruction acquiring module 1012 can pack the master-end marking information, and the packed master-end marking information is sent to the slave-end 200 through the master-end information sharing module 1014 .

Further, the slave information sharing module 2024 can receive the data sent by the master 100, such as the master marker information, in addition to sending the marker information and the initial medical image containing the marker information to the master 100, and The received master tag information is transmitted to the slave tag instruction acquiring module 2022 . Further, the slave tag instruction acquisition module 2022 can parse the master tag information to obtain the position information in the master tag information. Further, the slave tag instruction acquiring module 2022 can pack the slave tag information, and the packed slave tag information is sent to the master 100 through the slave information sharing module 2024 .

Without limitation, the main-end image data acquisition module 1011 can also establish initial medical images of predetermined objects such as tissues, organs or blood vessels according to the initial medical image data, and the main-end image data acquisition module 1011 preferably provides a three-dimensional modeling function , to obtain 3D medical images. In addition, the main-end display module 1015 is preferably a 3D display to display three-dimensional medical images stereoscopically, so as to improve a more accurate reference value, so that the application accuracy of remote surgery scenarios is greatly improved. In order to improve the accuracy of the operation, the image data acquisition module 1011 of the main terminal mainly uses 3D modeling and artificial intelligence algorithms to process the image data collected in real time to generate the 3D initial medical image. The artificial intelligence algorithm can realize fully automatic modeling without manual repair. It has the characteristics of high fidelity, no distortion, small size and convenient display, effectively reducing the modeling time of 3D models and improving the accuracy of 3D model modeling. Further, the slave display module 2025 may be a 2D display or a 3D display.

In order to further improve the smoothness of information exchange between the master terminal 100 and the slave terminal 200, in an embodiment, the mark sharing method further includes: real-time wireless transmission of audio data and/or video data from the slave terminal 200 to the master terminal 100 . In an embodiment, the mark sharing method further includes: the master terminal 100 wirelessly transmits audio data and/or video data to the slave terminal 200 in real time.

Further, the surgical robot system also includes an audio and/or video acquisition module, configured to acquire audio and/or video data.

As shown in Figure 2, the doctor's main console 101 is also configured with a main-end audio and/or video acquisition module 1016, which is used to collect audio and/or video data of the main end 100, and transmit the described data to the slave end 200 in real time. The audio and/or video data of the master terminal facilitates online communication between the master doctor and the slave assistant doctor through voice and/or video to further improve communication efficiency. The master end 100 can transmit audio and video data between the master end 100 and the slave end 200 through the master end data transmission module 1017, and provide a real-time audio and video low-latency transmission scheme to ensure low delay and high latency during data transmission. quality.

Similarly, the slave-side control device 202 is configured with a slave-side audio and/or video acquisition module 2026 for collecting audio and/or video data from the slave-side 200, and transmitting the slave-side audio and/or video data to the master 100 in real time. /or video data, so as to facilitate online communication between the master doctor and the slave doctor through voice and/or video, so as to further improve communication efficiency. The slave end 200 further transmits audio and video data between the master end 100 and the slave end 200 through the slave end data transmission module 2027, and provides a real-time audio and video low-latency transmission scheme to ensure low delay and high latency during data transmission. quality.

The master audio and/or video acquisition module 1016 and the slave audio and/or video acquisition module 2026 include but are not limited to audio and video equipment such as cameras and microphones. The master-end audio and/or video acquisition module 1016, and the slave-end audio and/or video acquisition module 2026 can be connected to a wired or wireless network, preferably connected to 5G Wi-Fi, with faster transmission speed and better transmission quality. Wherein, the slave-end audio and/or video acquisition module 2026 performs data communication with the surgical robot 201 to receive the video data transmitted by the image acquisition device.

The surgical robot system further includes an image acquisition device, and the initial medical image data can be determined by using an image acquisition device (such as an endoscope) to collect images in the body.

Further, considering that the image captured by the image acquisition device will change in real time, at this time, in order to improve the labeling efficiency, the primary end label information is fused in the initial medical image and/or all information is fused in the initial medical image. After the tag information of the slave end, it also includes: acquiring the image change information of the initial medical image currently generated, and updating the tag information of the master end and/or the slave end according to the image change information of the initial medical image mark information, so that the master-end mark information and/or the slave-end mark information follow the changes in the image of the initial medical image collected by the image acquisition device in real time; so that the mark dynamically follows the change of the image, making the mark more efficient , and the operation is more convenient.

Further, the marking sharing method further includes: before acquiring the first marking instruction input by the user, starting a marking mode, where the marking mode allows the user to input the first marking instruction. It can be understood that, when the first marking command is the marking command of the main end, before the main end 100 obtains the main end marking command input by the user, it also includes starting the marking mode, and only after the marking mode is started, the user is allowed to start the marking mode on the main end. The terminal 100 inputs the master label command. And when the first mark command is a slave mark command, before the slave terminal 200 obtains the slave mark command input by the user, it also includes a start mark mode, after the start mark mode, the user is allowed to input the slave end tag directive.

In a specific embodiment, the label sharing method further includes: acquiring the label status of the master terminal 100 and/or the slave terminal 200, and outputting a master-slave control instruction or Enable markup directives. The master-slave control command is used to control the motion state (including the operation state) of the master terminal 100 or the slave terminal 200, and the enable flag command is used to start the flag mode.

Further, the marking sharing method further includes: obtaining an exit marking instruction input by a user, and exiting the marking mode according to the exit marking instruction.

Furthermore, the tag sharing method further includes: according to the enable tag command, making the master 100 and/or the slave 200 enter an operation lock state, so as to prohibit the movement of the master 100 and/or the slave 200, so that The master end 100 and/or the slave end 200 are prohibited from controlling the movement of the surgical robot 201 . For example, when the surgical robot system enters the marking mode, the main operating unit of the doctor’s main console 101 is in an operation locked state, and the main operating unit cannot teleoperate the surgical robot 201, for example, the slave terminal 200 receives the motion state of the main operating unit, but The movement of the surgical robot cannot be controlled accordingly, that is, the operating handle of the main operating unit cannot realize teleoperation to control the movement and operation of the surgical robot, and/or, when the surgical robot system enters the marking mode, the surgical robot 201 is automatically locked and cannot move.

Furthermore, the tag sharing method further includes: according to the exit tag command, making the master terminal 100 and/or the slave terminal 200 enter the operation unlock state, so as to allow the movement of the master terminal 100 and/or the slave terminal 200, so that The master end 100 and/or the slave end 200 allow to control the movement of the surgical robot. For example, when the surgical robot system exits the marking mode, the main operating unit of the doctor’s main console 101 is in the operation unlock state, and the main operating unit can remotely operate the surgical robot 201, for example, the slave terminal 200 receives the motion state of the main operating unit, and According to this, the movement of the surgical robot is controlled, that is, the operating handle of the main operation unit can control the movement of the surgical robot and the surgical operation by remote operation, and/or, when the surgical robot system exits the marking mode, the surgical robot 201 is automatically unlocked and can move.

It should be understood that when the surgical robot system enters the operation lock state, the surgical robot system locks the master-slave mapping relationship to facilitate the marking operation of the slave end 200 or the master end 100; and when the surgical robot system is in the operation unlock state, the operation The robot system can reconstruct the master-slave mapping relationship to allow the master terminal 100 to perform teleoperation on the slave terminal 200 according to the determined master-slave control instructions, so that the surgical robot 201 can control the movement of surgical instruments or endoscopes according to the master-slave control instructions .

Referring to FIG. 3 , in an exemplary embodiment, the main terminal 100 includes a left foot pedal 1041 and a right foot pedal 1042 , and these foot pedals are integrated on the doctor's main console 101 . One of the left footrest 1041 and the right footrest 1042 is configured as a first footrest, and the other of the left footrest 1041 and the right footrest 1042 is configured as a second footrest. The first pedal is used to output the enable flag instruction, and the master terminal 100 starts the flag mode according to the enable flag instruction of the first pedal; The Exit Mark command of the second pedal exits Mark mode. When in use, the doctor at the main end can enter the marking mode or exit the marking mode by controlling the corresponding pedals with both feet. For example, the current operating state of the left foot pedal 1041 is to be stepped on and an enable flag instruction is output, then the master terminal 100 determines to enter the flag mode; otherwise, the current operating state of the right foot pedal 1042 is to be stepped on and output an exit flag instruction, then The master 100 determines to exit the marking mode.

Continuing to refer to FIG. 3 , in a specific embodiment, the master-side flag instruction acquiring module 1012 may include a left operating arm 1043 and a right operating arm 1044 . These manipulation arms are also integrated on the doctor's main console 101 . The left operating arm 1043 and the right operating arm 1044 constitute the main operating unit, which can accept external instructions to output motion information, and one of the left operating arm 1043 and the right operating arm 1044 is configured as a first operating arm, and the other is configured as a first operating arm. is the second operating arm. The first operating arm is used for selecting the main-end mark corresponding to the main-end mark information according to the operation instruction input by the user. The second operating arm is configured to use the selected main-end marker to create the main-end marker on the initial medical image according to an operation instruction input by a user.

As shown in FIG. 3 and FIG. 7 , the master display module 1015 includes a master display interface G1. By manipulating the first operating arm, the user can select the drawing tool on the interactive interface of the main terminal, and the drawing tool provides various drawing commands corresponding to the preset graphics, so as to select the graphics in the drawing tool to create a drawing corresponding to the main drawing tool. The master endian flag for the endian flag directive. In addition, by manipulating the second operating arm, the user can draw a main-end marking corresponding to the main-end marking instruction on the initial medical image according to the preset graphic determined by the first operating arm. For example, the left operating arm 1043 is used to select a drawing tool, and the right operating arm 1044 is used to draw graphics with the drawing tool.

In a specific embodiment, the doctor at the primary end controls the movement of the left operating arm 1043 to move the position of the cursor to select the desired graphic and the position to be marked, and determines that the selected image is selected via the left control handle 1045 of the left operating arm 1043. The selected graphic is operated by the fingers of the right control handle 1029 of the right operating arm 1044 to perform pinching selection (such as the function of the left mouse button) and drawing operations.

As shown in FIG. 3 and FIG. 7 , in a specific embodiment, the master-end interaction interface G1 provides a drawing tool G2 and a master-end undo button G8. The drawing tool G2 of the master-end interactive interface G1 provides various preset graphics, such as a rectangular preset master-end mark G3 and a circular preset master-end mark G4. Of course, the preset graphics are not limited to this example, and can also be are various other shapes, and the shapes are not limited to regular figures. The drawing tool G2 of the main-end interactive interface G1 can also provide a paintbrush to facilitate user-defined drawing of graphics. In this way, the user can use the mouse to operate the drawing tool, and use the drawing tool G2 to draw graphics on the initial medical image displayed on the main-end interaction interface G1, so as to obtain the main-end mark.

In the example shown in Figure 7, the user can draw a circular main end mark G5, a rectangular main end mark G7, and a user-defined main end mark G5 through the mouse and according to the preset graphics provided by the drawing tool G2. End labeled G6. In addition, the master cancel button G8 is used to provide an operation command for canceling the currently drawn graphic corresponding to the master marker, and the user can cancel the current graphic by clicking the master cancel button G8 with the mouse. Of course, in addition to displaying markers and medical images, the main-end interactive interface G1 is also used to display initial medical images containing markers.

Fig. 5 shows the workflow of master-side tag creation according to a preferred embodiment of the present application. As shown in Figure 5, the workflow of master-side marker creation includes the following steps:

Step A1: Start the main terminal marking command acquisition module 1012; since the main terminal marking command acquisition module 1012 is embedded in the operating program of the doctor's main console 101, after running the doctor's main console 101 program, the main terminal is automatically started Marking function, but the marking mode needs to be activated in order to carry out the marking operation.

Step A2: Determine whether the slave end is in the marking state; in one embodiment, the marking functions of the master end and the slave end are mutually exclusive, that is, at the same time, only one can perform marking operations; if the current slave end 200 is in the marking state, then The master 100 automatically enters step A10 to exit the master marking mode; if the current slave 200 is not in the marking state, the master 100 can enter the master marking mode in step A3. Therefore, the tag sharing device at the master end preferably also acquires the tag status of the slave end, and when the tag status of the slave end is currently in an unmarked state, it starts the tag mode of the master end, and then marks the initial medical image to obtain the tag status of the master end. terminal marking instruction, and preferably in step S4, send the local current marking status corresponding to the master terminal to the slave terminal 200 in real time.

Step A3: The master end enters the master end mark mode; after entering the master end mark mode, execute step A4: the master end 100 sends the local current mark state corresponding to the master end to the slave end 200 in real time, so that the master end 100 is marking The state is sent to the slave end 200, so that the slave end 200 can judge whether it can enter the mark mode according to the mark state of the master end.

After entering the main-end marking mode, step A5 is also performed: judging whether to select a preset marking graphic; in this step A5, the user can select a system-preset graphic to draw the main-end marking, or select a custom graphic to draw a graphic.

Go further to step A6: draw the main-end mark according to the drawing method selected in the previous step A5 to generate a main-end mark instruction; the user can draw the main-end mark on the initial medical image through the left and right control handles.

After completing the drawing of the main-end mark, step A7 can be used to further determine whether to cancel the drawn main-end mark; if yes, then flow to step A8 to delete the latest main-end mark, otherwise the program automatically flows to step A9.

Step A8: Delete the latest main-end mark, preferably only the latest main-end mark can be deleted each time.

Step A9: Determine whether to exit the master-end marking; if yes, exit the marking mode A10, if not, continue to loop the process A5 to A9.

In a preferred embodiment, when the main terminal 100 enters the main terminal marking mode, its main terminal interactive interface G1 automatically displays the preset main terminal marking graphics and the main terminal cancel button on the screen, and when the main terminal 100 exits the main terminal marking mode , the master-end interactive interface G1 automatically hides the preset master-end marker graphics and master-end cancel button on the screen.

In more detail, in a specific mode of operation, as shown in Figure 6, the doctor at the main end triggers a specific pedal B1 to output an instruction to enable the main end marking, thereby entering the main end marking mode A3, after entering the main end marking mode A3 , you can choose two marking methods, including:

The first marking method: the doctor at the main end determines to draw the preset graphic B2; then, the doctor at the main end can select the preset graphic B3 through the left operating arm 1043; after that, the doctor at the main end can draw the preset graphic B4 through the right operating arm 1044;

The second way of marking: the doctor at the main end determines to draw the custom graphic B5; then, the doctor at the main end can draw the custom graphic path B6 through the right operating arm 1044 .

The above marking methods can finally perform the undo operation B7; if the undo is determined, the main end undo button B8 can be selected through the left operating arm 1043 to undo the current main end marking graphics. After undoing, the preset graphics can be cyclically selected and drawn , or you can redraw a custom graphics path. For example, the user uses the left operating arm 1043 to move the cursor to the graphic selection window (ie, the drawing tool G2) of the main-end interactive interface G1, pinches the fingers of the left hand to select a preset graphic, and releases the left hand after selection. Next, the user moves the cursor to the position to be marked through the right operating arm 1044 , then pinches the fingers of the right hand and moves the position to perform the marking operation, and releases the fingers after the marking is completed.

In a specific embodiment, in the state of not selecting a preset marker graphic, the main terminal 100 draws a custom marker graphic by default, uses the right operating arm 1044 to move the cursor to the position to be marked, pinches the fingers to mark and move, releases the After completing a marking operation. In addition, the left operating arm 1043 moves the cursor to the position of the undo button G8 on the main end, and only needs to pinch the fingers to perform the undo operation.

What is provided above is that only one of the master end and the slave end can perform marking, but in other embodiments, the master end and the slave end can also perform marking at the same time and store their respective marking information. Further, when only one of the master end and the slave end can be marked, the surgical robot system can set the mark priority level, such as the master end and the slave end both issue mark instructions at the same time, at this time, determine the object to perform mark according to the priority level , if the marking priority of the master end is higher than that of the slave end, it allows the master end to perform marking operations first.

Fig. 8 shows the flow of the master end sharing the master end mark with the slave end according to a preferred embodiment of the invention. As shown in FIG. 8, in an embodiment, the process of sharing the master-end mark from the master end to the slave end includes the following steps:

Step 1: The main end completes the main end marking to generate the main end marking instruction; specifically, the main end doctor completes the main end marking operation by manipulating the right operating arm 1044 and the left operating arm 1043 and pinching fingers at the main end 100;

Step 2: The main end saves the position information of the main end mark, mainly the coordinate information; the main end mark instruction acquisition module 1012 will save the X and Y coordinate information of all the main end mark points in the screen of the main end interactive interface in real time, and consider The difference between the resolution of the master display screen and the slave display screen, preferably also save the resolution information of the master screen;

Step 3: Send the position information of the master end mark to the slave end; after the master end mark is completed, the master end information sharing module 1014 will send the position information of the master end mark to the slave end 200;

Step 4: The slave end receives the position information of the master mark; after receiving the position information of the master mark, the slave end 200 performs data analysis to obtain the coordinates of the master mark;

Step 5: The slave end reconstructs the master end mark according to the analysis data; during the drawing process, the slave end draws the master end mark proportionally according to the resolution of the slave end display screen;

Step 6: After the master marker is drawn from the slave end, data fusion is finally performed on the master marker, the slave marker and the medical image, and the initial medical image fused with the master marker and the slave marker is acquired and displayed .

Referring to FIG. 4 and FIG. 11 , in a specific embodiment, the slave display module 2025 includes a slave interactive interface H1 capable of displaying initial medical images and marked medical images. Further, the interactive interface H1 of the slave terminal provides a drawing tool H2 (ie, a graph selection window), a mark button H9 and a cancel button H8 of the slave terminal. The drawing tool H2 of the slave-end interactive interface H1 can also provide various preset graphics, such as a rectangular preset slave-end mark H3, and a circular preset slave-end mark H4. Of course, the preset graphics are not limited to this example. Various other shapes are also possible, and the shape is not limited to regular figures. The drawing tool H2 of the slave-side interactive interface H1 can also provide a paintbrush to facilitate user-defined drawing of graphics; the user can operate the drawing tool H2 and utilize the drawing tool H2 to perform an initial medical image display on the slave-side interactive interface H1. Drawing of a graph corresponding to the slave end marker to obtain the slave end marker.

In the example shown in Figure 11, the user can draw a circular slave mark H5, a rectangular slave mark H7, and a slave mark H7 according to the preset graphics provided by the drawing tool H2 of the slave interactive interface H1. The defined slave end is labeled H6. In addition, the slave cancel button H8 is used to provide an operation command for canceling the currently drawn figure corresponding to the slave mark, and the user can cancel the current figure by clicking the slave cancel button H8 with the left mouse button. Of course, in addition to displaying markers and medical images, the slave terminal interaction interface H1 is also used to display initial medical images containing markers.

In some embodiments, the slave terminal interaction interface H1 also provides an exit mark button (not shown), and the exit mark button is used to output an exit mark instruction, and the slave terminal 200 is configured according to the exit mark instruction of the exit mark button. Exit marker mode. In some embodiments, the marking button H9 is used for outputting an enabling marking instruction, and the slave terminal 200 starts the marking mode according to the marking enabling instruction of the marking button H9. In another embodiment, the mark button H9 and the exit mark button are integrated into one button. After entering the slave mark mode, the mark button H9 is transformed into an exit mark button, and after exiting the mark mode, the exit mark button is transformed into Label button H9.

As shown in Figure 4, in an exemplary embodiment, the slave-side mark instruction acquisition module 2022 includes a keyboard 2071 and a mouse 2072; the keyboard 2071 is provided for user key input and text input to facilitate generation of the slave-side mark in text form, thereby realizing Text annotation; the mouse 2072 is for the user to select and draw graphics, so as to facilitate the generation of slave-end marks in graphic form. In a specific embodiment, the slave control device 202 is integrated with the image trolley 203 , and the image trolley 202 can be configured with a partition 211 for placing a keyboard 2071 and a mouse 2072 .

Fig. 9 shows the workflow of slave-side tag creation according to a preferred embodiment of the present application. As shown in Figure 9, in a non-limiting embodiment, the process of creating a slave tag includes the following steps:

Step C1: Start the module for acquiring the slave marking instruction; since the slave marking instruction acquiring module 2022 is embedded in the running program of the surgical robot at the slave end, after running the surgical robot program, the slave marking function will be automatically started, and the follow-up needs to enter Only in the slave marking mode can the slave marking operation be performed.

Step C2: Determine whether the master terminal 100 is in the marking state; in one embodiment, the marking functions of the master and slave terminals are mutually exclusive, and at the same time, only one can perform the marking operation; if the current master terminal 100 is in the marking state, then the slave terminal The 200 automatically enters step C10 to exit the slave marking mode; if the current master 100 is not in the marking state, the slave 200 can enter the slave marking mode in step C3. Therefore, the slave-end control device 202 is preferably able to determine to mark the initial medical image to obtain the slave-end mark according to the non-marking state of the master end, and preferably sends a corresponding message to the master end 100 in step C4. The local current marking state on the slave.

Step C3: enter the slave end mark mode from the end; after entering the slave end mark mode, perform step C4: send the mark state corresponding to the slave end by the slave end 200 to the master end 100, thereby the state that the slave end 200 is marking is sent to the master terminal 100, so that the master terminal 100 can judge whether it can enter the slave terminal marking mode according to the marking status of the slave terminal. For example, the mouse 2072 can be used to click the mark button on the screen of the slave-end interactive interface to enter the slave-end mark mode, preferably after entering the slave-end mark mode, the slave-end interactive interface automatically displays a preset mark graphic and a slave-end cancel button on the screen.

After entering the slave marking mode, step C5 is also performed: judging whether to select the preset marking graphics; in this step C5, the user can select the graphics preset by the system to draw the slave markings, or select custom graphics to draw graphics.

Go further to step C6: draw the slave end mark according to the drawing method selected in the previous step C5; the user can draw the slave end mark on the medical image by moving the mouse.

After completing the drawing of the slave mark, step C7 can be used to further determine whether to cancel the drawn slave mark; if yes, then flow to step C8 to delete the latest slave mark, otherwise the program automatically flows to step C9.

Step C8: delete the latest slave mark, preferably only the latest slave mark can be deleted each time.

Step C9: Judging whether to exit the slave marking; if yes, then exit the marking mode C10, if not, then continue the loop process C5 to C9.

In a preferred embodiment, after the slave terminal 200 enters the slave terminal marking mode, its slave terminal interactive interface automatically displays a preset slave terminal marking graphic and a slave cancel button on the screen, and when the slave terminal 200 exits the slave terminal marking mode After that, the slave terminal interactive interface automatically hides the preset slave mark graphics and slave cancel button on the screen.

In more detail, in a specific operation mode, as shown in FIG. 10 , the slave-end assistant doctor clicks the mark button with the mouse to output the slave-end mark instruction, thereby entering the slave-end mark mode C3, and after entering the slave-end mark mode C3, and Similar to the main end, you can also choose two marking methods, including:

The first marking method: the assistant doctor at the slave end determines to draw the preset graphic D2; then, the assistant doctor at the slave end selects the preset graphic D3 with the left button of the mouse; after that, the assistant doctor at the slave end draws the preset graphic D4 with the left button of the mouse;

The second marking method: the assistant doctor from the terminal determines to draw the custom graphic D5; then, the assistant from the terminal can draw the custom graphic path D6 through the left button of the mouse.

The above marking methods can finally perform the undo operation D7; if the undo is confirmed, you can use the left mouse button to select the slave end undo button to cancel the current slave end marking graphics, and you can cycle through the preset graphics and draw them, or you can re- Draw a custom graphics path.

Therefore, the slave-side assistant doctor clicks the mark button on the slave-side interactive interface with the mouse to enter the slave-end mark mode C3. When drawing the preset mark graph, first move the cursor to the graph selection window, and click the left mouse button to select the preset mark graph , select and release the mouse. Then, use the mouse to move the cursor to the drawing area, click the left button of the mouse to move and draw, and release the mouse after the drawing is completed.

In a specific embodiment, in the state of not selecting a preset mark figure, the slave terminal 200 draws a custom mark figure by default, uses the mouse to move the cursor to the drawing area, clicks the left button of the mouse to perform a moving and drawing operation, and releases the Turn on the mouse. When undoing the operation, use the mouse to move the cursor to the position of the undo button in the drawing area, and click the left mouse button to undo the operation.

Fig. 12 shows a process of sharing a slave tag from a slave to a master according to a preferred embodiment of the present application. As shown in Figure 12, in one embodiment, the process of sharing the slave mark from the slave to the master includes the following steps:

Step 11: the slave end completes the slave end marking to generate a slave end marking instruction; specifically, the slave end assists the doctor to complete the slave end marking operation through the mouse and/or keyboard at the slave end 200;

Step 12: Save the position information of the slave mark from the end, mainly coordinate information; the slave mark instruction acquisition module 2022 will save the X and Y coordinate information of all slave mark points in the screen of the slave end interactive interface in real time, and take into account The difference between the resolutions of the master display screen and the slave display screen, preferably saving the resolution information of the slave screen;

Step 13: Send the position information of the slave mark to the master; after the slave mark is completed, the slave information sharing module 2024 will send the position information of the slave mark to the master 100;

Step 14: The master end receives the position information of the slave end mark; after receiving the position information of the slave end mark, the master end 100 performs data analysis to obtain the coordinates of the slave end mark.

Step 15: The master rebuilds the slave markers based on the analysis data. During the drawing process, the master draws the slave markers proportionally according to the resolution of the master display screen;

Step 16: After the master end draws the slave end mark, finally perform data fusion of the master end mark, the slave end mark and the initial medical image, and obtain all The initial medical image and display it.

More specifically, as shown in FIG. 13, the master terminal 100 obtains the slave-end label information Q1 shared by the slave-end 200, and utilizes the saved master-end label information and the established initial medical image Q2, based on the data fusion technology Q3 To obtain medical images containing markers and display Q4 on the master end, similarly, the slave end 200 acquires the master end marker information Q5 shared by the master end 100, and uses the saved slave end marker information and the established initial medical image The image Q2 is based on the data fusion technology Q3 to obtain the initial medical image containing the marker and display Q6 to the slave.

Therefore, the master end 100 and the slave end 200 respectively perform data fusion on the data acquired by them. The master-slave first obtains medical image data, including but not limited to endoscopic video data and CT images. Then obtain the tag information of the slave end and the tag information of the master end. After the master end 100 and the slave end 200 respectively obtain the position information of these tags, the medical image with the tag information is obtained through a data fusion algorithm. Finally, the master and slave ends respectively The display module displays the corresponding initial medical image containing the marker.

In more detail, as shown in Figure 14, in a specific embodiment, the principle of data fusion is:

After the self-defined master-end mark L2 in the process L1, the preset slave-end mark L4 in the process L3, and the initial medical image in the process L5, after the data fusion in the process L6, the master-end fused mark is obtained The medical image L7 is displayed through the display screen of the master end, and the marked medical image L8 fused by the slave end is also obtained and displayed through the display screen of the slave end. In this way, the doctor at the master end can watch the operation scene remotely through the display screen of the master end and communicate with the assistant doctor at the slave end, and the assistant doctor at the slave end can learn about the surgical guidance of the doctor at the master end in time through the display screen at the slave end in the operating room and communicate with the assistant doctor at the slave end. The host doctor communicates with each other to ensure the smooth and safe operation.

Furthermore, a preferred embodiment of the present application also provides a readable storage medium, on which a program is stored, and when the program is executed, the label sharing performed by the previous label sharing device is performed. method.

And, a preferred embodiment of the present application also provides an electronic device for performing a marker sharing method for a surgical robot, the electronic device includes a processor and the above-mentioned readable storage medium, the processor configured to execute the programs stored on the readable storage medium.

It should also be understood that any tag sharing method described above is also applicable to the tag sharing device provided in this application.

Although the present application discloses as above, it is not limited thereto. Those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

A tag sharing method for surgical robots, applicable to the first surgical robot end, characterized in that it includes:

Obtain the currently generated initial medical image;

After obtaining the first marking instruction input by the user, the first marking information is sent to the second surgical robot in real time, and the first marking information is fused in the initial medical image, wherein the first marking instruction including the first tag information;

After receiving the second marker information sent by the second surgical robot in real time, the second marker information is fused in the initial medical image.
The marker sharing method for surgical robots according to claim 1, wherein after receiving the second marker information sent by the second surgical robot in real time, after fusing the first marker information The second marker information is fused into the initial medical image.
The marking sharing method for a surgical robot according to claim 1 or 2, further comprising: starting a marking mode, the marking mode allows the user to input the First mark command.
The marker sharing method for surgical robots according to claim 1 or 2, wherein fusing the first marker information in the initial medical image comprises:

Reconstructing a first marker corresponding to the first marker information on the initial medical image according to the position information in the first marker information;

Fusing the second marker information in the initial medical image includes:

Reconstructing a second marker corresponding to the second marker information on the initial medical image according to the position information in the second marker information.
The marker sharing method for a surgical robot according to claim 1 or 2, wherein the first marker information is fused in the initial medical image and/or the first marker information is fused in the initial medical image After the second mark information, also include:

Acquiring image change information of the currently generated initial medical image, and updating the first marker information and/or the second marker information according to the image change information of the initial medical image, so that the first marker information and/or the second marker information follows changes in the initial medical image in real time.
The label sharing method for surgical robots according to claim 3, further comprising:

Obtain the flag state of the first surgical robot end and/or the second surgical robot end, and output master-slave control instructions or enable A marking command; the master-slave control command is used to control the movement of the first surgical robot end or the second surgical robot end; the enabling marking command is used to start the marking mode.
The marking sharing method for a surgical robot according to claim 6, further comprising: obtaining an exit marking instruction input by a user, and exiting the marking mode according to the exit marking instruction.
The label sharing method for surgical robots according to claim 7, further comprising:

Entering into an operation lock state according to the activation flag instruction, so as to prohibit the movement of the first surgical robot end and/or the second surgical robot end;

An operation unlock state is commanded according to the exit flag to allow movement of the first surgical robot end and/or the second surgical robot end.
The label sharing method for surgical robots according to claim 3, further comprising:

Acquiring the marking state of the second surgical robot, and starting the marking mode when it is determined that the second surgical robot is currently in a non-marking state, and sending the local current marking state to the second surgical robot in real time.
The label sharing method for surgical robots according to claim 3, further comprising:

Acquiring the operating state of the first foot pedal at the first surgical robot end, and starting the marking mode according to the operating state of the first foot pedal;

Obtaining the operating state of the second foot pedal on the first surgical robot end, and exiting the marking mode according to the operating state of the second foot pedal; or, further comprising:

Starting the marking mode according to an instruction output by a marking button on the interactive interface of the first surgical robot;

Exit the marking mode according to an instruction output by the exit marking button on the interaction interface of the first surgical robot.
A marker sharing device for a surgical robot, suitable for the first surgical robot end, characterized in that it includes:

An image data acquisition module, configured to acquire the currently generated initial medical image;

A marking instruction acquisition module, configured to acquire a first marking instruction input by a user, the first marking instruction including first marking information, and also used to acquire second marking information sent in real time by the second surgical robot end;

A data fusion module, configured to fuse the first marker information and/or the second marker information in the initial medical image.
A surgical robot system, comprising a first surgical robot end and a second surgical robot end, wherein the first surgical robot end comprises the marker sharing device for surgical robots according to claim 11 .
The surgical robot system according to claim 12, wherein the first surgical robot end further includes a first foot pedal and a second foot pedal, and the marking instruction acquisition module includes a first operating arm and a second operating arm. operating arm;

The first foot pedal is used to output an enable flag command to start the flag mode;

The second foot pedal is used to output an exit marking command to exit the marking mode;

The first operating arm is used to select a first marker corresponding to the first marker information according to an operation instruction input by a user;

The second operating arm is configured to adopt the selected first marker according to an operation instruction input by a user, and create the first marker on the initial medical image.
The surgical robot system according to claim 12, wherein the first surgical robot terminal further includes an interactive interface, the interactive interface includes a marking button and an exit marking button; the marking instruction acquisition module includes a keyboard and a mouse;

The interactive interface displays the initial medical image;

The mark button is used to output the enable mark instruction to start the mark mode;

The exit marking button is used to output an exit marking instruction to exit the marking mode;

The keyboard and the mouse are used to generate a first mark corresponding to the first mark information according to an instruction input by a user.
A readable storage medium, on which a program is stored, and when the program is executed, the marker sharing method for a surgical robot according to any one of claims 1-10 is executed.
An electronic device for performing a marker sharing method for a surgical robot, wherein the electronic device includes a processor and a readable storage medium as claimed in claim 15, the processor is configured to execute The program stored on the readable storage medium.