WO2023065988A1

WO2023065988A1 - Collision detection method and apparatus, device, and readable storage medium

Info

Publication number: WO2023065988A1
Application number: PCT/CN2022/121629
Authority: WO
Inventors: 苗燕楠; 彭晓宁; 李自汉; 江磊; 王家寅
Original assignee: 上海微创医疗机器人（集团）股份有限公司
Priority date: 2021-10-21
Filing date: 2022-09-27
Publication date: 2023-04-27

Abstract

A collision detection method and apparatus, a device, and a readable storage medium. The method comprises: acquiring a current medical image (S703); acquiring position information of a target part of a medical instrument (S704); acquiring a standard medical image obtained by means of pre-processing, the standard medical image carrying position information of a target object (S705); according to a first matching relationship between the current medical image and the standard medical image, converting the position information of the target object in the standard medical image into a target medical image, and according to a second matching relationship between the current medical image and the position information of the target part of the medical instrument, converting the position information of the target part of the medical instrument into the target medical image (S706); and performing collision detection according to the position information of the target object and the position information of the target part of the medical instrument in the target medical image (S707). This can improve detection accuracy, and is more intelligent.

Description

Collision detection method, device, equipment, readable storage medium

technical field

The present application relates to the field of intelligent medical technology, in particular to a collision detection method, device, equipment, and a readable storage medium.

Background technique

With the continuous development of robots, more and more robots are used in the field of surgery. Although the traditional rigid mechanism robot has been widely used in various operations in the medical field, its adaptability, safety and flexibility are relatively poor, and it is easy to cause damage to the internal tissues of the human body during the operation. Collision detection technology has a wide range of applications in the field of robotics. Existing collision detection technologies mainly include two types:

1) The OBB bounding box is used to detect the collision between the robot arm and the arm, and between the joints of the robotic arm. This collision detection method is mainly based on the establishment of a bounding box for the robotic arm, and is realized by detecting the collision between the bounding boxes.

2) External force collision detection using six-dimensional torque sensing or joint torque sensing at the end of the robot. This collision detection method is based on the torque information fed back by the torque sensor, and calculates the external force on the end of the robot so that the external force generated when the collision occurs can be detected. Variety.

However, OBB bounding box collision detection is mainly aimed at collision detection between rigid body structures with regular shapes of convex polyhedrons, and is not suitable for collision detection between instrument ends and soft tissues. The six-dimensional torque sensor or joint torque sensor is bulky and generally difficult to install at the end of the device. It is often used in the collision detection between collaborative robots and the external environment, and it is not suitable for the collision detection of narrow lumens in the body.

In summary, existing collision detection techniques cannot be effectively used for collision detection between microsurgical instruments and soft tissues, especially invisible blood vessels, nerves, and sensitive tissues behind endoscopic images.

Contents of the invention

Based on this, it is necessary to address the above technical problems and provide a collision detection method, device, equipment, and readable storage medium that can expand the scope of application of collision detection and detect collisions between microsurgical instruments and soft tissues.

In a first aspect, the present application provides a collision detection method, the method comprising: acquiring at least two image information of a target object under different viewing angles; obtaining the spatial position of the target object according to the at least two image information; Obtaining position information of the medical instrument connected to the end of the instrument arm of the surgical robot; determining a collision between the medical instrument and the target object according to the spatial position of the target object and the position information of the medical instrument.

In the second aspect, the present application provides another collision detection method, the method comprising: obtaining the current medical image; obtaining the position information of the target part of the medical device; obtaining a pre-processed standard medical image, the standard medical image carrying The position information of the target object; according to the first matching relationship between the current medical image and the standard medical image, the position information of the target object in the standard medical image is converted into the target medical image, and according to the current medical image and the The second matching relationship of the position information of the target part of the medical device is used to convert the position information of the target part of the medical device into the target medical image; according to the position information of the target object in the target medical image and the Collision detection is performed based on the position information of the target part of the medical device.

In a third aspect, the present application provides a collision detection system. The collision detection system includes a processor, a medical mirror, and a medical device. The medical device is provided with a sensor, and the sensor is used to collect the location information, and send the location information of the target part of the collected medical device to the processor; the medical mirror is used to collect the current medical image, and send the current medical image to the processor; the The processor is used to execute the above collision detection method.

In a fourth aspect, the present application provides a collision detection device, which includes: a current medical image acquisition module, used to acquire the current medical image; a position information acquisition module, used to acquire the position information of the target part of the medical device; The image acquisition module is used to acquire the pre-processed standard medical image, the standard medical image carries the position information of the target object; the matching relationship calculation module is used to calculate the current medical image and the first one of the standard medical image A matching relationship, calculating a second matching relationship between the current medical image and the position information of the target part of the medical device; a conversion module, configured to convert the position information of the target object in the standard medical image according to the first matching relationship Converting to the target medical image, converting the position information of the target part of the medical device into the target medical image according to the second matching relationship; a collision detection module, configured to Collision detection is performed on the position information of the object and the position information of the target part of the medical instrument.

In a fifth aspect, the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the method involved in any of the above-mentioned embodiments are implemented .

In a sixth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method involved in any one of the above-mentioned embodiments are implemented.

The above-mentioned collision detection method, device, equipment, and readable storage medium carry the position information of the target object in the standard medical image obtained by preprocessing, so that the target object in the standard medical image is The position information of the object is converted into the target medical image, and the position information of the target part of the medical device is converted into the target medical image according to the second matching relationship between the current medical image and the position information of the target part of the medical device, so that in the target Collision detection in medical images expands the scope of application of collision detection and is more intelligent, capable of detecting collisions between microsurgical instruments and soft tissues.

Description of drawings

Fig. 1 is a system diagram of a collision detection system in an embodiment;

Fig. 2 is a schematic diagram of the application scenario of the surgical robot system involved in the present application;

FIG. 3 is a schematic diagram of a master device according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a slave device according to an embodiment of the present application;

Figure 5a is a schematic diagram of a surgical instrument according to an embodiment of the present application;

Figure 5b is an enlarged view of part A of Figure 5a;

Fig. 6 is a schematic diagram of an operation field image of an embodiment of the present application;

7 is a flowchart of a collision detection method for a surgical robot according to an embodiment of the present application;

Fig. 8 is a flow chart of the marking process of feature points in the embodiment of the present application;

Fig. 9 is a schematic diagram of the binocular vision of the embodiment of the present application;

Fig. 10 is a calculation principle diagram of binocular vision in the embodiment of the present application;

Fig. 11 is a schematic diagram of master-slave operation control of a robot of the present application;

Fig. 12 is a schematic flow chart of a collision detection method in an embodiment;

Fig. 13 is a schematic diagram of a scene of standard medical image acquisition in an embodiment;

Fig. 14 is an embodiment of an endoscopic image, a preoperative medical image and an endoscopic image, and the standard medical image is a fusion flow chart of the preoperative medical image;

Fig. 15 is a schematic diagram of a collision between a medical device and a target object in one embodiment;

Fig. 16 is a schematic diagram of the spatial position of the medical device in the target operation area in one embodiment;

Fig. 17a is a schematic diagram of the spatial relationship between the surgical instrument and the tissue according to the embodiment of the present application;

Figure 17b is an enlarged view of part B of Figure 17a;

Fig. 18 is a flow chart of the collision detection of the embodiment of the present application;

Fig. 19 is a schematic diagram of a collision safety visual warning according to an embodiment of the present application;

Fig. 20 is a schematic diagram of a sound and light warning for collision safety in an embodiment of the present application;

Fig. 21 is a schematic diagram of a ray collision detection method in an embodiment;

Fig. 22 is a flowchart of a ray collision detection method in an embodiment;

Fig. 23 is a schematic diagram of a convex polygon collision detection method in an embodiment;

Fig. 24 is a flowchart of a convex polygon collision detection method in an embodiment;

Fig. 25 is a schematic diagram of a linear projection collision detection method in an embodiment;

Fig. 26 is a schematic diagram of a sound warning for collision safety in an embodiment;

Fig. 27 is a structural block diagram of a collision detection device in an embodiment;

Figure 28 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

As used in this application, the singular forms "a", "an" and "the" include plural objects, the term "or" is generally used in the sense of including "and/or", and the term "several" Usually, the term "at least one" is used in the meaning of "at least one", and the term "at least two" is usually used in the meaning of "two or more". In addition, the terms "first", "second "Two" and "third" are used for descriptive purposes only, and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or at least two of these features, the term "proximal end" is usually the end close to the operator, the term The "distal end" is usually the end close to the patient, that is, the end close to the lesion. "One end" and "the other end" as well as "proximal end" and "distal end" usually refer to the corresponding two parts, which not only include the end point, the term "installation" , "connected" and "connected" should be understood in a broad sense. For example, it can be fixed connection, detachable connection, or integrated; it can be mechanical connection or electrical connection; it can be direct connection or through The intermediary is indirectly connected, which can be the internal communication of two elements or the interaction relationship between two elements. In addition, as used in this application, an element is arranged on another element, usually only means that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the relationship between the two elements can be direct or indirect through an intermediate element. connection, coupling, fit or transmission, but cannot be understood as indicating or implying the positional information relationship between two elements, that is, one element can be in any orientation such as inside, outside, above, below or on one side of another element, unless the content Also clearly point out. Those of ordinary skill in the art can understand the specific meanings of the above terms in this application according to specific situations.

The collision detection method provided in the embodiment of the present application may be applied to the collision detection system shown in FIG. 1 . The collision detection system includes a processor, a medical mirror, and a medical device. The medical device is equipped with a sensor, and the sensor is used to collect the position information of the target part of the medical device, and send the collected position information of the target part of the medical device to the processor. ; The medical mirror is used to collect the current medical image and send the current medical image to the processor; the processor is used to execute the collision detection method below.

Figure 2 shows an application scenario of a surgical robot system, which includes a master-slave teleoperated surgical robot, that is, the surgical robot system includes a master device 100 (ie, a doctor-side control device), a slave device 200 (ie, the patient-side control device), a main controller, and a supporting device 400 (eg, an operating bed) for supporting the surgical object for surgery. It should be noted that, in some embodiments, the supporting device 400 may also be replaced by other surgical operation platforms, which is not limited in the present application.

Please refer to FIG. 3 , the master device 100 is an operation end of a teleoperated surgical robot, and includes a master operator 101 installed thereon. The main operating hand 101 is used to receive the operator's hand movement information, which can be input as the movement control signal of the whole system. Optionally, the master controller is also disposed on the master device 100 . Preferably, the master device 100 further includes an imaging device 102, the imaging device 102 can provide a stereoscopic image for the operator, and provide an image of the surgical field for the operator to perform a surgical operation. The image of the surgical field includes the type and quantity of surgical instruments, their posture in the abdomen, the shape and arrangement of blood vessels in the patient's organs and tissues, and the surrounding organs and tissues. Optionally, the master device 100 also includes a foot-operated surgical control device 103 , through which the operator can complete input of relevant operation instructions such as electrocution and electrocoagulation.

Please refer to FIG. 4 , the slave device 200 is a specific execution platform of a teleoperated surgical robot, and includes a base 201 and a surgical execution component installed thereon. The operation execution assembly includes an instrument arm 210 and instruments, and the instruments are hung or connected to the end of the instrument arm 210 . Further, the instruments include surgical instruments 221 (such as high-frequency electric knife, etc.) for specific surgical operations and endoscopes 222 for auxiliary observation; correspondingly, the instruments for connecting or mounting the endoscope 222 Arm 210 may be referred to as a mirror arm.

In one embodiment, the instrument arm 210 includes an adjustment arm and a working arm. The tool arm is a mechanical fixed point mechanism, which is used to drive the instrument to move around the mechanical fixed point, so as to perform minimally invasive surgical treatment or photographing on the patient 410 on the supporting device 400 . The adjustment arm is used to adjust the pose of the mechanical fixed point in the working space. In another embodiment, the instrument arm 210 is a spatially configured mechanism with at least six degrees of freedom, which is used to drive the surgical instrument 221 to move around an active fixed point under program control. The surgical instrument 221 is used to perform specific surgical operations, such as clamping, cutting, scissors and other operations, please refer to FIG. 5a and FIG. Agency 242. The surgical instrument 221 can telescopically move along the axial direction of the instrument shaft 241; it can perform autorotation around the circumference of the instrument shaft 241, that is, form an autorotation joint 251; the operating mechanism 242 can perform pitching, yaw, and opening and closing movements, Pitch joints 252, yaw joints 253, and opening and closing joints 254 are respectively formed so as to realize various applications in surgical operations. It should be noted that since the surgical instrument 221 and the endoscope 222 have a certain volume in practice, the above-mentioned "fixed point" should be understood as a fixed area. Of course, those skilled in the art can understand the "fixed point" according to the prior art.

The master controller is connected to the master device 100 and the slave device 200 respectively, and is used to control the movement of the operation execution component according to the movement of the master operator 101. Specifically, the master controller includes a master-slave mapping module, the The master-slave mapping module is used to obtain the terminal pose of the master operator 101 and the predetermined master-slave mapping relationship to obtain the desired terminal pose of the surgical actuator, and then control the instrument arm 210 to drive the instrument to the desired terminal pose . Further, the master-slave mapping module is also used to receive instrument function operation instructions (such as electric cutting, electrocoagulation and other related operation instructions), and control the energy driver of the surgical instrument 221 to release energy to implement electric cutting, electrocoagulation and other surgical operations. In some embodiments, the main controller also accepts the force information received by the surgical execution component (for example, the force information of the human tissue and organ on the surgical instrument), and feeds back the force information received by the surgical execution component to the main operator 101 , so that the operator can feel the feedback force of the surgical operation more intuitively.

Further, the medical robot system also includes an image trolley 300 . The image trolley 300 includes: an image processing unit (not shown) communicated with the endoscope 222 . The endoscope 222 is used to acquire intracavity (referring to the patient's body cavity) surgical field images. The image processing unit is used to perform image processing on the image of the operative field acquired by the endoscope 222 and transmit it to the imaging device 102 so that the operator can observe the image of the operative field. Optionally, the image trolley 300 further includes a display device 302 . The display device 302 is connected in communication with the image processing unit, and is used to provide an auxiliary operator (such as a nurse) with real-time display of an operation field image or other auxiliary display information.

Optionally, in some surgical application scenarios, the surgical robot system also includes auxiliary components such as a ventilator, an anesthesia machine 500 and an instrument table 600 for use in surgery. Those skilled in the art can select and configure these auxiliary components according to the prior art, and no further description will be given here.

Please refer to FIG. 6 , which shows a surgical operation space scene. In an example, 3 to 4 surgical holes can be drilled on the body surface of the patient 410, and a poking card with a through hole can be installed and fixed. The surgical instrument 221 and the inner The speculum 222 respectively enters the surgical operation space in the body through the poking holes.

During normal surgical operations, the operator (for example, the main operating doctor) controls the end pose of the surgical instrument 221 through master-slave teleoperation under the guidance of the surgical field image. During the operation, the operator sits in front of the main device 100 located outside the sterile field, observes the image of the surgical field returned by the imaging device 102, and controls the movement of the surgical execution component by operating the main operating hand 101 to complete various operations. Surgical operation. During the operation, the posture of the operation hole will remain still (i.e. form a fixed point) to avoid crushing and damaging the surrounding tissues. Under the guidance of the operation field image taken by the endoscope 222, the operator The instrument 221 cuts, cuts, electrocoagulates, and sutures the lesion tissue to complete the predetermined surgical goal. In the surgical space, as shown in Figure 6, a plurality of surgical instruments 221 and endoscopes 222 penetrate into the narrow space of the human body through poking holes; the endoscope 222 can provide real-time feedback of the surgical instruments 221 and tissue image information. During the operation, the surgical instrument 221 is likely to collide with vulnerable key tissue parts (ie, target objects), such as arteries, heart valves, and the like.

In order to solve the collision problem between the surgical instrument 221 and the target object, please refer to FIG. 7. This embodiment provides a collision detection method for a surgical robot, which includes:

Step S1: Obtain at least two image information of the target object in the surgical environment under different viewing angles;

Step S2: Obtain the spatial position of the target object according to the at least two image information;

Step S3: Obtain the position information of the surgical instrument connected to the end of the instrument arm of the surgical robot;

Step S4: Determine the collision between the surgical instrument and the target object according to the spatial position of the target object and the position information of the surgical instrument.

Optionally, the surgical robot system includes at least two image acquisition units and a collision processing unit, the at least two image acquisition units are used to acquire at least two image information from different viewing angles, the instrument arm 210 and at least two image acquisition units The units are respectively connected in communication with the collision processing unit; the collision processing unit is used to execute the above-mentioned collision detection method of the surgical robot. For example, the collision processing unit may be set on the slave device 200, or may be set on the master device 100, or be set independently. The present application makes no particular limitation on the specific location of the collision processing unit.

With such a configuration, based on at least two image information from different viewing angles and the acquired position information of the surgical instrument 221, the collision detection between the surgical instrument 221 and the target object can be realized, which can effectively improve the safety of the surgical operation and avoid accidentally injuring surrounding normal tissues , blood vessels and nerves to ensure the safety of surgical operations. Furthermore, due to the use of visual processing technology, the demand for sensing equipment is reduced, and the structure of the system is simplified.

Optionally, the target object has feature points, and the feature points are determined by modeling based on medical images. Please refer to FIG. 13 , in an alternative example, the medical image can be obtained by scanning the patient with a medical image scanning device (such as CT or MRI, etc.) before the operation. In an optional embodiment, after obtaining the medical images, image processing algorithms are used to complete the tissue modeling in the operation space and complete the three-dimensional reconstruction of the operation scene. According to the situation in the abdominal cavity, the operator can determine the outline of the key tissues that need special attention, and determine and mark the feature points before the operation.

Further, please refer to FIG. 8, in step S1, the marking process of feature points may include:

Step S11: Obtain medical images of organs and tissues in the operating space; use endoscope, CT or MRI and other medical image scanning devices to scan and obtain medical images of organs and tissues in the operating space in the preoperative preparation stage;

Step S12: Tissue modeling and image calibration; establish a tissue model through a visual processing algorithm to obtain spatial position information of organs and tissues, and further complete image calibration through specific calibration points;

Step S13: Mark the feature points of key tissues; identify the important key tissue parts that need special attention through the machine learning algorithm or the doctor, so as to determine the outline and feature points of the target object that needs to be detected for collision, and mark them for intraoperative The position of the feature point of the target object can be updated in real time. In one application scenario, for the organs and tissues in the surgical space, special attention needs to be paid to vulnerable target parts, such as arteries, heart valves, etc. For general tissues, even if they collide with surgical instruments, they will not cause harm , this part of the organization can be excluded from the target object without special attention.

In step S2, the step of obtaining the spatial position of the target object according to the at least two image information includes: establishing a real-time three-dimensional model of the target object according to the at least two image information, and communicating with the tissue After the model is registered, the real-time position information of the feature points in the coordinate system of the image acquisition device is obtained in real time, so as to obtain the spatial position of the target object.

Specifically, according to the at least two image information, the principle of establishing a real-time three-dimensional model of the target object may be based on the principle of binocular vision. The principle of binocular vision will be described below with reference to FIGS. 9 and 10 .

As shown in FIG. 9, the images P1 and P2 presented by two image acquisition units (respectively the first image acquisition unit 701 and the second image acquisition unit 702 in FIG. 9) to the same object P (in FIG. There is a difference between the first image 711 and the second image 712 ), which is also called "parallax". The greater the parallax, the smaller the depth that can be detected; on the contrary, the smaller the parallax, the greater the depth that can be detected. The size of the parallax corresponds to the distance between the object and the two image acquisition units.

Further, as shown in FIG. 10 , the distance between the optical centers of the two image acquisition units, that is, the baseline is denoted as b, and the focal lengths of the two image acquisition units are both f. The two image acquisition units observe the same point P(x, y, z) of the measured object at the same time, the first image 711 and the second image collected by the first image acquisition unit 701 and the second image acquisition unit 702 712. The images of point P are respectively P ₁ and P ₂ , where the coordinates of P ₁ are (x _l , y _l ), and the coordinates of P ₂ are (x _r +b, y _r ). According to the principle of similar triangles, it can be obtained as follows Relational formula:

From formula (1), the following relationship can be obtained:

According to the above formulas (2)-(4), the three-dimensional coordinate information of the point P on the measured object in the coordinate system of the image acquisition device can be obtained. Similarly, according to the above formulas (2)-(4), the three-dimensional coordinate information of any feature point on the measured object in the coordinate system of the binocular image acquisition device can be obtained, and then the three-dimensional model of the measured object can be constructed. After obtaining the real-time 3D model of the target object, it can be registered with the preoperative tissue model according to the common registration method in the field, so as to obtain the real-time position information of the feature points in the coordinate system of the image acquisition device , so that the spatial position of the target object can be obtained.

Optionally, in some embodiments, at least two of the image acquisition units are disposed on the endoscope 222, that is, the endoscope 222 is a 3D endoscope with at least two cameras. At this time, the coordinate system of the image acquisition device is the endoscope coordinate system. In some other embodiments, two 2D endoscopes with monocular cameras can be used as image acquisition units respectively. Of course, in some other embodiments, the image acquisition unit may also be independent from the endoscope 222, which is not limited in this embodiment.

In another embodiment, as shown in FIG. 11 , during normal surgical operations, the operator controls the position and posture of the end of the instrument through master-slave teleoperation under the guidance of the endoscopic image. The position of the end of the instrument includes the translational movement of the instrument along the X, Y, and Z directions, and the attitude includes the pitch, yaw, and rotation of the end of the instrument.

The present application also provides a collision detection method, which is illustrated in FIG. 12 by taking the method applied to the processor in FIG. 1 as an example, including the following steps:

S703: Acquire the current medical image.

Specifically, the current medical image is a real-time medical image collected by the medical mirror, which is collected by an image sensor of the medical mirror after the medical mirror is inserted into a target operation position.

S704: Obtain position information of the target part of the medical device.

Specifically, a medical device refers to a surgical operation device, and its target site may refer to an end of the surgical operation device. Wherein, the acquisition of the position information of the target part of the medical device may be obtained through kinematics calculation. For example, during master-slave surgery, based on the Cartesian end position and velocity of the master end, the response command Cartesian position and command velocity of the slave end instrument in the robot coordinate system are calculated.

S705: Acquire a pre-processed standard medical image, where the standard medical image carries position information of the target object.

S706: Convert the position information of the target object in the standard medical image into the target medical image according to the first matching relationship between the current medical image and the standard medical image, and according to the second matching relationship between the current medical image and the position information of the target part of the medical device The position information of the target part of the medical device is converted into the target medical image.

S707: Perform collision detection according to the position information of the target object in the target medical image and the position information of the target part of the medical device.

Specifically, the first matching relationship is the matching relationship between the current medical image and the standard medical image, which mainly completes the matching between the current medical image and the standard medical image by adapting the positions of key tissue labeling points in the current medical image. The second matching relationship is the matching between the current medical image and the position information of the target part of the medical device, which is mainly established through the derivation of robot kinematics. Specifically, the movement position information of the instrument arm and the endoscope arm, including the joints The position and speed information of the robot, through the kinematics of the robot and the kinematics mapping relationship, calculate the second matching relationship between the pre-medical image and the position information of the target part of the medical device.

In one of the embodiments, before converting the position information of the target object in the standard medical image into the target medical image according to the first matching relationship between the current medical image and the standard medical image, it also includes: identifying the target object in the current medical image Processing objects: matching and fusing the target object in the standard medical image with the object to be processed to obtain a first matching relationship, the first matching relationship is used to convert the position information of the target object in the standard medical image to the target medical image.

In one of the embodiments, before converting the position information of the target part of the medical device into the target medical image according to the second matching relationship between the current medical image and the position information of the target part of the medical device, it further includes: The motion information of the medical mirror of the medical image and the motion information of the target part of the medical device are calculated by kinematic mapping to obtain the second matching relationship, and the second matching relationship is used to convert the position information of the target part of the medical device into the target medical image .

Specifically, as shown in FIG. 14 , in practical applications, the current medical image is an endoscopic image, the standard medical image is a preoperative medical image, and the position information of the target part of the medical device is the position information of the end of the medical device. The first matching relationship is to obtain the preoperative medical image of the surgical space tissue first, then identify the tissue location information, and reconstruct the intra-abdominal 3D model according to the tissue location information, and combine the intra-abdominal 3D model with the intraoperative 3D endoscopic image The first matching relationship can be obtained by performing fusion.

The acquisition of the second matching relationship is as follows: first obtain the movement position information of the mechanical arm and the endoscope arm, and map the position of the end of the instrument according to the forward kinematics of the robot to obtain the position of the instrument in the endoscope image coordinate system Position information, so as to realize the fusion of 3D endoscopic image and robot coordinate system.

Specifically, the target medical image can be regarded as a fused image, which can be obtained by fusing target objects and medical instruments in the standard medical image into the current medical image. Or it is obtained by fusing the target objects and medical instruments in the standard medical image into a new space, and the space where the target medical image is located is not limited here.

Specifically, after obtaining the first matching relationship and the second matching relationship, according to the first matching relationship and the second matching relationship, the position information of the target object in the standard medical image and the position information of the target part of the medical device are converted into the same space, such as the target medical image space, so that the position information of the target object in the standard medical image and the position information of the target part of the medical device are comparable.

In the above-mentioned embodiment, a three-dimensional intra-abdominal model is first established through preoperative medical images, and the spatial position information of key tissues is marked, and then the positions of key tissue marking points under the 3D endoscopic image are adapted to complete the registration of the two coordinate systems Fusion realizes the fusion of preoperative medical images and intraoperative 3D endoscopic images. Then, the motion position information of the instrument arm and the endoscope arm, including the position and speed information of each joint, is calculated through the robot kinematics and kinematics mapping relationship to obtain the position system of the instrument endoscope image coordinate system, realizing intraoperative Fusion of 3D endoscopic images with robot coordinate system. Based on the above two steps, the fusion of preoperative medical images, intraoperative 3D endoscopic images, and robot coordinate systems is realized, and the three-dimensional spatial position of surgical instruments in the surgical area is calculated in real time.

Specifically, in the target medical image, the distance between the position information of the target object and the position information of the target part of the medical device is calculated to determine whether a collision occurs, and if not, continue the detection for a next cycle.

Specifically, as shown in Figures 15 and 16, Figure 15 is a schematic diagram of a collision between a medical instrument and a target object in an embodiment, and Figure 16 is a schematic diagram of the spatial position of a medical instrument in a target operating area in an embodiment, in which In the embodiment, during the operation, the doctor sees the real-time scene of the operation area through the endoscopic image. Within the field of view of the endoscope, the doctor can see the surgical lesion tissue (prostate as an example), and can also see some sensitive tissues and vascular tissues, but there are also other sensitive tissues and vascular and nerve tissues that are not within the field of view of the endoscope image. These sensitive tissues are easily injured if the instrument moves beyond the field of view of the endoscope. By fusing the three-dimensional intra-abdominal model established by preoperative medical images, endoscopic visual images, and instrument positions, it is possible not only to determine the relative spatial position of surgical instruments relative to lesions and sensitive tissues within the field of view, but also to determine The positional relationship of surgical instruments relative to sensitive tissues and peripheral nerves and blood vessels that doctors cannot see through the endoscope.

With reference to FIG. 15 , in this embodiment, collisions between surgical instruments moving beyond the field of view of the endoscope and invisible tissues and blood vessels can be effectively identified. Based on the spatial position of the surgical instrument in the surgical operation scene, the collision between the surgical instrument and sensitive tissues, blood vessels and nerves, etc. is detected in real time. When the spatial position of the instrument overlaps with the spatial position of the tissue, a collision warning is issued and the position of the collision point in the abdominal cavity is fed back.

In the above collision detection method, the pre-processed standard medical image carries the position information of the target object, so that the first matching relationship between the current medical image and the standard medical image is calculated, and the relationship between the current medical image and the position information of the target part of the medical device is calculated. The second matching relationship: converting the position information of the target object in the standard medical image into the target medical image according to the first matching relationship, and converting the position information of the target part of the medical device into the target medical image according to the second matching relationship, In this way, the collision detection in the target medical image can improve the detection accuracy and be more intelligent.

In one embodiment, the collision detection is performed according to the position information of the target object in the target medical image and the position information of the target part of the medical device, including at least one of the following: according to the position information of the target object in the target medical image and the position information of the medical device The position information of the target part is based on the ray collision detection method for collision detection; according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on the convex polygon collision detection method; according to the target medical image The position information of the target object and the position information of the target part of the medical device are subjected to collision detection based on a linear projection collision detection method.

The steps of acquiring the spatial position of the target object and the position information of the surgical instrument 221 will be described below with reference to FIG. 17a and FIG. 17b through an example. In the example shown in FIG. 17b, the target object includes four feature points O1, O2, O3, and O4. In this surgical application scenario, the slave device 200 includes three instrument arms 210 (respectively instrument arms 210-1, instrument arm 210-2 and instrument arm 210-3), two surgical instruments 221 (surgical instrument 221a and surgical instrument 221b respectively), and an endoscope 222 with two image acquisition units. The surgical instrument 221a has two instrument marking points T1 and T2, and the surgical instrument 221b has two instrument marking points T3 and T4. The surgical instrument 221a is mounted and connected to the instrument arm 210-1, and the surgical instrument 221b is mounted and connected to the instrument On the arm 210-2, the endoscope 222 is mounted and connected to the instrument arm 210-3. At this time, the instrument arm 210-3 is also called the mirror arm.

During the operation, the image information of the tissue in the operation space can be obtained in real time by the endoscope 222, and updated in real time by the image processing unit, and the position information of each feature point in the endoscope coordinate system {Oe} can be obtained;

The endoscope 222 and the

surgical instruments

221a and 221b are installed on the instrument arm 210, the base coordinate system of the base 201 of the slave device 200 is set to {Ob}, and the instrument arm base coordinate system of the instrument arm 210-1 is {Ob1} , the instrument arm base coordinate system of the instrument arm 210-2 is {Ob2}, and the instrument arm base coordinate system of the instrument arm 210-3 (ie, the scope arm) is {Ob3}.

Calculate the pose information of the endoscope 222 relative to the base coordinate system {Ob3} of the arm, that is, the position of Oe in the base coordinate system {Ob3} of the arm

Furthermore, the position information of the feature point O1 in the base coordinate system {Ob} can be obtained from the kinematic relationship

in,

is the pose information of the arm base coordinate system {Ob3} relative to the base coordinate system {Ob},

is the pose information of the feature point O1 in the endoscope coordinate system {Oe}. From this calculation, the position information of each instrument marker point of the surgical instrument 221 relative to the respective base coordinate system is obtained, that is, the position of the instrument marker points T1 and T2 in the instrument arm base coordinate system {Ob1}

The positions of instrument markers T3 and T4 in the instrument arm base coordinate system {Ob2}

Furthermore, the current position information of each instrument marker point in the base coordinate system {Ob} can be obtained from the kinematic relationship:

Furthermore, the current pose information of each surgical instrument 221 in the base coordinate system {Ob} can be obtained.

The motion information of the marker points of the instrument includes velocity information, acceleration information, direction information, and the like. The dotted line uses speed information as an example for illustration. Velocity information of instrument marking points can be obtained by differential kinematics:

Among them, J is the Jacobian matrix of the equipment markers relative to the base coordinate system of the equipment arm;

Indicates the influence matrix of joint i on the linear velocity of the instrument marker point;

Represents the influence matrix of joint i on the angular velocity of the instrument marker point;

Indicates the joint velocity of each joint in the instrument arm; v _e indicates the velocity of the instrument marker.

As in the above formula (7) and formula (8), the possible expected position information of the device marker point in the subsequent certain period of time can be obtained by integration:

Among them, p ₀ represents the current position of the instrument marking point;

Indicates the expected position of the instrument marker after time t _n elapses.

From this, the expected position information of the surgical instrument can be obtained, so as to determine the collision situation.

Since the position information of the surgical instrument 221 includes current position information and expected position information, correspondingly, the collision situation includes the current collision situation and the expected collision situation. By judging whether the position information of the feature point is within the enveloping area or the expected enveloping area of the instrument marker point, the current collision situation and the expected collision situation between the target object and the surgical instrument 221 can be obtained.

Please refer to FIG. 18 , in an example, the step of determining the collision situation in step S4 includes:

Step S41: Take the feature point O _o as the center of the sphere, the position of O _o is (xo ₀ , y _o0 , z _o0 ), and establish a sphere Co with Ro as the radius:

C _o : (xx _o0 ) ² +(yy _o0 ) ² +(zz _o0 ) ² ＝R _o ² (10)

Step S42: Take the instrument marking point T _o on the surgical instrument as the center of the sphere, the position of T _o is (x _t0 , y _t0 , z _t0 ), and use Rt as the radius to establish a sphere Ct:

C _t : (xx _t0 ) ² +(yy _t0 ) ² +(zz _t0 ) ² ＝R _t ² (11)

Step S43: If the distance D between the sphere Co and the sphere Ct<Ro+Rt, then mark the feature point in contact with the instrument marker point, which means that the surgical instrument 221 has collided with the target object; otherwise, mark the The feature point is not in contact with the device marker point. See the following formula (12) and formula (13) for details:

D<(R _o +R _t ) (13)

Furthermore, for a certain target object, there may be multiple feature points, and for this case, the collision rule for the target object can be formulated according to the actual setting of the threshold P. In one example, the target object includes M feature points, wherein N feature points are in contact with the instrument marker point, N is a natural number, M is a natural number not less than N, if the ratio of N to M is greater than Threshold P, P∈(0,1], then it is determined that the surgical instrument 221 will collide with the target object. The values of M, N, and P can be set according to the actual situation, and the smaller the P value, the The collision detection of the target object is more important.

Optionally, in step S4, the step of starting the security protection mechanism includes:

Step S44: setting a virtual boundary for the movement of the surgical instrument 221 according to the collision situation, and restricting the surgical instrument 221 from entering the range of the virtual boundary. In one example, after obtaining the expected motion information of the slave device 200 through the master-slave mapping according to the motion information of the master device 100, a virtual boundary limit is set according to the pre-collision information to prevent the surgical instrument 221 from moving to collide with the target object At the same time, according to the collision situation and the expected motion command of the slave device 200, the instrument arm 210 and the surgical instrument 221 of the slave device 200 are kept away from the collision position.

Optionally, if the collision situation includes the collision between the surgical instrument and the target object, at least one of an alarm, a reminder and a safety protection mechanism is activated. Preferably, in step S4, the step of alarming or prompting includes:

Step S45: In the imaging device 102 and/or the display device 302, add a text prompt of the collision information, and emphatically display the collision part between the surgical instrument 221 and the target object, such as red highlighting, etc., to help doctors and assistants Alarm or reminder is given by means of image display, as shown in Figure 19.

Optionally, in step S4, the step of alarming or prompting includes:

Step S46: flashing the warning light and/or prompting by sound. Please refer to FIG. 20 , in an example, a warning light is set at the instrument outer end of the tool arm of the instrument arm 210, if the surgical instrument 221 mounted or connected to the instrument arm 210 collides (that is, the current collision situation is a surgical instrument 221 collides with the target object), then flicker at a higher frequency, such as a 2Hz yellow light flicker. If the surgical instrument 221 mounted or connected to the instrument arm 210 is about to collide (that is, the expected collision situation is that the surgical instrument 221 collides with the target object), then flash at a lower frequency, such as a 1 Hz yellow light. Further, the instrument arm 210 can also be provided with an alarm sound prompting device to provide different levels of sound prompts, such as a 2 Hz sound prompt if a collision occurs, and a 1 Hz sound prompt if a collision is about to occur.

In order to describe the above collision detection in detail, the above three methods are described in detail below:

According to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on the ray collision detection method, including: generating the origin according to the position information of the target part of the medical device in the target medical image, and using the origin As the starting point, a ray is sent in the direction of movement of the medical device; according to the ray and the position information of the target object in the target medical image, it is judged whether the ray intersects the target object; when the ray intersects the target object, and the position of the intersection point and the position of the origin When the distance satisfies the preset condition, it is determined that the target object collides with the medical device.

In combination with Figures 21 and 22, Figure 21 is a schematic diagram of a ray collision detection method in an embodiment, and Figure 22 is a flowchart of a ray collision detection method in an embodiment, in this embodiment, as shown in Figure 21 , select a position as the origin, launch a ray from the origin in a certain direction, calculate whether the ray path intersects with the surface of the object, and if there is an intersection point, calculate the distance between the intersection point and the origin. Specifically, in this embodiment, the end of the instrument is taken as the origin, a ray is emitted toward the movement direction of the end of the instrument, and then the intersection point of the ray and the tissue surface is calculated and the distance between the intersection point and the origin is given. If the distance is positive or none The intersection point means that there is no collision between the instrument and the tissue. Among them, the distance between two points in three-dimensional space is calculated by the formula:

Where (x ₁ , y ₁ , z ₁ ) is the coordinates of point 1, and (x ₂ , y ₂ , z ₂ ) is the coordinates of point 2.

Among them, preferably, the calculation of the intersection point of the ray and the object can be solved by establishing the parametric equations of the straight line and the geometric body and solving the equation system formed by them simultaneously. If the equation system has no solution, then there is no intersection point. If there is a solution, all the solutions correspond to all of their intersection points. coordinate. The parameter equation of a sphere in three-dimensional space: f(X)＝||XC|| ² ＝R ² , the parameter equation of a ray in three-dimensional space:

Where X is the coordinates of the point on the sphere, C is the coordinates of the center of the sphere, R is the radius of the sphere; P is the coordinates of the starting point of the ray, t is the coefficient,

is a unit vector along the ray direction.

In one embodiment, according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on a convex polygon collision detection method, including: according to the position information of the target part of the medical device in the target medical image Generate the first geometric body based on the position information; generate the second geometric body according to the position information of the target object in the target medical image; calculate the Minkowski difference between the first geometric body and the second geometric body; judge whether the target object collides with the medical device according to the Minkowski difference .

23 and 24, FIG. 23 is a schematic diagram of a convex polygon collision detection method in an embodiment, and FIG. 24 is a flow chart of a convex polygon collision detection method in an embodiment. In this embodiment, by calculating two The Minkowski difference between two geometries to be collided with is judged whether there is a collision between the two geometries according to whether the difference contains the origin (0,0). As shown in this figure, S1 and S2 have collision overlapping parts, so the S3 geometry generated by their Minkowski difference includes the origin (0,0). The Minkowski difference is calculated by taking the difference between the coordinates of a point in one geometry and all points in another geometry.

Specifically, the cylinder at the end of the instrument is taken as one geometry, and the sensitive tissue is taken as another geometry, and then the Minkowski difference of the two geometry is calculated, if the origin is included, there is a collision, and if the origin is not included, there is no collision. The Minkowski difference calculation process of S1 and S2 geometry in the figure is: S1={a1,a2,a3,a4}; S2={b1,b2,b3,b4}; S3={(a1-b1),(a1-b2 ),(a1-b3),(a1-b4),(a2-b1),(a2-b2),(a2-b3),(a2-b4),(a3-b1),(a3-b2), (a3-b3),(a3-b4),(a4-b1),(a4-b2),(a4-b3),(a4-b4)}, where a1, a2, a3, a4 represent the vertices of geometry S1 , b1, b2, b3, b4 represent the vertices of geometry S2.

In one embodiment, according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on the linear projection collision detection method, including: according to the position information of the target part of the medical device in the target medical image Generating the first geometric body based on the position information; generating the second geometric body according to the position information of the target object in the target medical image; taking the moving direction of the medical device as the projection direction, calculating whether the projection parts of the first geometric body and the second geometric body overlap; when overlapping, then Determine the collision between the target object and the medical device.

As shown in FIG. 25, FIG. 25 is a schematic diagram of a linear projection collision detection method in an embodiment. In this embodiment, it is judged by calculating whether the projections of two geometric bodies to be collision detected on a straight line overlap. Whether the geometry collides in the direction pointed by the projected line. For surgical instruments and tissues, the end of the instrument can be regarded as a geometric body, and the sensitive tissue can be regarded as another geometric body, and then the instruction speed direction of the instrument is used as the direction of the projected line to project and calculate whether the projected parts of the two geometric bodies overlap. If so, then Indicates that the instrument collides with the tissue according to the current motion trend.

In one of the embodiments, the collision detection is performed according to the position information of the target object in the target medical image and the position information of the target part of the medical device, including: when the target object collides with the medical device, outputting a perceivable The collision warning includes at least one of a display reminder and a sound reminder.

As shown in Figure 26, this embodiment provides a collision safety sound alarm method, which detects whether there is a collision at the surgical instrument end, and if there is a collision, the robot operation master sends a buzzer to remind the doctor that the surgical instrument has collided with sensitive tissues , if there is no collision, no sound prompt will be made.

In one of the embodiments, before obtaining the pre-processed standard medical image, it also includes: obtaining an initial medical image through a medical image scanning device; identifying the target object in the initial medical image, and marking the position of the target object to obtain the target object Position information. A standard medical image is obtained according to the position information of the target object and the initial medical image.

Please continue to use the image information scanned by CT, MRI and other tomographic imaging technologies as shown in Figure 11, and complete the tissue modeling in the surgical space through image processing algorithms. Before operation, according to the tissue modeling information in the abdominal cavity, the key tissues that need special attention can be determined, and marker points can be established for 3D reconstruction of the surgical operation scene, that is, the tissues that need to be marked can be determined in the preoperative image, and the 3D model can be marked In this way, standard medical images with labeled objects of interest, including sensitive tissues, can be obtained.

In one embodiment, the above collision detection method further includes: displaying the current medical image and/or the target medical image through a display device and/or an augmented reality device.

Specifically, the display device may refer to a display device on a doctor's console, through which a current medical image and/or a target medical image may be displayed. In an optional embodiment, the system may also introduce an augmented reality device, through which the current medical image and/or the target medical image are displayed.

Among them, in this embodiment, a three-dimensional digital scene inside the abdominal cavity is reconstructed by obtaining preoperative CT images/MR images, and the tissues in the surgical area, hidden nerves, blood vessels and sensitive tissues are marked, and preoperative medical images, endoscopic The endoscope vision image and the position of the instrument in the robot coordinate system are fused to determine the spatial position and posture of the surgical instrument in the 3D surgical digital scene, so as to obtain the 3D spatial position relationship of the instrument, 3D endoscope, and human tissue and intuitively display it on the 3D surgical scene. When the surgical instrument collides with the sensitive tissue in the operation area, there is a risk of poking the sensitive tissue, especially when sensitive nerves, blood vessels, and tissues hidden in the invisible area of the endoscopic visual image, through the virtual surgical visual image (including AR\ VR) prompts the risk of collision and prompts the collision and depth of the instrument and tissue through the gradual color change of the visual image and the sound, and feeds back the collision force between the instrument and the tissue to the main operating terminal to ensure the safety of the surgical operation.

Specifically, as shown in FIG. 26 , the master operator forms a master-slave control relationship with the robotic arm 201 and surgical instruments, thereby forming a master-slave mapping, and then calculates the three-dimensional model of the abdominal cavity, as well as the matching of preoperative medical images and endoscopic images. relationship, so as to fuse the three of them to determine whether the position between the current position of the end of the device and the sensitive tissue is less than the threshold value, if so, the device collides with the tissue, and the processing is performed according to the above process to issue an alarm. If the position between the current position of the end of the device and the sensitive tissue is not less than the threshold, then the device has not collided with the tissue, and the next cycle of detection continues.

In one of the embodiments, the method further includes: acquiring the visual space coordinate system of the augmented reality device; calculating a third matching relationship between the visual space coordinate system of the augmented reality device and the image space coordinate system of the current medical image; according to the third The matching relationship displays the current medical image in the visual space of the augmented reality device; and/or displays the target medical image in the visual space of the augmented reality device according to the first matching relationship, the second matching relationship and the third matching relationship.

The visual space coordinate system is the coordinate system of the display space of the augmented display device. In order to display the current medical image and/or the target medical image in the visual space, in this embodiment, the visual space coordinate system of the augmented reality device and the current medical image are first calculated. The third matching relationship between the image space coordinate systems, and then display the current medical image in the visual space of the augmented reality device according to the third matching relationship; and/or according to the first matching relationship, the second matching relationship and the third matching relationship Display the target medical image in the visual space of the augmented reality device.

Optionally, when the target object collides with the medical device, the target medical image is displayed, so that the doctor can be prompted where the collision occurred by displaying the fused image; when the target object and the medical device do not collide, the current medical image is displayed .

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times, The execution order of these steps or stages is not necessarily performed sequentially, but may be performed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the same inventive concept, an embodiment of the present application further provides a collision detection device for implementing the above-mentioned collision detection method. The solution to the problem provided by the device is similar to the implementation described in the above method, so for the specific limitations in one or more embodiments of the collision detection device provided below, please refer to the definition of the collision detection method above, I won't repeat them here.

In one embodiment, as shown in FIG. 27 , a collision detection device is provided, including: a current medical image acquisition module 1901 , a location information acquisition module 1902 , a standard medical image acquisition module 1903 , a conversion module 1904 and a collision detection module 1905 ,in:

A current medical image acquisition module 1901, configured to acquire a current medical image;

A location information acquisition module 1902, configured to acquire the location information of the target part of the medical device;

A standard medical image acquisition module 1903, configured to acquire a pre-processed standard medical image, where the standard medical image carries the position information of the target object;

The conversion module 1904 is configured to convert the position information of the target object in the standard medical image into the target medical image according to the first matching relationship between the current medical image and the standard medical image, and according to the position information of the current medical image and the target part of the medical device The second matching relationship converts the position information of the target part of the medical device into the target medical image;

The collision detection module 1905 is configured to perform collision detection according to the position information of the target object in the target medical image and the position information of the target part of the medical device.

In one of the embodiments, the collision detection module 1905 is configured to perform collision detection according to at least one of the following: according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision is performed based on the ray collision detection method Detection; according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on the convex polygon collision detection method; according to the position information of the target object in the target medical image and the position of the target part of the medical device Information, based on the linear projection collision detection method for collision detection.

In one of the embodiments, the above-mentioned collision detection module 1905 includes: a ray generation unit, configured to generate an origin according to the position information of the target part of the medical device in the target medical image, and use the origin as a starting point to send a ray toward the moving direction of the medical device; The intersection judging unit is used to judge whether the ray intersects the target object according to the ray and the position information of the target object in the target medical image; the first collision result output unit is used for when the ray intersects the target object, and the position of the intersection point is the same as the origin When the distance between the positions satisfies the preset condition, it is determined that the target object collides with the medical device.

In one of the embodiments, the collision detection module 1905 includes: a first geometry generation unit, configured to generate a first geometry according to the position information of the target part of the medical device in the target medical image; a second geometry generation unit, configured to generate a first geometry according to the target The position information of the target object in the medical image generates a second geometric body; the Minkowski difference calculation unit is used to calculate the Minkowski difference between the first geometric body and the second geometric body; the second collision result output unit is used to calculate the Minkowski difference according to the Minkowski difference Determine whether the target object collides with the medical device.

In one of the embodiments, the collision detection module 1905 includes: a third geometry generation unit, configured to generate the first geometry according to the position information of the target part of the medical device in the target medical image; a fourth geometry generation unit, configured to generate the first geometry according to the target The position information of the target object in the medical image generates a second geometric body; the overlapping judging unit is used to use the moving direction of the medical device as the projection direction, and calculate whether the projection parts of the first geometric body and the second geometric body overlap;

The third collision result output unit is configured to determine that the target object collides with the medical instrument when they overlap.

In one embodiment, the above device further includes: an alarm module, configured to output a perceivable collision alarm when the target object collides with the medical device.

In one of the embodiments, the above-mentioned device further includes: an initial medical image acquisition module, configured to acquire an initial medical image through a medical image scanning device; a standard medical image generation module, configured to identify the target object in the initial medical image, and mark the target The position of the object is used to obtain the position information of the target object, and the standard medical image is obtained according to the position information of the target object and the initial medical image.

In one of the embodiments, the above device further includes: an identification unit, configured to identify the object to be processed in the current medical image; a first matching relationship generating unit, configured to match the target object in the standard medical image with the object to be processed Fusion obtains the first matching relationship.

In one of the embodiments, the above-mentioned device includes: a second matching relationship generating unit, configured to perform kinematic mapping calculation to obtain the second matching relationship.

In one of the embodiments, the above-mentioned device further includes: a first display module, configured to display the current medical image and/or the target medical image through a display device and/or an augmented reality device.

In one of the embodiments, the above device further includes: a first coordinate system acquisition module, configured to acquire the visual space coordinate system of the augmented reality device; a third matching relationship generation module, configured to calculate the visual space coordinate system of the augmented reality device and The third matching relationship between the image space coordinate systems of the current medical image; the second display module is used to display the current medical image in the visual space of the augmented reality device according to the third matching relationship; and/or according to the first matching relationship, The second matching relationship and the third matching relationship display the target medical image in the visual space of the augmented reality device.

In one embodiment, the above device further includes: a third display module, configured to display the target medical image when the target object collides with the medical instrument; and display the current medical image when the target object does not collide with the medical instrument.

Each module in the above-mentioned collision detection device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure may be as shown in FIG. 28 . The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies. When the computer program is executed by the processor, a collision detection method is realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.

Those skilled in the art can understand that the structure shown in Figure 28 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment on which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, there is also provided a computer device, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims

A collision detection method, characterized in that, comprising:

Acquiring at least two image information of the target object under different viewing angles;

Obtaining the spatial position of the target object according to the at least two image information;

Obtain the position information of the medical instrument connected to the end of the instrument arm of the surgical robot;

A collision situation between the medical instrument and the target object is determined according to the spatial position of the target object and the position information of the medical instrument.
The method according to claim 1, wherein the target object has feature points, and the feature points are determined by establishing a tissue model based on medical images.
The method according to claim 2, wherein the step of obtaining the spatial position of the target object according to the at least two image information comprises:

Establish a real-time three-dimensional model of the target object according to the at least two image information, and after registration with the tissue model, obtain real-time position information of the feature points in the coordinate system of the image acquisition device in real time, thereby The spatial position of the target object is obtained.
The method according to claim 3, characterized in that at least two image acquisition units are used to obtain the at least two image information, at least two of the image acquisition units are arranged on the endoscope, and the endoscope is connected to at the end of the arm of the surgical robot.
The method according to claim 4, wherein the step of obtaining the spatial position of the target object comprises:

Obtaining position information of the feature points in the image information in the endoscope coordinate system;

Obtaining the pose information of the endoscope in the arm base coordinate system;

According to the position information of the feature points in the endoscope coordinate system and the pose information of the endoscope in the arm base coordinate system, the position information of the feature points in the base coordinate system of the arm is obtained. , and then obtain the position information of the feature point in the base coordinate system, so as to obtain the spatial position of the target object.
The method according to claim 1, wherein the location information of the medical device includes current location information and expected location information; the collision situation includes the current collision situation and the expected collision situation.
The method according to claim 6, wherein the step of obtaining the expected location information of the medical device comprises:

Obtaining position information of at least two instrument marking points on the medical instrument in the base coordinate system of the instrument arm, and then obtaining position information of the instrument marking points in the base coordinate system;

Acquiring motion information of the marker points of the instrument;

Obtaining the expected spatial position of the instrument marking point according to the position information and motion information of the instrument marking point;

The expected position information of the medical device is obtained according to the expected spatial positions of at least two marking points of the device.
The method according to claim 2, wherein the step of determining the collision situation comprises:

Establishing a sphere Co with the feature point as the center and Ro as the radius;

Establishing a sphere Ct with the instrument marking point on the medical instrument as the center and Rt as the radius;

If the distance D between the sphere Co and the sphere Ct<Ro+Rt, then mark the feature point in contact with the instrument marker point.
The method according to claim 8, wherein the target object includes M feature points, wherein N feature points are in contact with the instrument marking points, N is a natural number, and M is a natural number not less than N, If the ratio of N to M is greater than the threshold P, P∈(0,1], it is determined that the medical instrument will collide with the target object.
The method according to claim 1, further comprising, when the target object collides with the medical device, at least one of giving an alarm, prompting and starting a safety protection mechanism.
The method according to claim 10, wherein the step of starting a security protection mechanism comprises:

A virtual boundary is set for the movement of the medical device according to the collision situation, and the medical device is restricted from entering the range of the virtual boundary.
A collision detection method, characterized in that, comprising:

Get the current medical image;

Obtain the location information of the target part of the medical device;

Acquiring a pre-processed standard medical image, the standard medical image carrying the position information of the target object;

Converting the position information of the target object in the standard medical image into the target medical image according to the first matching relationship between the current medical image and the standard medical image, and according to the relationship between the current medical image and the medical device The second matching relationship of the position information of the target part converts the position information of the target part of the medical device into the target medical image;

Collision detection is performed according to the position information of the target object in the target medical image and the position information of the target part of the medical device.
The method according to claim 12, wherein the collision detection according to the position information of the target object in the target medical image and the position information of the target part of the medical device comprises at least one of the following:

Performing collision detection based on a ray collision detection method according to the position information of the target object in the target medical image and the position information of the target part of the medical device;

Performing collision detection based on a convex polygon collision detection method according to the position information of the target object in the target medical image and the position information of the target part of the medical device;

According to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on a linear projection collision detection method.
The method according to claim 13, wherein the collision detection is performed based on a ray collision detection method according to the position information of the target object in the target medical image and the position information of the target part of the medical device, include:

generating an origin according to the position information of the target part of the medical device in the target medical image, and using the origin as a starting point to emit rays in the direction of movement of the medical device;

judging whether the ray intersects the target object according to the ray and the position information of the target object in the target medical image;

When the ray intersects the target object and the distance between the position of the intersection point and the position of the origin satisfies a preset condition, it is determined that the target object collides with the medical instrument.
The method according to claim 13, wherein the collision detection is performed based on a convex polygon collision detection method according to the position information of the target object in the target medical image and the position information of the target part of the medical device ,include:

generating a first geometric body according to position information of a target part of the medical device in the target medical image;

generating a second geometric body according to the position information of the target object in the target medical image;

calculating the Minkowski difference of the first geometry and the second geometry;

Whether the target object collides with the medical instrument is judged according to the Minkowski difference.
The method according to claim 13, characterized in that, according to the position information of the target object in the target medical image and the position information of the target part of the medical device, the collision detection is performed based on a linear projection collision detection method ,include:

generating a first geometric body according to position information of a target part of the medical device in the target medical image;

generating a second geometric body according to the position information of the target object in the target medical image;

Using the moving direction of the medical device as the projection direction, calculate whether the projection parts of the first geometric body and the second geometric body overlap;

When overlapping, it is determined that the target object collides with the medical instrument.
The method according to any one of claims 12 to 16, characterized in that before acquiring the pre-processed standard medical image, further comprising:

Obtain an initial medical image through a medical image scanning device;

identifying a target object in the initial medical image, and marking the location of the target object to obtain location information of the target object;

A standard medical image is obtained according to the position information of the target object and the initial medical image.
The method according to any one of claims 12 to 16, wherein the position of the target object in the standard medical image is calculated according to the first matching relationship between the current medical image and the standard medical image Before the information is converted into the target medical image, it also includes:

identifying an object to be processed in the current medical image;

According to the matching and fusion of the target object in the standard medical image and the object to be processed, a first matching relationship is obtained, and the first matching relationship is used to convert the position information of the target object in the standard medical image into a target medical images.
The method according to any one of claims 12 to 16, wherein the target part of the medical device is calculated according to the second matching relationship between the current medical image and the position information of the target part of the medical device Before converting the location information into the target medical image, it also includes:

According to the motion information of the medical mirror that collects the current medical image and the motion information of the target part of the medical device, perform kinematic mapping calculation to obtain a second matching relationship, and the second matching relationship is used to use the medical device The position information of the target part is converted into the target medical image.
The method according to any one of claims 12 to 16, further comprising:

Displaying the current medical image and/or the target medical image through a display device and/or an augmented reality device.
The method according to claim 20, further comprising:

Obtain the visual space coordinate system of the augmented reality device;

calculating a third matching relationship between the visual space coordinate system of the augmented reality device and the image space coordinate system of the current medical image;

Displaying the current medical image in the visual space of the augmented reality device according to the third matching relationship; and/or displaying the current medical image according to the first matching relationship, the second matching relationship and the third matching relationship The target medical image is displayed in the visual space of the augmented reality device.
The method according to claim 20, further comprising:

displaying the target medical image when the target object collides with the medical instrument;

When the target object and the medical instrument do not collide, the current medical image is displayed.
A collision detection system, characterized in that the collision detection system includes a processor, a medical mirror and a medical device, the medical device is provided with a sensor, the sensor is used to collect the position information of the target part of the medical device, And send the position information of the target part of the collected medical equipment to the processor; the medical mirror is used to collect the current medical image, and send the current medical image to the processor; the processor is used to Executing the collision detection method described in any one of claims 1 to 22.
The system according to claim 23, wherein the system further comprises a display device and/or an augmented reality device, and the display device and/or augmented reality device communicate with the processor; the display device and and/or an augmented reality device is configured to display the current medical image and/or the target medical image sent by the processor.
A collision detection device, characterized in that the device comprises:

The current medical image acquisition module is used to acquire the current medical image;

A position information acquisition module, configured to acquire position information of the target part of the medical device;

A standard medical image acquisition module, configured to acquire a pre-processed standard medical image, the standard medical image carrying the position information of the target object;

a conversion module, configured to convert the position information of the target object in the standard medical image into the target medical image according to the first matching relationship between the current medical image and the standard medical image; The second matching relationship of the position information of the target part of the medical device converts the position information of the target part of the medical device into the target medical image;

A collision detection module, configured to perform collision detection according to the position information of the target object in the target medical image and the position information of the target part of the medical device.
A computer device, comprising a memory and a processor, the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 22 when executing the computer program.
A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 22 are realized.