CN110353774B

CN110353774B - Auxiliary puncture robot, control method thereof, computer device, and storage medium

Info

Publication number: CN110353774B
Application number: CN201910412717.XA
Authority: CN
Inventors: 盖炜; 苏健
Original assignee: Shenzhen Mingjie Medical Technology Co ltd
Current assignee: Shenzhen Mingjie Medical Technology Co ltd
Priority date: 2018-12-15
Filing date: 2019-05-17
Publication date: 2023-05-12
Anticipated expiration: 2039-05-17
Also published as: CN110353774A

Abstract

The invention provides an auxiliary puncture robot, a control method thereof, computer equipment and a storage medium, wherein the auxiliary puncture robot control method comprises the following steps: receiving in-vivo image data of a patient including a body surface contour and a puncture planning path; scanning the body surface of the patient through a scanner installed on the robot to acquire the point cloud data of the body surface of the patient; registering the patient body surface point cloud data with the in-vivo image data of the patient; converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot; calculating the preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under the mechanical arm tool coordinate system; and controlling the puncture needle on the mechanical arm to move to a preset position. According to the technical scheme, the in-vivo three-dimensional data and the body surface three-dimensional data are fused, the reliability of visual guidance is improved, and an advantageous auxiliary tool is provided for puncture operation in medical operation.

Description

Auxiliary puncture robot, control method thereof, computer device, and storage medium

Technical Field

The present invention relates to the field of medical devices, and in particular, to an auxiliary piercing robot, a control method thereof, a computer device, and a storage medium.

Background

The robot auxiliary puncture is a technology that a mechanical hand holds a puncture needle to hover on the body surface of a patient, the needle insertion point and the needle insertion direction are completely fixed, a doctor only needs to confirm the correctness of the position, then the puncture needle is pushed into the body of the patient, and then the mechanical hand releases the puncture needle, so that the puncture process is completed. The auxiliary puncture function avoids the problem of inaccurate needle insertion direction caused by the fact that a doctor holds the puncture needle.

The existing vision guiding technology of the auxiliary puncture robot adopts a two-dimensional camera, identifies an object according to data imaged by one or more cameras, and calculates the pose of the object. The imaging data of the two-dimensional camera does not have depth information, and the pose of an object is required to be solved, so that the imaging data of a plurality of angles can be subjected to complex mathematical reconstruction. For simple geometric bodies with regular shapes, the three-dimensional contour can be reconstructed by means of recognition of characteristic quantities such as side lengths, radiuses and the like, while for non-special curved surfaces such as patient bodies, the three-dimensional contour cannot be reconstructed by a two-dimensional camera. Therefore, the visual guidance system of the robot cannot effectively fuse imaging information of the body surface of a patient with tomographic imaging data such as CT (computed tomography), MRI (magnetic resonance imaging) and the like, and a puncture path formulated based on the tomographic imaging data cannot be accurately reflected in visual guidance of the body surface.

Moreover, two-dimensional cameras recognize objects in dependence on differences in pixel values in the imaging data, which requires that the objects and the background have significantly different contrast or color differences in the imaging data, otherwise the contours of the objects may not be resolved or misjudgment may occur, and particularly for medical surgical scenes where color and ambient light are inconvenient to modify, it is more difficult to reliably recognize the objects and conduct visual guidance.

Disclosure of Invention

The present invention is achieved in that a first aspect of the present invention provides a method for controlling an auxiliary penetration robot, comprising:

receiving in-vivo image data of a patient including a body surface contour and a puncture planning path;

scanning the body surface of the patient through a scanner installed on the robot to acquire the point cloud data of the body surface of the patient;

registering the patient body surface point cloud data with the in-vivo image data of the patient;

converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot;

calculating the preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under the mechanical arm tool coordinate system; and controlling the puncture needle on the mechanical arm to move to a preset position.

Optionally, in the above auxiliary puncture robot control method, the step of receiving in-vivo image data including a body surface contour of the patient is receiving CT tomographic image data or B-mode image data or magnetic resonance image data.

Optionally, in the auxiliary puncture robot control method, the preset position is a position where the puncture needle hovers above the body surface of the patient.

Optionally, in the auxiliary puncture robot control method, the puncture plan path includes a puncture needle position coordinate and a needle direction vector.

Optionally, the above auxiliary puncture robot control method further includes the sub-steps after registering the patient body surface point cloud data with the in-vivo image data of the patient: calculating a conversion matrix M of a scanner coordinate system and an in-vivo image data coordinate system; the "converting the puncture plan path in the in-vivo image data coordinate system into the puncture plan path in the robot arm tool coordinate system" is a calculation conversion based on the conversion matrix N and the conversion matrix M of the scanner coordinate system and the robot arm tool coordinate system.

Optionally, the auxiliary puncture robot control method further includes the steps of: setting the axis of a gripper mounted on the mechanical arm to be parallel to the x-axis of the mechanical arm tool coordinate system; the aiming point on the axis of the gripper is set as the tool center point of the robotic arm.

Optionally, the auxiliary puncture robot control method further includes the steps of: a transformation matrix M between the scanner coordinate system and the robot tool coordinate system is obtained.

Optionally, in the above auxiliary penetration robot control method, the step of acquiring a transformation matrix M between a scanner coordinate system and a robot tool coordinate system includes: scanning at least three calibration pieces through a scanner to obtain coordinate data of the calibration pieces under a scanner coordinate system; and calculating to obtain a transformation matrix between the coordinate system of the scanner and the coordinate system of the mechanical arm tool according to the coordinate data of the calibration piece under the coordinate system of the mechanical arm tool and the coordinate data of the calibration piece under the coordinate system of the scanner.

Optionally, the auxiliary puncture robot control method further includes the steps of: converting the puncture needle position coordinate and the needle direction vector under the in-vivo image data coordinate system into the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system of the robot, comprising: acquiring a first transformation matrix and a second transformation matrix; obtaining a puncture needle position coordinate of T and a needle direction vector of V under an in-vivo image data coordinate system by homogeneous coordinate conversion; converting puncture needle position coordinates and needle direction vectors under an in-vivo image data coordinate system into puncture needle position coordinates T 'and needle direction vectors V' under a mechanical arm tool coordinate system according to the following calculation formula:

M is a first transformation matrix and N is a second transformation matrix.

Optionally, the auxiliary puncture robot control method further includes the steps of: calculating the preset position of the puncture needle arranged on the mechanical arm according to the puncture needle position coordinate and the needle inserting direction vector under the mechanical arm tool coordinate system; the puncture needle on the control mechanical arm moves to a preset position, comprising: the translation X, Y, Z of the gripper in the x, y, z axis directions and the rotation angle A, B, C about the z, y, x axis directions in the robotic arm tool coordinate system are calculated according to the following equations:

wherein t is _x1 、t _y1 、t _z1 For the puncture needle position coordinates T', S _x 、S _y 、S _Z Is the coordinate of the tool center point in the mechanical arm tool coordinate system, V _x1 、V _y1 、V _Z1 Is a needle-inserting direction vector V'; forming motion instructions (X, Y, Z, A, B, C) according to the translation amounts X, Y, Z along the X, Y and Z axis directions and the rotation angles A, B, C around the Z, Y and X axis directions; executing the motion instruction to control the puncture needle to move to a preset position to hover.

According to another aspect of the present invention, there is provided a method of controlling an auxiliary penetration robot, including:

receiving in-vivo image data of a patient, which comprises a body surface contour, and calculating a puncture planning path under an in-vivo image data coordinate system according to the in-vivo image data;

Scanning the body surface of the patient by a scanner to acquire 3D image data of the body surface of the patient; registering the patient body surface 3D image data with the in-vivo image data of the patient;

converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot; calculating the preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under the mechanical arm tool coordinate system;

and controlling the puncture needle on the mechanical arm to move to a preset position.

Optionally, the auxiliary puncture robot control method further includes the steps of: the preset position is a position where the puncture needle hovers above the body surface of the patient.

Optionally, the auxiliary puncture robot control method further includes the steps of: the puncture planning path includes puncture needle insertion position coordinates and a needle insertion direction vector.

According to still another aspect of the present invention, there is provided a method of controlling an auxiliary penetration robot, comprising:

receiving in-vivo image data of a patient, which comprises a body surface contour, and calculating a preset position of a puncture needle under an in-vivo image data coordinate system according to the in-vivo image data;

converting the preset position of the puncture needle under the in-vivo image data coordinate system into the preset position of the puncture needle under the mechanical arm tool coordinate system of the robot; and controlling the puncture needle on the mechanical arm to move to a preset position.

According to still another aspect of the present invention, there is provided a method of controlling an auxiliary penetration robot, including:

receiving in-vivo image data of a patient including a body surface contour and a puncture planning path; scanning a body surface of a patient by a scanner mounted on the robot to acquire 3D image data of the patient;

registering the patient 3D image data with the in-vivo image data of the patient; converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot;

acquiring in-vivo image data containing body surface contours of a patient and a puncture planning path;

According to another aspect of the present invention, there is provided an auxiliary penetration robot including:

a mechanical arm; the machine head is arranged on the mechanical arm and used for fixing the puncture needle;

the scanner is arranged on the mechanical arm and used for scanning 3D image data of the body surface of the patient;

A processing unit for registering 3D body surface image data of the patient from the scanner and in vivo image data of the patient including a body surface contour; the processing unit converts the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot; the processing unit calculates the preset position of the puncture needle arranged on the mechanical arm according to the puncture planning path under the mechanical arm tool coordinate system; and the control unit is used for controlling the puncture needle to move to a preset position according to the puncture planning path under the mechanical arm tool coordinate system.

Optionally, in the auxiliary puncture robot control device, an axis of the gripper is parallel to an x-axis of a mechanical arm tool coordinate system; the aiming point on the axis of the gripper is the tool center point of the robotic arm.

Optionally, the auxiliary puncture robot control device, wherein the puncture planning path comprises puncture needle position coordinates and a needle direction vector.

Optionally, in the auxiliary puncture robot control device, the processing unit is connected with the CT device in a data manner, and the in-vivo image imaging data and the puncture planning path come from the CT device.

According to another aspect of the present invention, a computer device is provided comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executing the computer program to perform steps based on a method of controlling an assisted penetration robot.

According to another aspect of the present invention, there is provided a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of a method of assisting in controlling a penetration robot.

The invention provides an auxiliary puncture robot control method, an auxiliary puncture robot control device, auxiliary puncture robot control equipment and a storage medium, wherein the auxiliary puncture robot control method comprises the following steps: receiving in-vivo image data of a patient including a body surface contour and a puncture planning path; scanning the body surface of the patient through a scanner installed on the robot to acquire the point cloud data of the body surface of the patient; registering the patient body surface point cloud data with the in-vivo image data of the patient; converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot; calculating the preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under the mechanical arm tool coordinate system; and controlling the puncture needle on the mechanical arm to move to a preset position. According to the technical scheme, the three-dimensional scanner capable of generating the point cloud is used for replacing a two-dimensional camera to serve as a visual guiding system of the robot, the three-dimensional point cloud with depth information can be obtained through single-angle scanning, large-scale scanning of the body surface of a patient can be easily achieved due to the fact that the scanning interval of the point cloud is short, then three-dimensional image fusion is carried out on the three-dimensional point cloud and tomographic imaging data such as CT and MRI, organs and planned needle inserting paths sketched based on the tomographic imaging data are directly marked into the body surface point cloud data, so that a puncture needle is guided to be placed at a correct position by a guiding manipulator to be pushed by a doctor to complete puncture, meanwhile, the three-dimensional data in a body are fused with the three-dimensional data in the body surface, complex operation and limitation of reconstructing a three-dimensional image through a two-dimensional image are omitted, the calculated amount of visual guiding is simplified, the reliability of visual guiding is improved, and a favorable auxiliary tool is provided for puncture operation in medical operation.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for controlling an auxiliary penetration robot according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating the operation of a method for controlling an auxiliary penetration robot according to the present invention;

FIG. 3 is a flowchart of a method for controlling an auxiliary penetration robot according to a second embodiment of the present invention;

FIG. 4 is a flowchart of a method for controlling an auxiliary penetration robot according to a third embodiment of the present invention;

FIG. 5 is a flowchart of a method for controlling an auxiliary penetration robot according to a fourth embodiment of the present invention;

FIG. 6 is a flowchart of a method for controlling an auxiliary penetration robot according to a fifth embodiment of the present invention;

FIG. 7 is a schematic view of a robot configuration for carrying out one embodiment of the auxiliary penetration robot control method of the first to fifth embodiments of the present invention;

FIG. 8 is an enlarged view of a portion of the handpiece of FIG. 7;

FIG. 9 is a schematic view of a robot configuration for implementing another embodiment of the auxiliary penetration robot control method of the first to fifth embodiments of the present invention;

FIG. 10 is a schematic illustration of coordinate calibration of the robotic arm of FIG. 9;

FIG. 11 is a preferred flow chart of a method of controlling the auxiliary lancing robot of FIG. 9 according to the present invention;

fig. 12 is a specific flowchart of step S10 in fig. 11;

fig. 13 is a further specific flowchart of step S10 in fig. 11;

FIG. 14 is a schematic view of a control device for an auxiliary penetration robot according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of a computer device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

In order to illustrate the technical scheme of the invention, the following description is made by specific examples.

The embodiment of the invention provides a control method of an auxiliary puncture robot, as shown in fig. 1 and 2, the control method comprises the following steps:

S11, receiving in-vivo image data containing body surface contours of a patient and a puncture planning path.

In step S11, receiving in-vivo image data including body surface contours of a patient means receiving CT tomographic image data or B-mode image data or magnetic resonance image data, and the puncture planning path means combining actual conditions and personal experiences of a puncture subject by a doctor, and integrating the puncture path from the body surface of the patient to a focus target.

And S12, scanning the body surface of the patient through a scanner arranged on the robot to acquire the point cloud data of the body surface of the patient.

In step S12, the scanner is mounted on the mechanical arm, and the scanner may be a laser scanner, and the distance from the target point on the body surface of the patient to the central point of the scanner is scanned by using the scanner as the center and using the laser pulse ranging method, so as to obtain the point cloud data of the body surface of the patient under the coordinate system of the scanner.

And S13, registering the body surface point cloud data of the patient with in-vivo image data of the patient.

In step S13, the computer automatically completes the image fusion and reconstruction between the surface point cloud data of the patient and the in-vivo image data of the patient, and a three-dimensional fusion image of the surface point cloud data and the in-vivo image data of the patient is obtained.

S14, converting the puncture planning path under the in-vivo image data coordinate system into the puncture planning path under the mechanical arm tool coordinate system of the robot.

In step S14, the puncture plan path in the in-vivo image data coordinate system is converted into the puncture plan path in the mechanical arm tool coordinate system of the robot, which is calculated and converted according to the conversion matrix N of the scanner coordinate system and the mechanical arm tool coordinate system and the conversion matrix M of the scanner coordinate system and the in-vivo image data coordinate system, and the puncture plan path includes the puncture needle position coordinate and the needle insertion direction vector, which are obtained by the calculation and conversion of the conversion matrix N and the conversion matrix M, so that the puncture needle position coordinate and the needle insertion direction vector in the mechanical arm tool coordinate system can be obtained.

S15, calculating the preset position of the puncture needle arranged on the mechanical arm according to the puncture planning path under the mechanical arm tool coordinate system,

in step S15, a preset position of the puncture needle mounted on the mechanical arm is calculated according to the coordinate of the puncture needle position and the vector of the needle insertion direction under the coordinate system of the mechanical arm tool, wherein the preset position is a position where the puncture needle hovers above the body surface of the patient.

And S16, controlling the puncture needle on the mechanical arm to move to a preset position.

In step S16, the translation and rotation of the mechanical arm are obtained according to the preset position coordinates, and then the puncture needle on the mechanical arm is moved to the preset position.

The embodiment of the invention uses a three-dimensional scanner capable of generating the point cloud to replace a two-dimensional camera as a visual guiding system of the robot, and can acquire the three-dimensional point cloud with depth information through single-angle scanning. Because the scanning interval of the point cloud is very short, the large-scale scanning of the body surface of a patient can be easily realized, then three-dimensional image fusion is carried out on the patient body surface and tomographic imaging data such as CT (computed tomography), MRI (magnetic resonance imaging) and the like, and organs and planned needle insertion paths which are drawn based on the tomographic imaging data are directly marked into the body surface point cloud data, so that a mechanical arm is guided to place a puncture needle at a correct position for a doctor to push the needle to complete puncture.

Fig. 3 is a flowchart of a control method of an auxiliary piercing robot according to a second embodiment of the present invention, where the control method includes:

s21, receiving in-vivo image data of the body surface contour of the patient, and calculating a puncture planning path under an in-vivo image data coordinate system according to the in-vivo image data.

In step S21, receiving in-vivo image data including a body surface contour of a patient means receiving CT tomographic image data or B-mode image data or magnetic resonance image data, and the puncture planning path means obtaining a puncture path from the body surface of the patient to a focus target point according to an actual condition of the puncture target.

And S22, scanning the body surface of the patient through a scanner to acquire 3D image data of the body surface of the patient.

S23, registering the 3D image data of the body surface of the patient with the in-vivo image data of the patient;

in step S23, the computer automatically completes the image fusion and reconstruction between the surface point cloud data of the patient and the in-vivo image data of the patient, and a three-dimensional fusion image of the surface point cloud data and the in-vivo image data of the patient is obtained.

S24, converting a puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot;

in step S24, the puncture plan path in the in-vivo image data coordinate system is converted into the puncture plan path in the mechanical arm tool coordinate system of the robot, which is calculated and converted according to the conversion matrix N of the scanner coordinate system and the mechanical arm tool coordinate system and the conversion matrix M of the scanner coordinate system and the in-vivo image data coordinate system, and the puncture plan path includes the puncture needle position coordinate and the needle insertion direction vector, which are obtained by the calculation and conversion of the conversion matrix N and the conversion matrix M, so as to obtain the puncture needle position coordinate and the needle insertion direction vector in the mechanical arm tool coordinate system.

S25, calculating a preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under a mechanical arm tool coordinate system;

In step S25, a preset position of the puncture needle mounted on the mechanical arm is calculated according to the coordinate of the puncture needle position and the vector of the needle insertion direction in the mechanical arm tool coordinate system, wherein the preset position is a position where the puncture needle hovers above the body surface of the patient.

And S26, controlling the puncture needle on the mechanical arm to move to a preset position.

In step S26, the translation and rotation amounts of the mechanical arm are obtained according to the preset position coordinates, and then the puncture needle on the mechanical arm is moved to the preset position.

According to the embodiment of the invention, a three-dimensional scanner is used for replacing a two-dimensional camera as a visual guiding system of the robot, 3D image data of the body surface of a patient is obtained, then three-dimensional image fusion is carried out on the 3D image data and tomographic imaging data such as CT (computed tomography) and MRI (magnetic resonance imaging) and the like, and organs and planned needle insertion paths which are sketched based on the tomographic imaging data are directly marked into the body surface image, so that a mechanical arm is guided to place a puncture needle at a correct position, and a doctor pushes the puncture needle to complete puncture.

Fig. 4 is a flowchart of a control method of an auxiliary piercing robot according to a third embodiment of the present invention, where the control method includes:

s31, receiving in-vivo image data containing body surface contours of a patient, and calculating the preset position of the puncture needle under the in-vivo image data coordinate system according to the in-vivo image data.

In step S31, receiving in-vivo image data of a patient including a body surface contour means receiving CT tomographic image data or B-mode image data or magnetic resonance image data, and the preset position of the puncture needle means obtaining a needle insertion position coordinate and a needle insertion direction vector from the body surface of the patient to a focus target point according to an actual condition in the puncture subject in-vivo image data.

S32, scanning the body surface of the patient through a scanner to acquire 3D image data of the body surface of the patient;

s33, registering the 3D image data of the body surface of the patient with the in-vivo image data of the patient;

in step S33, the computer automatically completes the image fusion and reconstruction between the surface point cloud data of the patient and the in-vivo image data of the patient, and a three-dimensional fusion image of the surface point cloud data and the in-vivo image data of the patient is obtained.

S34, converting the preset position of the puncture needle under the in-vivo image data coordinate system into the preset position of the puncture needle under the mechanical arm tool coordinate system of the robot;

the puncture planning path under the in-vivo image data coordinate system is converted into the puncture planning path under the mechanical arm tool coordinate system of the robot, the puncture planning path comprises puncture needle position coordinates and needle direction vectors which are obtained through calculation and conversion of the conversion matrix N and the conversion matrix M according to the conversion matrix N of the scanner coordinate system and the mechanical arm tool coordinate system and the conversion matrix M of the scanner coordinate system and the in-vivo image data coordinate system, and the puncture needle position coordinates and the needle direction vectors under the mechanical arm tool coordinate system can be obtained.

And S35, controlling the puncture needle on the mechanical arm to move to a preset position.

Fig. 5 is a flowchart of a control method of an auxiliary piercing robot according to a fourth embodiment of the present invention, where the control method includes:

s41, receiving in-vivo image data containing body surface contours of a patient and a puncture planning path;

in step S41, receiving in-vivo image data including body surface contours of a patient means receiving CT tomographic image data or B-mode image data or magnetic resonance image data, and the puncture planning path means combining actual conditions and personal experiences of a puncture subject by a doctor, and integrating the puncture path from the body surface of the patient to a focus target.

S42, scanning the body surface of the patient through a scanner installed on the robot to acquire 3D image data of the patient;

S43, registering 3D image data of a patient with in-vivo image data of the patient;

in step S43, the computer automatically completes the image fusion and reconstruction between the surface point cloud data of the patient and the in-vivo image data of the patient, and a three-dimensional fusion image of the surface point cloud data and the in-vivo image data of the patient is obtained.

S44, converting the puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot;

in step S44, the puncture plan path in the in-vivo image data coordinate system is converted into the puncture plan path in the mechanical arm tool coordinate system of the robot, which is calculated and converted according to the conversion matrix N of the scanner coordinate system and the mechanical arm tool coordinate system and the conversion matrix M of the scanner coordinate system and the in-vivo image data coordinate system, and the puncture plan path includes the puncture needle position coordinate and the needle insertion direction vector, which are obtained by the calculation and conversion of the conversion matrix N and the conversion matrix M, so that the puncture needle position coordinate and the needle insertion direction vector in the mechanical arm tool coordinate system can be obtained.

S45, calculating a preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under a mechanical arm tool coordinate system;

In step S45, a preset position of the puncture needle mounted on the mechanical arm is calculated according to the coordinate of the puncture needle position and the vector of the needle insertion direction under the coordinate system of the mechanical arm tool, wherein the preset position is a position where the puncture needle hovers above the body surface of the patient.

And S46, controlling the puncture needle on the mechanical arm to move to a preset position.

In step S46, the translation and rotation amounts of the mechanical arm are obtained according to the preset position coordinates, and then the puncture needle on the mechanical arm is moved to the preset position.

According to the embodiment of the invention, a three-dimensional scanner is used for replacing a two-dimensional camera as a visual guiding system of the robot, 3D image data of a patient is obtained, then three-dimensional image fusion is carried out on the 3D image data and tomographic imaging data such as CT (computed tomography) and MRI (magnetic resonance imaging) and the like, and organs and planned needle insertion paths which are sketched based on the tomographic imaging data are directly marked into body surface point cloud data, so that a mechanical arm is guided to place a puncture needle at a correct position, and a doctor pushes the puncture needle to complete puncture.

Fig. 6 is a flowchart of a control method of an auxiliary piercing robot according to a fifth embodiment of the present invention, where the control method includes:

s51, acquiring in-vivo image data containing body surface contours of a patient and a puncture planning path;

In step S51, acquiring in-vivo image data including a body surface contour of a patient refers to acquiring CT tomographic image data or B-mode image data or magnetic resonance image data, and the puncture planning path refers to integrating a doctor with the actual condition and personal experience of a puncture object to obtain a puncture path from the body surface of the patient to a focus target point.

S52, scanning the body surface of the patient through a scanner arranged on the robot to acquire the point cloud data of the body surface of the patient;

in step S52, the scanner is mounted on the mechanical arm, and the scanner may be a laser scanner, and the distance from the target point on the body surface of the patient to the central point of the scanner is scanned by using the scanner as the center and using the laser pulse ranging method, so as to obtain the point cloud data of the body surface of the patient under the coordinate system of the scanner.

S53, registering the surface point cloud data of the patient with in-vivo image data of the patient;

in step S53, the computer automatically completes the image fusion and reconstruction between the body surface point cloud data of the patient and the in-vivo image data of the patient, and a three-dimensional fusion image of the body surface point cloud data and the in-vivo image data of the patient is obtained.

S54, converting a puncture planning path under the in-vivo image data coordinate system into a puncture planning path under the mechanical arm tool coordinate system of the robot;

S55, calculating a preset position of a puncture needle arranged on the mechanical arm according to a puncture planning path under a mechanical arm tool coordinate system;

and S56, controlling the puncture needle on the mechanical arm to move to a preset position.

Fig. 7 and 8 are schematic structural views of a robot for implementing the above-described auxiliary penetration robot control method. Referring to fig. 3 and 4, the robot includes a driving mechanism 100, a handpiece 1, and a visual positioning device. The robot is connected to the control terminal 200.

The driving mechanism 100 is used for driving the handpiece 1 to move relative to the patient 300, so that the beam needle 2 on the handpiece 1 accurately reaches a preset position and enters the patient 300 according to a planned path. In this embodiment, the drive mechanism is a six-axis crank arm. It will be appreciated that the configuration of the drive mechanism 100 is not limited to the solution of this embodiment, as long as precise movement in three dimensions relative to the patient 300 within a region is achieved.

The handpiece 1 is connected to a drive mechanism 100 by a mounting structure. The mounting structure includes a base plate 18, a sliding structure, and a linkage structure C. The handpiece 1 is mounted on the bottom plate 18 by a sliding structure so as to be relatively slidable up and down. The machine head 1 is provided with an electron accelerator, and the accelerator comprises an electron emission gun, an accelerating tube and a microwave generator. The electron emission gun is used for emitting electron beams. A magnetron is commonly used as a microwave generator, which is in communication with an accelerating tube for providing microwaves for accelerating electrons. The beam needle 2 is fixedly connected with the accelerating tube. The electron emission gun emits electron beams according to the instruction sent by the control terminal 200, the electron beams are accelerated by the accelerating tube, and finally the electron beams are emitted through the beam needle 2. The control terminal 200 can control the dose of the patient 300 to be treated by controlling the firing time of the electron gun.

The control terminal 200, for example, a computer, is connected to the drive mechanism 100 and the handpiece 1 by a wire or wirelessly, and transmits a command to the drive mechanism 100 and the handpiece 1. The control terminal 200 receives real-time information, such as position information, image information, dose information, etc., of the driving mechanism 100 and the handpiece 1 at the same time.

The vision positioning device acquires a body surface 3D image corresponding to the tumor focus part of the patient 300 through the 3D scanner 19 and transmits the body surface 3D image to the control terminal 200. The control terminal 200 matches the acquired 3D image of the body surface of the patient 300 with a pre-stored tomographic image of the tumor focus position of the patient 300, acquires a coordinate system based on the tumor focus position of the patient 300 and a coordinate system based on the driving mechanism 100, and controls the driving mechanism 100 to drive the beam needle 2 to move according to a preset treatment path plan, so that the beam needle 2 is directly inserted into a preset position in the body of the patient 300. When the beam needle 2 reaches a preset position in the patient 300, the control terminal 200 starts the electron gun to emit electron beams according to preset dosage, and the electron beams are accelerated by the accelerating tube and then are driven into the patient 300 through the beam needle 2 to ablate bad tissues.

Fig. 9 is a schematic view of a robot for implementing the auxiliary piercing robot control method.

Wherein the robot comprises a scanner 4, a gripper 3 and a mechanical arm 1, the gripper 3 is arranged on an end flange 2 of the mechanical arm 3, the scanner 4 can be arranged on the gripper 3 or on the mechanical arm 1, for example, the scanner 3 is arranged on an end joint of the mechanical arm, the position of the scanner 4 is not particularly limited herein, the gripper 3 is provided with a puncture needle, a calibration piece is required to be arranged on the gripper 3 to obtain a transformation matrix, the calibration piece can be various regular objects with a center point, such as a sphere, a cube, a cone and the like, the regular objects with the center point can be arranged on the calibrator, the calibration piece can be a light spot, the light spot can be projected on the calibrator or can be suspended in the air, no matter what type the calibrator selects, the coordinates of the calibration piece under the mechanical arm tool coordinate system are known in advance, for example, the calibration piece is arranged on the calibration device, the coordinates of the calibration piece under the mechanical arm tool coordinate system can be obtained according to the relative positions of the calibration device and the end flange of the mechanical arm and the positions of the calibration piece on the calibration device, the central coordinates of the calibration piece under the scanner coordinate system are obtained by scanning the calibration piece through the scanner, specifically, the point cloud data of three spheres are obtained through the scanner, spherical fitting is carried out on the point cloud data of the three spheres, the spherical center coordinates of the three spheres under the scanner coordinate system are obtained, and the spherical center coordinates of the three spheres under the scanner coordinate system can be obtained by carrying out spherical fitting on the point cloud data by adopting a least square method.

And calculating to obtain a transformation matrix between the coordinate system of the scanner and the coordinate system of the mechanical arm tool according to the coordinate data of the calibration piece under the coordinate system of the mechanical arm tool and the coordinate data of the calibration piece under the coordinate system of the scanner.

In the step, a center coordinate of the calibration piece under the mechanical arm tool coordinate system and a center translational vector of the center coordinate of the calibration piece under the scanner coordinate system are calculated, and the center translational vector is calculated according to a singular value decomposition method to obtain a transformation matrix between the scanner coordinate system and the mechanical arm tool coordinate system.

As an embodiment, referring to fig. 10, the steps include:

spherical center coordinate vector of three spheres under mechanical arm tool coordinate system

And the spherical center coordinate vector of the three spheres in the scanner coordinate system +.>

Respectively calculating arithmetic mean values to obtain gravity center coordinate vectors of the three spheres under a mechanical arm tool coordinate system>

And the barycentric coordinate vector of the three spheres in the scanner coordinate system +.>

Calculating the spherical center coordinate vector according to the following formula

Center of gravity translation vector +.>

Center of gravity translation vector +.>

Shift the gravity center by a vector

Conversion to a 3X 3 matrix X ₀ :

Shift the gravity center by a vector

Conversion to a 3 x 3 matrix Y ₀ :

Computing a matrix

And singular value decomposition of matrix H>

Wherein U is H.H ^T Is set as H ^T Feature vector of H, H ^T And H is ^T H has the same characteristic value, and the characteristic value is set as lambda ₁ ,...,λ _r R represents the rank of matrix H; diagonal element +.>

And diagonal elements in Λ are arranged in descending order, and the rest elements in Λ are 0;

establishing a rotation matrix R and a translation matrix T between a scanner coordinate system and a mechanical arm tool coordinate system, and acquiring a transformation matrix according to the following formula:

wherein, set up

According to the formula->

Obtaining a transformation matrix:

further, the auxiliary puncture robot control method further comprises the steps of:

acquiring a second transformation matrix of a scanner coordinate system and a tomography data coordinate system;

wherein, the registration of the body surface outline and the perspective data of the patient is automatically completed by a computer to obtain a second transformation matrix of the CT data coordinate system and the body surface point cloud coordinate system

The computer acquires at least three point coordinates of a body surface point cloud coordinate system on the body surface outline of the patient, acquires coordinates of the three points in a CT data coordinate system, and acquires a second transformation matrix by calculating translation quantity and performing singular value decomposition.

Further, converting the puncture needle position coordinate and the needle direction vector under the in-vivo image data coordinate system into the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system of the robot, including:

Acquiring a first transformation matrix and a second transformation matrix;

obtaining a puncture needle position coordinate of T and a needle direction vector of V under an in-vivo image data coordinate system by homogeneous coordinate conversion;

converting puncture needle position coordinates and needle direction vectors under an in-vivo image data coordinate system into puncture needle position coordinates T 'and needle direction vectors V' under a mechanical arm tool coordinate system according to the following calculation formula:

m is a first transformation matrix and N is a second transformation matrix.

Further, the control of the movement of the puncture needle on the mechanical arm to the preset position includes:

the translation X, Y, Z of the gripper in the x, y, z axis directions and the rotation angle A, B, C about the z, y, x axis directions in the robotic arm tool coordinate system are calculated according to the following equations:

/>

wherein t is _x1 、t _y1 、t _z1 For the puncture needle position coordinates T', S _x 、S _y 、S _Z Is the coordinate of the tool center point in the mechanical arm tool coordinate system, V _x1 、V _y1 、V _Z1 Is a needle-inserting direction vector V';

forming motion instructions (X, Y, Z, A, B, C) according to the translation amounts X, Y, Z along the X, Y and Z axis directions and the rotation angles A, B, C around the Z, Y and X axis directions;

executing the motion instruction to control the puncture needle to move to a preset position to hover.

In the above steps, a movement instruction is generated according to the translation amount and the rotation angle of the gripper, and after the gripper and the scanner are rotated according to the movement instruction, the movement is performed along a certain coordinate axis according to the translation amount in the movement instruction, so that the puncture needle can be moved to the hovering position.

A sixth embodiment of the present invention provides an auxiliary penetration robot control device, including:

a mechanical arm;

the machine head is arranged on the mechanical arm and used for fixing the puncture needle;

a processing unit for registering the 3D body surface image data of the patient from the scanner and the in vivo image data of the patient comprising the body surface contours. The processing unit converts the puncture planning path in the in-vivo image data coordinate system into a puncture planning path in the mechanical arm tool coordinate system of the robot. The processing unit calculates the preset position of the puncture needle arranged on the mechanical arm according to the puncture planning path under the mechanical arm tool coordinate system.

And the control unit is used for controlling the puncture needle to move to a preset position according to the puncture planning path under the mechanical arm tool coordinate system.

The puncture planning path includes puncture needle insertion position coordinates and a needle insertion direction vector.

The processing unit is in data connection with the CT equipment, and the in-vivo image imaging data and the puncture planning path come from the CT equipment.

According to the embodiment, a three-dimensional scanner is used for replacing a two-dimensional camera to serve as a visual guiding system of the robot, 3D image data of the body surface of a patient are obtained, then three-dimensional image fusion is carried out on the three-dimensional image data and tomographic imaging data such as CT and MRI, organs and planned needle insertion paths which are sketched based on the tomographic imaging data are directly marked in the body surface image, and accordingly a mechanical arm is guided to place a puncture needle at a correct position for a doctor to push the needle to complete puncture.

The embodiment of the invention provides an auxiliary puncture robot control method, as shown in fig. 11, which comprises the following steps:

s10, setting a scanner coordinate system and a mechanical arm tool coordinate system, and acquiring a first transformation matrix between the scanner coordinate system and the mechanical arm tool coordinate system.

In step S10, the robot includes the scanner 4, the gripper 3 and the mechanical arm 1, the gripper 3 is disposed on the end flange 2 of the mechanical arm 3, the scanner 4 may be disposed on the gripper 3 or may be disposed on the mechanical arm 1, for example, the scanner 3 is disposed on the end joint of the mechanical arm, the position of the scanner 4 is not specifically limited herein, the gripper 3 is provided with a puncture needle, the transformation matrix is required to be acquired, the calibration member may be various regular objects with a center point, such as a sphere, a cube, a cone, or the like, the regular objects with a center point may be disposed on the calibrator, the calibration member may be a light spot, the light spot may be projected on the calibrator or may be suspended in the air, no matter what type of the calibrator is selected, its coordinates under the tool coordinate system of the mechanical arm are known in advance, for example, the calibration member is disposed on the calibrator, and the coordinate system under the tool coordinate system of the mechanical arm may be derived according to the relative positions of the calibrator and the end flange of the mechanical arm and the position of the calibration member on the calibrator.

As one way, as shown in fig. 12, setting the scanner coordinate system and the robot tool coordinate system in step S10 includes:

step s101. The axis of the gripper is arranged parallel to the x-axis of the robot tool coordinate system.

Step S102, setting an aiming point on the axis of the gripper as a tool center point of the mechanical arm.

In the above steps S101 and S102, a three-dimensional point cloud scanner (scanner for short) and a round tube gripping device (gripper for short) are first fixed at the end of a six-axis mechanical arm (mechanical arm for short), so that the scanner, the gripper and the flange at the end of the mechanical arm remain relatively stationary in motion, the scanner and the gripper form a robot hand eye system, then the axis of the gripper is set to be parallel to the x axis of the mechanical arm tool coordinate system, the aiming point of the gripper is set as the tool center point (Tool Central Point, TCP) of the mechanical arm, and the position of the point under the mechanical arm tool coordinate system is recorded as S (S) _x ,s _y ,s _z )。

In step S10, as one way, as shown in fig. 13, a first transformation matrix between the scanner coordinate system and the robot tool coordinate system is acquired, including:

s103, obtaining the center coordinates of the calibration piece under the mechanical arm tool coordinate system.

In step S103, as an embodiment, the coordinates of the centers of the three spheres on the marker in the tool coordinate system of the mechanical arm are obtained according to the relative positions of the marker and the mechanical arm. Wherein the calibrator is mounted to the end of the robotic arm, wherein the calibrator meets the following conditions: the calibrating device is provided with three calibrating pieces of spheres, the three side lengths of a triangle formed by the sphere centers of the three spheres are mutually unequal, and the three spheres can be stably arranged at the tail end of the mechanical arm and keep relative static with the flange at the tail end of the mechanical arm; after the installation, the spherical center coordinates of the three spheres can be obtained in a mechanical arm tool coordinate system at the tail end of the mechanical arm; when the scanner scans the calibrator, the three spheres are required to be in the effective field of view of the scanner and are not shielded from each other, wherein the calibrator can be connected to the tail end flange of the grasping device through a connecting rod, the three spheres are arranged on the calibrator according to preset conditions, and the coordinates of the sphere centers of the three spheres under the tool coordinate system of the mechanical arm are obtained according to the tool center point of the mechanical arm and the arrangement positions of the three spheres on the calibrator.

S104, scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system.

In step S104, specifically, as shown in fig. 13, the point cloud data of the three spheres are obtained by the scanner, spherical fitting is performed on the point cloud data of the three spheres to obtain spherical center coordinates of the three spheres under the coordinate system of the scanner, and spherical fitting may be performed on the point cloud data by using the least square method to obtain spherical center coordinates of the three spheres under the coordinate system of the scanner.

S105, calculating center coordinates of the calibration piece under a mechanical arm tool coordinate system and gravity center translation vectors of the center coordinates of the calibration piece under a scanner coordinate system.

As one embodiment, step S105 includes:

And three spheres in the scanner coordinate systemLower spherical center coordinate vector ∈ ->

Center of gravity translation vector +.>

Center of gravity translation vector +.>

/>

And S106, calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between the scanner coordinate system and the mechanical arm tool coordinate system.

As one embodiment, step S106 includes:

shift the gravity center by a vector

Conversion to3X 3 matrix X ₀ :

Shift the gravity center by a vector

Conversion to a 3 x 3 matrix Y ₀ :

Computing a matrix

And singular value decomposition of matrix H>

wherein, set up

According to the formula->

Obtaining a transformation matrix:

s20, acquiring tomographic imaging data of a patient, and acquiring puncture needle position coordinates and needle direction vectors under a tomographic imaging data coordinate system according to the tomographic imaging data.

In step S20, three-dimensional image fusion may be performed on tomographic data such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging ) and the like to obtain tomographic data of the patient, and a puncture path is drawn according to a puncture scheme formulated by a doctor for the tomographic data, as shown in fig. 2, and a puncture needle position coordinate is recorded as T (T) _x ,t _y ,t _z ) The needle insertion direction vector is V (V _x ,v _y ,v _z )。

S30, controlling a scanner to scan the body surface of the patient to obtain point cloud data of the body surface outline of the patient, registering the point cloud data of the body surface outline with tomographic imaging data, and obtaining a second transformation matrix of a scanner coordinate system and a tomographic imaging data coordinate system.

In step S30, the registration of the body surface contour and the perspective data of the patient is automatically completed by the computer to obtain a second transformation matrix of the CT data coordinate system and the body surface point cloud coordinate system

S40, converting the puncture needle position coordinate and the needle direction vector under the fault imaging data coordinate system into the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system according to the first transformation matrix and the second transformation matrix.

As one embodiment, step S40 includes:

acquiring a first transformation matrix

Obtaining a second transformation matrix

Converting the puncture needle position coordinate and the needle direction vector by using the homogeneous coordinate to obtain a puncture needle position as

The direction vector of needle insertion is->

Converting the puncture needle position coordinate and the needle direction vector into a puncture needle position coordinate T 'and a needle direction vector V' under a mechanical arm tool coordinate system according to the following calculation formula:

/>

and S50, controlling the puncture needle to move to the corresponding needle inserting position to hover according to the coordinate of the needle inserting position and the needle inserting direction vector under the coordinate system of the mechanical arm tool.

As one embodiment, step S50 includes:

step S501, calculating translation X, Y, Z of the gripper along the x, y and z axis directions and rotation angles A, B, C around the z, y and x axis directions under a mechanical arm tool coordinate system according to the following formula:

wherein t is _x1 、t _y1 、t _z1 For the puncture needle position coordinates T', S _x 、S _y 、S _Z Is the coordinate of the tool center point in the mechanical arm tool coordinate system, V _x1 、V _y1 、V _Z1 Is the needle insertion direction vector V'.

Step S502, a motion instruction (X, Y, Z, A, B and C) is formed according to the translation amount X, Y, Z along the X, Y and Z axis directions and the rotation angle A, B, C around the Z, Y and X axis directions.

Step S503, executing a motion instruction to control the puncture needle to move to a corresponding needle inserting position to hover.

In the above steps S501 to S503, as shown in fig. 6 and 7, a movement command is generated according to the translational amount and the rotation angle of the gripper, and after the rotational movement command is given to the gripper and the scanner according to the movement command, the movement is performed along a coordinate axis according to the translational amount in the movement command, so that the puncture needle can be moved to the hovering position.

According to the technical scheme, the three-dimensional scanner capable of generating the point cloud is used for replacing a two-dimensional camera to serve as a visual guiding system of the robot, and the three-dimensional point cloud with depth information can be obtained through single-angle scanning. Because the scanning interval of the point cloud is very short, the large-scale scanning of the body surface of a patient can be easily realized, then three-dimensional image fusion is carried out on the patient body surface and tomographic imaging data such as CT (computed tomography), MRI (magnetic resonance imaging) and the like, and organs and planned needle insertion paths which are drawn based on the tomographic imaging data are directly marked into the body surface point cloud data, so that a mechanical arm is guided to place a puncture needle at a correct position for a doctor to push the needle to complete puncture. According to the technical scheme, the in-vivo three-dimensional data and the body surface three-dimensional data are fused, so that complex operation and limitation of reconstructing a three-dimensional image through a two-dimensional image are avoided, the calculated amount of visual guidance is simplified, the reliability of visual guidance is improved, and an advantageous auxiliary tool is provided for puncture operation in medical operation.

Another embodiment of the present invention provides an auxiliary penetration robot control apparatus 60, as shown in fig. 14, the auxiliary penetration robot includes a scanner, a gripper and a mechanical arm, the gripper and the scanner are disposed at the end of the mechanical arm, a penetration needle is gripped on the gripper, and the auxiliary penetration robot control apparatus 60 includes:

A first transformation matrix obtaining unit 601, configured to set a scanner coordinate system and a robot tool coordinate system, and obtain a first transformation matrix between the scanner coordinate system and the robot tool coordinate system;

a puncture location information acquiring unit 602, configured to acquire tomographic imaging data of a patient, and acquire a puncture needle insertion location coordinate and a needle insertion direction vector in a tomographic imaging data coordinate system according to the tomographic imaging data;

a second transformation matrix obtaining unit 603, configured to control the scanner to scan the body surface of the patient to obtain point cloud data of the body surface contour of the patient, register the point cloud data of the body surface contour with tomographic imaging data, and obtain a second transformation matrix of the scanner coordinate system and the tomographic imaging data coordinate system;

a puncture location information conversion unit 604, configured to convert the puncture location coordinate and the puncture direction vector under the tomographic imaging data coordinate system into a puncture location coordinate and a puncture direction vector under the mechanical arm tool coordinate system according to the first transformation matrix and the second transformation matrix;

and the movement control unit 605 is used for controlling the puncture needle to move to the corresponding needle inserting position to hover according to the puncture needle inserting position coordinates and the needle inserting direction vector under the mechanical arm tool coordinate system.

Further, the first transformation matrix acquisition unit 601 is further configured to:

setting the axis of the gripper parallel to the x-axis of the robotic tool coordinate system;

the aiming point on the axis of the gripper is set as the tool center point of the robotic arm.

Further, the first transformation matrix acquisition unit 601 is specifically configured to:

acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system;

scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system;

calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system;

and calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system.

Further, the puncture location information conversion unit 604 is specifically configured to:

acquiring a first transformation matrix

Obtaining a second transformation matrix

The direction vector of needle insertion is->

Converting the puncture needle position coordinate and the puncture needle direction vector into a puncture needle position coordinate T 'and a puncture needle direction vector V' under a mechanical arm tool coordinate system according to the following calculation formula:

/>

Further, the mobile control unit is specifically configured to:

executing the motion instruction to control the puncture needle to move to the corresponding needle inserting position to hover.

The present embodiment provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements a method for controlling an auxiliary penetration robot in embodiment 1, and in order to avoid repetition, a detailed description is omitted.

Fig. 15 is a schematic diagram of a computer device in an embodiment of the invention. As shown in fig. 15, the computer device 6 includes a processor 63, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 63. The processor 63 implements the steps of a control method of the auxiliary penetration robot in the above embodiment, such as steps S10, S20, S30, S40 and S50 shown in fig. 1, when executing the computer program 62. Alternatively, processor 63 implements an auxiliary penetration robot control device of the above-described embodiment when executing computer program 62.

By way of example, the computer program 62 may be divided into one or more modules/units, which are stored in the memory 61 and executed by the processor 63 to perform the data processing procedure of the present invention. One or more of the modules/units may be a series of computer program segments capable of performing particular functions to describe the execution of the computer program 62 in the computer device 6.

The terminal device 6 may be a computing device such as a desktop computer, a notebook computer, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor 63, a memory 61. It will be appreciated by those skilled in the art that fig. 7 is merely an example of the terminal device 6 and does not constitute a limitation of the terminal device 6, and may include more or less components than illustrated, or may combine certain components, or different components, e.g., the terminal device may also include an input-output device, a network access device, a bus, etc.

The processor 63 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or a memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk provided on the terminal device 6, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like. Further, the memory 61 may also include both an internal storage unit and an external storage device of the terminal device 6. The memory 61 is used to store computer programs and other programs and data required by the terminal device. The memory 61 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the steps of each method embodiment described above may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of the jurisdiction's jurisdiction and the patent practice, for example, in some jurisdictions, the computer readable medium does not include electrical carrier signals and telecommunication signals according to the jurisdiction and the patent practice.

The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.

Claims

1. A method of assisting a penetration robot in controlling, the method comprising:

receiving in-vivo image data containing body surface contours of a patient and a puncture planning path, wherein the puncture planning path comprises puncture needle position coordinates and a needle direction vector;

scanning the body surface of a patient through a scanner installed on the robot to acquire the point cloud data of the body surface of the patient;

registering the body surface point cloud data of the patient with the in-vivo image data of the patient so as to carry out image fusion reconstruction on the body surface point cloud data of the patient and the in-vivo image data, thereby obtaining a three-dimensional fusion image of the body surface point cloud data of the patient and the in-vivo image data;

Converting the puncture needle position coordinate and the needle direction vector under the in-vivo image data coordinate system into the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system of the robot;

calculating a preset position of a puncture needle arranged on the mechanical arm according to the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system;

controlling a puncture needle on the mechanical arm to move to the preset position, wherein the preset position is a position where the puncture needle hovers above the body surface of a patient;

the step of converting the puncture needle position coordinate and the needle direction vector under the in-vivo image data coordinate system into the puncture needle position coordinate and the needle direction vector under the mechanical arm tool coordinate system of the robot comprises the following steps: acquiring a first transformation matrix and a second transformation matrix; obtaining a puncture needle position coordinate of T and a needle direction vector of V under an in-vivo image data coordinate system by homogeneous coordinate conversion; converting puncture needle position coordinates and a needle direction vector under an in-vivo image data coordinate system into puncture needle position coordinates T 'and a needle direction vector V' under a mechanical arm tool coordinate system according to the following calculation formula;

M is a first transformation matrix, and N is a second transformation matrix, wherein the first transformation matrix is obtained by the following method: acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system; scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system; calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system; calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system; the second transformation matrix is obtained by: acquiring at least three point coordinates of a body surface point cloud coordinate system on the body surface profile of a patient, acquiring coordinates of the three points in a CT data coordinate system, and acquiring the coordinates by calculating translation quantity and performing singular value decomposition;

calculating a preset position of a puncture needle arranged on the mechanical arm according to the puncture needle position coordinate and the needle inserting direction vector under the mechanical arm tool coordinate system; controlling the puncture needle on the mechanical arm to move to the preset position comprises the following steps: calculating translation X, Y, Z of the gripper in the x, y and z axis directions and rotation angle A, B, C around the z, y and x axis directions under the mechanical arm tool coordinate system according to the following formula; forming motion instructions (X, Y, Z, A, B, C) according to the translation amounts X, Y, Z along the X, Y and Z axis directions and the rotation angles A, B, C around the Z, Y and X axis directions; executing a motion instruction to control the puncture needle to move to a preset position to hover;

2. The method of controlling an auxiliary puncture robot according to claim 1, wherein the step of receiving in-vivo image data including a body surface contour of the patient is receiving CT tomographic image data or B-mode image data or magnetic resonance image data.

3. The method of controlling the auxiliary puncture robot according to claim 1, wherein the step of registering the patient body surface point cloud data with the in-vivo image data of the patient further comprises the sub-step of: calculating a conversion matrix M of the scanner coordinate system and the in-vivo image data coordinate system;

"convert puncture planning path under the in-vivo image data coordinate system into puncture planning path under the robot arm tool coordinate system" is to calculate conversion based on the conversion matrix N and the conversion matrix M of the scanner coordinate system and the robot arm tool coordinate system.

4. The method for controlling an auxiliary penetration robot according to claim 3, further comprising: and acquiring a transformation matrix M between the scanner coordinate system and the mechanical arm tool coordinate system.

5. A method of assisting a penetration robot in controlling, the method comprising:

receiving in-vivo image data of a patient, which comprises a body surface contour, and calculating a puncture planning path under an in-vivo image data coordinate system according to the in-vivo image data, wherein the puncture planning path comprises puncture needle position coordinates and a needle direction vector;

scanning the body surface of the patient by a scanner to acquire 3D image data of the body surface of the patient;

registering the body surface 3D image data of the patient with the in-vivo image data of the patient so as to carry out image fusion reconstruction on the body surface point cloud data of the patient and the in-vivo image data, thereby obtaining a three-dimensional fusion image of the body surface point cloud data of the patient and the in-vivo image data;

m is a first transformation matrix, and N is a second transformation matrix, wherein the first transformation matrix is obtained by the following method: acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system; scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system; calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system; calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system; the second transformation matrix is obtained by: acquiring at least three point coordinates of a body surface point cloud coordinate system on the body surface profile of a patient, acquiring coordinates of the three points in a CT data coordinate system, calculating translation amount and performing singular value division Obtaining solution;

6. The method of controlling an auxiliary puncture robot according to claim 5, wherein the step of receiving in-vivo image data including a body surface contour of the patient is receiving CT tomographic image data or B-mode image data or magnetic resonance image data.

7. A method of assisting a penetration robot in controlling, the method comprising:

scanning a body surface of a patient by a scanner mounted on the robot to acquire 3D image data of the patient;

registering the 3D image data of the patient with the in-vivo image data of the patient so as to carry out image fusion reconstruction on the body surface point cloud data of the patient and the in-vivo image data, thereby obtaining a three-dimensional fusion image of the body surface point cloud data of the patient and the in-vivo image data;

M is a first transformation matrix, and N is a second transformation matrix, wherein the first transformation matrix is obtained by the following method: acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system; scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system; calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system; calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system; second variationThe matrix is obtained by: acquiring at least three point coordinates of a body surface point cloud coordinate system on the body surface profile of a patient, acquiring coordinates of the three points in a CT data coordinate system, and acquiring the coordinates by calculating translation quantity and performing singular value decomposition;

8. A method of assisting a penetration robot in controlling, the method comprising:

acquiring in-vivo image data containing body surface contours of a patient and a puncture planning path, wherein the puncture planning path comprises puncture needle position coordinates and a needle direction vector;

m is a first transformation matrix, and N is a second transformation matrix, wherein the first transformation matrix is obtained by the following method: acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system; scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system; calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system; calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system; the second transformation matrix is obtained by: acquiring at least three body surface point cloud coordinate systems on body surface contours of patients The point coordinates are obtained, coordinates of the three points in a CT data coordinate system are obtained through calculating translation quantity and singular value decomposition;

9. An auxiliary piercing robot, comprising:

a mechanical arm;

The processing unit is used for registering the 3D body surface image data of the patient from the scanner and the in-vivo image data of the patient containing the body surface outline so as to carry out image fusion reconstruction on the body surface point cloud data of the patient and the in-vivo image data to obtain a three-dimensional fusion image of the body surface point cloud data of the patient and the in-vivo image data; the processing unit converts a puncture planning path under an in-vivo image data coordinate system into a puncture planning path under a mechanical arm tool coordinate system of the robot, wherein the puncture planning path comprises puncture needle position coordinates and a needle inserting direction vector; the processing unit calculates the preset position of the puncture needle arranged on the mechanical arm according to the puncture planning path under the mechanical arm tool coordinate system;

the control unit is used for controlling the puncture needle to move to the preset position according to a puncture planning path under the mechanical arm tool coordinate system, and the preset position is a position where the puncture needle hovers above the body surface of a patient;

M is a first transformation matrix, and N is a second transformation matrix, wherein the first transformation matrix is obtained by the following method: acquiring a center coordinate of the calibration piece under a mechanical arm tool coordinate system; scanning the calibration piece through a scanner to obtain the center coordinate of the calibration piece under the scanner coordinate system; calculating the center coordinate of the calibration piece under the mechanical arm tool coordinate system and the gravity center translation vector of the center coordinate of the calibration piece under the scanner coordinate system; calculating the gravity center translation vector according to a singular value decomposition method to obtain a transformation matrix between a scanner coordinate system and a mechanical arm tool coordinate system; the second transformation matrix is obtained by: acquiring at least three point coordinates of a body surface point cloud coordinate system on the body surface profile of a patient, acquiring coordinates of the three points in a CT data coordinate system, and acquiring the coordinates by calculating translation amount and performing singular value decomposition；

10. The auxiliary penetration robot of claim 9, wherein the processing unit is in data connection with a CT device, the in-vivo image data and the penetration plan path being from the CT device.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 8 when the computer program is executed.

12. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 8.