CN115778554B - Catheter robot, registration method thereof and readable storage medium - Google Patents

Abstract

The application provides a catheter robot, a registration method thereof and a readable storage medium, wherein the method comprises the following steps: acquiring an actual path point of the catheter and/or the instrument by means of a sensor arranged on the catheter and/or the instrument on which the catheter is mounted; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in the anatomical model according to the first transformation matrix; determining a target pipeline centerline segment corresponding to the simulated path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided; calculating the centroid of the center line section of the target pipeline; if the centroid is different from the plurality of skeleton points included in the target pipeline center line segment, determining a connecting line between the centroid and the simulation path point, and selecting one of the plurality of skeleton points with the shortest distance from the connecting line as a matching point of the actual path point. By means of the method, accuracy of point cloud registration can be improved, and therefore accuracy of navigation in catheterization is improved.

Description

Catheter robot, registration method thereof and readable storage medium

Technical Field

The present application relates to the technical field of surgical robots, and in particular, to a catheter robot, a registration method of the catheter robot, and a readable storage medium.

Background

Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during a medical procedure to reduce patient recovery time, discomfort, and adverse side effects. In minimally invasive medical techniques it is often necessary to insert a catheter through a natural orifice in the patient's anatomy or through a surgical incision, through the complex structure of the body canal to reach or be adjacent to the target under the cooperation of an intraoperative navigation system.

In most cases, the catheter is moved along the body duct (e.g. bronchi, vessels, ureters, etc.) without freely shuttling in a manner that would damage the duct, so it is considered that the path of movement of the catheter/instrument is limited to the body duct, has a similarity to the centre line of the body duct, and in particular for smaller ducts, has a higher similarity to the centre line due to the limited range of radial movement.

In accordance with this principle, the intraoperative navigation system can employ point cloud registration to detect and display the position of the catheter tip and/or the catheter-mounted instrument tip in real-time. Specifically, the path data of the catheter/instrument movement can be acquired through a sensor and matched with the pipeline central line data in the anatomical model of the patient, and the coordinate transformation between the two is determined through point cloud registration, so that the real-time position of the catheter/instrument is obtained.

The point cloud registration in the related art adopts the criterion of shortest Euclidean distance to find the corresponding point pair, and then calculates coordinate conversion according to the corresponding point pair. However, due to the complexity of the human body pipeline and the influence of the patient movement, the error corresponding point pair may occur often by adopting the criterion of shortest euclidean distance, for example, the point on the pipeline central line corresponding to the path point of the catheter is not in the pipeline where the catheter is currently positioned, and the like, so that the accuracy of point cloud registration is not high, and the accuracy of navigation in the catheterization is affected.

Disclosure of Invention

The embodiment of the application provides a registration method, a registration device and terminal equipment of a catheter robot, which can solve the problem that accurate registration of path data is difficult to carry out in the related technology.

In a first aspect, embodiments of the present application provide a catheter robot comprising a robotic arm, a catheter instrument engaged with a power section of the robotic arm, a processor communicatively coupled to the robotic arm, the catheter instrument comprising an instrument pod configured to engage with the power section and a catheter coupled to the instrument pod, the catheter and/or catheter-mounted instrument being provided with a sensor for measuring a position of the catheter and/or instrument, the processor being configured to perform the steps of: acquiring actual path points of the catheter and/or instrument with the sensor; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to the first transformation matrix, wherein the anatomical model comprises a pipeline central line; determining a target pipeline centerline segment corresponding to the simulated path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided; calculating the centroid of the center line section of the target pipeline; if the centroid is different from the plurality of skeleton points included in the target pipeline center line segment, determining a connecting line between the centroid and the simulation path point, and selecting one of the plurality of skeleton points with the shortest distance from the connecting line as a matching point of the actual path point.

Wherein the processor is configured to perform the following steps after determining the connection between the centroid and the simulated waypoint: if the centroid is one of a plurality of skeleton points, a first foot from a simulation path point to a projection line is obtained, a second foot from each skeleton point to the projection line is obtained, and the projection line is a straight line passing through two end points of a target pipeline center line segment; and selecting a skeleton point corresponding to a second foot drop with the shortest distance between the first foot drops as a matching point of the actual path point.

Wherein the processor is configured to perform the steps of: updating the point pair set by utilizing a first point pair formed by the actual path points and the matching points thereof; if the point pair set meets a first condition, updating the first transformation matrix by using the point pair set, wherein the first condition comprises that the number of point pairs in the point pair set is larger than a preset value.

Wherein the processor is configured to perform the following steps in a set of matching point pair update point pairs consisting of an actual path point and its matching point: judging whether a second point pair comprising a matching point exists in the point pair set; if the first distance is smaller than the second distance, the second point pair is replaced with the first point pair.

Wherein the processor is configured to perform the following steps in updating the first transformation matrix using the set of point pairs: substituting the point pairs in the point pair set into an objective function to calculate a latest second transformation matrix; the first transformation matrix is updated according to the latest second transformation matrix.

The first transformation matrix is a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix, wherein the weight of the latest second transformation matrix is larger than that of the historical second transformation matrix.

Wherein the processor is configured to perform the following steps prior to determining a target pipe centerline segment corresponding to the simulated waypoint in a plurality of pipe centerline segments into which the pipe centerline is divided: extracting feature skeleton points from a plurality of skeleton points included in a pipeline center line, wherein positions of the skeleton points are represented by codes, and the codes of the skeleton points are s=x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ Wherein x is ₁ 、x ₂ 、x ₃ Respectively the coordinates of skeleton points in the first dimension, the second dimension and the third dimension of the coding sequence, R ₁ 、R ₂ The size of the anatomical model in the first dimension and the second dimension of the coding sequence are respectively; the pipeline center line is divided into a plurality of pipeline center line segments according to the characteristic framework points.

Wherein the processor is configured to perform the following steps in extracting feature skeleton points from a plurality of skeleton points comprised by the pipeline centerline: determining a starting point; starting from a starting point, searching skeleton points by using a region growing method, and counting the number of connected skeleton points, wherein the number of connected skeleton points is the number of all child nodes in the neighborhood of the skeleton points; if the number of connected skeleton points is greater than 1 and the number of connected parent nodes of the skeleton points is equal to 1, the skeleton points are bifurcation points in the characteristic skeleton points, and if the number of connected skeleton points is equal to 1 and the number of connected parent nodes of the skeleton points is equal to 0, the skeleton points are tail end points in the characteristic skeleton points.

Wherein the processor is configured to perform the following steps in selecting the starting point: randomly selecting an integrated layer; counting the total number of skeleton points in the layers at the two sides of the body layer in the direction perpendicular to the cross section to determine the extending direction of the air passage; and searching skeleton points of the tracheal entrance as a starting point according to the total number of skeleton points in the body layer and the extending direction of the airway.

Wherein the processor is configured to perform the steps of: counting the number of skeleton points in the pipeline center line segment; deleting the pipeline center line segments with the number smaller than a preset threshold value.

In a second aspect, embodiments of the present application provide a registration method of a catheter robot, the method including: acquiring an actual path point of the catheter and/or the instrument by means of a sensor arranged on the catheter and/or the instrument on which the catheter is mounted; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to the first transformation matrix, wherein the anatomical model comprises a pipeline central line; determining a target pipeline center line segment corresponding to the simulated path point in a plurality of pipeline center line segments into which the pipeline center line is divided, wherein the target pipeline center line segment comprises a plurality of skeleton points; calculating the centroid of the center line section of the target pipeline; if the centroid is different from the plurality of skeleton points included in the target pipeline center line segment, determining a connecting line between the centroid and the simulation path point, and selecting one of the plurality of skeleton points with the shortest distance from the connecting line as a matching point of the actual path point.

In a third aspect, embodiments of the present application provide a computer-readable storage medium having stored thereon computer program instructions for execution by a processor to perform the method of the second aspect of the present application.

Compared with the prior art, the embodiment of the application has the beneficial effects that: by the embodiment of the application, the actual path point of the catheter and/or the instrument is acquired by using a sensor; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to the first transformation matrix, wherein the anatomical model comprises a pipeline central line; determining a target pipeline centerline segment corresponding to the simulated path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided; calculating the centroid of the center line section of the target pipeline; determining a connection line between the centroid and the simulated path point; if the centroid is different from the plurality of skeleton points included in the target pipeline center line segment, selecting one of the plurality of skeleton points, which is the shortest in distance from the connecting line, as a matching point of the actual path point. In order to adapt to complex human body environment, the method and the device firstly determine the target pipeline center line segment corresponding to the simulation path point so as to reduce errors that the matching point is not in the pipeline where the catheter is currently located, select a skeleton point with the shortest connecting line between the simulation path point and the centroid of the target pipeline center line segment, approximately consider the skeleton point as an intersection point of the connecting line and the target pipeline center line segment, take the intersection point as a matching point of an actual path point, more accurately reflect the matching relationship between the path point of the motion of the catheter in the pipeline and the pipeline center line, improve the accuracy of point cloud registration, namely obtain a more accurate first transformation matrix, and further improve the navigation accuracy in catheterization.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only embodiments of the invention and that other drawings may be obtained based on the drawings provided without inventive effort for a person skilled in the art.

FIG. 1 is a schematic view of a catheter robot according to an embodiment of the present application;

FIG. 2 is a schematic illustration of a catheter apparatus and a power section according to one embodiment of the present application;

fig. 3 is a flowchart of a registration method of a catheter robot according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of determining a target pipeline centerline segment corresponding to a simulated waypoint according to an embodiment of the present application;

FIG. 5 is a schematic illustration of determining matching points in the case where the centroid is not a skeleton point on a target pipe centerline segment in an embodiment of the present application;

FIG. 6 is a schematic illustration of a centroid being a skeleton point on a target pipe centerline segment in an embodiment of the present application;

FIG. 7 is a schematic illustration of determining matching points where the centroid is a skeleton point on a target pipe centerline segment in an embodiment of the present application;

FIG. 8 is a schematic diagram of the specific flow of S17 in FIG. 3;

FIG. 9 is a schematic diagram showing a specific flow of S18 in FIG. 3;

FIG. 10 is a flow chart of intra-operative navigation according to an embodiment of the present application;

FIG. 11 is a schematic flow chart of pipeline centerline segmentation according to an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of a pixel location represented by a single number code in an embodiment of the present application;

FIG. 13 is a schematic diagram showing a specific flow of S21 in FIG. 11;

FIG. 14 is a schematic diagram of the specific flow of S211 in FIG. 12;

FIG. 15 is a schematic structural diagram of a control system of a catheter robot according to an embodiment of the present disclosure;

fig. 16 is a schematic structural diagram of a computer readable storage medium according to an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

It is noted that unless otherwise indicated, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs.

The registration method of the catheter robot, and the computer-readable storage medium according to the embodiments of the present application are described below with reference to the accompanying drawings.

Fig. 1 illustrates a catheter system 1000 provided in an embodiment of the present application. Catheter system 1000 includes an imaging cart 100, a cart 200 and a master controller 300 respectively connected to the imaging cart 100, a catheter instrument 400 that can be coupled to the cart 200, a sensor system 500 connected to the cart 200, a control system 600 for effecting control among the catheter instrument 400, the master controller 300, the sensor system 500, and the imaging cart 100, and the like. Wherein, the master controller 300 may be connected with the cart 200 by wire or wirelessly. When an operator performs various procedures on a patient beside the cart 200, the main controller 300 is operated to trigger control instructions, and the catheter instrument 400 is driven by the cart 200 to advance, retract, bend, and turn.

The trolley 200 may be generally moved to the side of the operating table for engaging the catheter instrument 400 and controlling the catheter instrument 400 to be raised and lowered in a vertical direction or to be translated in a horizontal direction or to be moved in a non-vertical and non-horizontal direction under control instructions, thereby providing a better pre-operative preparation angle for the operation of the catheter instrument 400. The control command may be a command triggered by an operator by operating the master controller 300, or may be a command triggered by the operator directly clicking or pressing a key provided on the cart 200. Of course, in other embodiments, the control command may also be a voice control or a command triggered by a force feedback mechanism.

As shown in fig. 1, the trolley 200 may further include a base 210, a sliding base 220 capable of moving up and down along the base 210, and 2 mechanical arms 230 fixedly connected to the sliding base 220. The robotic arm 230 may include a plurality of arm segments coupled at joints that provide the robotic arm 230 with a plurality of degrees of freedom, e.g., seven degrees of freedom corresponding to seven arm segments. The distal end of the mechanical arm 230 is provided with a power portion (not shown in the figure), and the power portion of the mechanical arm 230 is used for engaging the catheter apparatus 400 and controlling the distal end of the catheter apparatus 400 to bend and turn correspondingly under the driving action of the power portion. Wherein the 2 mechanical arms 230 may be identical or partially identical in structure, one mechanical arm 230 for engaging the inner catheter instrument 410 and the other mechanical arm 230 for engaging the outer catheter instrument 420. When the outer catheter device 420 is installed, the catheter of the inner catheter device 410 may be inserted into the catheter of the outer catheter device 420 after the outer catheter device 420 is installed.

The sensor system 500 has one or more subsystems for receiving information about the catheter instrument 400. The subsystem may include: a position sensor system; a shape sensor system for determining the position, orientation, speed, velocity, pose, and/or shape of the tip of the catheter instrument 400 and/or along one or more sections of a catheter that may constitute the catheter instrument 400; and/or a visualization system for capturing images from the distal end of the catheter instrument 400.

The imaging vehicle 100 may be provided with a display system 110, a flushing system (not shown), and the like. The display system 110 is used to display images or representations of the surgical site and catheter instrument 400 generated by the subsystems of the sensor system 500. Real-time images of the surgical site and catheter instrument 400 captured by the visualization system may also be displayed. Image data from imaging techniques such as Computed Tomography (CT), magnetic Resonance Imaging (MRI), optical Coherence Tomography (OCT), ultrasound, and the like may also be used to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intra-operative image data may be presented as two-dimensional, three-dimensional or four-dimensional (e.g., time-based or rate-based information) images and/or as images from models created from the preoperative or intra-operative image dataset, and virtual navigation images may also be displayed. In the virtual navigation image, the actual position of the catheter instrument 400 is registered with the pre-operative image to externally present the virtual image of the catheter instrument 400 within the surgical site to the operator.

The control system 600 includes at least one memory and at least one processor. It is understood that the control system 600 may be integrated into the cart 200 or the imaging cart 100, or may be provided separately. The control system 600 may support wireless communication protocols such as IEEE 802.11, irDA, bluetooth, homeRF, DECT, and wireless telemetry, among others. The control system 600 may transmit one or more signals indicative of movement of the catheter instrument 400 by the power section to move the catheter instrument 400. The catheter apparatus 400 may extend to a surgical site within the body via an opening or surgical incision of a natural orifice of the patient.

Further, the control system 600 may include a mechanical control system (not shown) for controlling the movement of the catheter instrument 400 and an image processing system (not shown), and thus may be integrated in the trolley 200. The image processing system is used for virtual navigation path planning and thus may be integrated in the video car 100. Of course, the subsystems of the control system 600 are not limited to the specific cases listed above, and may be reasonably configured according to actual situations. The image processing system can image the surgical site by using the imaging technology based on the image of the surgical site recorded before or during the operation. Software used in conjunction with manual input may also convert the recorded images into two-or three-dimensional composite images of portions or whole anatomical organs or segments. During the virtual navigation procedure, the sensor system 500 may be used to calculate the position of the catheter instrument 400 relative to the patient's anatomy, which may be used to generate external tracking images and internal virtual images of the patient's anatomy, enabling registration of the actual position of the catheter instrument 400 with the preoperative images so that the virtual images of the catheter instrument 400 within the surgical site may be presented to the operator from the outside.

The inner catheter device 410 and the outer catheter device 420 are generally identical in structure composition and have an inner catheter 41 and an outer catheter 42, respectively, which are elongated and flexible, wherein the outer catheter 42 has a diameter slightly larger than the inner catheter 41 so that the inner catheter 41 may pass through the outer catheter 42 and provide a certain support for the inner catheter 41 so that the inner catheter 41 may reach a target site within a patient for tissue or cell sampling from the target site.

Certain movements of the master 300 may cause corresponding movements of the catheter instrument 400. For example, as an operator manipulates the directional shifter lever of master controller 300 upward or downward, the movement of the directional shifter lever of master controller 300 may be mapped to a corresponding pitching movement of the tip of catheter instrument 400; when the operator manipulates the directional shifter lever of the master controller 300 to move left or right, the movement of the directional shifter lever of the master controller 300 may be mapped to a corresponding yaw movement of the tip of the catheter instrument 400. In this embodiment, the master controller 300 may control the movement of the distal end of the catheter instrument 400 over a 360 ° spatial range.

Fig. 2 illustrates a catheter apparatus 400 provided in an embodiment of the present application. The catheter instrument 400 is configured to engage the power section 240 of the robotic arm 230, the catheter instrument 400 including an instrument pod 45 configured to engage the power section 240 and a catheter 48 coupled to the instrument pod 45. Wherein, the "engagement" refers to a state in which the driving force of the power unit 240 can be transmitted into the instrument box 45 and the catheter 48 can be normally moved when the instrument box 45 is mounted to the power unit 240. For example, the distal end of the catheter 48 may be bent, turned, or the like, by the driving force of the power unit 240.

The tip, which may also be referred to herein as a distal end or head, refers to the end that is distal from the cartridge 45; the front end, which may also be referred to as the proximal end or the tail end, refers to the end that is proximal to the cartridge 45.

The processor of the control system 600 is configured to perform the following steps to implement the registration method of the catheter robot provided in an embodiment of the present application. As shown in fig. 3, the method includes:

s11: the actual path point of the catheter and/or instrument is acquired with the sensor.

The actual path point is the path point of the tip of the catheter and/or the instrument carried by the catheter, which is acquired by the sensor, and is generally described by coordinates of the tip of the catheter and/or the instrument in the world coordinate system, or the surgical environment coordinate system. The sensor may comprise a position sensor and/or a shape sensor.

The position sensor may be provided on the catheter and/or the instrument. A position sensor may be a component of an electromagnetic positioning system. The electromagnetic positioning system may further comprise a magnetic field generating means and a magnetic field detecting means. The magnetic field generating component is used for generating a magnetic field, the position sensor can induce magnetic field change in the magnetic field, the magnetic field detecting component can detect the magnetic field change so as to detect the pose of the position sensor relative to the magnetic field/magnetic field generating component, the position of the catheter tail end/instrument tail end relative to the magnetic field/magnetic field generating component can be calculated by combining the coordinate conversion relation between the position sensor and the catheter tail end/instrument tail end, which are preset or calibrated, and the position of the catheter tail end/instrument tail end in the world coordinate system, namely the actual path point, can be calculated by combining the coordinate conversion relation between the magnetic field/magnetic field generating component and the world coordinate system.

A shape sensor system may be used to obtain the position of the catheter tip/instrument tip. For example, the shape sensor may comprise an optical fiber aligned with the catheter, the optical fiber bending sensor formed by the optical fiber may feed back the shape of the catheter, from which the position of the catheter tip/instrument tip relative to the base of the shape sensor may be calculated, and in combination with the position of the base of the shape sensor in the world coordinate system, the position of the catheter tip/instrument tip in the world coordinate system, i.e. the actual path point, may be calculated.

The actual path point reflects the position of the tail end of the catheter/the tail end of the instrument when the sensor feeds back, and the actual path point obtained by the repeated feedback of the same sensor is arranged according to the feedback time, so that the moving path of the tail end of the catheter/the tail end of the instrument can be obtained. The set of these actual waypoints may be referred to as a set of waypoints or a cloud of waypoints.

S12: a simulated waypoint of the catheter and/or instrument in the anatomical model corresponding to the actual waypoint is obtained from the first transformation matrix.

Medical images (such as CT, MRI, etc.) can be obtained by scanning a target area of a patient before an operation, a three-dimensional preoperative image is obtained by three-dimensional reconstruction of the preoperative image, and an anatomical model is obtained by appropriate image processing of the three-dimensional preoperative image. The image processing may include image segmentation for extracting a human body duct involved in an operation from a three-dimensional image, and skeleton extraction for the human body duct to obtain a duct center line for facilitating navigation in the operation. The pipeline center line is the center line of the human body pipeline, can also be called as the skeleton of the human body pipeline, is a curve for describing part of the geometric characteristics of the human body pipeline, is positioned at the middle part of the human body pipeline, has the same topological structure as the human body pipeline, and has a width of a single pixel. The central line of the pipeline can provide information required by navigation in operation, and meanwhile, compared with the original human body pipeline, the data volume of the pipeline is obviously reduced, the processing is more convenient, and the real-time navigation is facilitated.

The pipeline center line is a three-dimensional curve, and in practical application, the pipeline center line is often stored and used in a mode of a plurality of three-dimensional points, in other words, the pipeline center line comprises the three-dimensional points, the three-dimensional points can be called skeleton points, and a set of skeleton points can be called a skeleton point set or skeleton point cloud.

The anatomical model is a three-dimensional binary image in which the pixel value of each pixel (which may also be referred to as a voxel) is 0 or 1, which is used to indicate whether the pixel is a skeleton point. In the embodiments of the present application, an example in which the pixel value of a skeleton point is 1 and the pixel value of a non-skeleton point is 0 will be described. In fact, the values may be reversed in other embodiments, which are not limited in this regard.

The pixel value range is determined by the image bit depth supported by the display system, and the common value range is 0-255 or 0-1023. In displaying an anatomical model, in order to intuitively distinguish between skeleton points and non-skeleton points, the pixel values of the anatomical model are generally mapped, for example, the pixel values of the skeleton points are mapped to a maximum value (255 or 1023), displayed as white, and the pixel values of the non-skeleton points are mapped to 0, displayed as black. The reverse is of course possible, i.e. skeleton points are shown in black and non-skeleton points in white.

To facilitate subsequent registration, the channel centerlines may be preprocessed. The preprocessing may include resampling to extract feature skeleton points and/or tube centerlines from a set of skeleton points. Feature skeleton points generally include bifurcation points and end points for segmenting a conduit centerline. Whether the skeleton point is a characteristic skeleton point can be judged by counting the number of the skeleton points in the neighborhood of the certain skeleton point. The application provides a feature skeleton point extraction method based on coding, and the method is concretely described in the following embodiments. The bifurcation point and the end point reflect the structural characteristics of the human body duct itself, and in addition, for the segment with too long length/too large bending degree, new characteristic points can be introduced to further segment the segment to improve the registration accuracy.

To avoid disrupting the connectivity of the pipeline centerline, resampling is typically upsampling, and specific resampling parameters, such as resampling interval, number of interpolations, may be determined as needed. The order of the feature skeleton points is not limited if the preprocessing includes extraction and resampling.

The first transformation matrix may be a transformation matrix between the world coordinate system and the anatomical model, and the actual path points may be transformed from the world coordinate system into the coordinate system of the anatomical model using the first transformation matrix to obtain simulated path points.

The simulated path points reflect the position of the catheter end/instrument end in the anatomical model, and the position of the catheter end/instrument end in the displayed surgical site can be obtained by combining the display parameters of the surgical site, so that the surgical site and the catheter/instrument in the surgical site are fused and displayed, and the intra-operative navigation is realized. The surgical site may be displayed as a preoperative image and/or an intra-operative image. If preoperative image data is used for displaying the surgical site, as the anatomical model is obtained by processing the three-dimensional preoperative image, the position of the catheter tip/instrument tip in the surgical site can be obtained according to the coordinate conversion relation of the displayed surgical site relative to the three-dimensional preoperative image. If the operative site is displayed using intra-operative image data, registration of the intra-operative image with the three-dimensional pre-operative image/anatomical model is required to obtain a coordinate transformation relationship of the displayed operative site relative to the three-dimensional pre-operative image, and then to obtain the position of the catheter tip/instrument tip in the operative site.

In order to realize intra-operative navigation, the method calculates a transformation matrix between a world coordinate system and an anatomical model, namely a first transformation matrix by carrying out point cloud registration on a skeleton point set and a path point set.

The representative algorithm of point cloud registration is a nearest point iterative (Iterative closest point, ICP) algorithm, the core idea of which is to minimize the distance between two point sets, which are brought close to each other by iteration. The principle of ICP is briefly described below.

The basic method of ICP comprises two steps: 1. matching the point cloud Q and the point cloud P, and finding out a corresponding point pair between the point cloud Q and the point cloud P; 2. and calculating a transformation matrix between the point clouds Q and P according to the corresponding point pairs. One corresponding pair of points consists of two points, one from the point cloud P and the other from the point cloud Q, which are mutually matched points, and which are considered to be corresponding, i.e. which are substantially identical.

If the true and accurate corresponding point pairs can be directly found, the above process needs to be performed only once, so that a sufficiently accurate transformation matrix can be directly calculated. However, in practical application, it is difficult to directly find an accurate corresponding point pair, so that ICP can match according to the principle of closest distance (generally euclidean distance), that is, for each point in the point cloud P, a point closest to the point cloud Q is found as its matching point. A transformation matrix is then calculated based on the pairs of points. After completing one round of calculation, the ICP judges whether an iteration stop condition is met, if not, the conversion matrix obtained by the round of calculation is used for carrying out conversion updating on the point cloud P, and then the point cloud Q and the converted point cloud P are used for repeating the process until iteration is stopped.

Since ICP is relatively sensitive to initial values, if the point clouds P and Q initially differ significantly, direct use of ICP may not guarantee convergence. Therefore, the point cloud registration can be divided into two parts of coarse registration and fine registration, an initial transformation matrix is obtained by adopting the coarse registration, and then the fine registration is carried out by using ICP, and the ICP of the first round is used for matching the point cloud Q and the point cloud P transformed by using the initial transformation matrix.

In the process of moving the catheter along the human body pipeline, a movable range exists in the radial direction of the pipeline, so that the catheter possibly deviates from the central line of the pipeline, the thicker the pipeline is, the larger the movable range is, and the influence of organ movement is added, so that the movement path of the catheter and the shape of the central line of the framework possibly are different. Therefore, for the registration of the path point set and the skeleton point set, the optimal matching point pair is difficult to accurately find by adopting classical ICP, and the obtained registration result tends to fall into local optimum. For this purpose, the present application proposes an improved ICP fine registration method, see in particular the following description. Prior to performing the ICP fine registration method provided herein, coarse registration may be performed to obtain an initial value of the first transformation matrix.

Since in the surgical procedure, it is not necessary to match each actual path point, for example, in the case of ending ICP iteration, only virtual path points need to be obtained to achieve intra-operative navigation, and no subsequent registration step needs to be performed. Therefore, after this step, it can be determined whether the registration condition is satisfied, and if so, the subsequent step is executed again, otherwise, the following step is not executed. Registration conditions may include ICP iteration in progress, starting an ICP iteration, etc.

S13: a target pipe centerline segment corresponding to the simulated path point is determined among the plurality of pipe centerline segments into which the pipe centerline is divided.

The pipeline centerline may be divided into a plurality of pipeline centerline segments based on the feature skeleton points. Each pipe center line segment has two end points, one of which is a starting point and the other of which is a finishing point, and which is the starting point and which is the finishing point can be set according to anatomical characteristics, a motion path of a catheter robot, actual requirements and the like. For example, if the body duct is an airway, since the airway is a multi-stage branched tree structure, one branch point of the two endpoints, which is higher (i.e., closer to the trachea), is generally taken as a starting point, and one branch point/endpoint of the two endpoints, which is lower (i.e., closer to the alveoli), is taken as an ending point, which can be set inversely as required.

Based on the motion characteristic of the catheter robot in the human body pipeline, the target pipeline center line segment can be determined first to improve the accuracy of subsequent matching, and the target pipeline center line segment is the pipeline center line segment where the estimated simulation path point is located.

First, the simulated waypoints are within the target pipe centerline segment, meaning that the simulated waypoints must be between the two endpoints of the target pipe centerline segment. Is converted into a relative position relation of space, and is segmented for the central line of the pipeline

Only satisfy the analog path point +.>

At the start->

Is +.>

At the same time at the end +.>

Is +.>

Is possible for the target pipe centerline segment. From this, the +.>

And->

：

(1)

(2)

Only satisfy

And->

Is the line centerline segment of the pipeline, which is the analog path point +.>

The segments where the pipeline is likely to be located (possible segments for short) are primarily screened accordingly. If no meeting is found->

And is also provided with

The line center line of the line is segmented, the previous analog path point +.>

Is taken as the analog path point +.>

Is defined by a target pipe centerline segment.

For the possible sections which are primarily screened, if the number of the possible sections is 1, the possible sections are directly used as the center line sections of the target pipeline; if the number is greater than 1, further screening is required. In particular, an analog waypoint may be calculated

Third distance between projection line segments to possible segmentation +.>

And selecting the smallest third distance as the target pipeline center line segment. For the pipeline centerline segment->

The projection line of the lens is the starting point +.>

And end->

Straight line segments therebetween.

As illustrated in the accompanying drawings, the broken lines in the figure represent the pipe centerlines, a, b, c, of the three pipe centerline segments, satisfying the requirement for the simulated path point A

And->

And (2) the segments a and b, and comparing the third distance between the projection line segments of the simulation path point A and the segments a and b to obtain the segment a of the center line of the target pipeline corresponding to the simulation path point A.

If the simulated path point is close to the bifurcation of the human body pipeline, more than one minimum third distance may occur, or there may be a third distance very close to the minimum third distance, the above-mentioned manner of determining the centerline segment of the target pipeline may have a certain error. For this purpose, the above-described determination may be supplemented to some extent.

Specifically, for the possible sections which are primarily screened, judging whether to screen again, if so, screening again from the possible sections to obtain candidate sections, calculating the moving direction of the catheter, and selecting one closest to the moving direction from the candidate sections as a target pipeline center line section; otherwise, directly selecting the smallest third distance as the target pipeline center line segment.

When determining whether a rescreen is required, there may be two ways of determining the position based on the third distance and based on the simulated path point. The two modes can be used independently, or can be used in a combined mode or in a combined mode.

Based on the third distance, a small range can be set, if the difference value between the third distance and the minimum third distance falls into the range except the minimum third distance, the pipeline center line segment corresponding to the third distance of which the difference value falls into the range is the candidate segment. The difference here may be an absolute value, i.e. the difference itself, or a relative value, i.e. the ratio of the difference to the minimum third distance.

Based on the position of the simulated path point, it may be determined whether an endpoint exists among the endpoints of the possible segments, which belongs to the bifurcation point and whose distance from the simulated path point is smaller than a threshold value, and if so, all segments to which the endpoint belongs are selected as candidate segments. The distance between the simulated path point and the end point may be a euclidean distance between the two, or a ratio of the euclidean distance to a length of a projected line segment to which the end point belongs, or a projected distance between the simulated path point and the end point (a euclidean distance between a foot of the simulated path point on the projected line segment and the end point), or a ratio of the projected distance to a length of a projected line segment to which the end point belongs, or a ratio of a corresponding point of the simulated path point (i.e., an intersection point of a centroid of the simulated path point to the pipe centerline segment and the pipe centerline segment) to the end point, or a ratio of the distance to a length of the projected line segment to which the end point belongs.

The prior simulated waypoints can be calculated

Point to the current simulated path point +.>

The vector of (a) is a positive integer and h is a positive integer, and can be set experimentally, empirically, or the like. The direction of the candidate segment may be a vector pointing from its start point to its end point. And then calculating the included angle between the moving direction of the catheter and the candidate segment in a vector inner product mode, and selecting one with the smallest included angle as the center line segment of the target pipeline.

S14: the centroid of the target pipe centerline segment is calculated.

The centroid, i.e. the center of the shape, may have its coordinates as an average of the coordinates of all skeleton points that make up the target pipe centerline segment. Since the target tube centerline segment is processed from the medical image, its density can be considered uniform, and thus the centroid of the target tube centerline segment is the centroid.

If the centroid is different from the skeleton points forming the target pipeline center line segment, jumping to S15; if the centroid is one of the skeleton points, the process proceeds to S16.

S15: and determining a connecting line between the centroid and the simulated path point, and selecting one with the shortest distance from the connecting line from a plurality of skeleton points as a matching point of the actual path point.

Because the pipeline center line reflects the shape of the human body pipeline, the pipeline center line is generally stored and used in a point set mode, often cannot be expressed in an analytic mode, an analytic solution of an intersection point cannot be obtained, and even if the analytic solution can be obtained, the pipeline center line is not necessarily just a skeleton point. Thus, in this application, one skeleton point on the target pipe centerline segment that is the shortest distance from the connection line is approximated as the intersection between the target pipe centerline segment and the connection line.

The intersection point between the center line segment of the target pipeline and the connecting line is mainly adopted as a matching point, the registration mode is more in line with the motion mode of the catheter robot in the human body pipeline, and the found matching point is more accurate.

For example, as shown in FIG. 5, the dashed line in the figure represents the target pipe centerline segment, s _c The centroid of the centerline segment for the target conduit, the solid line represents the path of the catheter robot in the anatomical model, simulating the path point t _n And centroid s _c The intersection point of the connecting line of (a) and the central line segment of the target pipeline is approximately the skeleton point q _k Skeleton point q _k I.e. the matching point of the actual path point corresponding to tn.

There is a special case: the centroid of a certain target pipe centerline segment is just one skeleton point thereon. As shown in FIG. 6, the dashed line in the figure represents the target tubular centerline segment, the solid line represents the path of the catheter robot in the anatomical model, s _c Centroid and s for segmenting the centerline of a target pipeline _c Is a skeleton point on the centerline segment of the target pipeline, in which case the intersection of the line with the centerline segment of the target pipeline must be the centroid s, no matter where the simulated path point is located _c That is, the matching points of all the path points corresponding to the centerline segment of the target pipeline are centroid s _c This obviously violates the intent of the match. In this case, therefore, the intersection point cannot be adopted as the matching point.

S16: acquiring a first foot from a simulation path point to a projection line, and acquiring a second foot from each framework point to the projection line, wherein the projection line is a straight line passing through two end points of a target pipeline center line segment; and selecting a skeleton point corresponding to a second foot drop with the shortest distance between the first foot drops as a matching point of the actual path point.

If the centroid of the target pipeline center line segment is just one skeleton point on the centroid, the matching point is selected according to the principle that the projection distance is nearest. Specifically, a first foot of a simulation path point to a projection line is obtained, the projection line is a straight line passing through two end points of a target pipeline center line segment, and a second foot of each skeleton point to the projection line is obtained. And selecting a skeleton point corresponding to a second foot drop with the shortest distance between the first foot drops as a matching point of the actual path point. Since the second foot drop and the fourth distance of each skeleton point are determined after the segmentation is completed, in order to shorten the calculation time, the fourth distance between the second foot drop of all skeleton points on the pipeline center line and the start point/end point of the belonging pipeline center line segment can be calculated and stored in advance. When the projection distance is needed to be used later, the projection distance can be obtained by only calling the fourth distance of all skeleton points on the center line section of the target pipeline, calculating the fifth distance between the first drop foot and the starting point/ending point of the center line section of the target pipeline and calculating the difference between the fourth distance and the fifth distance.

For example, as shown in FIG. 7, the solid lines in the figure represent the path of the catheter robot in the anatomical model, the dashed lines represent the target tube centerline segments, the dashed lines represent projection lines, and the simulated path points

The first drop to projection line is +.>

And start->

The fifth distance between is->

Skeleton point->

The second drop foot to projection line is +.>

And start->

The fourth distance between is->

Analog Path Point->

And skeleton point->

The projection distance between them is->

。

S17: and updating the point pair set by using the first point pair formed by the actual path points and the matching points.

Since the transformation matrix generally has 6 degrees of freedom, a single point pair does not solve for a unique solution, and therefore a set of point pairs is required to temporarily store a plurality of point pairs to calculate the transformation matrix. After matching is completed, an attempt may be made to update the set of point pairs with the resulting point pairs. One way to update is to add the first pair directly to the set of pairs. However, due to the complexity of the catheter robot motion, it may happen that multiple path points are matched to the same skeleton point, which is detrimental to subsequent calculations. To address this problem, a first pair of points may be determined prior to joining the set of point pairs, as follows.

As shown in fig. 8, in some embodiments, the steps specifically include:

s171: a determination is made as to whether a second point pair exists in the set of point pairs that includes a matching point.

If not, jumping to S172; if so, the process proceeds to S173.

S172: the first pair is added to the set of pairs.

S173: and calculating a first distance according to the actual path point and the matching point, calculating a second distance according to the path point and the matching point in the second point pair, and replacing the second point pair with the first point pair if the first distance is smaller than the second distance.

The first distance may be referred to as the distance of the first pair of points and the second distance may be referred to as the distance of the second pair of points. First, theA distance may be the current actual path point

Corresponding analog path point->

The euclidean distance between the second distance and the matching point may be the euclidean distance between the simulated path point corresponding to the path point in the second pair of points and the matching point. Alternatively, the first distance may be the current actual path point +.>

Corresponding analog path point->

The second distance may be a projection distance between the simulated path point corresponding to the path point in the second point pair and the matching point.

If the first distance is greater than or equal to the second distance, the set of point pairs is not modified.

It may then be determined whether the updated set of point pairs satisfies a first condition, the first condition being a condition for updating the transformation matrix. The first condition may include the number of pairs in the set of pairs being greater than a preset value. In theory, only 3 point pairs are needed to calculate the unique solution of the transformation matrix, and in order to reduce the influence of errors, a plurality of point pairs are often used to estimate the optimal solution of the transformation matrix by a least square method in practical application. The preset value is the minimum value of the number of points needed by the set estimated transformation matrix. The preset value must be greater than or equal to 3, and the specific value can be set as required, for example, 100, 200, etc.

If the set of point pairs satisfies the first condition, the process proceeds to S18.

S18: the first transformation matrix is updated using the set of point pairs.

As shown in fig. 9, in some embodiments, the steps specifically include:

s181: substituting the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix.

An objective function may be designed in advance, where the objective function generally reflects the error of the matching point pair, and then each point pair in the point pair set is substituted into the objective function, and a transformation matrix that can make the objective function minimum is estimated as the latest second transformation matrix, or the second transformation matrix of the iteration of this round.

The usual objective function of ICP can be used:

（3）

where N is the total number of pairs in the set of pairs, i is the sequence number of pairs in the set of pairs, i=1, 2, …, N,

for the coordinates of the matching point in the anatomical model in the ith point pair, +.>

The coordinates of the actual path point in the world coordinate system in the ith point pair are given by T, which is a transformation matrix.

Because the human body duct is thick and thin, the larger the deviation range of the path relative to the central line in the thicker duct. In order to obtain a more accurate transformation matrix, the radius parameters of the human body pipeline can be introduced into an objective function, and the improved objective function is as follows:

（4）

wherein, the liquid crystal display device comprises a liquid crystal display device,

and (3) the average radius of the target pipeline center line segment n corresponding to the simulated path point corresponding to the actual path point in the ith point pair.

S182: the first transformation matrix is updated according to the latest second transformation matrix.

The latest second transformation matrix may be directly used as the updated first transformation matrix. Alternatively, to accommodate the dynamic local registration case, a historical second transformation matrix may be introduced at the time of computing the first transformation matrix. Optionally, the first transformation matrix is a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix, wherein the weight of the latest second transformation matrix is greater than the weight of the historical second transformation matrix.

For example, a specific calculation formula of the first transformation matrix is:

（5）

where M represents the sequence number of the last iteration that has been completed, M represents the sequence number of the historical iteration (the iteration preceding the last iteration), m=1, 2, …, M-1.

After each round of ICP iteration is completed, the actual path points are next transformed using the updated first transformation matrix.

Through implementation of the embodiment, in order to adapt to a complex human body environment, the method and the device firstly determine the target pipeline center line segment corresponding to the simulation path point so as to reduce errors that the matching point is not in the pipeline where the catheter is currently located, select a skeleton point with the shortest connecting line between the simulation path point and the centroid of the target pipeline center line segment, and approximately consider the skeleton point as an intersection point of the connecting line and the target pipeline center line segment, and take the intersection point as a matching point of an actual path point, so that the matching relationship between the path point of the motion of the catheter in the pipeline and the pipeline center line can be reflected more accurately, the accuracy of point cloud registration is improved, namely a more accurate first transformation matrix is obtained, and the navigation accuracy in a catheterization is improved.

The following description, in conjunction with the accompanying drawings, exemplifies the complete process of intra-operative navigation based on an anatomical model extracted from a three-dimensional lung CT image (obtained by three-dimensional reconstruction of a plurality of lung tomographic CT images) and the airway centerline therein, wherein the same parts as in the previous embodiments are not repeated.

The airways include the trachea and bronchi, which are generally in a stepwise bifurcated tree structure, and thus may also be referred to as airway trees. The trachea splits into left and right main bronchi at the extremities, which are distributed deep into the left and right lungs, into lobar bronchi (typically three branches of the left lung and two branches of the right lung) which split into segment bronchi, and finally into alveoli step by step.

As shown in fig. 10, the process of intra-operative navigation in an embodiment of the present application includes:

s101: an anatomical model is acquired that contains the airway centerline.

S102: resampling the airway centerline.

S103: bifurcation and end points are extracted from a plurality of skeletal points included in the airway centerline.

S104: the airway centerline is divided into a plurality of segments using bifurcation and end points.

S105: coarse registration is performed on the world coordinate system and the anatomical model.

Coarse registration is typically performed by acquiring coordinates of a small number of points with distinct features (e.g., the main carina, the first level, second level carina of the left and right lobes, etc.) in the world coordinate system and in the anatomical model.

S106: the actual path point of the catheter tip is acquired with a sensor.

S107: and carrying out coordinate transformation on the actual path points by using the first transformation matrix to obtain corresponding simulation path points.

S108: and judging whether the registration condition is satisfied.

If so, jumping to S109; otherwise, the process goes to S106.

S109: a target segment corresponding to the simulated waypoint is determined.

S110: the centroid of the target segment is calculated.

S111: and judging whether the centroid is a skeleton point on the target segment.

If yes, go to S112, otherwise go to S113.

S112: acquiring a first foot from a simulation path point to a projection line, and acquiring a second foot from each skeleton point included in a target segment to the projection line, wherein the projection line is a straight line passing through two endpoints of the target segment; and selecting a skeleton point corresponding to a second foot drop with the shortest distance between the first foot drops as a matching point of the actual path point.

Jump to S114.

S113: and determining a connecting line between the centroid and the simulated path point, and selecting one with the shortest distance from the connecting line from a plurality of skeleton points included in the target segment as a matching point of the actual path point.

S114: a determination is made as to whether a second point pair exists in the set of point pairs that includes a matching point.

If not, jumping to S115; otherwise, the process goes to S116.

S115: and adding the first point pair formed by the actual path points and the matching points into the point pair set.

Jump to S118.

S116: and calculating a first distance according to the actual path point and the matching point, calculating a second distance according to the path point and the matching point in the second point pair, and comparing the magnitudes of the first distance and the second distance.

If the first distance is smaller than the second distance, jumping to S117; otherwise, the process goes to S118. In some embodiments of the present invention, in some embodiments,

s117: the second point pair is replaced with the first point pair.

S118: a determination is made as to whether the set of pairs of points satisfies a first condition.

If so, jumping to S119; otherwise, the process goes to S106.

S119: substituting the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix.

S120: a weighted average of the most recent second transformation matrix and at least a portion of the historical second transformation matrix is calculated as an updated first transformation matrix.

Jump to S106.

The process of S106-S120 may be referred to as dynamic fine registration of the world coordinate system and the anatomical model.

As shown in fig. 11, in an embodiment of the present invention, the process of segmenting the center line of the pipeline specifically includes:

s21: feature skeleton points are extracted from a plurality of skeleton points that make up a centerline of the pipeline.

In a conventional three-dimensional image, the position of a pixel is represented by three-dimensional coordinates (x, y, z), 3 numbers are required to represent the position of the pixel, a large storage space is occupied, and the search speed is adversely affected because the 3 numbers are required to be compared in the search process. To reduce the space occupied by storing three-dimensional images such as anatomical models, and to increase query speed, the locations of skeleton points in this embodiment are represented by single-digit encodings.

Specifically, the code s=x for each pixel (including skeleton points) in the anatomical model ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ Wherein x is ₁ 、x ₂ 、x ₃ The coordinates of the pixels in the first, second and third dimensions of the coding sequence, R ₁ 、R ₂ The dimensions of the anatomical model in the first and second dimensions of the coding sequence, respectively.

As illustrated in the accompanying drawings, as shown in fig. 12, the anatomical model has three dimensions X, Y, Z and the coding order is X, Y, Z in order, s=x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ ，x ₁ Is the coordinate of the pixel on the X axis, X ₂ X is the coordinate of the pixel on the Y-axis ₃ R is the coordinate of the pixel on the Z axis ₁ For the size of the anatomical model on the X-axis, i.e. the total number of pixels, R ₂ Is the dimension of the anatomical model on the Y-axis.

In the related art, the feature points are generally extracted from a two-dimensional skeleton image by screening candidate feature points by a table look-up method. The principle of the table look-up method is as follows: the two-dimensional skeleton image is a binary image, then consider an 8 neighborhood of one pixel, a total of 2 for the 3×3 region including itself ⁹ 512 possible permutations, each representing a distribution of skeleton points, wherein which permutations represent that pixel is known as an endpoint/intersection point. Design a 3×3, wherein each position is set to a different value, for example, 1, 2, 4, 8, 16, 32, 64, 128, 256, the value after convolution is a scalar, the convolution values corresponding to the end points and the cross points are calculated and stored as a feature point correspondence table in one-to-one correspondence with the arrangement of the 3 x 3 region centered on the pixel. When the method is used, the convolution is only needed to be performed on the two-dimensional skeleton image by using the convolution check, and then the convolution value in the feature point corresponding table is searched in the obtained convolution image, so that all candidate feature points can be found out at one time.

However when a look-up table is applied to a three-dimensional skeleton image (e.g. an anatomical model in the present application), since the neighborhood extends from a two-dimensional 8 neighborhood to a 26 neighborhood, the 3 x 3 region extends to a 3 x 3 region, the possible permutation is from 2 ⁹ Becomes 2 ²⁷ = 134217728, the generation and lookup of tables becomes very complex, and the lookup method is not suitable for extracting feature skeleton points from three-dimensional skeleton images.

As shown in fig. 13, in an embodiment of the present application, it is proposed to extract feature skeleton points based on a region growing method, which specifically includes:

s211: a starting point is determined.

The starting point, also called a seed point, is a skeleton point that serves as a starting point for the growth of the region. According to the structural characteristics of the airway tree, the starting point of the airway tree, namely the skeleton point of the tracheal entrance, is generally selected as the starting point. In the related art, a manual selection of a starting point is generally required, and a method for automatically selecting the starting point is provided in the application, which is specifically as follows.

As shown in fig. 14, in an embodiment of the present application, the steps specifically include:

s2111: the integration layer is randomly selected.

The three-dimensional CT image is reconstructed from a plurality of tomographic CT images, wherein each tomographic CT image corresponds to an anatomical layer. The anatomical model is obtained by performing image processing on a three-dimensional CT image, and correspondingly, the anatomical model is composed of a plurality of body layers, and each body layer corresponds to a processing result of a tomographic CT image. One dimension of the anatomical model (typically z) is used to distinguish between the different volume layers.

S2112: the total number of skeleton points in the layers on both sides of the body layer in the direction perpendicular to the cross section is counted to determine the direction in which the airway extends.

Tomographic CT images are those that image cross-sections by projection reconstruction. According to the structural characteristics of the air passage, the position of the air passage inlet can be known, only a small number of skeleton points are started on the same side as the extending direction of the air passage (only one skeleton point is arranged on each layer of the air passage), then the number of skeleton points is rapidly increased (including the part of the bronchus), and no skeleton point or only a small number of skeleton points are started on the opposite side to the extending direction of the air passage, and then the air passage is changed into no skeleton point; the number of skeleton points near the end of the airway tree is significantly reduced, but the number of skeleton points for a single layer is generally greater than the tracheal entrance location. From which the direction of airway extension can be determined.

S2113: and searching skeleton points of the tracheal entrance as a starting point according to the total number of skeleton points in the body layer and the extending direction of the airway.

Starting from a randomly selected volume layer, searching towards the tracheal entrance according to the direction of airway extension until the skeleton point of the tracheal entrance is found.

S212: starting from the starting point, searching skeleton points by using a region growing method and counting the connected number of the skeleton points.

The connected number is the number of all child nodes in the neighborhood of the skeleton point. In the traditional three-dimensional coordinate system, the relative coordinate relation between the skeleton point and the pixel points in the neighborhood is known, namely, the three-dimensional coordinate of a certain skeleton point is known, the three-dimensional coordinate of all the pixel points in the neighborhood can be determined, and the relative coordinate relation is substituted into the coding calculation formula s=x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ And obtaining the relative coding relation between the skeleton point and the pixel points in the neighborhood of the skeleton point.

Starting from a starting point, searching skeleton points in the neighborhood of the starting point, wherein the found skeleton points are child nodes of the starting point, and the starting point is a father node of the skeleton points. Similarly, for each skeleton point except for the starting point, a parent node is needed in the neighborhood of the skeleton point, child nodes except for the parent node in the skeleton points found in the neighborhood are child nodes of the skeleton point, and the number of the child nodes is counted to obtain the connection number of the skeleton point.

S213: and determining characteristic skeleton points according to the connection numbers of the skeleton points and the father nodes thereof.

If the number of connected skeleton points is greater than 1 and the number of connected parent nodes is equal to 1, the skeleton points are bifurcation points in the characteristic skeleton points. If the number of connected skeleton points is equal to 1 and the number of connected parent nodes is equal to 0, the skeleton points are end points in the characteristic skeleton points.

S22: the pipeline center line is divided into a plurality of pipeline center line segments according to the characteristic framework points.

And determining father characteristic points of the characteristic skeleton points except the starting point, wherein two end points of each pipeline center line segment are one characteristic skeleton point and father characteristic points thereof. The process of determining a father characteristic point for a characteristic skeleton point comprises the following steps: searching along the direction of the father node of the feature skeleton point, namely searching the father node of the feature skeleton point, searching the father node of the father node, and the like, wherein the first feature skeleton point is the father feature point of the feature skeleton point.

Since there may be burrs in the pipeline centerline, errors in the extracted feature skeleton points and the segmented pipeline centerline segments may occur, and optionally, the next two steps are performed to filter the erroneous segments and feature skeleton points.

S23: and counting the number of skeleton points in the pipeline center line segment.

S24: deleting the pipeline center line segments with the number smaller than a preset threshold value.

A parameter for representing the level may be set for the feature skeleton points, where the level parameter of each feature skeleton point is 1 added to the level parameter of its parent feature point, and the smaller the level parameter, the higher the level of the feature skeleton point, and the closer to the trachea. The preset threshold may be determined based on a level parameter of an endpoint of the pipe center segment. In general, the larger the level parameter is, the shorter the branch pipe section corresponding to the feature skeleton point is, and correspondingly, the smaller the preset threshold value is. Optionally, a class range is set, which generally excludes larger class parameters, and only segments of the pipeline centerline with endpoints within the class range are filtered.

If a certain pipeline center line segment is deleted, the starting point s is divided _i The other skeleton points except (i.e., the one of the two endpoints with the smaller level parameter) are changed to non-skeleton points, and all the pipe centerline segments with the non-skeleton points as the starting points are deleted. If after deleting s _i The number of pipe centerline segments as starting point is equal to 1, then s will be _i Modifying to common skeleton points, if s is deleted _i The number of pipe centerline segments as starting point is equal to 0, then s will be _i Modified to the end point.

The embodiment of the application also provides a control system of the catheter robot. Referring to fig. 15, a schematic structural diagram of a control system of a catheter system according to an embodiment of the present application is shown. As shown in fig. 15, the control system 600 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 being connected by the bus 62; the memory 61 stores computer program instructions executable by the processor 60, and the processor 60 executes the computer program instructions to perform the method for registering a catheter robot according to any of the foregoing embodiments of the present application.

The memory 61 may include a high-speed random access memory (RAM: random Access Memory), and may further include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. The communication connection between the device network element and the at least one other network element is achieved via at least one communication interface 63 (which may be wired or wireless), the internet, a wide area network, a local network, a metropolitan area network, etc. may be used.

Bus 62 may be an ISA bus, a PCI bus, an EISA bus, or the like. The buses may be classified as address buses, data buses, control buses, etc. The memory 61 is configured to store a program, and the processor 60 executes the program after receiving an execution instruction, and the registration method of the catheter robot disclosed in any of the foregoing embodiments of the present application may be applied to the processor 60 or implemented by the processor 60.

The processor 60 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuitry in hardware or instructions in software in the processor 60. The processor 60 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory 61 and the processor 60 reads the information in the memory 61 and in combination with its hardware performs the steps of the method described above.

The control system of the catheter robot provided by the embodiment of the application and the registration method of the catheter robot provided by the embodiment of the application have the same beneficial effects as the method adopted, operated or realized by the same inventive concept.

The present application further provides a computer readable storage medium corresponding to the method for registering a catheter robot provided in the foregoing embodiments, please refer to fig. 16, which illustrates a computer readable storage medium 6 having stored thereon computer program instructions which, when executed by a processor, implement the method for registering a catheter robot provided in any of the foregoing embodiments.

It should be noted that examples of the computer readable storage medium may include, but are not limited to, an optical disc, a phase change memory (PRAM), a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), other types of Random Access Memories (RAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash memory, or other optical or magnetic storage medium, which will not be described in detail herein.

The computer readable storage medium provided by the above embodiment of the present application and the registration method of the catheter robot provided by the embodiment of the present application have the same advantageous effects as the method adopted, operated or implemented by the application program stored therein, because of the same inventive concept.

Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that may be performed in the various method embodiments described above.

It should be noted that:

in the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the application and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the following schematic diagram: i.e., the claimed application requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this application.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the present application and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

The foregoing is merely a preferred embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

It will be apparent to those skilled in the art that the above-described functional units and modules are merely illustrated for convenience and brevity of description, and that in practical applications, the above-described functional units and modules may be allocated to different functional units or modules, that is, the internal structure of the apparatus may be divided into different functional units or modules, so as to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present invention. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium may include content that is subject to appropriate increases and decreases based on jurisdictional legislation and patent practice requirements, such as in some jurisdictions, based on jurisdictional and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a condition or event is determined" or "if a condition or event is detected" may be interpreted in the context to mean "upon determination" or "in response to determination" or "upon detection of a condition or event, or" in response to detection of a condition or event.

In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.

Claims (12)

1. A catheter robot comprising a robotic arm, a catheter instrument engaged with a power section of the robotic arm, a processor communicatively coupled to the robotic arm, the catheter instrument comprising an instrument pod configured to engage with the power section and a catheter coupled to the instrument pod, the catheter and/or the catheter-mounted instrument being provided with a sensor for measuring a position of the catheter and/or the instrument, the processor being configured to perform the steps of:

acquiring an actual path point of the catheter and/or the instrument with the sensor;

obtaining simulated waypoints of the catheter and/or the instrument corresponding to the actual waypoints in an anatomical model according to a first transformation matrix, the anatomical model comprising a pipeline centerline;

determining a target pipeline centerline segment corresponding to the simulated path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided;

calculating the centroid of the target pipeline center line segment;

and if the centroid is different from a plurality of skeleton points included in the target pipeline center line segment, determining a connecting line between the centroid and the simulation path point, and selecting one of the skeleton points with the shortest distance from the connecting line as a matching point of the actual path point.

2. The catheter robot of claim 1, wherein the processor is configured to perform the following steps after the calculation of the centroid of the target pipe centerline segment:

if the centroid is one of the skeleton points, a first foot from the simulation path point to a projection line is obtained, and a second foot from each skeleton point to the projection line is obtained, wherein the projection line is a straight line passing through two end points of the center line segment of the target pipeline; and selecting a skeleton point corresponding to a second foot drop with the shortest distance between the first foot drops as a matching point of the actual path point.

3. The catheter robot of claim 2, wherein the processor is configured to perform the steps of:

updating the point pair set by utilizing a first point pair formed by the actual path points and the matching points;

and if the point pair set meets a first condition, updating the first transformation matrix by using the point pair set, wherein the first condition comprises that the number of point pairs in the point pair set is larger than a preset value.

4. A catheter robot according to claim 3, wherein the processor is configured to perform the following steps in the set of matching point pair update point pairs consisting of the actual path point and its matching point:

Judging whether a second point pair comprising the matching point exists in the point pair set or not;

and if the first distance is smaller than the second distance, replacing the second point pair with the first point pair.

5. The catheter robot of claim 3, wherein the processor is configured to perform the following steps in the updating the first transformation matrix using the set of point pairs:

substituting the point pairs in the point pair set into an objective function to calculate a latest second transformation matrix;

and updating the first transformation matrix according to the latest second transformation matrix.

6. The catheter robot of claim 5, wherein the first transformation matrix is a weighted average of the most recent second transformation matrix and at least a portion of a historical second transformation matrix, wherein the most recent second transformation matrix has a weight greater than a weight of the historical second transformation matrix.

7. The catheter robot of claim 1, wherein the processor is configured to perform the following steps prior to the determining a target pipe centerline segment corresponding to the simulated path point among the plurality of pipe centerline segments into which the pipe centerline is divided:

extracting feature skeleton points from a plurality of skeleton points included in the pipeline center line, wherein the positions of the skeleton points are represented by codes, and the codes of the skeleton points are s=x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ Wherein x is ₁ 、x ₂ 、x ₃ The coordinates of the skeleton point in the first dimension, the second dimension and the third dimension of the coding sequence are respectively R ₁ 、R ₂ The size of the anatomic model in the first dimension and the second dimension of the coding sequence is respectively;

dividing the pipeline center line into a plurality of pipeline center line segments according to the characteristic skeleton points.

8. The catheter robot of claim 7, wherein the processor is configured to perform the following steps in the extracting feature skeleton points from the tube centerline including a plurality of skeleton points:

determining a starting point;

starting from the starting point, searching the skeleton point by using a region growing method and counting the number of the connected skeleton points, wherein the number of the connected skeleton points is the number of all child nodes in the neighborhood of the skeleton point;

If the number of the connected skeleton points is greater than 1 and the number of the connected parent nodes is equal to 1, the skeleton points are bifurcation points in the characteristic skeleton points, and if the number of the connected skeleton points is equal to 1 and the number of the connected parent nodes is equal to 0, the skeleton points are tail end points in the characteristic skeleton points.

9. The catheter robot of claim 8, wherein the processor is configured to perform the following steps in the determining a starting point:

randomly selecting an integrated layer;

counting the total number of skeleton points in the layers of the designated number on two sides of the body layer in the direction perpendicular to the cross section so as to determine the extending direction of the air passage;

and searching skeleton points of the tracheal entrance according to the total number of the skeleton points in the body layer and the extending direction of the airway to serve as the starting point.

10. The catheter robot of claim 7, wherein the processor is configured to perform the steps of:

counting the number of skeleton points in the pipeline center line segment;

deleting the number of pipeline center line segments smaller than a preset threshold value.

11. A method of registration of a catheter robot, comprising:

acquiring an actual path point of a catheter and/or an instrument carried by the catheter using a sensor disposed on the catheter and/or the instrument;

Obtaining simulated waypoints of the catheter and/or the instrument corresponding to the actual waypoints in an anatomical model according to a first transformation matrix, the anatomical model comprising a pipeline centerline;

determining a target pipeline centerline segment corresponding to the simulated path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided, the target pipeline centerline segment comprising a plurality of skeletal points;

calculating the centroid of the target pipeline center line segment;

and if the centroid and the plurality of skeleton points are different, determining a connecting line between the centroid and the simulation path point, and selecting one of the plurality of skeleton points, which has the shortest distance from the connecting line, as a matching point of the actual path point.

12. A computer readable storage medium storing computer program instructions configured to be loaded by a processor and to perform steps implementing the method of claim 11.

Images