CN109567942B - Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology - Google Patents
Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology Download PDFInfo
- Publication number
- CN109567942B CN109567942B CN201811288413.9A CN201811288413A CN109567942B CN 109567942 B CN109567942 B CN 109567942B CN 201811288413 A CN201811288413 A CN 201811288413A CN 109567942 B CN109567942 B CN 109567942B
- Authority
- CN
- China
- Prior art keywords
- osteotomy
- mechanical arm
- module
- points
- robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/70—Manipulators specially adapted for use in surgery
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/108—Computer aided selection or customisation of medical implants or cutting guides
Abstract
The invention relates to a craniomaxillofacial surgery robot auxiliary system adopting an artificial intelligence technology, which comprises a preoperative surgical scheme planning subsystem, an intraoperative navigation positioning subsystem and a robot control subsystem, wherein the robot control subsystem is connected with the preoperative surgical scheme planning subsystem; the preoperative surgical scheme planning subsystem carries out design of an osteotomy surgical scheme by using preoperative imaging data; the intraoperative navigation positioning subsystem carries out real-time obstacle avoidance on an anatomical structure and an instrument human body of an operation area under the assistance of a machine vision technology so as to automatically reach a designed navigation area; and the robot control subsystem optimizes the motion trail of the mechanical arm according to the obtained navigation area. The invention can generate a corresponding operation osteotomy scheme before an operation, and identify an anatomical region and a working region needing obstacle avoidance in real time during the operation.
Description
Technical Field
The invention relates to the technical field of surgical robot assistance, in particular to a craniomaxillofacial surgical robot assistance system adopting an artificial intelligence technology.
Background
Craniomaxillofacial plastic surgery achieves the aim of correcting craniomaxillofacial deformity by moving craniomaxillofacial bones in a subarea mode through methods of traction bone formation, bone cutting, repositioning bone transplantation and the like. The difficulty of the operation is mainly shown in that: complex anatomical structures, dense surrounding tissue structures such as blood vessels and nerves of interest, narrow or nearly blind surgical fields, and physical tremor of the surgeon and fatigue from long-term surgical procedures. The craniomaxillofacial operation has very high requirement on operation precision, and the traditional operation has larger risk and relatively high technical threshold compared with other surgical operations because the technical defects are difficult to meet the operation precision requirement.
The current surgical robot products mainly focus on master-slave general purpose, the development of the special surgical robot is lagged, and the artificial intelligence technology is applied to the surgical robot less. The craniomaxillofacial plastic operation mainly operates craniomaxillary bones, the manual error is large during preoperative planning, an optimal operation planning scheme is difficult to select, and a narrow operation space in the operation needs a doctor to avoid a plurality of obstacles for operation.
The research of the prior patent technology shows that the robot arm auxiliary equipment of the craniofacial plastic surgery auxiliary device disclosed by the Chinese patent publication No. CN103598916A is provided with a force feedback device, but lacks multiple degrees of freedom, can only move under the operation of a surgeon, cannot automatically position according to a navigation system, and has lower intelligence degree of a control system. The utility model discloses a supplementary arm based on seven degrees of freedom craniomaxillofacial operations of optical navigation that chinese patent publication number is CN202146362U, this utility model patent only introduced the component of a section of arm to and the fixed mounting mode of surgical instruments's optical navigation marker among the end effector. For the entire system, the application of artificial intelligence techniques is not detailed. Chinese patent publication No. CN107280764A discloses a craniomaxillofacial plastic surgery robot, which mainly describes the mechanical configuration and control part, and does not describe the cooperation with navigation system or the application of other artificial intelligence technology. The robot is still in a relatively independent mechanical device state, and the clinical use requirement of further intelligent control cannot be met.
Disclosure of Invention
The invention aims to solve the technical problem of providing a craniomaxillofacial surgery robot auxiliary system adopting an artificial intelligence technology, which can automatically utilize preoperative imaging data to design an osteotomy scheme, and utilize navigation equipment in the operation to carry out real-time obstacle avoidance on an anatomical structure and an instrument human body in an operation area under the assistance of a machine vision technology so as to complete the functions of automatically reaching the designed navigation area and carrying out operation.
The technical scheme adopted by the invention for solving the technical problems is as follows: the craniomaxillofacial surgery robot auxiliary system adopting the artificial intelligence technology is provided, and comprises a preoperative surgical scheme planning subsystem, an intraoperative navigation positioning subsystem and a robot control subsystem; the preoperative surgical scheme planning subsystem carries out design of an osteotomy surgical scheme by using preoperative imaging data; the intraoperative navigation positioning subsystem carries out real-time obstacle avoidance on an anatomical structure and an instrument human body of an operation area under the assistance of a machine vision technology so as to automatically reach a designed navigation area; and the robot control subsystem optimizes the motion trail of the mechanical arm according to the obtained navigation area.
The preoperative surgical scheme planning subsystem comprises an imaging database, a training sample set module, a test sample set module and an osteotomy scheme determining module; the imaging database excludes data irrelevant to the corresponding operation from the imaging data of the corresponding operation patient, and establishes database data taking a DICOM data source as a core; the training sample set module is used for establishing a point cloud data model; the test sample set module is used for completing osteotomy design; and the osteotomy operation scheme determining module carries out parameter correction on the osteotomy design obtained by the test sample set module according to the data of the training sample set module.
The training sample set module carries out 3D reconstruction on the mandible in a layer-by-layer marking mode, obtains a mandible angle osteotomy module by calibrating an osteotomy face, and establishes a point cloud data model for the mandible angle osteotomy module.
The test sample set module accurately segments the osteotomy point cloud from the original point cloud data through a deep neural network model, wherein 128 pieces of characteristic information are extracted from each point, and a multilayer perceptron is used for fusion to segment the mandible point cloud; acquiring edge points by calculating Euclidean distances between the points, namely, dividing original point cloud data into non-feature points and feature points through a deep neural network model, calculating the Euclidean distances between all the points in the feature points and all the points in the non-feature points, and taking the points with the distances smaller than a certain value as the edge points; dividing edge points at two ends by a k-means clustering algorithm; the plane is obtained by fitting the junction points, namely after the left and right groups of junction points are obtained, the two groups of junction points need to be fitted respectively, so that the osteotomy plane is obtained, the osteotomy plane is fitted by utilizing a least square method through the sum of squares of the distances between the minimum point and the plane, and the osteotomy design is completed.
The osteotomy operation scheme determination module compares the accuracy of artificial osteotomy and point cloud osteotomy in the training sample set module, establishes the relation between the accuracy and the point cloud model point number, and corrects parameters of an osteotomy plane fitted by the test sample set module by comparing the speed of artificial osteotomy and point cloud osteotomy.
The intraoperative navigation positioning subsystem comprises a magnetic navigation module and an automatic obstacle avoidance module, wherein the magnetic navigation module automatically identifies the space coordinates of metal balls in the automatic registration dental appliance set under an image space coordinate system by using a mode identification algorithm, obtains the space coordinates of the metal balls in the automatic registration dental appliance set under a magnetic locator coordinate system by using a space transformation method, and establishes a registration transformation relation between the image space coordinate system and the magnetic locator coordinate system; the conversion relation between the image space coordinate system and the magnetic positioning instrument coordinate system is calculated and updated in real time along with the movement of the patient, and the correct registration relation is always kept; the automatic obstacle avoidance module enables the space multi-degree-of-freedom mechanical arm to autonomously search an optimal path which does not collide with the obstacle from an initial point to a target point by using an obstacle avoidance path algorithm.
The automatic registration tooth socket device comprises a tooth socket body, six metal balls are arranged in the tooth socket body, a pluggable 5-degree-of-freedom magnetic positioning sensor is further arranged in the tooth socket body, the 5-degree-of-freedom magnetic positioning sensor and the six metal balls are fixed in position relation, and the mutual space distance among the six metal balls is unique and not repeated.
The automatic obstacle avoidance module performs collision detection by adopting a collision model detection method based on space geometry envelope, and specifically comprises the following steps: modeling the mechanical arm according to a D-H representation method, and enveloping each connecting rod of the mechanical arm by using a cylinder with the minimum diameter; for irregular three-dimensional obstacles, a spherical envelope is used; when collision detection is carried out, the radial maximum radius of the mechanical arm connecting rod is superposed on the thickness of the barrier, so that the collision problem between the mechanical arm and the barrier is converted into the position relation judgment between a space straight line segment and a sphere; when a plurality of obstacles exist in the operation, the method is further converted into a sphere sequence described based on the sphere center and the sphere radius, so that the position relation judgment between the space straight-line segment of each connecting rod of the mechanical arm and the obstacle entity is converted into the distance judgment between the space straight-line segment and a series of sphere center points; if the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the spherical barrier in the motion range of the mechanical arm is larger than the radius of the sphere, the mechanical arm and the barrier cannot collide; if the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the sphere obstacle in the motion range of the mechanical arm is smaller than the radius of the sphere, the mechanical arm collides with the obstacle.
The automatic obstacle avoidance module adopts a three-dimensional path planning based on an A-x algorithm, and specifically comprises the following steps: defining an OPEN list and a CLOSE list to store node information, wherein the OPEN list stores nodes to be tested as points on a possible path; the CLOSE list stores the detected nodes; the algorithm starts from the initial position of the working space, expands the initial position node and the surrounding directions thereof, and stores the initial position node and the surrounding directions in an OPEN list; if the grid of the obstacle exists, the nodes do not need to be considered, namely, the heuristic value is defined to be infinite, then a node with the minimum heuristic value is selected from the OPEN list, then the surrounding neighbors are considered, judgment is carried out through the valuation function, circulation processing is carried out until the set target point is added into the CLOSE list, and algorithm searching is finished; or in the process of loop judgment, if the OPEN list is empty, the path is not found, and the algorithm is ended.
The robot control subsystem optimizes the track of the mechanical arm joint by adopting a particle swarm optimization algorithm, and specifically comprises the following steps: dividing the motion trail of each joint of the mechanical arm into three sections, wherein the first section and the third section are expressed by a cubic polynomial, the second section is expressed by a quintic polynomial, and time is used as an independent variable; deducing the relation between the coefficient of the polynomial interpolation function and the running time of the three-segment track according to the joint angle value of the joint of the mechanical arm at each point and the constraint relation at each point; initializing parameters of a particle swarm optimization algorithm; calculating a polynomial coefficient matrix according to the generated N groups of combined values of 3-5-3 polynomial interpolation time; obtaining a polynomial representing the joint position corresponding to each section of track according to the polynomial coefficient matrix, obtaining a polynomial of joint speed through first-order derivation, calculating whether the maximum speed corresponding to each section of track meets the constraint condition, and if not, discarding corresponding particles; calculating corresponding fitness of the particles meeting the constraint condition by taking the running time sum of the three sections of tracks as a fitness function; comparing the obtained fitness and selecting an individual optimal value and a group optimal value; updating the position and the speed of each particle according to a particle swarm optimization algorithm; and judging whether the ending condition is met, if so, exiting the circulation, and otherwise, recalculating the polynomial coefficient matrix.
Advantageous effects
Due to the adoption of the technical scheme, compared with the prior art, the invention has the following advantages and positive effects: the invention can carry out automatic point cloud analysis before an operation according to the imaging data to generate a corresponding operation osteotomy scheme. In the operation, an anatomical region and a working region which need to avoid an obstacle are identified in real time, and under the guidance of a navigation system, the robot finishes the positioning of an operation part under the condition of avoiding the obstacle, and performs corresponding operation.
Drawings
FIG. 1 is a flow diagram of a training sample set module;
FIG. 2 is a diagram of a deep neural network model employed by a test sample set module;
FIG. 3 is a flow diagram of a test sample set module;
FIG. 4 is a schematic view of the use of a magnetic navigation module;
fig. 5 is a schematic view of a craniomaxillofacial orthopaedic robotic system.
Detailed Description
The invention will be further illustrated with reference to the following specific examples. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and such equivalents may fall within the scope of the present invention as defined in the appended claims.
The embodiment of the invention relates to a craniomaxillofacial surgical robot auxiliary system adopting an artificial intelligence technology, which comprises a preoperative surgical scheme planning subsystem, an intraoperative navigation positioning subsystem and a robot control subsystem, wherein the preoperative surgical scheme planning subsystem comprises a craniomaxillofacial surgical robot control subsystem and a craniomaxillofacial surgical robot control subsystem; the preoperative surgical scheme planning subsystem carries out design of an osteotomy surgical scheme by using preoperative imaging data; the intraoperative navigation positioning subsystem carries out real-time obstacle avoidance on an anatomical structure and an instrument human body of an operation area under the assistance of a machine vision technology so as to automatically reach a designed navigation area; and the robot control subsystem optimizes the motion trail of the mechanical arm according to the obtained navigation area.
The preoperative surgical scheme planning subsystem comprises an imaging database, a training sample set module, a test sample set module and an osteotomy scheme determining module; the imaging database mainly selects imaging data of a patient in a corresponding operation, excludes data irrelevant to the corresponding operation, and establishes database data taking a DICOM data source as a core; the training sample set module is used for establishing a point cloud data model; the test sample set module is used for completing osteotomy design; and the osteotomy operation scheme determining module carries out parameter correction on the osteotomy design obtained by the test sample set module according to the data of the training sample set module.
As shown in fig. 1, the training sample set module performs 3D reconstruction on the mandible in a layer-by-layer labeling manner, obtains a mandibular angle osteotomy module by calibrating an osteotomy plane, and establishes a point cloud data model for the mandibular angle osteotomy module.
The test sample set module mainly comprises: and (4) transfer learning (feature extraction), namely, accurately segmenting the osteotomy point cloud by the original point cloud data through a deep neural network model (see figure 2). And extracting 128 pieces of characteristic information from each point, fusing by using a multilayer perceptron, and finally realizing the segmentation of the mandible point cloud. As shown in fig. 3, the edge points are obtained by calculating the euclidean distances between the points, that is, the original point cloud data is divided into non-feature points and feature points by the deep neural network model, the euclidean distances between all the points in the feature points and all the points in the non-feature points are calculated, and the points whose distances are smaller than a certain value are taken as the edge points. The edge points at the two ends are segmented through a k-means clustering algorithm, namely the focus obtained in the previous step is a segmentation plane focus at two sides of the mandible, so that two groups of points need to be divided according to a certain rule, and the k-means algorithm clusters the points through distance. And fitting the boundary points to obtain a plane. After the left and right groups of boundary points are obtained, the two groups of boundary points need to be fitted respectively, so that an osteotomy plane is obtained. The osteotomy plane is fitted using a least squares method by minimizing the sum of the squares of the distances between the points and the plane. And finishing the osteotomy design.
The osteotomy operation scheme determination module compares the accuracy of artificial osteotomy and point cloud osteotomy in the training sample set module, establishes the relation between the accuracy and the point cloud model point number, and corrects parameters of an osteotomy plane fitted by the test sample set module by comparing the speed of artificial osteotomy and point cloud osteotomy.
The intraoperative navigation positioning subsystem comprises a magnetic navigation module and an automatic obstacle avoidance module.
In order to realize the positioning function of the magnetic navigation module, the embodiment designs an automatic registration tooth socket device. The automatic registration tooth socket device is a tooth socket with six built-in metal balls and a pluggable 5-degree-of-freedom magnetic positioning sensor, the position relation between the 5-degree-of-freedom magnetic positioning sensor and the six built-in metal balls is fixed, and the mutual space distance between the six metal balls is unique and not repeated. As shown in fig. 4, the patient wears the mouthpiece device to perform CT scan to generate a DICOM image, and the position of the center of the six metal balls in the image space is automatically identified by a pattern recognition algorithm. The automatic registration tooth socket device is provided with a pluggable 5-degree-of-freedom magnetic positioning sensor, so that the automatic registration tooth socket device has a coordinate system belonging to the automatic registration tooth socket device, and a fixed position relation is established between six built-in metal balls with the diameters of 2mm and the 5-degree-of-freedom magnetic positioning sensor, so that the coordinate position of the metal ball center in the 5-degree-of-freedom magnetic positioning sensor coordinate system can be obtained through the fixed position relation between the metal balls and the 5-degree-of-freedom magnetic positioning sensor.
Automatically identifying the space coordinates of six metal balls in the automatic registration dental appliance set under an image space coordinate system by using a pattern identification algorithm; and obtaining the space coordinates of six metal balls in the automatic registration dental appliance set under a coordinate system of the magnetic locator by using a space transformation method. Then, the existing algorithm (such as ICP algorithm) can be used to establish the registration transformation relationship between the image space coordinate system and the magnetic locator coordinate system. The conversion relation between the image space coordinate system and the magnetic positioning instrument coordinate system is calculated and updated in real time along with the movement of the patient, and the correct registration relation is always kept, so that accurate positioning can be realized.
In order to achieve the purpose of automatic obstacle avoidance of the surgical robot in the operation, an obstacle avoidance path algorithm is needed to enable the spatial multi-degree-of-freedom mechanical arm to autonomously search an optimal path which does not collide with an obstacle from an initial point to a target point. Therefore, the problem of collision detection of the mechanical arm, namely whether each joint of the mechanical arm collides with an obstacle in a working environment in the motion process, must be considered when planning an obstacle avoidance path of the robot.
In the embodiment, the automatic obstacle avoidance module detects the obstacle by adopting a collision model based on space geometry envelope. In order to facilitate calculation and improve the efficiency of a collision detection algorithm, the problem of entity collision between each moving joint of the mechanical arm and the obstacle can be converted through simplification of the mechanical arm model and simplification of the obstacle model. The mechanical arm is modeled according to a D-H representation method, and each connecting rod of the mechanical arm is enveloped by a cylinder with the minimum diameter. For irregular three-dimensional obstacles, a spherical envelope is used, thereby greatly simplifying the obstacle model. When collision detection is carried out, the radial maximum radius of the mechanical arm connecting rod is superposed on the thickness of the barrier, so that the collision problem between the mechanical arm and the barrier is converted into position relation judgment between a space straight line segment and a sphere. Meanwhile, under the condition that a plurality of obstacles exist in the operation, the method can be further converted into a sphere sequence described based on the sphere center and the sphere radius, so that the position relation judgment between the space straight-line segment of each connecting rod of the mechanical arm and the obstacle entity can be converted into the distance judgment between the space straight-line segment and a series of sphere center points. If the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the spherical barrier in the motion range of the mechanical arm is larger than the radius of the sphere, the mechanical arm and the barrier cannot collide; if the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the sphere obstacle in the motion range of the mechanical arm is smaller than the radius of the sphere, the mechanical arm collides with the obstacle.
And the automatic obstacle avoidance module adopts mechanical arm obstacle avoidance path planning based on an A-x algorithm. The a-algorithm is a typical graph-based search method in artificial intelligence, and heuristic information related to problems is added in the search to guide the search to be carried out towards the most promising direction for searching the shortest path of a state space.
When the A-x algorithm is adopted, firstly, environment modeling is needed, namely, the working environment is gridded, and the gridding needs to be refined as much as possible in the gridding process considering that the obstacles have irregularity. And then, describing the position information of the obstacle point and the non-obstacle point in the Cartesian coordinate space by adopting a matrix consisting of 0, l elements (wherein 1 represents an obstacle point, and 0 represents a non-obstacle point). In a three-dimensional cartesian coordinate space, a three-dimensional matrix can be used for environment modeling, specifically, a process of changing 0 element in an initial zero matrix corresponding to an obstacle in an obstacle space into 1 element, so that a region is divided into a plurality of cubes to replace a search space.
When planning a three-dimensional path, firstly defining an OPEN list and a CLOSE list to store node information, wherein the OPEN list stores nodes to be detected as possible points on the path; the CLOSE list stores the detected nodes. The algorithm starts from the initial position of the working space, expands the initial position node and the surrounding directions thereof, and stores the initial position node and the surrounding directions in an OPEN list. If the obstacle squares exist, the obstacle squares do not need to be considered, namely, the heuristic values are defined to be infinite, then a node with the minimum heuristic value is selected from the OPEN list, then the surrounding neighbors are considered, judgment is carried out through an evaluation function, circulation processing is carried out until the set target point is added into the CLOSE list, and algorithm searching is finished; or in the process of loop judgment, if the OPEN list is empty, the path is not found, and the algorithm is ended.
The optimal obstacle avoidance path searched by the A-star algorithm is a space broken line segment, each joint of the mechanical arm can generate a shaking phenomenon in the motion process, and the obstacle avoidance path is subjected to smooth optimization processing through a second-order B-spline curve, so that the mechanical arm can work continuously and stably.
The implementation process of the mechanical arm obstacle avoidance path algorithm is as follows:
(1) initializing the initial position and the end position of the mechanical arm end effector and the initial joint angle of each joint, and establishing an obstacle model in a three-dimensional space.
(2) And planning a three-dimensional path by adopting an A-x algorithm, and smoothing the obtained path by using a quadratic B-spline curve.
(3) And performing inverse solution of mechanical arm kinematics on the three-dimensional path obtained in the last step. If the inverse solution fails, the point is not in the range of the working space of the mechanical arm end effector, and the previous step is returned.
(4) And carrying out kinematics positive solution on the mechanical arm to obtain the space coordinate of each joint.
(5) And determining a line segment equation between two adjacent joints of the mechanical arm according to the space position coordinates of each joint obtained in the previous step, and detecting the collision condition between the mechanical arm and the barrier. And if collision occurs, returning to the step (4) and combining the multi-solution property of the inverse solution of the robot, and reselecting a group of feasible solutions in other solutions. Instead, the solution is an optimal set of joint angles.
Path planning ultimately results in a series of discrete path points. In order to meet the running conditions of the mechanical arm, the position, the speed and the acceleration of each joint need to be controlled in the process that the mechanical arm runs from the starting position to the tail end position, so that the track planning is realized. The optimal time trajectory planning means that under the condition of meeting the constraint conditions of speed, acceleration change, moment and the like, the time corresponding to each section of path is adjusted, so that the motion time of the mechanical arm is shortest, and the operation time is shortened.
The robot control subsystem in this embodiment uses polynomial interpolation, which means that the motion trajectory of each joint of the mechanical arm is divided into three sections, the first section and the third section are represented by a cubic polynomial, the second section is represented by a quintic polynomial, and time is used as an independent variable. According to the joint angle values of the joints of the mechanical arm at each point and the constraint relation at each point, the relation between the coefficient of the polynomial interpolation function and the running time of the three-segment track can be deduced.
The robot control subsystem is further optimized through a track based on a particle swarm optimization algorithm. The particle swarm algorithm is characterized in that a group of random particles in a feasible solution space is regarded as a group, the fitness value of each particle is set by a corresponding function value, the particles are searched in the feasible solution space under the guidance of the current optimal particles, and the optimal position of the group is finally searched through iteration. In the whole process, the particles utilize the two extrema found: the optimal solution pBest (individual extremum) searched by the particle itself and the optimal solution gBest (global extremum) searched by the group are used for updating the particle itself. The specific steps of planning the optimal time trajectory of one joint of the mechanical arm are as follows:
(1) initializing parameters of the particle swarm optimization algorithm, including inertia weight, acceleration factor, particle number N and initial position and speed of the particles.
(2) And calculating a polynomial coefficient matrix according to the generated N groups of combined values of the 3-5-3 polynomial interpolation time.
(3) And obtaining a polynomial representing the joint position for each section of track pair according to the coefficient matrix, obtaining a polynomial of joint speed through first-order derivation, calculating whether the maximum speed corresponding to each section of track meets the constraint condition, and if not, discarding the corresponding particles.
(4) And calculating the corresponding fitness of the particles meeting the constraint condition by taking the running time sum of the three sections of tracks as a fitness function.
(5) And comparing the fitness obtained in the last step and selecting an individual optimal value pBest and a population optimal value gBest.
(6) Updating the position and speed of each particle according to a particle swarm optimization algorithm
(7) And (4) judging whether the ending condition is met, if so, exiting the circulation, and otherwise, turning to the step (2).
The invention is further illustrated by the following specific example.
As shown in fig. 5, the hardware of the craniomaxillofacial plastic surgery robot system is mainly composed of four parts, which are respectively: a craniomaxillofacial plastic surgery robot, a visual positioning navigation camera, a workstation and a craniomaxillofacial plastic surgery terminal system.
The craniomaxillofacial plastic surgery robot system helps a doctor to complete guiding, positioning and other operation operations. Aiming at the clinical requirements of craniomaxillofacial plastic surgery, the design of a stable, flexible, reliable and high-precision robot mechanism is the key for realizing the success of the puncture operation. On the basis of comprehensively considering various machine configurations, a three-dimensional frame movable lower jaw operation robot with six degrees of freedom is designed, a UR5 robot of Danish Youhao company is adopted, the robot has 6 degrees of freedom, 3 degrees of freedom at the bottom are used for initial positioning, the repeated positioning precision of the robot reaches +/-0.1 mm, the robot is fixed on a mechanical structure, and wheels are mounted at the bottom of the mechanical structure to form a movable trolley. The space size of an operating room and the safety of robot fixation are considered, and the fixing mode of the robot is a horizontal fixing mode; in consideration of the carrying convenience of the robot and the stability of the robot in the moving process, the robot fixing system adopts an aluminum profile mechanism and aluminum plate and 304 stainless steel load.
Considering the medical environment, according to the actual requirements of craniomaxillofacial plastic surgery, the ENSENSO series N35 visual positioning navigation camera of Germany IDS company is designed and adopted, the visual positioning navigation camera is widely applied in the research field of computer-assisted surgery navigation systems, and the visual positioning navigation camera of the company is also adopted in the operation navigation systems proposed by a plurality of famous international medical equipment companies. In the craniomaxillofacial plastic surgery, the visual positioning navigation camera can obtain the pose information of the lower jaw and soft tissue protection device of a patient in the craniomaxillofacial plastic surgery in real time, and the pose information is transmitted to the visual software of a workstation in real time and used for positioning and planning the drilling and milling path of the robot. The maximum precision of a visual positioning navigation camera in an effective area adopted by the system design is 2mm, in order to enable the camera to have a better visual field, a camera fixing mechanism at the top end of a trolley is designed, and pitching adjustment and height adjustment are considered in the camera debugging process, so that the camera fixing mechanism adopts a pitching and height adjustable design.
The system design adopts the Apoqi AB0X-700 as a workstation of the whole craniomaxillofacial plastic surgery robot system, the CPU of the system is Intel I7, the dominant frequency is 3.40GHz, the memory is 8G, and the operating system adopts Windows 10, so that the performance requirement of the system can be met. The workstation has the main functions of planning the preoperative surgical path, recording surgical drilling and milling data in the operation and realizing the evaluation of the surgical quality according to the drilling and milling data after the operation.
The craniomaxillofacial plastic surgery tail end system consists of drilling and milling equipment and a drill clamp; the system designs a clamp at the tail end of a robot arm, and an adjustable 304 stainless steel wrapping and clamping mechanism is adopted.
The software system connects the robot, the visual imaging equipment and the patient and is responsible for tasks such as data acquisition, image processing, operation planning, robot control and the like, the system uses CT images to carry out preoperative three-dimensional modeling, and spatial transformation and operation instrument monitoring of the operation system are realized through registration with intraoperative real-time images.
The software part can be divided into four modules of visual equipment data processing and navigation, a mandible robot motion planning and control module, master-slave control and operation safety monitoring.
And the visual equipment data processing and navigation module. The module receives and processes data of a visual system in real time, obtains coordinates of real-time target points through an image segmentation and recognition technology, registers the coordinates with target points in craniomaxillofacial CT data obtained before an operation, establishes a conversion relation among different coordinate spaces including a navigation space, an image space, a patient coordinate space and a craniomaxillofacial plastic surgery robot coordinate space, and then calculates and displays the relative positions of a drilling and milling probe, the craniomaxillofacial and protection mechanism in real time in the drilling and milling process, so that a doctor is guided to complete operation.
And a craniomaxillofacial plastic surgery robot motion planning and control module. The module aims to design an operation planning path for the craniomaxillofacial plastic surgery robot and control the craniomaxillofacial plastic surgery robot to assist a doctor in completing operation. Because the rigid soft tissue protection and installation positioning structure has assembly errors in the actual installation process, the module calculates the positions of an actual surgical cutting line and a drilling and milling point in a navigation space based on target registration information, and calculates the mandible tangential direction of the drilling and milling point at each drilling and milling point position as the entering direction of the robot; for each cutting point, calculating whether the drilling and milling probe intersects with the protection structure on the path along the entering direction of the currently designed robot: and if so, calculating an included angle between the cutting point and a connecting line of the corresponding point at the upper end of the protection structure, and setting the connecting line with the included angle slightly increased based on the set value as a new cutting path. After the path design is completed, the module sends the operation planning path information of the drilling and milling points to the robot, and the robot finishes posture transformation and automatically reaches the drilling and milling points.
And the master-slave control module. The medical robot system needs to interact with a doctor to enhance the safety of the operation, and the master-slave control module can receive and process control instructions sent by the drilling and milling pedal end and control the drilling and milling or stopping of the robot end effector.
And a safety monitoring module. In the aspect of safety, an autonomous multi-layer safety monitoring method aiming at craniomaxillofacial plastic surgery is adopted. Firstly, the detection information adopted by the monitoring system is determined, a mathematical model reflecting the relation between data input and final safety decision output is established, and the monitoring in various aspects such as position, speed and the like is realized, so that the safety problem of a patient in an operation is solved.
As can be easily found, the invention can carry out automatic point cloud analysis before operation according to the imaging data to generate a corresponding operation osteotomy scheme. In the operation, an anatomical region and a working region which need to avoid an obstacle are identified in real time, and under the guidance of a navigation system, the robot finishes the positioning of an operation part under the condition of avoiding the obstacle, and performs corresponding operation.
Claims (7)
1. A craniomaxillofacial surgery robot auxiliary system adopting artificial intelligence technology is characterized by comprising a preoperative surgical scheme planning subsystem, an intraoperative navigation positioning subsystem and a robot control subsystem; the preoperative surgical scheme planning subsystem carries out design of an osteotomy surgical scheme by using preoperative imaging data; the intraoperative navigation positioning subsystem carries out real-time obstacle avoidance on an anatomical structure and an instrument human body of an operation area under the assistance of a machine vision technology so as to automatically reach a designed navigation area; the robot control subsystem optimizes the motion trail of the mechanical arm according to the obtained navigation area; the preoperative surgical scheme planning subsystem comprises an imaging database, a training sample set module, a test sample set module and an osteotomy scheme determining module; the imaging database excludes data irrelevant to the corresponding operation from the imaging data of the corresponding operation patient, and establishes database data taking a DICOM data source as a core; the training sample set module is used for establishing a point cloud data model, specifically, 3D reconstruction is carried out on a mandible in a layer-by-layer marking mode, a mandibular angle osteotomy module is obtained by calibrating an osteotomy surface, and the point cloud data model is established for the mandibular angle osteotomy module; the test sample set module is used for completing osteotomy design; the osteotomy operation scheme determining module carries out parameter correction on the osteotomy design obtained by the test sample set module according to the data of the training sample set module; the test sample set module accurately segments the osteotomy point cloud from the original point cloud data through a deep neural network model, wherein 128 pieces of characteristic information are extracted from each point, and a multilayer perceptron is used for fusion to segment the mandible point cloud; acquiring edge points by calculating Euclidean distances between the points, namely, dividing original point cloud data into non-feature points and feature points through a deep neural network model, calculating the Euclidean distances between all the points in the feature points and all the points in the non-feature points, and taking the points with the distances smaller than a certain value as the edge points; dividing edge points at two ends by a k-means clustering algorithm; the plane is obtained by fitting the junction points, namely after the left and right groups of junction points are obtained, the two groups of junction points need to be fitted respectively, so that the osteotomy plane is obtained, the osteotomy plane is fitted by utilizing a least square method through the sum of squares of the distances between the minimum point and the plane, and the osteotomy design is completed.
2. The robot assistance system for craniomaxillofacial surgery using artificial intelligence technology as claimed in claim 1, wherein said osteotomy procedure plan determination module compares the accuracy of artificial osteotomy in the training sample set module with the point cloud osteotomy accuracy, and establishes a relationship between the accuracy and the point cloud model point number, and performs parameter correction on the osteotomy plane fitted to the test sample set module by comparing the artificial osteotomy speed with the point cloud osteotomy speed.
3. The craniomaxillofacial surgery robot assistance system adopting the artificial intelligence technology as claimed in claim 1, wherein the intraoperative navigation positioning subsystem comprises a magnetic navigation module and an automatic obstacle avoidance module, the magnetic navigation module automatically identifies the space coordinates of the metal balls in the automatic registration dental appliance set under an image space coordinate system by using a mode identification algorithm, obtains the space coordinates of the metal balls in the automatic registration dental appliance set under a magnetic locator coordinate system by using a space transformation method, and establishes a registration transformation relationship between the image space coordinate system and the magnetic locator coordinate system; the conversion relation between the image space coordinate system and the magnetic positioning instrument coordinate system is calculated and updated in real time along with the movement of the patient, and the correct registration relation is always kept; the automatic obstacle avoidance module enables the space multi-degree-of-freedom mechanical arm to autonomously search an optimal path which does not collide with the obstacle from an initial point to a target point by using an obstacle avoidance path algorithm.
4. The craniomaxillofacial surgery robotic assistance system adopting artificial intelligence technology as claimed in claim 3, wherein said automatic registration mouthpiece means comprises a mouthpiece body, six metal balls are built in said mouthpiece body, a pluggable 5-degree-of-freedom magnetic positioning sensor is further arranged in said mouthpiece body, the positional relationship between said 5-degree-of-freedom magnetic positioning sensor and said six metal balls is fixed, and the mutual spatial distance between said six metal balls is unique and non-repetitive.
5. The craniomaxillofacial surgery robot auxiliary system adopting the artificial intelligence technology as recited in claim 3, wherein the automatic obstacle avoidance module performs collision detection by adopting a collision model detection method based on spatial geometry envelope, specifically: modeling the mechanical arm according to a D-H representation method, and enveloping each connecting rod of the mechanical arm by using a cylinder with the minimum diameter; for irregular three-dimensional obstacles, a spherical envelope is used; when collision detection is carried out, the radial maximum radius of the mechanical arm connecting rod is superposed on the thickness of the barrier, so that the collision problem between the mechanical arm and the barrier is converted into the position relation judgment between a space straight line segment and a sphere; when a plurality of obstacles exist in the operation, the method is further converted into a sphere sequence described based on the sphere center and the sphere radius, so that the position relation judgment between the space straight-line segment of each connecting rod of the mechanical arm and the obstacle entity is converted into the distance judgment between the space straight-line segment and a series of sphere center points; if the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the spherical barrier in the motion range of the mechanical arm is larger than the radius of the sphere, the mechanical arm and the barrier cannot collide; if the vertical distance between each rod piece of the mechanical arm and the sphere center of the sphere sequence of the sphere obstacle in the motion range of the mechanical arm is smaller than the radius of the sphere, the mechanical arm collides with the obstacle.
6. The craniomaxillofacial surgical robot assistance system adopting an artificial intelligence technology as claimed in claim 3, wherein the automatic obstacle avoidance module adopts a three-dimensional path planning based on an a-algorithm, specifically: defining an OPEN list and a CLOSE list to store node information, wherein the OPEN list stores nodes to be tested as points on a possible path; the CLOSE list stores the detected nodes; the algorithm starts from the initial position of the working space, expands the initial position node and the surrounding directions thereof, and stores the initial position node and the surrounding directions in an OPEN list; if the grid of the obstacle exists, the nodes do not need to be considered, namely, the heuristic value is defined to be infinite, then a node with the minimum heuristic value is selected from the OPEN list, then the surrounding neighbors are considered, judgment is carried out through the valuation function, circulation processing is carried out until the set target point is added into the CLOSE list, and algorithm searching is finished; or in the process of loop judgment, if the OPEN list is empty, the path is not found, and the algorithm is ended.
7. The craniomaxillofacial surgery robot assistance system adopting the artificial intelligence technology as claimed in claim 1, wherein the robot control subsystem performs trajectory optimization on a mechanical arm joint by adopting a particle swarm optimization algorithm, and specifically comprises: dividing the motion trail of each joint of the mechanical arm into three sections, wherein the first section and the third section are expressed by a cubic polynomial, the second section is expressed by a quintic polynomial, and time is used as an independent variable; deducing the relation between the coefficient of the polynomial interpolation function and the running time of the three-segment track according to the joint angle value of the joint of the mechanical arm at each point and the constraint relation at each point; initializing parameters of a particle swarm optimization algorithm; calculating a polynomial coefficient matrix according to the generated N groups of combined values of 3-5-3 polynomial interpolation time; obtaining a polynomial representing the joint position corresponding to each section of track according to the polynomial coefficient matrix, obtaining a polynomial of joint speed through first-order derivation, calculating whether the maximum speed corresponding to each section of track meets the constraint condition, and if not, discarding corresponding particles; calculating corresponding fitness of the particles meeting the constraint condition by taking the running time sum of the three sections of tracks as a fitness function; comparing the obtained fitness and selecting an individual optimal value and a group optimal value; updating the position and the speed of each particle according to a particle swarm optimization algorithm; and judging whether the ending condition is met, if so, exiting the circulation, and otherwise, recalculating the polynomial coefficient matrix.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811288413.9A CN109567942B (en) | 2018-10-31 | 2018-10-31 | Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811288413.9A CN109567942B (en) | 2018-10-31 | 2018-10-31 | Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109567942A CN109567942A (en) | 2019-04-05 |
| CN109567942B true CN109567942B (en) | 2020-04-14 |
Family
ID=65921019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811288413.9A Active CN109567942B (en) | 2018-10-31 | 2018-10-31 | Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109567942B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NL2032742A (en) * | 2022-08-12 | 2022-09-20 | Univ Lishui | A surgical navigation system based on artificial intelligence and graph theory algorithm |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109907827B (en) * | 2019-04-16 | 2020-07-14 | 北京大学第三医院（北京大学第三临床医学院） | Operation navigation system for mandibular angle osteotomy |
| CN109875683B (en) * | 2019-04-16 | 2020-06-12 | 北京大学第三医院（北京大学第三临床医学院） | Method for establishing osteotomy face prediction model in mandibular angle osteotomy |
| CN110613511B (en) * | 2019-10-16 | 2021-03-16 | 武汉联影智融医疗科技有限公司 | Obstacle avoidance method for surgical robot |
| EP4062854A4 (en) * | 2019-11-21 | 2023-02-22 | Microport Navibot (Suzhou) Co., Ltd. | Osteotomy verification method and verification apparatus, readable storage medium, and orthopedic surgery system |
| CN110960321B (en) | 2019-12-18 | 2021-07-02 | 苏州微创畅行机器人有限公司 | Registration target, registration method, registration device, electronic equipment and storage medium |
| CN111179350B (en) * | 2020-02-13 | 2022-04-08 | 张逸凌 | Hip joint image processing system |
| CN111803211B (en) * | 2020-07-13 | 2021-11-16 | 北京大学口腔医学院 | Jaw positioning system and method for maxillofacial surgery |
| CN112060093B (en) * | 2020-09-10 | 2022-08-02 | 云南电网有限责任公司电力科学研究院 | Path planning method for overhead line maintenance mechanical arm |
| CN112057166A (en) * | 2020-09-24 | 2020-12-11 | 中国科学院自动化研究所 | Experimental system and method for simulating skull surgery |
| CN112380207A (en) * | 2020-10-20 | 2021-02-19 | 哈尔滨医科大学 | Liver cancer intervention planning informatization system |
| CN112245013B (en) * | 2020-11-13 | 2021-10-15 | 南京佗道医疗科技有限公司 | Quick-release mechanism, tail end execution instrument and robot |
| CN116686053A (en) * | 2020-12-01 | 2023-09-01 | 直观外科手术操作公司 | System and method for planning a medical environment |
| CN113081615B (en) * | 2021-03-23 | 2021-11-23 | 四川大学 | Intelligent robot for maxillofacial plastic surgery |
| CN113555109A (en) * | 2021-07-08 | 2021-10-26 | 南通罗伯特医疗科技有限公司 | Preoperative planning device based on improved PCT neural network |
| CN113893049A (en) * | 2021-10-26 | 2022-01-07 | 上海优医基医疗影像设备有限公司 | Oral dental implant surgery system registration device, method and manufacturing method |
| CN114404047A (en) * | 2021-12-24 | 2022-04-29 | 苏州微创畅行机器人有限公司 | Positioning method, system, device, computer equipment and storage medium |
| CN114800556B (en) * | 2022-04-26 | 2023-10-24 | 广州东焊智能装备有限公司 | Automatic processing production line based on industrial robot |
| CN115409838B (en) * | 2022-11-01 | 2023-06-09 | 北京精准医械科技有限公司 | Registration method and system of instruments in medical image and related equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102551892A (en) * | 2012-01-17 | 2012-07-11 | 王旭东 | Positioning method for craniomaxillofacial surgery |
| CN104644266A (en) * | 2013-11-18 | 2015-05-27 | 上海交通大学医学院附属第九人民医院 | Method for measuring jaw cutting forces of different surgical methods |
| US20170000497A1 (en) * | 2013-11-29 | 2017-01-05 | The Johns Hopkins University | Cranial reference mount |
| CN106251284B (en) * | 2016-08-16 | 2019-08-20 | 边专 | Medical image registration method based on facing |
| CN107280764B (en) * | 2017-05-12 | 2020-11-06 | 上海交通大学 | Craniomaxillofacial plastic surgery operation robot |
| CN108324378A (en) * | 2018-02-11 | 2018-07-27 | 浙江工业大学 | Based on the whole accurate surgery systems of the co-operating craniofacial deformity reduction of major-minor robot and its operation method |
-
2018
- 2018-10-31 CN CN201811288413.9A patent/CN109567942B/en active Active
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NL2032742A (en) * | 2022-08-12 | 2022-09-20 | Univ Lishui | A surgical navigation system based on artificial intelligence and graph theory algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109567942A (en) | 2019-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109567942B (en) | Craniomaxillofacial surgical robot auxiliary system adopting artificial intelligence technology | |
| US11179219B2 (en) | Surgical robot system for stereotactic surgery and method for controlling stereotactic surgery robot | |
| CN112155729B (en) | Intelligent automatic planning method and system for surgical puncture path and medical system | |
| CN110051436B (en) | Automated cooperative work assembly and application thereof in surgical instrument | |
| CN112370163B (en) | Fibula transplantation surgical robot for mandible reconstruction | |
| CN110613511B (en) | Obstacle avoidance method for surgical robot | |
| Deacon et al. | The Pathfinder image-guided surgical robot | |
| Li et al. | Autonomous multiple instruments tracking for robot-assisted laparoscopic surgery with visual tracking space vector method | |
| CN112006777A (en) | Nailing surgical robot system based on surface tracking and control method | |
| JP2022501090A (en) | Medical robot with automatic positioning means | |
| CN115179294A (en) | Robot control method, system, computer device, and storage medium | |
| Gonçalves et al. | A vision system for robotic ultrasound guided orthopaedic surgery | |
| CN109806004A (en) | A kind of surgical robot system and operating method based on cloud data technique | |
| Jerbić et al. | RONNA G4—Robotic Neuronavigation: A novel robotic navigation device for stereotactic neurosurgery | |
| Nguyen et al. | Optimization of layout and path planning of surgical robotic system | |
| US11523868B2 (en) | Bone registration methods for robotic surgical procedures | |
| Meng et al. | Development of an intra-operative active navigation system for robot-assisted surgery | |
| CN114886567A (en) | Method for calibrating hands and eyes of surgical robot with telecentric motionless point constraint | |
| Tan et al. | Automatic generation of autonomous ultrasound scanning trajectory based on 3-d point cloud | |
| Wang et al. | Image-based trajectory tracking control of 4-DOF laparoscopic instruments using a rotation distinguishing marker | |
| CN116793334A (en) | Precision test system and test method of bone surgery navigation system | |
| Huang et al. | Robot-Assisted Deep Venous Thrombosis Ultrasound Examination Using Virtual Fixture | |
| Yasin et al. | Using continuum robots for force-controlled semi autonomous organ exploration and registration | |
| Lu et al. | Spatial mapping method of craniosynostosis surgical robot based on point cloud registration | |
| Greene et al. | dVPose: Automated Data Collection and Dataset for 6D Pose Estimation of Robotic Surgical Instruments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221116 Address after: No. 888, Zhujiang Road, juegang Town, Rudong County, Nantong City, Jiangsu Province, 226499 Patentee after: NANTONG ROBERT MEDICAL TECHNOLOGY Co.,Ltd. Address before: 201318 1, Lane 588, Tianxiong Road, Pudong New Area, Shanghai_ Room 402, building 10, No. 28 Patentee before: SHANGHAI PANYAN ROBOT TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |