CN112382359A - Patient registration method and device, electronic equipment and computer readable medium - Google Patents
Patient registration method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN112382359A CN112382359A CN202011449427.1A CN202011449427A CN112382359A CN 112382359 A CN112382359 A CN 112382359A CN 202011449427 A CN202011449427 A CN 202011449427A CN 112382359 A CN112382359 A CN 112382359A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- image
- coordinate system
- patient
- probe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 239000000523 sample Substances 0.000 claims abstract description 178
- 230000009466 transformation Effects 0.000 claims abstract description 73
- 230000001815 facial effect Effects 0.000 claims abstract description 24
- 230000000007 visual effect Effects 0.000 claims abstract description 20
- 239000011159 matrix material Substances 0.000 claims description 93
- 239000013598 vector Substances 0.000 claims description 54
- 239000003550 marker Substances 0.000 claims description 35
- 238000013519 translation Methods 0.000 claims description 32
- 238000012795 verification Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 230000001131 transforming effect Effects 0.000 claims description 10
- 238000000354 decomposition reaction Methods 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 210000003128 head Anatomy 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 8
- 230000005484 gravity Effects 0.000 description 8
- 238000001356 surgical procedure Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000014509 gene expression Effects 0.000 description 3
- 238000002324 minimally invasive surgery Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000000203 mixture Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004984 smart glass Substances 0.000 description 2
- 101100290429 Mus musculus Mcm3 gene Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007675 cardiac surgery Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 238000011902 gastrointestinal surgery Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 210000000115 thoracic cavity Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
- G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
Abstract
The embodiment of the application provides a patient registration method, a patient registration device, electronic equipment and a computer readable medium, and relates to the field of artificial intelligence. Wherein the method comprises the following steps: acquiring a probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; and determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result. Through the embodiment of the application, not only can the patient be conveniently and quickly registered, but also the registration accuracy of the patient can be effectively improved.
Description
Technical Field
The embodiment of the application relates to the field of artificial intelligence, in particular to a patient registration method, a patient registration device, an electronic device and a computer readable medium.
Background
Since the last 90 s, robot-assisted surgery has gradually become a significant trend. The successful application of a large number of surgical robot systems in clinic has attracted great interest in the medical and scientific fields at home and abroad. Currently, minimally invasive surgical robots are gradually becoming the leading edge of the international robot field and a research hotspot. The minimally invasive surgery robot system integrates a plurality of emerging disciplines, and realizes the minimally invasive, intelligent and digital surgery. Until now, minimally invasive surgical robots have been widely used all over the world, and the types of the applied surgery include urology, obstetrics and gynecology, cardiac surgery, thoracic surgery, hepatobiliary surgery, gastrointestinal surgery, otorhinolaryngology and other subjects.
When the minimally invasive surgery robot is used for assisting surgery, a doctor stands beside a control table and is dozens of centimeters away from an operation table, and looks inwards through a visiting mirror to study a three-dimensional image sent by a camera in a patient body. The three-dimensional image shows the surgical site and the surgical instruments affixed to the end points of the rod. The surgeon operates the surgical instrument using a control handle located directly below the screen. When the surgeon moves the control handle, the computer sends an electronic signal to the surgical instrument, which moves in synchronism with the control handle.
In order to realize the synchronous moving process, the registration of the patient needs to be completed first, and the relative position relationship between the minimally invasive surgery robot and the patient can be obtained in real time. In the prior art, the head of the patient needs to be positioned by a fixing tool to realize the registration of the patient, or some registration markers are pasted on the head of the patient to realize the registration of the patient. Patient registration methods in the prior art are complicated, errors can be caused due to the influence of external factors, the registration precision of a patient is reduced, and the operation precision of the patient is influenced. Therefore, how to simply perform patient registration and effectively improve the accuracy of patient registration is a technical problem to be solved at present.
Disclosure of Invention
The present application aims to provide a patient registration method, a patient registration device, an electronic device, and a computer-readable medium, which are used to solve the technical problem in the prior art of how to easily perform patient registration and effectively improve the precision of patient registration.
According to a first aspect of embodiments herein, a patient registration method is provided. The method comprises the following steps: acquiring a probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; and determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
According to a second aspect of embodiments herein, a patient registration apparatus is provided. The device comprises: the first acquisition module is used for acquiring probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot; the second acquisition module is used for acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; the registration module is used for registering the probe point cloud and the image point cloud so as to obtain a transformation relation between the camera coordinate system and the image coordinate system; the first determination module is used for determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: one or more processors; a storage configured to store one or more programs; when executed by the one or more processors, cause the one or more processors to implement a patient registration method as described in the first aspect of embodiments herein.
According to a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a patient registration method as described in the first aspect of embodiments of the present application.
According to the technical scheme of patient registration provided by the embodiment of the application, probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of a medical robot is obtained; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; compared with the prior other modes, the method has the advantages that the head of the patient is not required to be positioned by a fixing tool, markers do not need to be pasted on the head of the patient, any marker points do not need to be added on the face of the patient, and the patient can be conveniently and quickly registered. In addition, the influence of external factors is eliminated, and the registration precision of the patient is effectively improved.
Drawings
Some specific embodiments of the present application will be described in detail hereinafter by way of illustration and not limitation with reference to the accompanying drawings. The same reference numbers in the drawings identify the same or similar elements or components. Those skilled in the art will appreciate that the drawings are not necessarily drawn to scale. In the drawings:
fig. 1A is a flowchart illustrating steps of a patient registration method according to a first embodiment of the present application;
FIG. 1B is a schematic diagram of an anatomical coordinate system provided in accordance with an embodiment of the present application;
FIG. 2 is a flowchart illustrating steps of a patient registration method according to a second embodiment of the present application;
fig. 3 is a schematic structural diagram of a patient registration apparatus according to a third embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present application;
fig. 5 is a hardware structure of an electronic device according to a fifth embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application shall fall within the scope of the protection of the embodiments in the present application.
In the surgical procedure, the medical robot generally needs to obtain visual information related to the patient according to the marker or scene modeling, and the visual information is correlated with information in the medical image, so as to realize the positioning or navigation function for the patient. Visual information (such as pose information of markers, etc.) is obtained by some visual sensors, and is stored in the coordinate system of the visual sensors themselves, referred to as a camera coordinate system for short, and the space represented by the coordinate system is referred to as a camera space. The information in the medical image is obtained by processing the medical image on a computer. Medical images (such as CT, MR, etc.) are obtained by scanning a patient before or during surgery, and have a reference coordinate system attached to the patient, which is called an image coordinate system, and a correspondingly represented space is called an image space. The process of patient registration is the process of solving the transformation relationship between two spatial coordinate systems. The following will describe in detail the patient registration method provided in the embodiments of the present application.
Referring to fig. 1A, a flowchart illustrating steps of a patient registration method according to a first embodiment of the present application is shown.
Specifically, the patient registration method provided in this embodiment includes the following steps:
in step S101, a probe point cloud in which a probe tip sliding on the face of a patient is formed in a camera coordinate system of a vision sensor of a medical robot is acquired.
In this embodiment, the probe tip is placed on the patient's face, the position of the probe tip is continuously adjusted, the probe tip is always located on the patient's face, and coordinate data of the probe tip in the camera coordinate system is collected to form the probe point cloud. Since the placement of the probe tip on the patient's face is likely to cause deformation of the patient's face, resulting in the coordinate data of the probe tip in the camera coordinate system being collected not to coincide with the coordinate data of the patient's face in the camera coordinate system in a normal state, the tap position of the probe tip is usually selected to be a forehead and nose portion that is not easily deformed. The probe tip touches the face of the patient, the probe is moved to enable the probe tip to slide on the face of the patient, and the marker at the tail of the probe is kept in the visual field of the visual sensor and is identified all the time. Coordinate data of the probe tip in the camera coordinate system is calculated and saved in set a. After a period of time, the probe point cloud is formed by collecting the A. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In some optional embodiments, when acquiring a probe point cloud in which a probe tip sliding on a patient's face is configured in a camera coordinate system of a vision sensor of a medical robot, acquiring coordinate offset data of the probe tip relative to a marker in a coordinate system of the marker at a probe tail; acquiring a coordinate transformation matrix from a coordinate system of the marker to a coordinate system of the camera, wherein the coordinate transformation matrix is obtained by identifying the marker by the visual sensor; determining coordinate data of the probe tip in the camera coordinate system according to the coordinate offset data and the coordinate transformation matrix; and forming the probe point cloud according to the coordinate data of the probe point in the camera coordinate system. Thereby, the coordinate data of the probe tip in the camera coordinate system can be accurately determined by the coordinate offset data and the coordinate transformation matrix. Further, the probe point cloud can be effectively constructed through the coordinate data of the probe point in the camera coordinate system. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In one specific example, the probe is a needle-type tool with a needle-type tip (probe tip) at the end and a fixed marker at the end that is recognized by a visual sensor. The vision sensor obtains the position and posture data of the marker by identifying the marker and then passes through the probe calibrated in advanceAnd calculating the coordinate data of the probe tip under the camera coordinate system according to the offset relation between the probe tip and the marker. There are a total of three coordinate systems in the environment: the coordinate system of the Marker of the probe tail is marked as { Marker }; the coordinate system of the vision sensor, namely a Camera coordinate system { Camera }; image coordinate system { patent }. Coordinate offset data of probe tip under { Marker } coordinate systemMptipAs known, the data directly obtained under the vision sensor is a coordinate transformation matrix from a coordinate system { Marker } to a coordinate system { Camera } at a certain time
Then the coordinate data of the probe tip at that moment under { Camera } can be deduced:
it should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In step S102, an image point cloud of the face image of the patient extracted from the medical image of the patient in an image coordinate system is acquired.
In some optional embodiments, when acquiring an image point cloud formed in an image coordinate system of a facial image of the patient extracted from a medical image of the patient, extracting the facial image to obtain a facial model corresponding to the facial image; according to the direction vector of the face of the patient provided by the face image, emitting rays to the face model to obtain an intersection point of the rays and the face model; and forming the image point cloud according to the coordinate data of the intersection point of the ray and the face model in the image coordinate system. Therefore, the image point cloud can be effectively formed through the face model corresponding to the face image and the direction vector of the face of the patient provided by the face image. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In a specific example, when a ray is emitted to the face model according to a direction vector of the face of the patient provided by the face image to obtain an intersection of the ray and the face model, a ray position is moved on a plane perpendicular to the direction vector of the face of the patient by a specified step size to emit the ray to the face model from different ray positions and obtain a plurality of intersections of the ray and the face model. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In one specific example, a facial image is extracted from a patient image to form an image point cloud. The patient image to be registered is selected and a facial model, denoted SurfB, is extracted from the facial image by either an automatic method (e.g., Marching Cubes method) or a manual method (e.g., a thresholding method). A direction vector of the patient's face is determined from information provided by the facial image. As shown in fig. 1B, the a- > P direction of the anatomical coordinate system. The direction indicated by the direction vector along the patient's face rays from infinity, the intersection with SurfB, is considered a facial point. The ray positions are shifted in the plane of the direction vector perpendicular to the patient's face by a specified density (step size) to obtain a plurality of intersections, which are stored in the set B. And stopping collecting the intersection points when the ray range exceeds the area of the face image, wherein the set B at the moment forms an image point cloud. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In step S103, the probe point cloud and the image point cloud are registered to obtain a transformation relationship between the camera coordinate system and the image coordinate system.
In some optional embodiments, in registering the probe point cloud and the image point cloud, coarsely registering the probe point cloud and the image point cloud according to coordinate data of points in the probe point cloud and the image point cloud to obtain a coarse registration matrix for transforming the camera coordinate system and the image coordinate system; according to the rough registration matrix, performing fine registration on the probe point cloud and the image point cloud to obtain a fine registration matrix for transforming the camera coordinate system and the image coordinate system; and determining the fine registration matrix as a transformation relation between the camera coordinate system and the image coordinate system. Therefore, the probe point cloud and the image point cloud are roughly registered through the coordinate data of the points in the probe point cloud and the image point cloud, and the rough registration matrix can be accurately obtained. In addition, the probe point cloud and the image point cloud are precisely registered through the coarse registration matrix, so that the precise registration matrix can be accurately obtained, and the transformation relation between the camera coordinate system and the image coordinate system can be accurately determined. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In some optional embodiments, when coarsely registering the probe point cloud and the image point cloud according to the coordinate data of the points in the probe point cloud and the image point cloud, determining normal vectors of the points in the probe point cloud and the image point cloud according to the coordinate data of the points in the probe point cloud and the image point cloud; determining characteristic values of points in the probe point cloud and the image point cloud according to normal vectors of the points in the probe point cloud and the image point cloud; and carrying out coarse registration on the probe point cloud and the image point cloud according to the characteristic values of the points in the probe point cloud and the image point cloud so as to obtain a coarse registration matrix. Therefore, the probe point cloud and the image point cloud are roughly registered through the characteristic values of the points in the probe point cloud and the image point cloud, and the rough registration matrix can be accurately obtained. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In one specific example, the feature values of the points in the probe point cloud and the image point cloud may be fast point feature histograms of the points in the probe point cloud and the image point cloud. When determining the fast point feature histogram of the points in the probe point cloud and the image point cloud, firstly calculating the relative relation between each point to be calculated and k field points of the point according to the normal vectors of the point and the k field points, establishing a simplified point feature histogram, then calculating the point feature histogram of the k field points, and finally obtaining the fast point feature histogram through calculation, wherein the calculation expression is
Wherein, S (p)q) Representing the point p to be calculatedqSimplified point feature histogram of (1), F (p)q) Representing the point p to be calculatedqFast point feature histogram of (1), wiA weight value representing a simplified point feature histogram for the ith domain point. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In some optional embodiments, when the probe point cloud and the image point cloud are precisely aligned according to the coarse registration matrix, respectively initializing an optimal rotation matrix and an optimal translation vector according to a rotation matrix and a translation vector included in the coarse registration matrix; iteratively updating the initialized optimal rotation matrix and the optimal translation vector according to the coordinate data of the points in the probe point cloud and the image point cloud; and if the iteration termination condition is met, determining the fine registration matrix according to the optimal rotation matrix and the optimal translation vector. Therefore, the initialized optimal rotation matrix and the initialized optimal translation vector are iteratively updated through the coordinate data of the points in the probe point cloud and the image point cloud, and the fine registration matrix can be accurately determined. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In a specific example, when the initialized optimal rotation matrix and the optimal translation vector are iteratively updated according to the coordinate data of the points in the probe point cloud and the image point cloud, the probe point cloud is transformed according to the optimal rotation matrix and the optimal translation vector, and the transformed probe point cloud is compared with the image point cloud to find out the nearest neighbor point of the points in the probe point cloud in the image point cloud; under the condition that the nearest neighbor point of a point in the probe point cloud in the image point cloud is found, respectively removing the mass center of the probe point cloud and the image point cloud, and determining the covariance matrix of the probe point cloud after the mass center is removed and the covariance matrix of the image point cloud after the mass center is removed; and carrying out singular value decomposition on the covariance matrix, and updating the optimal rotation matrix and the optimal translation vector according to a left singular matrix and a right singular matrix obtained by decomposition. Wherein the iteration termination condition comprises at least one of: the variation of the optimal rotation matrix obtained by the current iteration updating relative to the optimal rotation matrix obtained by the last iteration updating is smaller than a first preset value, and the variation of the optimal translation vector obtained by the current iteration updating relative to the optimal translation vector obtained by the last iteration updating is smaller than a second preset value; and the iteration updating times of the optimal rotation matrix and the optimal translation vector reach the preset maximum iteration times. The first preset value and the second preset value can be set by a person skilled in the art according to actual needs, and this embodiment does not limit this. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In a specific example, the probe point cloud and the image point cloud are registered, and the transformation obtained after the registration is the transformation relation between the camera coordinate system and the image coordinate system, that is, the registration is completed. The size of the probe point cloud is usually smaller than that of the image point cloud, and the probe point cloud in the camera coordinate system is used as the source point cloud PsTaking the image point cloud in the image coordinate system as the target point cloud PtThen the point cloud registration problem can be described as:
wherein p iss，ptRespectively, corresponding points, | P, in the source point cloud and the target point cloudsAnd | is the size of the source point cloud, i.e. the number of points. The two point clouds are registered by a plurality of methods, which are generally divided into a coarse registration step and a fine registration step. Crude preparationThe quasi-registration method includes the steps of firstly, respectively calculating a normal vector of each point in two point clouds, then calculating a characteristic value (such as FPFH (field-programmable gate flash)) according to the normal vector, and matching the two point clouds according to the characteristic value to obtain a rough registration matrix T0(by the rotation matrix R0Sum vector t0Composition). Then use T0Inputting the initial value into a fine registration algorithm (such as ICP and the like), and performing iterative computation to obtain a fine registration matrix T (formed by a rotation matrix R)*And a translation vector t*Composition). The algorithm flow of ICP is as follows: initialization: k is iteration times, and k is initialized to be 0; given an initial transformation R0，t0(ii) a a. Using an initial transformation R0，t0Or R obtained from the last iterationk-1，tk-1As the current optimal transformation, transforming the source point cloud to obtain a temporary transformation point cloud, comparing the temporary transformation point cloud with the target point cloud, and finding out the nearest neighbor point of each point in the source point cloud in the target point cloud; b. if the point correspondence is known, let
Respectively representing the centroids of the source point cloud and the target point cloud, and calculating the point cloud without the centroids
Then, a covariance matrix is calculated
The covariance matrix H is a 3x3 matrix, and is subjected to singular value decomposition to obtain H ═ U ∑ VTIf the optimal rotation corresponding to the current point cloud is: rk＝VUTThe optimal translation is:
c. and continuously iterating the steps a and b until an iteration termination condition (namely a convergence state) is met: rk，tkIs less than a certain value or reaches a set maximum number of iterations. Final result R*,t*I.e. R at convergencek，tk. It will be appreciated that the above description is merely exemplary and that the present embodiment does not do soAny limitation is imposed.
In step S104, the transformation relationship between the camera coordinate system and the image coordinate system is determined as a patient registration result.
In this embodiment, the transformation relationship between the camera coordinate system and the image coordinate system may be the fine registration matrix. Accordingly, the fine registration matrix may be determined as a patient registration result. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In some optional embodiments, after obtaining the transformation relationship between the camera coordinate system and the image coordinate system, the method further comprises: determining registration errors of the probe point cloud and the image point cloud according to a transformation relation between the camera coordinate system and the image coordinate system, coordinate data of points in the probe point cloud, and coordinate data of points in the image point cloud; and verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system. Therefore, the conversion relation between the camera coordinate system and the image coordinate system is verified through the registration error of the probe point cloud and the image point cloud, and the verification result of the conversion relation between the camera coordinate system and the image coordinate system can be accurately obtained. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In a specific example, if the registration error satisfies a condition, the registration is successful, otherwise, the registration fails. Registration error is typically measured by the root mean square error Err. The smaller Err indicates the higher similarity of the two point clouds, the more successful the registration. Of course, for some registration algorithms there may be an own error metric, which is not described in detail here.
Wherein the content of the first and second substances,
is a point in the source point cloud
And transforming the nearest neighbor point in the target point cloud. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
According to the patient registration method provided by the embodiment of the application, probe point cloud formed by the probe point sliding on the face of a patient in the camera coordinate system of the visual sensor of the medical robot is obtained; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; compared with the prior other modes, the method has the advantages that the head of the patient is not required to be positioned by a fixing tool, markers do not need to be pasted on the head of the patient, any marker points do not need to be added on the face of the patient, and the patient can be conveniently and quickly registered. In addition, the influence of external factors is eliminated, and the registration precision of the patient is effectively improved.
The patient registration method provided by the present embodiment may be performed by any suitable device having data processing capabilities, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 2, a flowchart illustrating steps of a patient registration method according to a second embodiment of the present application is shown.
Specifically, the patient registration method provided in this embodiment includes the following steps:
in step S201, a probe point cloud in which a probe tip sliding on the face of a patient is configured in a camera coordinate system of a vision sensor of a medical robot is acquired.
Since the specific implementation of step S201 is similar to the specific implementation of step S101 in the first embodiment, it is not repeated herein.
In step S202, an image point cloud of the face image of the patient extracted from the medical image of the patient in an image coordinate system is acquired.
Since the specific implementation of step S202 is similar to the specific implementation of step S102 in the first embodiment, it is not repeated herein.
In step S203, removing outliers from the points in the image point cloud to obtain a filtered and denoised point cloud.
In this embodiment, an mcmd (maximum Consistency with Minimum distance) -Z automatic denoising algorithm may be adopted:
to the image point cloud PxCalculating a k neighborhood point cloud set P of any point PkIt is obvious that
Wherein k adjacent point clouds are point clouds formed by k points nearest to the p points;
calculating the center point of the point
Sum normal vector
Traversing neighborhood point cloud set PkCalculating the orthogonal distance OD from each point to the fitting plane, wherein the set of the orthogonal distances OD from each point to the fitting planej：
Wherein p isjRepresenting a cloud set of points PkAt any point, N (P)k) To representPoint cloud set PkThe number of point clouds in (a);
calculate Rz-core values for all points in the neighborhood:
wherein MAD represents the mean of absolute deviations;
expressing the average value of OD values of all points in the neighborhood, wherein Rz-core is a self-defined variable and is used for measuring whether a certain point is an abnormal point;
judging the Rz-core value, if the Rz-core value is less than 2.5, determining the Rz-core value as a normal point, otherwise determining the Rz-core value as an abnormal point, eliminating all abnormal points after traversing is completed, and obtaining the filtered and denoised point cloud Pm. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In step S204, down-sampling the filtered and denoised point cloud to obtain a simplified point cloud.
In this embodiment, a three-dimensional voxel grid is created for each point of the input filtered and denoised point cloud, coordinate values of all the points are searched, and the maximum value X in the direction of X, Y, Z is foundmax、Ymax、ZmaxAnd the minimum value Xmin、Ymin、ZminDetermining the side length L of the large cubic grid, if the side length L of the large cubic grid is larger than the preset side length L0Dividing a plurality of voxel grids along the direction X, Y, Z; presetting point cloud number NoSequentially comparing the point cloud number n in the plurality of voxel grids with a preset point cloud number threshold, if the point cloud number n exceeds the preset value, executing the following steps, otherwise deleting the voxel grid;
the side lengths L of several small cubic grids are compared againiWith a predetermined side length L0If the side length is greater than L0Continuously dividing a plurality of small cubes, and if the small cubes are less than or equal to L0Traversing the points in the voxel grid, and replacing other points in the voxel grid by the center of gravity of the voxel grid approximately, wherein the calculation formula of the center of gravity is as follows:
wherein d isiIndicating point (x)i，yi，zi) Distance to the center of the region of each voxel grid, diRepresents the minimum value of the distance, when the minimum value is reached (x)i，yi，zi) I is more than or equal to 0 and less than or equal to n as the gravity center;
wherein d isjIndicating point (x)j，yj，zj) To the region center of gravity (x) of each voxel grid0，y0，z0) Distance of dmaxIndicates the maximum value of the distance, the corresponding point being the farthest point found, max { d }jDenotes { d }jJ is more than or equal to 0 and less than or equal to n-1;
further, the center of gravity point (x) in the voxel grid is preserved0，y0，z0) Processing all voxel grids to obtain simplified point clouds, setting a threshold tau, and if tau is less than or equal to dmaxThen, it remains in accordance with djThe j point, and the gravity point, otherwise, only the gravity point is reserved; the center of gravity point and the point satisfying the distance maximum are the remaining points. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In step S205, the probe point cloud and the condensed point cloud are registered to obtain a transformation relationship between the camera coordinate system and the image coordinate system.
Since the specific implementation of step S205 is similar to the specific implementation of step S103 in the first embodiment, it is not repeated herein.
In step S206, the transformation relationship between the camera coordinate system and the image coordinate system is determined as a patient registration result.
Since the specific implementation of step S206 is similar to the specific implementation of step S104 in the first embodiment, it is not repeated herein.
In some optional embodiments, after obtaining the transformation relationship between the camera coordinate system and the image coordinate system, the method further comprises: acquiring first coordinate data of a marker of the head of the patient touched by the probe tip in the camera coordinate system and second coordinate data of the marker of the head of the patient in the medical image in the image coordinate system; determining a registration error of the probe point cloud and the image point cloud according to a transformation relation between the camera coordinate system and the image coordinate system, the first coordinate data and the second coordinate data; and verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system. Therefore, the conversion relation between the camera coordinate system and the image coordinate system is verified through the registration error of the probe point cloud and the image point cloud, and the verification result of the conversion relation between the camera coordinate system and the image coordinate system can be accurately obtained. It should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
In one specific example, the error metric is implemented by adding a verification point. Before the medical image of the patient is scanned, a marker which can be identified in the medical image can be attached to the head of the patient, so that the marker can be clearly identified in the medical image of the patient and the three-dimensional position of the marker in the image space can be recoveredPpverf. After the registration is finished, the marker is touched by the probe tip, and the three-dimensional position of the marker in the camera space is obtained by the vision sensorCpverf. The error is measured by a transformation of two points:
Err(R*,t*)＝||R*Cpverf+t*-Ppverf||2
it should be understood that the above description is only exemplary, and the present embodiment is not limited thereto.
On the basis of the first embodiment, abnormal points in the image point cloud are removed to obtain a filtered and denoised point cloud, the filtered and denoised point cloud is downsampled to obtain a simplified point cloud, and the probe point cloud and the simplified point cloud are registered to obtain a transformation relation between the camera coordinate system and the image coordinate system. In addition, the point cloud after filtering and denoising can be effectively simplified by down-sampling the point cloud after filtering and denoising, so that the registration speed of the simplified point cloud is effectively improved.
The patient registration method provided by the present embodiment may be performed by any suitable device having data processing capabilities, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 3, a schematic structural diagram of a patient registration apparatus according to a third embodiment of the present application is shown.
The patient registration apparatus provided in this embodiment includes: a first acquisition module 301, configured to acquire a probe point cloud formed by a probe point sliding on a patient's face in a camera coordinate system of a vision sensor of a medical robot; a second obtaining module 302, configured to obtain an image point cloud formed in an image coordinate system of a facial image of the patient extracted from a medical image of the patient; a registration module 303, configured to register the probe point cloud and the image point cloud to obtain a transformation relationship between the camera coordinate system and the image coordinate system; a first determining module 304, configured to determine a transformation relationship between the camera coordinate system and the image coordinate system as a patient registration result.
Optionally, the first obtaining module 301 is specifically configured to: acquiring coordinate offset data of a probe tip relative to a marker in a coordinate system of the marker at the tail of the probe; acquiring a coordinate transformation matrix from a coordinate system of the marker to a coordinate system of the camera, wherein the coordinate transformation matrix is obtained by identifying the marker by the visual sensor; determining coordinate data of the probe tip in the camera coordinate system according to the coordinate offset data and the coordinate transformation matrix; and forming the probe point cloud according to the coordinate data of the probe point in the camera coordinate system.
Optionally, the second obtaining module 302 includes: the extraction submodule is used for extracting the face image to obtain a face model corresponding to the face image; the transmitting sub-module is used for transmitting rays to the face model according to the direction vector of the face of the patient provided by the face image so as to obtain an intersection point of the rays and the face model; and the forming submodule is used for forming the image point cloud according to the coordinate data of the intersection point of the ray and the face model in the image coordinate system.
Optionally, the transmitting submodule is specifically configured to: the method includes moving a ray position in a plane perpendicular to a direction vector of the patient's face by a specified step size to emit rays from different ray positions toward the face model and obtaining a plurality of intersections of the rays with the face model.
Optionally, after the second obtaining module 302, the apparatus further includes: the removing module is used for removing abnormal points from the points in the image point cloud to obtain the point cloud after filtering and denoising; a down-sampling module, configured to down-sample the filtered and denoised point cloud to obtain a simplified point cloud, where the registration module 303 is specifically configured to: and registering the probe point cloud and the simplified point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system.
Optionally, the registration module 303 includes: the rough registration sub-module is used for carrying out rough registration on the probe point cloud and the image point cloud according to the coordinate data of the points in the probe point cloud and the image point cloud so as to obtain a rough registration matrix for transforming the camera coordinate system and the image coordinate system; the fine registration sub-module is used for performing fine registration on the probe point cloud and the image point cloud according to the coarse registration matrix so as to obtain a fine registration matrix for transforming the camera coordinate system and the image coordinate system; and the determining submodule is used for determining the fine registration matrix as a transformation relation between the camera coordinate system and the image coordinate system.
Optionally, the coarse registration sub-module is specifically configured to: determining normal vectors of points in the probe point cloud and the image point cloud according to coordinate data of the points in the probe point cloud and the image point cloud; determining characteristic values of points in the probe point cloud and the image point cloud according to normal vectors of the points in the probe point cloud and the image point cloud; and carrying out coarse registration on the probe point cloud and the image point cloud according to the characteristic values of the points in the probe point cloud and the image point cloud so as to obtain a coarse registration matrix.
Optionally, the fine registration sub-module comprises: the initialization unit is used for respectively initializing an optimal rotation matrix and an optimal translation vector according to the rotation matrix and the translation vector included by the coarse registration matrix; the iterative updating unit is used for iteratively updating the initialized optimal rotation matrix and the optimal translation vector according to the coordinate data of the points in the probe point cloud and the image point cloud; and the determining unit is used for determining the fine registration matrix according to the optimal rotation matrix and the optimal translation vector if the iteration termination condition is met.
Optionally, the iteration update unit is specifically configured to: transforming the probe point cloud according to the optimal rotation matrix and the optimal translation vector, and comparing the transformed probe point cloud with the image point cloud to find out the nearest neighbor point of the point in the probe point cloud in the image point cloud; under the condition that the nearest neighbor point of a point in the probe point cloud in the image point cloud is found, respectively removing the mass center of the probe point cloud and the image point cloud, and determining the covariance matrix of the probe point cloud after the mass center is removed and the covariance matrix of the image point cloud after the mass center is removed; and carrying out singular value decomposition on the covariance matrix, and updating the optimal rotation matrix and the optimal translation vector according to a left singular matrix and a right singular matrix obtained by decomposition.
Optionally, the iteration termination condition comprises at least one of: the variation of the optimal rotation matrix obtained by the current iteration updating relative to the optimal rotation matrix obtained by the last iteration updating is smaller than a first preset value, and the variation of the optimal translation vector obtained by the current iteration updating relative to the optimal translation vector obtained by the last iteration updating is smaller than a second preset value; and the iteration updating times of the optimal rotation matrix and the optimal translation vector reach the preset maximum iteration times.
Optionally, after the registration module 303, the apparatus further comprises: a second determining module, configured to determine a registration error between the probe point cloud and the image point cloud according to a transformation relationship between the camera coordinate system and the image coordinate system, coordinate data of a point in the probe point cloud, and coordinate data of a point in the image point cloud; the first verification module is used for verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system.
Optionally, after the registration module 303, the apparatus further comprises: the third acquisition module is used for acquiring first coordinate data of a marker of the head of the patient touched by the probe tip in the camera coordinate system and second coordinate data of the marker of the head of the patient in the medical image in the image coordinate system; a third determining module, configured to determine a registration error between the probe point cloud and the image point cloud according to a transformation relationship between the camera coordinate system and the image coordinate system, the first coordinate data, and the second coordinate data; and the second verification module is used for verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system.
The patient registration apparatus of this embodiment is used to implement the corresponding patient registration method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again.
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present application; the electronic device may include:
one or more processors 401;
a computer-readable medium 402, which may be configured to store one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a patient registration method as described in embodiment one or embodiment two above.
Fig. 5 is a hardware structure of an electronic device according to a fifth embodiment of the present application; as shown in fig. 5, the hardware structure of the electronic device may include: a processor 501, a communication interface 502, a computer-readable medium 503, and a communication bus 504;
wherein the processor 501, the communication interface 502 and the computer readable medium 503 are communicated with each other through a communication bus 504;
alternatively, the communication interface 502 may be an interface of a communication module, such as an interface of a GSM module;
the processor 501 may be specifically configured to: acquiring a probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; and determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
The Processor 501 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The computer-readable medium 503 may be, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code configured to perform the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code configured to carry out operations for the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may operate over any of a variety of networks: including a Local Area Network (LAN) or a Wide Area Network (WAN) -to the user's computer, or alternatively, to an external computer (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions configured to implement the specified logical function(s). In the above embodiments, specific precedence relationships are provided, but these precedence relationships are only exemplary, and in particular implementations, the steps may be fewer, more, or the execution order may be modified. That is, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a first acquisition module, a second acquisition module, a registration module, and a first determination module. The names of these modules do not in some cases constitute a limitation on the module itself, and for example, the first acquisition module may also be described as "a module that acquires a probe point cloud constituted by a probe point tip sliding on a patient's face in a camera coordinate system of a vision sensor of a medical robot".
As another aspect, the present application also provides a computer-readable medium on which a computer program is stored, which program, when executed by a processor, implements the patient registration method as described in the first or second embodiment.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: acquiring a probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot; acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system; registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system; and determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
The expressions "first", "second", "said first" or "said second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance, but these expressions do not limit the respective components. The above description is only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure.
When an element (e.g., a first element) is referred to as being "operably or communicatively coupled" or "connected" (operably or communicatively) to "another element (e.g., a second element) or" connected "to another element (e.g., a second element), it is understood that the element is directly connected to the other element or the element is indirectly connected to the other element via yet another element (e.g., a third element). In contrast, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is interposed therebetween.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (15)
1. A method of patient registration, the method comprising:
acquiring a probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot;
acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system;
registering the probe point cloud and the image point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system;
and determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
2. The patient registration method according to claim 1, wherein the acquiring a probe point cloud composed of a probe point slid on a patient's face in a camera coordinate system of a vision sensor of a medical robot comprises:
acquiring coordinate offset data of a probe tip relative to a marker in a coordinate system of the marker at the tail of the probe;
acquiring a coordinate transformation matrix from a coordinate system of the marker to a coordinate system of the camera, wherein the coordinate transformation matrix is obtained by identifying the marker by the visual sensor;
determining coordinate data of the probe tip in the camera coordinate system according to the coordinate offset data and the coordinate transformation matrix;
and forming the probe point cloud according to the coordinate data of the probe point in the camera coordinate system.
3. The method of claim 1, wherein the obtaining of the image point cloud in the image coordinate system of the facial image of the patient extracted from the medical image of the patient comprises:
extracting the facial image to obtain a facial model corresponding to the facial image;
according to the direction vector of the face of the patient provided by the face image, emitting rays to the face model to obtain an intersection point of the rays and the face model;
and forming the image point cloud according to the coordinate data of the intersection point of the ray and the face model in the image coordinate system.
4. The patient registration method according to claim 3, wherein the emitting a ray to the face model based on a direction vector of the patient's face provided by the facial image to obtain an intersection of the ray and the face model comprises:
the method includes moving a ray position in a plane perpendicular to a direction vector of the patient's face by a specified step size to emit rays from different ray positions toward the face model and obtaining a plurality of intersections of the rays with the face model.
5. The method of claim 1, wherein after acquiring the image point cloud of the image of the patient's face extracted from the patient's medical image in an image coordinate system, the method further comprises:
removing abnormal points from the points in the image point cloud to obtain a filtered and denoised point cloud;
down-sampling the filtered and de-noised point cloud to obtain a simplified point cloud,
the registering the probe point cloud and the image point cloud comprises:
and registering the probe point cloud and the simplified point cloud to obtain a transformation relation between the camera coordinate system and the image coordinate system.
6. The patient registration method of claim 1, wherein the registering the probe point cloud and the image point cloud to obtain a transformation relationship between the camera coordinate system and the image coordinate system comprises:
according to the coordinate data of the points in the probe point cloud and the image point cloud, carrying out coarse registration on the probe point cloud and the image point cloud so as to obtain a coarse registration matrix for transforming the camera coordinate system and the image coordinate system;
according to the rough registration matrix, performing fine registration on the probe point cloud and the image point cloud to obtain a fine registration matrix for transforming the camera coordinate system and the image coordinate system;
and determining the fine registration matrix as a transformation relation between the camera coordinate system and the image coordinate system.
7. The patient registration method of claim 6, wherein the coarse registration of the probe point cloud and the image point cloud according to coordinate data of points in the probe point cloud and the image point cloud comprises:
determining normal vectors of points in the probe point cloud and the image point cloud according to coordinate data of the points in the probe point cloud and the image point cloud;
determining characteristic values of points in the probe point cloud and the image point cloud according to normal vectors of the points in the probe point cloud and the image point cloud;
and carrying out coarse registration on the probe point cloud and the image point cloud according to the characteristic values of the points in the probe point cloud and the image point cloud so as to obtain a coarse registration matrix.
8. The patient registration method of claim 6, wherein the fine registration of the probe point cloud and the image point cloud according to the coarse registration matrix comprises:
respectively initializing an optimal rotation matrix and an optimal translation vector according to the rotation matrix and the translation vector included by the coarse registration matrix;
iteratively updating the initialized optimal rotation matrix and the optimal translation vector according to the coordinate data of the points in the probe point cloud and the image point cloud;
and if the iteration termination condition is met, determining the fine registration matrix according to the optimal rotation matrix and the optimal translation vector.
9. The patient registration method of claim 8, wherein the iteratively updating the initialized optimal rotation matrix and the optimal translation vector according to the coordinate data of the points in the probe point cloud and the image point cloud comprises:
transforming the probe point cloud according to the optimal rotation matrix and the optimal translation vector, and comparing the transformed probe point cloud with the image point cloud to find out the nearest neighbor point of the point in the probe point cloud in the image point cloud;
under the condition that the nearest neighbor point of a point in the probe point cloud in the image point cloud is found, respectively removing the mass center of the probe point cloud and the image point cloud, and determining the covariance matrix of the probe point cloud after the mass center is removed and the covariance matrix of the image point cloud after the mass center is removed;
and carrying out singular value decomposition on the covariance matrix, and updating the optimal rotation matrix and the optimal translation vector according to a left singular matrix and a right singular matrix obtained by decomposition.
10. The patient registration method of claim 8, wherein the iteration termination condition comprises at least one of:
the variation of the optimal rotation matrix obtained by the current iteration updating relative to the optimal rotation matrix obtained by the last iteration updating is smaller than a first preset value, and the variation of the optimal translation vector obtained by the current iteration updating relative to the optimal translation vector obtained by the last iteration updating is smaller than a second preset value;
and the iteration updating times of the optimal rotation matrix and the optimal translation vector reach the preset maximum iteration times.
11. The patient registration method of claim 1, wherein after obtaining the transformation relationship between the camera coordinate system and the image coordinate system, the method further comprises:
determining registration errors of the probe point cloud and the image point cloud according to a transformation relation between the camera coordinate system and the image coordinate system, coordinate data of points in the probe point cloud, and coordinate data of points in the image point cloud;
and verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system.
12. The patient registration method of claim 1, wherein after obtaining the transformation relationship between the camera coordinate system and the image coordinate system, the method further comprises:
acquiring first coordinate data of a marker of the head of the patient touched by the probe tip in the camera coordinate system and second coordinate data of the marker of the head of the patient in the medical image in the image coordinate system;
determining a registration error of the probe point cloud and the image point cloud according to a transformation relation between the camera coordinate system and the image coordinate system, the first coordinate data and the second coordinate data;
and verifying the transformation relation between the camera coordinate system and the image coordinate system according to the registration error of the probe point cloud and the image point cloud so as to obtain a verification result of the transformation relation between the camera coordinate system and the image coordinate system.
13. A patient registration apparatus, characterized in that the apparatus comprises:
the first acquisition module is used for acquiring probe point cloud formed by a probe point sliding on the face of a patient in a camera coordinate system of a visual sensor of the medical robot;
the second acquisition module is used for acquiring an image point cloud formed by the facial image of the patient extracted from the medical image of the patient in an image coordinate system;
the registration module is used for registering the probe point cloud and the image point cloud so as to obtain a transformation relation between the camera coordinate system and the image coordinate system;
the first determination module is used for determining the transformation relation between the camera coordinate system and the image coordinate system as a patient registration result.
14. An electronic device, characterized in that the device comprises:
one or more processors;
a computer readable medium configured to store one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a patient registration method as claimed in any one of claims 1-12.
15. A computer-readable medium, in which a computer program is stored which, when being executed by a processor, carries out the patient registration method according to any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011449427.1A CN112382359B (en) | 2020-12-09 | 2020-12-09 | Patient registration method and device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011449427.1A CN112382359B (en) | 2020-12-09 | 2020-12-09 | Patient registration method and device, electronic equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112382359A true CN112382359A (en) | 2021-02-19 |
| CN112382359B CN112382359B (en) | 2022-08-26 |
Family
ID=74590966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011449427.1A Active CN112382359B (en) | 2020-12-09 | 2020-12-09 | Patient registration method and device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112382359B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114098985A (en) * | 2021-11-29 | 2022-03-01 | 北京柏惠维康科技有限公司 | Method, device, equipment and medium for spatial matching of patient and medical image of patient |
| CN114587593A (en) * | 2022-03-18 | 2022-06-07 | 华科精准（北京）医疗科技有限公司 | Surgical navigation positioning system and use method thereof |
| CN115775266A (en) * | 2023-02-13 | 2023-03-10 | 北京精准医械科技有限公司 | Registration method applied to real-time puncture surgical robot |
| CN116563561A (en) * | 2023-07-06 | 2023-08-08 | 北京优脑银河科技有限公司 | Point cloud feature extraction method, point cloud registration method and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5999840A (en) * | 1994-09-01 | 1999-12-07 | Massachusetts Institute Of Technology | System and method of registration of three-dimensional data sets |
| CN106780459A (en) * | 2016-12-12 | 2017-05-31 | 华中科技大学 | A kind of three dimensional point cloud autoegistration method |
| CN107292925A (en) * | 2017-06-06 | 2017-10-24 | 哈尔滨工业大学深圳研究生院 | Based on Kinect depth camera measuring methods |
| CN110946659A (en) * | 2019-12-25 | 2020-04-03 | 武汉中科医疗科技工业技术研究院有限公司 | Registration method and system for image space and actual space |
| CN111311651A (en) * | 2018-12-11 | 2020-06-19 | 北京大学 | Point cloud registration method and device |
| CN111724420A (en) * | 2020-05-14 | 2020-09-29 | 北京天智航医疗科技股份有限公司 | Intraoperative registration method and device, storage medium and server |
-
2020
- 2020-12-09 CN CN202011449427.1A patent/CN112382359B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5999840A (en) * | 1994-09-01 | 1999-12-07 | Massachusetts Institute Of Technology | System and method of registration of three-dimensional data sets |
| CN106780459A (en) * | 2016-12-12 | 2017-05-31 | 华中科技大学 | A kind of three dimensional point cloud autoegistration method |
| CN107292925A (en) * | 2017-06-06 | 2017-10-24 | 哈尔滨工业大学深圳研究生院 | Based on Kinect depth camera measuring methods |
| CN111311651A (en) * | 2018-12-11 | 2020-06-19 | 北京大学 | Point cloud registration method and device |
| CN110946659A (en) * | 2019-12-25 | 2020-04-03 | 武汉中科医疗科技工业技术研究院有限公司 | Registration method and system for image space and actual space |
| CN111724420A (en) * | 2020-05-14 | 2020-09-29 | 北京天智航医疗科技股份有限公司 | Intraoperative registration method and device, storage medium and server |
Non-Patent Citations (5)
| Title |
|---|
| 吴秀杰 税午阳: "从奇和洞人颅面复原我们的祖先张什么样", 《化石》 * |
| 李健 等: "一种应用于大角度变换点云的配准方法", 《图书学报》 * |
| 赵夫群: "《点云模型的优化配准方法研究》", 31 July 2018 * |
| 陈学伟 等: "基于FPFH特征的点云配准技术", 《电脑知识与技术》 * |
| 陈立亮 等: "《铸造凝固模拟技术研究 华铸文集 第2卷》", 30 November 2015 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114098985A (en) * | 2021-11-29 | 2022-03-01 | 北京柏惠维康科技有限公司 | Method, device, equipment and medium for spatial matching of patient and medical image of patient |
| CN114587593A (en) * | 2022-03-18 | 2022-06-07 | 华科精准（北京）医疗科技有限公司 | Surgical navigation positioning system and use method thereof |
| CN115775266A (en) * | 2023-02-13 | 2023-03-10 | 北京精准医械科技有限公司 | Registration method applied to real-time puncture surgical robot |
| CN116563561A (en) * | 2023-07-06 | 2023-08-08 | 北京优脑银河科技有限公司 | Point cloud feature extraction method, point cloud registration method and readable storage medium |
| CN116563561B (en) * | 2023-07-06 | 2023-11-14 | 北京优脑银河科技有限公司 | Point cloud feature extraction method, point cloud registration method and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112382359B (en) | 2022-08-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112382359B (en) | Patient registration method and device, electronic equipment and computer readable medium | |
| CN110956635B (en) | Lung segment segmentation method, device, equipment and storage medium | |
| CN103793915B (en) | Inexpensive unmarked registration arrangement and method for registering in neurosurgery navigation | |
| EP1903972A2 (en) | Methods and systems for mapping a virtual model of an object to the object | |
| CN108154531B (en) | Method and device for calculating area of body surface damage region | |
| CN112734776B (en) | Minimally invasive surgical instrument positioning method and system | |
| CN113397704B (en) | Robot positioning method, device and system and computer equipment | |
| CN111260704A (en) | Vascular structure 3D/2D rigid registration method and device based on heuristic tree search | |
| CN113143459B (en) | Navigation method and device for laparoscopic augmented reality operation and electronic equipment | |
| CN116612166A (en) | Registration fusion algorithm for multi-mode images | |
| CN114931435B (en) | Three-dimensional model processing method and device and electronic equipment | |
| US20220249174A1 (en) | Surgical navigation system, information processing device and information processing method | |
| CN116327362A (en) | Navigation method, device, medium and electronic equipment in magnetic probe auxiliary bronchus operation | |
| CN113974834B (en) | Method and device for determining sleeve pose of surgical robot system | |
| CN113012230B (en) | Method for placing surgical guide plate under auxiliary guidance of AR in operation | |
| US20210287434A1 (en) | System and methods for updating an anatomical 3d model | |
| CN115272356A (en) | Multi-mode fusion method, device and equipment of CT image and readable storage medium | |
| CN112562070A (en) | Craniosynostosis operation cutting coordinate generation system based on template matching | |
| CN113256693A (en) | Multi-view registration method based on K-means and normal distribution transformation | |
| CN112258647A (en) | Map reconstruction method and device, computer readable medium and electronic device | |
| CN114098985A (en) | Method, device, equipment and medium for spatial matching of patient and medical image of patient | |
| CN114299127B (en) | Oral cavity image registration method and device, electronic equipment and computer storage medium | |
| US20240005503A1 (en) | Method for processing medical images | |
| CN116245920A (en) | Point cloud registration method, device, equipment and medium for patient face | |
| WO2024055493A1 (en) | Heterogeneous and three-dimensional observation registration method based on deep phase correlation, and medium and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100191 Room 501, floor 5, building 9, No. 35 Huayuan North Road, Haidian District, Beijing Applicant after: Beijing Baihui Weikang Technology Co.,Ltd. Address before: 100191 Room 608, 6 / F, building 9, 35 Huayuan North Road, Haidian District, Beijing Applicant before: Beijing Baihui Wei Kang Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |