CN116828160A - Projection method and operation auxiliary system - Google Patents
Projection method and operation auxiliary system Download PDFInfo
- Publication number
- CN116828160A CN116828160A CN202310778887.6A CN202310778887A CN116828160A CN 116828160 A CN116828160 A CN 116828160A CN 202310778887 A CN202310778887 A CN 202310778887A CN 116828160 A CN116828160 A CN 116828160A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- image
- coordinate system
- projected
- projector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012545 processing Methods 0.000 claims description 69
- 238000006243 chemical reaction Methods 0.000 claims description 26
- 230000009466 transformation Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 5
- 230000002093 peripheral effect Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 claims description 2
- 230000004044 response Effects 0.000 claims 1
- 230000001131 transforming effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 18
- 230000000007 visual effect Effects 0.000 abstract description 8
- 230000008569 process Effects 0.000 description 12
- 206010028980 Neoplasm Diseases 0.000 description 6
- 210000004556 brain Anatomy 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 239000004973 liquid crystal related substance Substances 0.000 description 4
- 210000001519 tissue Anatomy 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 210000004204 blood vessel Anatomy 0.000 description 2
- 238000007428 craniotomy Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000000725 suspension Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000002513 implantation Methods 0.000 description 1
- 238000007917 intracranial administration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000001147 pulmonary artery Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 230000002792 vascular Effects 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
The invention provides a projection method and an operation auxiliary system, wherein the method comprises the following steps: registering, namely unifying the structured light module and the model point cloud to the same coordinate system; generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system; the first image is projected by the projector onto a target object surface. According to the method, the position of the target object point cloud is rapidly determined through structured light registration, the first image corresponding to the point cloud to be projected is generated by combining the model point cloud to project, whether the projection falling point accords with the expectation or not is not detected through the tracking equipment, the angle of the projector is not required to be adjusted, the projection effect is corrected, and the accuracy and the projection efficiency of projection are improved. Through the point cloud projection, a user does not need to switch the sight between the concerned part and the screen, so that visual discussion, teaching, operation planning, operation navigation and the like are facilitated, and the visual field range is wider.
Description
Technical Field
The invention relates to the technical field of medical instruments, in particular to a projection method and an operation auxiliary system.
Background
At present, when a treatment scheme is formulated for a patient, a surgical scheme is generally discussed and planned in a three-dimensional model established based on preoperative images, the display field of view in a screen is limited, and visual difference exists between the surgical scheme and a real patient, so that a doctor cannot feel the surgical scheme intuitively. In addition, during surgery, a doctor is usually guided through a surgical navigation positioning system, for example, a surgical instrument is displayed in a three-dimensional model to guide the doctor to adjust the position of the surgical instrument, and in this guiding manner, the doctor needs to continuously switch the line of sight between a surgical site and a screen, so that the surgical operation is inconsistent and easy to fatigue.
In view of the above-mentioned drawbacks of the prior art, the present invention provides a point cloud projection method and an operation assisting system to solve or at least partially solve the above-mentioned drawbacks of the prior art.
Disclosure of Invention
The invention provides a point cloud projection system which is used for solving the defects that in the prior art, a doctor is not intuitive enough to observe a three-dimensional model, and needs to repeatedly switch between an operation part and a screen, so that operation is easy to be incoherent.
The invention provides a projection method, which comprises the following steps:
registering, namely unifying the structured light module and the model point cloud to the same coordinate system; the structural light module comprises a camera and a projector, and the model point cloud comprises a point cloud to be projected;
generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system;
the first image is projected by the projector onto a target object surface.
According to the projection method provided by the invention, the registering, unifying the structured light module and the model point cloud to the same coordinate system, comprises the following steps:
acquiring a point cloud of a target object through the structural optical module;
registering the point cloud of the target object with the model point cloud to obtain a conversion relation between a structured light coordinate system and a model point cloud coordinate system;
and unifying the structured light module and the model point cloud to the same coordinate system according to the conversion relation between the structured light coordinate system and the model point cloud coordinate system.
According to the projection method provided by the invention, the registering, unifying the structured light module and the model point cloud to the same coordinate system, comprises the following steps:
the mechanical arm is controlled to drive the structured light module to register a patient, and the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system is obtained;
and unifying the structured light module and the model point cloud to the same coordinate system according to the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system and the current mechanical arm projection gesture.
According to the projection method provided by the invention, the generating the first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system comprises:
generating a third image according to the point cloud to be projected and the position of the projector;
performing distortion transformation on the third image to obtain a second image;
and processing the second image according to the internal parameters of the projector to obtain the first image corresponding to the point cloud to be projected.
According to the projection method provided by the invention, the third image is generated according to the point cloud to be projected and the position of the projector, and the method comprises the following steps:
converting the point cloud to be projected into a projector coordinate system;
and carrying out normalization operation on the point cloud to be projected under the projector coordinate system to obtain the third image.
According to the projection method provided by the invention, the third image is subjected to distortion transformation to obtain the second image, and the second image is realized according to the following formula:
x′＝x(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 xy+p 2 (r 2 +2x 2 )；
y′ ＝ y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 2 xy+p 1 (r 2 +2y 2 )；
wherein x and y are point coordinates in the third image, x ', y' are point coordinates in the second image obtained by distortion transformation, and r 2 ＝x 2 +y 2 ；k 1 、k 2 、k 3 For radial distortion parameter, p 1 、p 2 The tangential distortion parameters are obtained by pre-calibration.
According to the projection method provided by the invention, the processing of the second image according to the internal reference of the projector to obtain the first image corresponding to the point cloud to be projected comprises the following steps:
and processing the second image according to the internal parameters of the projector to obtain a two-dimensional image under a projection image plane, and obtaining the first image according to projection resolution.
According to the projection method provided by the invention, the generating the first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system comprises:
when the point is required to be projected, highlighting the point required to be projected in the first image; and/or the number of the groups of groups,
when the outline is required to be projected, connecting peripheral points in the first image to form a closed outline; and/or the number of the groups of groups,
and when the area is required to be projected, carrying out morphological processing on the first image.
According to the projection method provided by the invention, the processing module comprises a memory and a processor, and the processor loads a computer program stored in the memory to realize the projection method according to any one of the above.
The present invention also provides a surgical assistance system comprising: a structured light module and the processing module described above;
the structural light module comprises a camera and a projector, and is used for collecting point clouds of a target object;
the processing module is in communication connection with the structured light module and is used for generating a first image corresponding to the point cloud to be projected according to the model point cloud and the point cloud of the target object acquired by the structured light module, and projecting the first image to the surface of the target object through the projector; the model point cloud comprises the point cloud to be projected.
The surgical auxiliary system provided by the invention further comprises a fixing device, wherein the fixing device is a fixed connection structure or a mounting structure matched with a mounting position and is used for fixing the structured light module at a required position.
According to the surgical auxiliary system provided by the invention, the system comprises a plurality of groups of the fixed structures and the structured light modules so as to realize multi-angle point cloud projection.
According to the surgical auxiliary system provided by the invention, the system further comprises a mechanical arm, wherein the structural light module is installed or integrated on the mechanical arm, and the mechanical arm is in communication connection with the processing module and is used for driving the structural light module to move to a required position according to a control instruction of the processing module.
The projection method and the operation auxiliary system provided by the invention have at least the following beneficial effects:
1. the method has the advantages that information (inside or surface) to be observed can be projected to the surface of the target object, a user does not need to switch the sight between a concerned part and a screen, visual study, teaching, operation planning, operation navigation and the like are convenient, and the visual field range is wider;
2. the structured light module has the dual functions of collecting point cloud and point cloud projection, the processing module processes data to generate a projection image, and the whole projection system has simple structure and low cost;
3. patient registration based on a structured light module has high registration precision and more accurate projection;
4. by converting the coordinate system of the point cloud to be projected and combining the projector parameters to carry out distortion transformation and imaging, the image (first image) to be projected, which is more adaptive to the point cloud to be projected and the projector, is obtained, and the projection effect is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic illustration of a surgical assistance system according to the present invention;
FIG. 2 is a schematic flow chart of a projection method according to the present invention;
FIG. 3 is a second schematic view of a surgical assistance system according to the present invention;
FIG. 4 is a third schematic view of a surgical assistance system according to the present invention;
FIG. 5 is a schematic view of a projection effect of a point cloud projection method according to the present invention;
fig. 6 is a schematic diagram of a second projection effect of the point cloud projection method according to the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
A point cloud projection method and surgical assistance system of the present invention are described below in conjunction with fig. 1-6.
To facilitate understanding of the present invention, a description will be given of a structure of an operation assisting system corresponding to a projection method of the present invention with reference to fig. 1. As shown in fig. 1, the system includes a structured light module 100, and a processing module 200. The structured light module 100 comprises a camera 110 and a projector 120, the structured light module 100 being capable of employing a three-dimensional point cloud based on structured light principles. The structural light point cloud acquisition is an active type point cloud acquisition technology and has the advantages of non-contact, large field of view, high precision (large data volume of the point cloud), good instantaneity and the like. In the process of collecting the point cloud, the structured light module 100 can project the coded two-dimensional pattern to the target object through the projector 120, the pattern can be deformed due to inconsistent height of the object surface, the camera 110 shoots and collects the deformed pattern, then decodes the pattern to obtain the corresponding relation between the object surface and the projected pattern point, and calculates the three-dimensional point cloud on the surface of the target object based on the triangulation principle. The structured light module 100 may also be projected towards a target object by a projector 120. The processing module 200 performs data interaction with the structured light module 100 for performing data processing. The processing module 200 may be integrated with the aforementioned structured light module 100, or may be an external independent device, such as a notebook computer, a workstation, a PDA, etc., and the present invention is not limited to the form of the processing module 200 and the form of hardware.
The method of the present invention may be performed in the foregoing processing module 200, and fig. 2 is a schematic flow chart of a projection method provided by the present invention, as shown in fig. 2, and the method includes:
s21, registering, wherein the structured light module 100 and the model point cloud are unified into the same coordinate system; the structured light module 100 includes a camera 110 and a projector 120, and the model point cloud includes a point cloud to be projected;
specifically, the model point cloud is a three-dimensional point cloud corresponding to the target object, for example, a three-dimensional point cloud generated based on medical images (magnetic resonance images, functional magnetic resonance images, CT images, phase contrast magnetic resonance vascular imaging, and the like) of the target object, and for example, the target object is a physical demonstration model, and the model point cloud is a three-dimensional point cloud generated based on a corresponding three-dimensional design model. The model point cloud comprises point cloud data such as a puncture point, an anatomical line, a tumor, a blood vessel, a brain region and the like.
Registration is to establish a mapping relation between a real space and a model point cloud space. The position of the structured light module 100 in the real space is determined, and the structured light module 100 collects the point cloud of the target object, so that the point cloud position of the target object in the real space is determined. The point cloud of the target object may be used to register with the model point cloud, so as to establish a mapping relationship between the real space and the model point cloud space, and the processing module 200 unifies the structured light module 100 and the model point cloud to the same coordinate system, for example, to the structured light coordinate system, to the hospital bed coordinate system, to the operation table coordinate system, to the model point cloud coordinate system, and so on.
S22, generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system;
in particular, the projector parameters include internal parameters, i.e. the position of the projector, and external parameters, i.e. the projector device internal parameters, such as the focal length, the corresponding physical length of the unit pixels in the projection image plane, etc. The aforementioned position of the structured light module 100 in the same coordinate system is determined, and the projector 120 is an integral part of the structured light module 100, so that the position of the projector 120 (projector external reference) in the same coordinate system can be determined, and in addition, the projector internal reference is known by calibration. According to the model point cloud and projector parameters under the same coordinate system, a first image corresponding to the point cloud to be projected under the 'view angle of the projector 120' can be generated, that is, the projector projects according to the first image, and the 'point cloud to be projected' can be displayed on the surface of the target object.
S23, projecting the first image to the target object surface by the projector 120.
Specifically, the processing module 200 sends the first image to the projector 120 so that the projector 120 projects the first image onto the target object surface, displaying a "point cloud to be projected" on the target object surface. For example, intracranial tumors, blood vessels are projected onto the head of a patient, and for example, brain regions are projected onto the surface of a physical presentation model.
According to the projection method, the position of the target object point cloud is quickly and accurately determined through structured light registration, the first image corresponding to the point cloud to be projected is generated by combining the model point cloud for projection, whether the projection falling point accords with the expectation or not is not detected through tracking equipment, the angle of a projector is not required to be adjusted, the projection effect is not required to be corrected, and the accuracy and the projection efficiency of projection are improved. By projecting the point cloud to be projected (the internal or surface information to be observed) onto the surface of the target object, the user does not need to switch the line of sight between the concerned part and the screen, so that visual discussion, teaching, operation planning, operation navigation and the like are facilitated, and the visual field range is wider.
Based on any of the foregoing embodiments, in one embodiment, the S21 includes:
acquiring a point cloud of a target object through the structured light module 100;
registering the point cloud of the target object with the model point cloud to obtain a conversion relation between a structured light coordinate system and a model point cloud coordinate system;
and unifying the structured light module 100 and the model point cloud to the same coordinate system according to the conversion relation between the structured light coordinate system and the model point cloud coordinate system.
Specifically, the target object is subjected to point cloud collection by the structured light module 100, and it is understood that what is collected here is a visually visible surface point cloud of the target object, such as facial skin data, and also, for example, tissues in an exposed state in craniotomies. The processing module 200 receives the obtained point cloud data and registers the obtained point cloud data with the model point cloud to obtain a conversion relationship between the structured light coordinate system and the model point cloud coordinate system, where the position of the structured light module 100 in the structured light coordinate system is known, so that the structured light module 100 and the model point cloud can be converted into the same coordinate system, for example, into the structured light coordinate system and, for example, into the model point cloud coordinate system. It will be appreciated that the position of the collection and projection of the structured light spot cloud in this embodiment is the same position, and accordingly, the system structure may be that the structured light module 100 is placed at a fixed position, for example, the structured light module is fixed at a desired position by a fixing device (a fixed suspension structure, a fixed support structure referring to fig. 3, etc.), and for example, the structured light module 100 is mounted on a mechanical arm, and the processing module 200 controls and adjusts the position of the mechanical arm so that the structured light module 100 performs the collection and projection of the spot cloud at the desired position.
FIG. 5 is a schematic view of the projection effect of the point cloud projection method of the present invention, and FIG. 5 illustrates the projection effect of projecting the position of a tumor region onto the head of a patient; fig. 6 is a second schematic view of a projection effect of the point cloud projection method provided by the present invention, and fig. 6 illustrates a projection effect of projecting a tumor and surrounding tissues to a head of a patient (different colors may be rendered for different tissues, and fig. 5 and 6 are gray-scale diagrams, and color effects are missing).
In the embodiment, the point cloud acquired by the structural light module is directly registered with the model point cloud, so that the mapping relation between the model point cloud space and the real space is rapidly and accurately determined, the structural light module and the model point cloud are unified to the same coordinate system, the first image is conveniently generated for projection, whether the projection falling point accords with the expectation or not is not detected by the tracking equipment, the projector angle is not required to be adjusted, the projection effect is not required to be corrected, and the whole system is simple in structure, low in cost and convenient for market popularization.
Based on any of the foregoing embodiments, in one embodiment, the S21 includes:
the mechanical arm is controlled to drive the structured light module to register a patient, and the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system is obtained;
and unifying the structured light module and the model point cloud to the same coordinate system according to the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system and the current mechanical arm projection gesture.
Specifically, referring to fig. 4, the system structure is described first, the structured light module 100 is mounted or integrated on the mechanical arm 400, the processing module 200 controls the mechanical arm 400 to drive the structured light module 100 to collect a point cloud for a target object at one or more positions, and the processing module 200 registers the point cloud of one position or a fusion point cloud of a plurality of positions under the mechanical arm coordinate system with the model point cloud, so as to obtain a conversion relationship between the mechanical arm coordinate system and the model point cloud coordinate system. The processing module 200 controls the mechanical arm 400 to drive the structured light module 100 to move to a required projection pose, and the projection pose of the mechanical arm 400 corresponds to the conversion relation between the mechanical arm coordinate system and the structured light coordinate system. According to the two conversion relationships, the structured light module 100 and the model point cloud can be converted into the same coordinate system, for example, into the mechanical arm coordinate system, the model point cloud coordinate system, the sickbed coordinate system, and for example, into the structured light coordinate system under the projection posture.
In this embodiment, only one patient registration is required by the mechanical arm, the projection can be flexibly performed according to the current projection pose of the mechanical arm, the projection position and angle are more diversified, and the data processing (projection) efficiency is higher.
The following describes the process of unifying the structured light module and the model point cloud to the same coordinate system by a preferred embodiment:
(1) Calibration before projection.
Obtaining a conversion matrix cam_to_proj from the camera 110 to the projector 120, an internal reference matrix intrinsic_proj of the projector and a conversion relation flan_to_cam from a tail flange of the mechanical arm to the camera 110 through calibration of a structured light module;
(2) Structured light registration.
The mechanical arm drives the structural light module 100 to collect the structural light point cloud of the target object at one or more positions, and the structural light point cloud of the target object is converted into a mechanical arm coordinate system according to the calibrated parameters;
registering the structural point cloud and the model point cloud under the mechanical arm coordinate system to obtain the conversion relation from the model point cloud to the mechanical arm, so that the model point cloud is converted into the mechanical arm coordinate system.
(3) Model point cloud conversion.
According to the current gesture matrix of the mechanical arm, the conversion relation base_to_flan from the mechanical arm to the flange under the current projection gesture can be obtained;
the conversion relationship from the mechanical arm to the projector under the current projection posture is as follows:
base_to_proj＝cam_to_proj*flan_to_cam*base_to_flan；
the model point cloud under the robot arm coordinate system is converted to the projector coordinate system according to the following mode:
point_in_proj＝base_to_proj*point_in_base。
it should be noted that, in the structured light module 100, the positional relationship between the camera 110 and the projector 120 is fixed, and a camera coordinate system is generally used as the structured light coordinate system, however, a projector coordinate system may be used as the structured light coordinate system, and in this embodiment, the structured light module and the model point cloud are unified into the projector coordinate system, so as to generate the first image.
Based on any of the above embodiments, in one embodiment, the S22 includes:
generating a third image according to the point cloud to be projected and the position of the projector;
performing distortion transformation on the third image to obtain a second image;
and processing the second image according to the internal parameters of the projector to obtain the first image corresponding to the point cloud to be projected.
Specifically, a third image is generated according to the point cloud to be projected and the position of the projector, so as to realize conversion from the three-dimensional point cloud to the two-dimensional image, for example, the point cloud to be projected can be converted to a projector coordinate system, and the third image is a sheet-shaped point cloud (i.e. a two-dimensional image) obtained by normalizing the three-dimensional point cloud to be projected along the main axis direction (Z-axis direction) of the projector. Because distortion exists in the projection process of the projector 120, the third image is further subjected to distortion transformation to obtain a corrected second image, and the second image is processed by combining with the internal parameters of the projector to obtain a first image corresponding to the point cloud to be projected. The first image is a pixel image, which is convenient for the projector 120 to process and project based on the first image.
According to the method, the third image is generated according to the point cloud to be projected and the position of the projector, distortion transformation is conveniently carried out on the third image, interference of projection distortion of the projector on a projection effect is corrected, the first image is obtained by processing the second image according to internal parameters of the projector, projection is conveniently carried out by the projector, the position of the point cloud to be projected and the position of the projector are considered in the process of generating the first image through data processing, efficient and accurate projection can be achieved, follow-up detection of whether a projection falling point accords with expectations or not is not needed, the angle of the projector is not needed to be adjusted, projection effect is corrected, and projection accuracy and projection efficiency are improved.
Based on any of the foregoing embodiments, in one embodiment, the generating a third image according to the point cloud to be projected and the position of the projector includes:
converting the point cloud to be projected into a projector coordinate system;
and carrying out normalization operation on the point cloud to be projected under the projector coordinate system to obtain the third image.
Specifically, the point cloud to be projected and the structured light module 100 have been converted into the same coordinate system, where the structured light module 100 includes the camera 110 and the projector 120, i.e. the position of the projector module 120 is known, and the point cloud to be projected is converted into the projector coordinate system for further processing to obtain the third image. Preferably, the above processing is implemented according to the following formula:
wherein, the liquid crystal display device comprises a liquid crystal display device,for the point coordinates in the point cloud to be projected under the projector coordinate system, the +.>Sitting for the corresponding point in the third imageAnd (5) marking.
According to the embodiment, the point cloud to be projected is converted into the projector coordinate system and normalized to obtain the third image, the point cloud to be projected is subjected to preliminary imaging, and the third image is located in the plane of the unit distance in front of the projector, so that distortion transformation is facilitated, and the projection effect is optimized.
Based on any of the foregoing embodiments, in one embodiment, the performing distortion transformation on the third image to obtain a second image is implemented according to the following formula:
x′＝x(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 xy+p 2 (r 2 +2x 2 )；
y′ ＝ y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 2 xy+p 1 (r 2 +2y 2 )；
wherein x and y are point coordinates in the third image, x ', y' are point coordinates in the second image obtained by distortion transformation, and r 2 ＝x 2 +y 2 ；k 1 、k 2 、k 3 For radial distortion parameter, p 1 、p 2 The tangential distortion parameters are obtained by pre-calibration.
According to the embodiment, the second image is obtained by carrying out distortion transformation on the third image, so that the finally generated first image can adapt to the distortion of the projector, and the projector can project the point cloud to be projected to the required position more accurately.
Based on any of the foregoing embodiments, in one embodiment, the processing the second image according to the internal reference of the projector to obtain the first image corresponding to the point cloud to be projected includes:
and processing the second image according to the internal parameters of the projector to obtain a two-dimensional image under a projection image plane, and obtaining the first image according to projection resolution.
Specifically, according to the focal length f of the camera, the second image after distortion transformation is converted into physical imaging with (physical) length in the projection image plane, and then the first image under the pixel coordinate system is obtained by combining the resolution of the projector, and the first image can be used for projection by the projector. Preferably, the above procedure is performed according to the following formula:
wherein, the liquid crystal display device comprises a liquid crystal display device,is the point coordinates in the second image, f is the focal length, d x 、d y For the physical length of the unit pixel in the first image corresponding to the coordinate axis direction, +.>For the point coordinates of the second image corresponding to the origin of coordinates in the first image,/->Is the point coordinates in the second image.
In the embodiment, the first image is accurately generated by combining the internal parameters of the projector to process and convert the second image, so that the point cloud to be projected is convenient to project to the required position, the projection process does not need to detect whether the projection falling point accords with the expectation or not through the tracking equipment, the angle of the projector does not need to be adjusted, the projection effect is corrected, and the accuracy and the projection efficiency of projection are improved.
Based on any of the above embodiments, in one embodiment, the S22 includes:
when the point is required to be projected, highlighting the point required to be projected in the first image; and/or the number of the groups of groups,
when the outline is required to be projected, connecting peripheral points in the first image to form a closed outline; and/or the number of the groups of groups,
and when the area is required to be projected, carrying out morphological processing on the first image.
Specifically, if the point to be projected is a discrete point such as a needle insertion point, a bone screw implantation point and the like, the point to be projected in the two-dimensional image to be projected is highlighted, for example, a cross is displayed by taking the point as the center, and the point is enlarged and displayed; if the outline is required to be projected, such as a tumor outline and a brain area outline, connecting peripheral points in the two-dimensional image to be projected, and projecting the peripheral outline to the surface of the target object; if the whole area is required to be projected, morphological processing is performed on the two-dimensional image to be projected, specifically, the point cloud density of the point cloud to be projected may not meet the requirement, the pixel points in the first image generated correspondingly may be sparse and do not accord with the projection resolution of the projector, missing pixels in the pixel area to be projected of the first image may be supplemented through morphological processing (such as closed operation), and the point cloud projection effect is further optimized.
The processing module provided by the invention is described below, and the processing module described below and the projection method described above can be referred to correspondingly.
The processing module 200 provided by the invention comprises a memory and a processor, wherein the processor loads a computer program stored in the memory to realize the projection method of any one of the above.
Specifically, the processing module 200 may be a separate device outside the structural light module 100, such as a notebook computer, a workstation, a PDA, etc., and the processing module 200 may be integrated with the structural light module 100, so that the present invention is not limited to the existence form and hardware form of the processing module 200. The processor of the processing module 200 loads the computer program in the running memory, controls the structured light module 100 to collect the point cloud data of the target object, and the processing module 200 receives the point cloud data from the structured light module 100, processes the point cloud data in combination with the model point cloud data to generate a first image, and sends the first image to the structured light module 100, so that the projector of the structured light module 100 projects the point cloud to the required position of the target object.
The processing module of the embodiment can generate a first image according to the point cloud collected by the structural light module and the model point cloud, so that the structural light module can project the point cloud to be projected to a required position of the target object according to the first image. The projection process does not need to detect whether the projection falling point accords with the expectation or not through the tracking equipment, does not need to adjust the angle of the projector and correct the projection effect, and improves the accuracy and the projection efficiency of projection.
In some embodiments, the structured light module 100 is installed/integrated on a mechanical arm, and the processor of the processing module 200 loads the computer program in the operation memory to further realize that the mechanical arm is controlled to move, so as to drive the structured light module 100 to move to a required point cloud acquisition position and a point cloud projection position.
The following describes the surgical assistance system provided by the present invention, and the surgical assistance system described below and the projection method and the processing module 200 described above may be referred to correspondingly.
Still referring to fig. 1, the surgical assistance system provided by the present invention includes: a structured light module 100, and any of the processing modules 200 described above.
The structured light module 100 includes a camera 110 and a projector 120, and the structured light module 100 is capable of capturing a three-dimensional point cloud of a target object based on structured light principles. The structural light point cloud acquisition is an active type point cloud acquisition technology and has the advantages of non-contact, large field of view, high precision (large data volume of the point cloud), good instantaneity and the like. In the process of collecting the point cloud, the structured light module 100 can project the coded two-dimensional pattern to the target object through the projector 120, the pattern can be deformed due to inconsistent height of the object surface, the camera 110 shoots and collects the deformed pattern, then decodes the pattern to obtain the corresponding relation between the object surface and the projected pattern point, and calculates the three-dimensional point cloud on the surface of the target object based on the triangulation principle. The structured light module 100 may also project a first image to the target object through the projector 120 according to the instructions of the processing module 200.
In this embodiment, the structured light module 100 may be directly placed at a certain fixed position, for example, on a certain platform beside a hospital bed; the structured light module 100 may also be mounted or fixed in a certain position, for example, suspended above a hospital bed, mounted on the hospital bed, and fixedly supported on the ground by a support structure; the structured light module 100 may also be integrated or mounted on other devices, such as a trolley of a surgical navigation system, etc., and the present invention is not limited in the manner in which the structured light module 100 is placed/secured. It will be appreciated that the structured light module 100 collects visually visible surface point cloud data of the target object, for example, facial skin data, and tissue data in an exposed state during craniotomies, for example. .
The processing module 200 is in communication connection with the structured light module 100 for data interaction. The processing module 200 may be integrated with the aforementioned structured light module 100, or may be an external independent device, such as a notebook computer, a workstation, a PDA, etc., and the present invention is not limited to the form of the processing module 200 and the form of hardware. The processing module 200 can generate a first image corresponding to the point cloud to be projected according to the model point cloud and the point cloud of the target object acquired by the structural light module 100, send the first image to the structural light module 100, and project the first image to the surface of the target object through the projector 120. It uses a scene such as projection of anatomical lines, needle insertion points, tumors, pulmonary vessels, divided brain regions, etc. to a patient, and such as projection of divided brain regions to a teaching model, etc.
According to the operation auxiliary system, the position of the target object can be rapidly and accurately determined through the structure light module, the processing module efficiently and accurately generates a first image corresponding to the point cloud to be projected according to the collected point cloud and the model point cloud, and the projector of the structure light module accurately projects the first image, so that the projection process does not need to detect whether the projection falling point accords with the expectation or not through the tracking equipment, does not need to adjust the angle of the projector, corrects the projection effect, and improves the accuracy and the projection efficiency of the projection. The system not only can assist related discussion, teaching, operation planning and the like, but also can provide visual, accurate and efficient operation navigation for doctors through point cloud projection.
Based on the above embodiments, in one embodiment, the system further includes a fixing device 300, where the fixing device 300 is a fixed connection structure or a mounting structure matched with a mounting position, and is used to fix the structural light module in a desired position.
Specifically, the fixing device may be a fixed connection structure, one end of the fixed connection structure is connected to the structural light module 100, the other end of the fixed connection structure is connected to or supported by a floor, a wall, a ceiling or other equipment, fig. 3 is a second schematic structural view of a point cloud projection system provided by the present invention, fig. 3 illustrates a state that the structural light module 100 is supported on the floor by the fixing device 300, and at this time, the fixed connection structure is a tripod, a support rod, a folding bracket, or the like, in addition, the structural light module 100 may be suspended at a desired position by the fixed connection structure, for example, a suspension connection structure is provided at a wall, a ceiling, or the like, so that the structural light module 100 is located above a patient bed, further, the structural light module 100 is suspended above a patient bed by using a universal arm, and the structural light module 100 is connected by a ball head, so as to achieve adjustment and fixing of the position of the structural light module 100, and the fixing device may be a mounting structure that is matched with a mounting position on the patient bed, for example.
In the embodiment, the structural light module is fixed at a required position through the fixing device, so that the surgical auxiliary system is constructed with simple structure and low cost, and the use cost is reduced. And the structural light module has the dual functions of point cloud acquisition and point cloud projection, and the whole system is simple in structure, low in cost and favorable for market popularization.
Based on any of the above embodiments, in one embodiment, the system includes a plurality of sets of the fixed structure and the structured light modules to achieve multi-angle point cloud projection.
Based on any of the foregoing embodiments, in one embodiment, the system further includes a mechanical arm, where the mechanical arm is communicatively connected to the processing module, and is configured to drive the structured light module to move to a desired position according to a control instruction of the processing module.
Specifically, fig. 4 is a third schematic structural diagram of a point cloud projection system provided by the present invention, where the structural light module 100 is installed or integrated on the mechanical arm 400, the processing module 200 controls the mechanical arm 400 to drive the structural light module 100 to collect point clouds of a target object at one or more positions, and the processing module 200 registers the point clouds of one position or the fused point clouds of a plurality of positions under the mechanical arm coordinate system with the model point clouds to obtain a conversion relationship between the mechanical arm coordinate system and the model point cloud coordinate system (i.e. to realize patient registration). The processing module 200 controls the mechanical arm 400 to drive the structured light module 100 to move to a required projection pose, and the projection pose of the mechanical arm 400 corresponds to the conversion relation between the mechanical arm coordinate system and the structured light coordinate system. According to the two conversions, the structured light module 100 and the model point cloud can be converted into the same coordinate system, for example, into the mechanical arm coordinate system, the model point cloud coordinate system and the sickbed coordinate system, and for example, into the structured light coordinate system under the projection posture, and the first image is generated in the same coordinate system. Of course, the processing module 200 may also control the mechanical arm to perform point cloud collection and projection at the same position, and may directly perform registration processing during data processing, so that the structured light module 100 and the model point cloud are converted into the same coordinate system, for example, the structured light coordinate system and the model point cloud coordinate system, without passing through the mechanical arm coordinate system.
In this embodiment, only one patient registration is required by the mechanical arm, and the projection gesture of the mechanical arm can be adjusted flexibly according to the instruction according to the user operation or input, and projection is performed according to the current projection gesture of the mechanical arm, so that the projection position and angle are more diversified, and the data processing (projection) efficiency is higher.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (13)
1. A projection method, comprising:
registering, namely unifying the structured light module and the model point cloud to the same coordinate system; the structural light module comprises a camera and a projector, and the model point cloud comprises a point cloud to be projected;
generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters under the same coordinate system;
the first image is projected by the projector onto a target object surface.
2. The projection method of claim 1, wherein the registering, unifying the structured light module and the model point cloud to the same coordinate system comprises:
acquiring a point cloud of a target object through the structural optical module;
registering the point cloud of the target object with the model point cloud to obtain a conversion relation between a structured light coordinate system and a model point cloud coordinate system;
and unifying the structured light module and the model point cloud to the same coordinate system according to the conversion relation between the structured light coordinate system and the model point cloud coordinate system.
3. The projection method of claim 1, wherein the registering, unifying the structured light module and the model point cloud to the same coordinate system comprises:
the mechanical arm is controlled to drive the structured light module to register a patient, and the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system is obtained;
and unifying the structured light module and the model point cloud to the same coordinate system according to the conversion relation between the mechanical arm coordinate system and the model point cloud coordinate system and the current mechanical arm projection gesture.
4. The projection method according to claim 1, wherein the generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters in the same coordinate system comprises:
generating a third image according to the point cloud to be projected and the position of the projector;
performing distortion transformation on the third image to obtain a second image;
and processing the second image according to the internal parameters of the projector to obtain the first image corresponding to the point cloud to be projected.
5. The projection method of claim 4, wherein the generating a third image from the point cloud to be projected and the position of the projector comprises:
converting the point cloud to be projected into a projector coordinate system;
and carrying out normalization operation on the point cloud to be projected under the projector coordinate system to obtain the third image.
6. The projection method of claim 4, wherein the distortion transforming the third image results in a second image according to the following formula:
x′＝x(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 xy+p 2 (r 2 +2x 2 )；
y′ ＝ y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 2 xy+p 1 (r 2 +2y 2 )；
wherein x and y are point coordinates in the third image, x ', y' are point coordinates in the second image obtained by distortion transformation, and r 2 ＝x 2 +y 2 ；k 1 、k 2 、k 3 For radial distortion parameter, p 1 、p 2 The tangential distortion parameters are obtained by pre-calibration.
7. The projection method according to claim 4, wherein the processing the second image according to the internal reference of the projector to obtain the first image corresponding to the point cloud to be projected includes:
and processing the second image according to the internal parameters of the projector to obtain a two-dimensional image under a projection image plane, and obtaining the first image according to projection resolution.
8. The projection method according to claim 1, wherein the generating a first image corresponding to the point cloud to be projected according to the model point cloud and projector parameters in the same coordinate system comprises:
when the point is required to be projected, highlighting the point required to be projected in the first image; and/or the number of the groups of groups,
when the outline is required to be projected, connecting peripheral points in the first image to form a closed outline; and/or the number of the groups of groups,
and when the area is required to be projected, carrying out morphological processing on the first image.
9. A processing module comprising a memory, a processor, the processor loading a computer program stored in the memory to implement the projection method of any of claims 1-8.
10. A surgical assistance system, comprising: a structured light module as claimed in claim 9;
the structural light module comprises a camera and a projector, and is used for collecting point clouds of a target object;
the processing module is in communication connection with the structured light module and is used for generating a first image corresponding to the point cloud to be projected according to the model point cloud and the point cloud of the target object acquired by the structured light module, and projecting the first image to the surface of the target object through the projector; the model point cloud comprises the point cloud to be projected.
11. The surgical assist system of claim 10 further comprising a securing means, either a fixed connection or a mounting structure mated with a mounting location, for securing the structured light module in a desired position.
12. The surgical assist system of claim 11 wherein system comprises a plurality of sets of the fixed structure and the structured light modules to achieve multi-angle point cloud projection.
13. The surgical assist system of claim 10 further comprising a robotic arm on which the structured light module is mounted or integrated, the robotic arm being communicatively coupled to the processing module for moving the structured light module to a desired position in response to control commands from the processing module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310778887.6A CN116828160A (en) | 2023-06-29 | 2023-06-29 | Projection method and operation auxiliary system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310778887.6A CN116828160A (en) | 2023-06-29 | 2023-06-29 | Projection method and operation auxiliary system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116828160A true CN116828160A (en) | 2023-09-29 |
Family
ID=88127113
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310778887.6A Pending CN116828160A (en) | 2023-06-29 | 2023-06-29 | Projection method and operation auxiliary system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116828160A (en) |
-
2023
- 2023-06-29 CN CN202310778887.6A patent/CN116828160A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114711969B (en) | Surgical robot system and application method thereof | |
| JP6367905B2 (en) | Surgical robot system for stereotactic surgery and control method for stereotactic robot | |
| JP4822634B2 (en) | A method for obtaining coordinate transformation for guidance of an object | |
| US11944390B2 (en) | Systems and methods for performing intraoperative guidance | |
| US11416995B2 (en) | Systems, devices, and methods for contactless patient registration for a medical procedure | |
| JP2019030702A (en) | Surgical robot for stereotactic surgery and control method of surgical robot for stereotactic surgery | |
| US20180085175A1 (en) | Determining a configuration of a medical robotic arm | |
| CN101375805A (en) | Method and system for guiding operation of electronic endoscope by auxiliary computer | |
| WO2016095168A1 (en) | Tablet computer-based body data visualization method for surgical navigation | |
| US20240164848A1 (en) | System and Method for Registration Between Coordinate Systems and Navigation | |
| US20120004541A1 (en) | Surgery assistance system | |
| CN113143463B (en) | Operation navigation device, system, calibration method, medium and electronic equipment | |
| US20230410331A1 (en) | Imaging system | |
| CN116823898A (en) | Mixed reality augmented display method and system | |
| CN209826968U (en) | Surgical robot system | |
| Haliburton et al. | A visual odometry base-tracking system for intraoperative C-arm guidance | |
| CN114073579A (en) | Operation navigation method, device, electronic equipment and storage medium | |
| CN116828160A (en) | Projection method and operation auxiliary system | |
| CN114191078B (en) | Endoscope operation navigation robot system based on mixed reality | |
| CN114587592B (en) | Surgical navigation system and use method thereof | |
| CN114041876A (en) | Augmented reality orthopedic perspective navigation method and system based on structured light | |
| CN110368026B (en) | Operation auxiliary device and system | |
| CN116824054A (en) | Medical information projection system and method | |
| CN116823897A (en) | Mixed reality enhanced display system and method | |
| CN116807617A (en) | Surgical robot system and surgical navigation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |