TWI839311B - Three-dimensional real-time positioning compensation method - Google Patents

Abstract

A three-dimensional real-time positioning compensation method comprises to obtain an initial pose of several marking devices in a surgical image at world coordinates system respectively. A brightness normalization is performed on the surgical image to produce a normalized image. Several texture images with different resolution are generated for all marks of the marking devices, and the texture image with the closest resolution is taken as a standard image. At least N sampling points are selected from the standard image and the corresponding reference points are obtained in the normalized image. At least N optimization points with the minimum brightness error with the at least N reference points are obtained and an error value is calculated respectively. According to these error values, the initial attitude is modified to produce a compensating attitude.

Description

Compensation method for real-time 3D positioning during surgery

The present invention is mainly a three-dimensional real-time positioning compensation method for surgery, and in particular, a three-dimensional real-time positioning compensation method for surgery that increases the number of sampling points and corrects the trackball posture using image brightness and actual brightness.

In recent years, the development of computer-assisted positioning technology has enabled medical personnel to use imaging equipment to obtain a two-dimensional image of the patient's lesion location during precision surgical operations such as orthopedics or spine surgery, and then use a computer to reconstruct a three-dimensional image of the lesion based on the two-dimensional image and coordinate the lesion. Under the guidance of the computer, medical personnel can accurately implant the implant in the correct position, which has the advantage of greatly improving the accuracy of surgical positioning.

Patent Publication No. I708591 of the Republic of China discloses a three-dimensional real-time positioning method for orthopedic surgery, which fixes a three-dimensional marker device on the surgical site and sets up a camera to shoot a photographic image. Subsequently, the photographic image is identified to obtain the two-dimensional corner point position of the three-dimensional marker device. However, the photographic image itself is composed of pixels, which appear granular under microscopic conditions, and image gradients will be generated at the boundaries. Even if the three-dimensional marker device in the photographic image is in a static state, its two-dimensional corner point will still produce an error of ±3 pixels, resulting in a large error in calculating the posture of the three-dimensional marker device, and has a problem of poor accuracy.

In view of this, it is necessary to provide a three-dimensional real-time positioning compensation method for surgery to solve the above problems.

The purpose of the present invention is to provide a three-dimensional real-time positioning compensation method for surgery, which can correct the trackball posture by increasing the number of sampling points and using image brightness and actual brightness.

The "mechanical corner point" mentioned in the present invention refers to a regular polyhedron of a marking device having a central origin, and the actual coordinates of the four corners of each marked graphic on the regular polyhedron relative to the central origin.

To achieve the above-mentioned purpose, the present invention provides a method for real-time three-dimensional positioning compensation during surgery, comprising: obtaining a surgical image, wherein the image comprises a plurality of marking devices, wherein the marking device comprises a regular polyhedron, wherein the regular polyhedron comprises at least four geometrical figures, wherein the geometrical figure comprises a marker, wherein the marker comprises a border frame and a figure, wherein the figure is located inside the border frame and is used for identification to obtain a unique identification code; inputting the surgical image into an object detection model to detect a first border frame information of each of the plurality of marking devices, and a first identification code of each of the marking devices; The method comprises: obtaining a second bounding frame information of a bounding frame possessed by the plurality of marking devices, an identification code represented by a graphic possessed by each of the marking devices, and a third bounding frame information; obtaining four corner points of the corresponding bounding frame according to the second bounding frame information, performing a projection transformation calculation according to the first bounding frame information of each of the plurality of marking devices and the four corner points of the bounding frame possessed by the plurality of marking devices, and matching the corresponding mechanical corner points, so as to obtain an initial posture of each of the plurality of marking devices in the world coordinate system; performing a brightness normalization on the surgical image to generate a normalized image; performing a projection transformation calculation on the plurality of marking devices ... Each of the marks generates a plurality of texture images of equal length and width but different resolutions; a fuzziness calculation is performed on each of the texture images with the same identification code according to the identification code represented by the graphics of each of the marking devices, and the texture image with the closest resolution is used as a standard image; at least N sampling points other than the four corner points are selected from the border frame or graphics of the standard image according to the second bounding frame information and the third bounding frame information, and the projection transformation calculation is performed on the at least N sampling points in combination with the corresponding mechanical corner points, Then, a camera parameter is used to calculate to obtain a reference point corresponding to each of the at least N sampling points in the normalized image; an optimal point is obtained within an allowable range with the reference point as the origin, and the optimal point has a minimum brightness error with the reference point; an error value between each reference point and the corresponding optimal point is calculated by a minimizing error function, and then the initial posture is corrected according to the error value to generate a compensation posture; and the compensation postures of each of the several marking devices in the world coordinate system are drawn and displayed in real time on a display screen.

In some embodiments, 100≦N≦300.

In some embodiments, the tolerance range is ±1 pixel.

In some embodiments, the object detection model is YoLov5.

The three-dimensional real-time positioning compensation method for surgery of the present invention has the following characteristics: by increasing the number of sampling points on the border or figure of the mark, and using the minimum error between the image brightness of the sampling point and the actual brightness to correct the posture of each of the several marking devices in the world coordinate system. Thus, the three-dimensional real-time positioning compensation method for surgery of the present invention, under the accuracy verification using ASTM2554, the average error of the marking device is reduced from 0.8547 to 0.1493, which can achieve the effect of improving and stabilizing the positioning accuracy of the marking device.

[The present invention]

1: Marking device

11: Regular polyhedron

11a: Geometric drawing

12: Nail-like body

13: Logo

13a: Border

13b: Graphics

2: Camera

3: Computer host

4: Display screen

P: Sampling point

S: Spine

S1: Image acquisition step

S2: Object detection step

S3: Posture acquisition step

S4: Brightness normalization step

S5: Image generation step

S6: Fuzzy comparison step

S7: Sampling step

S8: Error minimization step

S9: Posture compensation step

[Figure 1] is a step flow chart of the three-dimensional real-time positioning compensation method for surgery of the present invention; [Figure 2] is a three-dimensional diagram of the marking device of the three-dimensional real-time positioning compensation method for surgery of the present invention; [Figure 3] is a diagram of the use of the three-dimensional real-time positioning compensation method for surgery of the present invention; [Figure 4] is a schematic diagram of the sampling points of the three-dimensional real-time positioning compensation method for surgery of the present invention.

The embodiments of the present invention are described in detail with the help of the drawings. The attached drawings are mainly simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner. Therefore, only the components related to the present invention are marked in the drawings, and the components shown are not drawn in the number, shape, size ratio, etc. during implementation. The actual specifications and dimensions during implementation are actually a selective design, and the layout of the components may be more complicated.

The following descriptions of the embodiments refer to the attached drawings to illustrate specific embodiments according to which the present invention can be implemented. The directional terms mentioned in the present invention, such as "upper", "lower", "front", "back", etc., are only with reference to the directions of the attached drawings. Therefore, the directional terms used are used to illustrate and understand the present application, rather than to limit the present application. In addition, in the specification, unless explicitly described to the contrary, the word "comprising" will be understood to mean including the elements described, but not excluding any other elements.

Please refer to FIG1, which is a flow chart of the steps of the three-dimensional real-time positioning compensation method for surgery of the present invention, which includes the following steps: Please also refer to FIG2, image acquisition step S1: obtain a surgical image, the image includes a plurality of marking devices 1, in this embodiment, the marking device 1 is a tracking ball, and a rigid dodecahedron ball is used. Specifically, the marking device 1 has a regular polyhedron 11 and a nail body 12, the regular polyhedron 11 has at least four geometric figures 11a showing regular pentagons, the geometric figure 11a has a mark 13, the mark 13 is composed of a frame 13a and a figure 13b, wherein the figure 13b is located inside the frame 13a, and is used for identification to obtain a unique identification code. For example, the mark 13 can be one of an AR-ToolKit mark, an ARTag mark, an ArUco mark or an AlVar mark. In this embodiment, the AlVar mark is used as the mark 13 on the geometric drawing 11a. The nail-shaped body 12 is used to fix the marking device 1 to at least one surgical site of a surgical instrument or a patient/teaching aid.

Please refer to FIG3 . When the surgical site is the spine S, the nail-shaped body 12 is a spinal nail, which is within the technical field of the present invention and can be understood by those with ordinary knowledge. It is worth noting that the plurality of marking devices 1 do not have the same identification code pattern 13b. For example, the identification code represented by the pattern 13b of the mark 13 of the marking device 1 of the surgical instrument can be 0 to 8, and the identification codes represented by the patterns 13b of the marks 13 of the three marking devices 1 of the spine can be 9 to 17, 18 to 26, and 27 to 35, respectively. It is worth mentioning that when the marking device 1 is inserted into the surgical site, the three geometric figures 11a face downwards towards the surgical site and cannot be identified. Therefore, the identification code represented by the figure 13b of the mark 13 can be set to 9 mark numbers.

Based on the above, in the present invention, at least one camera 2 is set up and shoots toward the plurality of marking devices 1 to generate the surgical image. In this embodiment, the camera 2 is a camera with high-definition function and six degrees of freedom (DOF) motion posture.

Object detection step S2: Input the surgical image into an object detection model in a computer host 3 to detect a first bounding box information of each of the plurality of marking devices 1, a second bounding box information of the border 13a of each of the marking devices 1, and an identification code represented by a graphic 13b of each of the marking devices 1 and a third bounding box information. In this embodiment, the identification code represented by the graphic 13b of at least three markings 13 of the marking device 1 can be identified, which has the effect of improving positioning accuracy.

In the present invention, the object detection model can be a deep learning model with parameter settings downloaded from the Internet to the computer host 3. The computer host 3 trains the deep learning model based on several surgical images previously taken in the C++ development environment to obtain the object detection model. In this embodiment, YoLov5 is used for illustration, which belongs to the common knowledge in the technical field of the present invention and will not be elaborated here.

Posture acquisition step S3: Obtain the four corner points of the corresponding border frame 13a according to the second bounding frame information. Subsequently, perform projection transformation calculation according to the first bounding frame information of each of the plurality of marking devices 1 and the four corner points of the border frame 13a they possess, and match the corresponding mechanical corner points to obtain an initial posture of each of the plurality of marking devices 1 in the world coordinate system. In this embodiment, the computer host 3 can complete the calculation by calling the solvepnp() function in the OpenCV computer vision library, which belongs to the common knowledge in the relevant field of the present invention and will not be elaborated here.

Brightness normalization step S4: Perform brightness normalization on the surgical image to generate a normalized image. For example, the standard image can be subjected to a brightness histogram warp using a contrast stretching method and then evenly stretched to a brightness range of 0 to 255.

Image generation step S5: Generate several texture images with the same length and width ratio but different resolutions for all the marks 13 of the several marking devices 1. In this embodiment, the original resolution of the surgical image taken by the camera 2 is 640*640, and the computer host 3 performs image processing methods such as reduction, enlargement and blurring on the surgical image to generate several texture images with resolutions of 320*320, 160*160, 80*80, 40*40, 20*20 and 10*10.

Fuzzy comparison step S6: According to the identification code represented by the graphic 13b of each marking device 1, a fuzziness calculation is performed on several texture images with the same identification code, and the texture image with the closest resolution is used as a standard image. In this embodiment, the fuzziness calculation is performed using an energy gradient function.

Please refer to FIG. 4 , sampling step S7: According to the second bounding box information and the third bounding box information, select at least N sampling points P other than the four corner points from the border frame 13a or the figure 13b of the standard image, in this embodiment, 100≦N≦300; perform projection transformation calculation on the at least N sampling points P in combination with the corresponding mechanical corner points, and then perform calculation through the camera parameters of the camera 2 to obtain a reference point corresponding to each of the at least N sampling points P in the normalized image. Among them, the camera parameters of the camera 2 belong to the common knowledge in the technical field of the present invention.

Error minimization step S8: Take the reference point as the origin and obtain an optimal point within an allowable range, where the optimal point has the minimum brightness error with the reference point. The allowable range can be ±1 pixel, and the formula for the brightness error can be as shown below, which belongs to the common knowledge in the technical field of the present invention.

Subsequently, an error value between each reference point and the corresponding optimization point is calculated by a minimization error function, and then the initial posture is corrected according to the error value to generate a compensation posture. The minimization error function can be shown as follows, which belongs to the common knowledge in the technical field of the present invention.

Posture compensation step S9: The compensation postures of the multiple marking devices 1 in the world coordinate system are drawn and displayed in real time on a display screen 4.

As mentioned above, the surgical three-dimensional real-time positioning compensation method of the present invention increases the number of sampling points on the border or figure of the mark, and uses the minimum error between the image brightness of the sampling point and the actual brightness to correct the posture of each of the several marking devices in the world coordinate system. In this way, the surgical three-dimensional real-time positioning compensation method of the present invention, under the accuracy verification using ASTM2554, the average error of the marking device is reduced from 0.8547 to 0.1493, which can achieve the effect of improving and stabilizing the positioning accuracy of the marking device.

The above disclosed implementation forms are only exemplary of the principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone familiar with this technology can modify and change the above implementation forms without violating the spirit and scope of the present invention. Any equivalent changes and modifications completed by using the content disclosed in the present invention should still be covered by the scope of the patent application below.

S1: Image acquisition step

S2: Object detection step

S3: Posture acquisition

S4: Brightness normalization step

S5: Image generation

S6: Fuzzy comparison step

S7: Sampling step

S8: Error minimization step

S9: Posture compensation step

Claims (4)

A method for real-time three-dimensional positioning compensation during surgery, comprising: Obtaining a surgical image, wherein the image comprises a plurality of marking devices, wherein the marking device comprises a regular polyhedron, wherein the regular polyhedron comprises at least four geometric figures, wherein the geometric figure comprises a mark, wherein the mark comprises a border frame and a figure, wherein the figure is located inside the border frame and is used for identification to obtain a unique identification code; Inputting the surgical image into an object detection model to detect a first border frame information of each of the plurality of marking devices, a second border frame information of the border frame of each of the marking devices, and an identification code represented by the figure of each of the marking devices and a third border frame information; According to the second bounding box information, the four corner points of the corresponding bounding box are obtained. According to the first bounding box information of each of the plurality of marking devices and the four corner points of the bounding box, projection transformation calculation is performed in combination with the corresponding mechanical corner points to obtain an initial posture of each of the plurality of marking devices in the world coordinate system; Perform a brightness normalization on the surgical image to generate a normalized image; Generate a plurality of texture images with equal length and width ratios but different resolutions for all the markers of the plurality of marking devices; According to the identification code represented by the graphics of each of the marking devices, a blur calculation is performed on the plurality of texture images with the same identification code, and the texture image with the closest resolution is used as a standard image; According to the second bounding box information and the third bounding box information, at least N sampling points other than the four corner points are selected from the border or figure of the standard image, and the at least N sampling points are matched with the corresponding mechanical corner points to perform projection transformation calculation, and then a camera parameter is used to perform operation to obtain a reference point corresponding to each of the at least N sampling points in the normalized image; Taking the reference point as the origin, an optimal point is obtained within an allowable range, and there is a minimum brightness error between the optimal point and the reference point. An error value between each reference point and the corresponding optimal point is calculated through a minimization error function, and then the initial posture is corrected according to the error value to generate a compensation posture; and The compensation postures of the multiple marking devices in the world coordinate system are drawn and displayed in real time on a display screen. The three-dimensional real-time positioning compensation method for surgery as described in claim 1, wherein 100≦N≦300. A surgical three-dimensional real-time positioning compensation method as described in claim 1, wherein the allowable range is ±1 pixel. A three-dimensional real-time positioning compensation method for surgery as described in claim 1, wherein the object detection model is YoLov5.