WO2023168682A1

WO2023168682A1 - Method and device for determining pose of surgical instrument in space

Info

Publication number: WO2023168682A1
Application number: PCT/CN2022/080286
Authority: WO
Inventors: 蒋俊锋; 陈亮; 赵晓萌
Original assignee: 常州锦瑟医疗信息科技有限公司
Priority date: 2022-03-11
Filing date: 2022-03-11
Publication date: 2023-09-14

Abstract

The present application relates to a method for determining a pose of a surgical instrument in a space, comprising: providing a target instrument image of a surgical instrument; extracting a relative position image of at least one feature through hole and a through hole image of the at least one feature through hole from the target instrument image; adjusting an orientation of a digital model of the surgical instrument relative to an imaging plane for multiple times, and performing simulation imaging on the digital model of the surgical instrument after each adjustment, so as to generate a plurality of simulated instrument images of the surgical instrument corresponding to the orientation; processing the through hole image and the relative position image of the at least one feature through hole on the basis of the plurality of simulated instrument images, so as to determine an orientation of the surgical instrument relative to the imaging plane during imaging of the target instrument image; and determining a pose of the surgical instrument in a space on the basis of the orientation of the surgical instrument relative to the imaging plane.

Description

Method and device for determining the posture of surgical instruments in space

Technical field

The present invention relates to the technical field of medical instruments, and in particular to a method and device for determining the posture of surgical instruments in space.

Background technique

Orthopedic surgical instruments are widely used in fracture treatment. However, for surgical instruments such as interlocking intramedullary nails, since they will be accommodated in the bone cavity during the operation, it is difficult to accurately position them, so they need to be continuously X-rayed and imaged. This excessive X-ray exposure can significantly affect the health of doctors and patients.

Therefore, it is necessary to provide an improved method and device for spatially orienting surgical instruments.

Contents of the invention

An object of the present application is to provide a method and device for determining the posture of a surgical instrument in space to avoid continuous X-ray imaging of the surgical instrument during surgery.

In one aspect of the present application, a method for determining the posture of a surgical instrument in space is provided, characterized in that the surgical instrument is formed of an X-ray imageable material, the surgical instrument has a predetermined instrument outer contour, and One or more characteristic through holes, wherein each characteristic through hole has a predetermined through hole profile, the method includes: providing a target instrument image of the surgical instrument, wherein the target instrument image includes an instrument outer contour of the surgical instrument At least part of the outer contour image and the through-hole image of at least one characteristic through-hole; extracting the relative position image of the at least one characteristic through-hole and the through-hole image of the at least one characteristic through-hole from the target device image, wherein the relative position image reflects the position of the through hole image of the at least one characteristic through hole relative to the at least part of the outer contour image; and the orientation of the digital model of the surgical instrument relative to the imaging plane is adjusted multiple times, and perform simulated imaging on the digital model of the surgical instrument after each adjustment to generate multiple simulated instrument images of the surgical instrument corresponding to the orientation; process the at least one simulated instrument image based on the multiple simulated instrument images The through hole image and the relative position image of a characteristic through hole are used to determine the orientation of the surgical instrument relative to the imaging plane when the target instrument image is imaged; and based on the relative position of the surgical instrument relative to the imaging plane The orientation determines the posture of the surgical instrument in space.

The above is an overview of the present application, and there may be situations where simplifications, generalizations, and details are omitted. Therefore, those skilled in the art should realize that this part is only illustrative and is not intended to limit the scope of the present application in any way. This Summary is neither intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Description of the drawings

The above and other features of the present application will be more fully understood from the following description and appended claims, taken in conjunction with the accompanying drawings. It will be understood that these drawings only depict several embodiments of the present application and therefore should not be considered as limiting the scope of the present application. By using the accompanying drawings, the contents of this application will be explained more clearly and in detail.

Figure 1a shows a schematic diagram of an instrument image of a surgical instrument 10 according to an embodiment of the present application;

Figure 1b shows a perspective view of the surgical instrument 10 shown in Figure 1a;

Figure 2 illustrates a method 200 for determining the posture of a surgical instrument in space according to one embodiment of the present application;

Figures 3a and 3b show the front and back of an exemplary physical calibration plate; Figures 3c and 3d show the front and back images of the calibration plate that is exposed and imaged together with the surgical instrument;

Figures 4a to 4d show schematic diagrams of relative position images and through-hole images extracted from target device images; Figure 4e shows an example of device images;

Figures 5a and 5b illustrate a method for filtering image data according to an embodiment of the present application;

Figures 6a and 6b show schematic diagrams of changes in the instrument image of the surgical instrument with the posture of the surgical instrument in space;

Figures 7a to 7d illustrate the process of further modifying the posture of the surgical instrument according to an embodiment of the present application.

Detailed ways

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter of this application. It will be understood that various configurations, substitutions, combinations, and designs of various aspects of the present application generally described in the present application and illustrated in the accompanying drawings are possible, and all of these are expressly constitutive of the present application. part.

Generally speaking, orthopedic surgical instruments such as intramedullary nails, compression locking bone plates or other surgical instruments (such as stents, endoscopes, etc.) need to be placed inside the patient's body to perform surgical operations. However, due to the obstruction of tissues in the patient's body cavity, surgical instruments may not be completely visible to doctors and other surgical operators, which affects the precise positioning of surgical instruments. Therefore, X-ray imaging devices such as C-arm X-ray machines are used to perform intraoperative imaging of surgical instruments to allow the operator to understand the orientation and position of the surgical instruments in the patient's body by observing two-dimensional images of the instruments.

Some existing image positioning solutions require taking two or more instrument images of surgical instruments (hereinafter referred to as target instrument images) during the operation, and positioning the surgical instruments based on these target instrument images. This method is also known as It is a bi-plane method or a multi-plane method. The biplane method increases the number of irradiations for the patient during one operation, and the operation is also more complicated. In order to solve this problem, the inventor of the present application has designed a method of spatially orienting surgical instruments using a single target instrument image, which effectively reduces the number of irradiations of the patient during the operation, and the implementation method is relatively simple. .

Figure 1a shows a schematic diagram of an instrument image of a surgical instrument 10 according to an embodiment of the present application. In the example shown in Figure 1a, the surgical instrument 10 is an interlocking intramedullary nail having an elongated configuration and having a predetermined outer instrument profile. For example, one end 14 of the interlocking intramedullary nail in Figure 1a has a generally trapezoidal cross-section (see the perspective view of the surgical instrument shown in Figure 1b, the trapezoidal cross-section is that of the conical tip), and in The side of the intramedullary nail near this end has a concave incision. It should be noted that the target instrument image shown in FIG. 1a only includes a part of the image of the outer contour of the surgical instrument 10, but does not include the complete image of the outer contour. In practical applications, as long as a part of the image of the outer contour can reflect the general shape or morphological trend of the surgical instrument, a complete image of the surgical instrument is not required.

In addition to the outer contour of the surgical instrument, the surgical instrument shown in FIGS. 1 a and 1 b also includes two characteristic through holes of the interlocking intramedullary nail, namely, the circular through hole 16 and the oblong through hole 18 . These through

holes

16 and 18 have relatively regular shapes. When the position and/or orientation of the surgical instrument relative to the imaging plane changes, the shape of the through holes on the image of the target instrument obtained by imaging will also change accordingly. Figures 6a and 6b are schematic diagrams showing changes in the instrument image of the surgical instrument as a function of the posture of the surgical instrument in space. Among them, in Figure 6a, the surgical instrument is located in front and right of the X-ray source, so the instrument image of the surgical instrument is biased to the right front at this time; in Figure 6b, the surgical instrument is located in front and left of the The instrument image is biased to the left front. As the position of the device image changes, the through-hole image of the characteristic through-hole in the device image will also change accordingly. It can be understood that Figures 6a and 6b only schematically illustrate the image changes caused by changes in the spatial posture of the surgical instrument. In actual applications, such changes can take on more or more complex forms.

Generally speaking, the image changes of regular-shaped through holes with changes in orientation are relatively regular and are more suitable as characteristic through holes, so they can be used to identify the posture of surgical instruments in space in subsequent processing processes. The via-based pose or orientation recognition will be explained in detail below. It can be understood that in some examples, the characteristic through hole may have other suitable shapes, such as a through hole with a regular polygonal cross-section, an elliptical through hole, a pressure locking hole (having two circular holes that partially overlap or a circular hole and a oval hole), etc. It should be noted that these characteristic through holes penetrate at least a certain area of the surgical instrument, so that when the surgical instrument is imaged by the X-ray imaging device, it appears different from the non-through hole area of the surgical instrument (that is, the X-ray opaque area). image. In the image of the instrument shown in Figure 1a, through-

holes

16 and 18 are shown as exposed areas (light gray) because they allow transmission of X-rays, while non-through-hole areas of surgical instrument 10 (formed from X-ray imageable material, Materials such as stainless steel) are shown as unexposed areas (black).

Figure 2 illustrates a method 200 for determining the posture of a surgical instrument in space according to one embodiment of the present application. This method can be used to process target instrument images such as the surgical instrument shown in Figure 1a to determine the posture of the surgical instrument in space. In the following, the method 200 shown in FIG. 2 will be further described in conjunction with the surgical instruments shown in FIGS. 1 a and 1 b.

As shown in Figure 2, the method 200 begins with step 202 of providing a target instrument image of the surgical instrument. Wherein, the target instrument image includes at least a part of the outer contour image of the instrument outer contour of the surgical instrument, and a through-hole image of at least one characteristic through-hole. For example, the target instrument image may include the outer contour of the instrument as shown in FIG. 1 a and the through hole image of at least one of the circular through hole 16 or the oblong through hole 18 . In some preferred embodiments, the target instrument image to be processed may include images of all characteristic through holes of the surgical instrument.

It can be understood that the target instrument image of the surgical instrument provided in step 202 is the actual image of the surgical instrument and part of the patient's body area at a certain moment during the operation. The X-ray imaging device may record or determine the position/orientation of the imaging plane or other similar datum plane or reference plane of the X-ray imaging device during imaging, such as recording the orientation parameters of the imaging plane or the X-ray source. In this way, with the help of the datum plane, the posture of the surgical instrument relative to a certain reference coordinate system (eg, the operating table) can be determined. The imaging plane is, for example, a focal plane with the X-ray irradiation direction as the normal line. In some embodiments, the orientation and position of the imaging plane can be adjusted and determined by the X-ray imaging device, which can be an equipment parameter of the X-ray imaging device. In some embodiments, the X-ray imaging device may be a C-arm X-ray machine.

In practical applications, the target instrument image of the surgical instrument provided in step 202 may have image distortion caused by exposure of the X-ray imaging device or other reasons. In some examples, in order to avoid the impact of image distortion on the subsequent gesture recognition of the surgical instrument, the image distortion of the instrument image can be corrected after step 202. For example, a calibration plate can be used to collect distortion parameters during image imaging of the target device. Figures 3a and 3b show the front and back of an exemplary physical calibration plate, while Figures 3c and 3d show the front and back images of the calibration plate being exposed and imaged together with the surgical instrument. In this way, by using the front image or back image of the calibration plate and the corresponding physical image, combined with the exposure parameters of the X-ray imaging device (such as focal length, pixel spacing of the imaging plane, etc.), the image distortion in the target instrument image of the surgical instrument can be corrected Make corrections. Those skilled in the art can understand that any suitable correction algorithm can be used for image distortion correction, which will not be described again here.

Next, in step 204, the relative position image of the characteristic through-hole and the through-hole image of the characteristic through-hole are extracted from the target instrument image of the surgical instrument. Wherein, the relative position image of the characteristic through hole reflects the position of the through hole image of the characteristic through hole relative to at least a part of the outer contour image.

Referring to Figures 4a to 4d, schematic diagrams of relative position images and through-hole images extracted from target device images are shown. Among them, Figure 4a is a schematic diagram of the target instrument image of an unprocessed surgical instrument, and Figure 4b is a part of the area of Figure 4a. This area includes a part of the instrument outer contour image and the through-hole image of two characteristic through-holes, that is, Relative position image. Figure 4c is a through-hole image of an oblong feature through-hole, and Figure 4d is a through-hole image of a circular feature through-hole. In some embodiments, the operation of extracting Fig. 4b from Fig. 4a may include the process of intercepting the area of interest, and the process of extracting contour information through image edge recognition processing; in other embodiments, extracting Fig. 4b from Fig. 4a The operation may only include the process of intercepting the region of interest, but not the process of extracting contour information through edge recognition (for example, an example of an actual instrument image as shown in Figure 4e). In some embodiments, the through hole images shown in Figure 4c and Figure 4d can be obtained by directly processing the instrument image of Figure 4a, or can also be obtained by processing the relative position image of Figure 4b.

As mentioned above, characteristic through holes usually have a relatively regular shape, such as a circular or oval shape, and the shape of the through hole image formed after changing with the actual spatial orientation of the surgical instrument is also relatively regular, so it is easy to identify. Therefore, in subsequent processing, images of characteristic through holes are often mainly used to determine the orientation of surgical instruments. However, it can be understood that this application does not limit the shape of the characteristic through holes to be regular. Irregular characteristic through holes can also be used for subsequent orientation and spatial posture recognition.

In addition, in step 204, the relative position image of the characteristic through hole is also extracted. The relative position image can be used to represent the location of the characteristic through hole in the surgical instrument. In the case of a surgical instrument having a plurality of characteristic through-holes, in particular a plurality of characteristic through-holes of the same shape, these characteristic through-holes change with the orientation of the surgical instrument depending on the deflection angle relative to the X-ray source. There may be different through-hole images; and the relative position image is conducive to identifying specific characteristic through-holes, so that corresponding identification processing can be performed for the specific characteristic through-holes.

Thereafter, in step 206, the orientation of the digital model of the surgical instrument relative to the imaging plane is adjusted multiple times, and simulated imaging is performed on the digital model of the surgical instrument after each adjustment to generate an image of the surgical instrument corresponding to the orientation. Multiple simulated device images.

It can be understood that the target instrument image of the surgical instrument is an image generated by the surgical instrument being oriented in a certain orientation relative to the imaging plane during the actual imaging process. In order to identify the orientation or posture of the surgical instrument, the inventor creatively performed simulated imaging on the three-dimensional digital model of the surgical instrument to generate multiple simulated instrument images, and then used these simulated instrument images to identify the orientation of the actual target instrument image. Among them, the three-dimensional digital model of the surgical instrument is the same digital virtual model as the physical model of the surgical instrument. By repeatedly adjusting the orientation of the three-dimensional digital model of the surgical instrument relative to the imaging plane in the virtual space and performing simulated imaging on it, the simulated instrument images of the surgical instrument in these orientations can be determined.

In some embodiments, depending on the accuracy requirements of the surgical instrument posture recognition and the computing power of the processing device, the size of the digital model orientation of each adjustment of the surgical instrument can be adaptively set/determined. For example, the digital model of the surgical instrument and the virtual light source point can be set in the same virtual digital space, and based on the maximum distance from the digital model of the surgical instrument to the virtual light source point (the maximum distance can be determined based on the maximum distance in the physical imaging space) The virtual digital space is divided into multiple virtual imaging areas, in which the distance between the center point of each imaging area and the virtual light source point increases in sequence. In some embodiments, the virtual digital space may be divided into 3, 4, 5, 6 or more virtual imaging areas. Then, for each virtual digital space, the digital model of the surgical instrument can be set at a certain reference point in it, such as the center point; then the deflection angle of the digital model in different directions can be gradually changed, and the orientation can be changed at the same time. The final digital model is subjected to simulated imaging to generate a simulated instrument image of the digital model at the deflection angle (orientation). It can be understood that in each simulated instrument image, the relative position image of the characteristic through-hole in the digital model of the surgical instrument and the through-hole image are different. In some embodiments, each adjustment may change the deflection angle by 1, 2, 3, 5 or more degrees. Depending on the embodiment, the simulated instrument image of the surgical instrument after each adjustment may or may not be stored.

Next, in step 208, based on the multiple simulated instrument images generated in step 206, the through-hole image and the relative position image of the characteristic through-hole extracted from the target instrument image in step 204 may be processed to determine whether The orientation of the surgical instrument relative to the imaging plane during instrument imaging.

In some embodiments, especially when the simulated instrument image of the surgical instrument is not stored after each orientation adjustment in step 206, each of the multiple simulated instrument images of the surgical instrument can be extracted from step 204. The through-hole images and relative position images are compared for similarity. Since the through-hole images are mainly compared during the similarity comparison process, the amount of calculation is relatively controllable. In practical applications, various commonly used comparison algorithms can be used for similarity comparison, such as algorithms that extract multiple feature points for comparison. This application does not limit this. Preferably, a predetermined similarity threshold can be set, such as 0.95 (assuming the same value is 1), for similarity comparison. In this way, a simulated instrument image including a through-hole image whose similarity to the through-hole image extracted from the target instrument image is higher than the predetermined similarity threshold can be selected, and the orientation corresponding to the simulated instrument image can be used as the surgical instrument when imaging the target instrument image. Orientation relative to the imaging plane. Alternatively, simulated device images below a predetermined similarity threshold can also be excluded. It can be understood that since the similarity threshold is set, adjusting the orientation of the digital model of the surgical instrument relative to the imaging plane in step 206 does not require traversing all orientations, but the simulation can be stopped as long as there is a simulated instrument image with a similarity that meets the requirements. Imaging. In addition, in some preferred embodiments, multiple rounds of adjustment can be performed on the digital model of the surgical instrument, wherein each round of adjustment adopts different orientation adjustment accuracy. For example, the first round of orientation adjustment can set the adjustment accuracy to 10 degrees, and after traversing all orientations, select only a part of the orientations (for example, under these orientations, the similarity of the characteristic through-hole images in the simulated device image is high Perform a second round of orientation adjustment at a certain threshold); the second round of orientation adjustment can set the adjustment accuracy to 2 degrees or less, and then if the simulation device image corresponding to a certain orientation during the orientation adjustment process has a similarity higher than the predetermined If the similarity threshold is the characteristic through-hole image, then the corresponding orientation of the simulated instrument image is determined as the orientation of the surgical instrument relative to the imaging plane when the target instrument image is imaged. It can be understood that the relative position image corresponding to the characteristic through hole can be processed similarly, which will not be described again here.

In other embodiments, neural network model technology may also be used to implement the processing of step 208. For example, the simulated instrument images generated in step 206 may be stored as a predetermined image database, where each simulated instrument image is associated with the orientation (in virtual space) of the digital model of the surgical instrument relative to the imaging plane when it was simulated to be imaged. Afterwards, the predetermined image database can be used to train the neural network model, thereby enabling it to have the ability to recognize and process similar image data. Subsequently, the trained neural network model can be used to process the through hole image and the relative position image of at least one characteristic through hole to determine the orientation of the surgical instrument relative to the imaging plane when the instrument image is imaged. In some embodiments, the neural network model can be any suitable neural network model, such as a convolutional neural network model.

In some other embodiments, the correspondence between the instrument image and the orientation data can also be obtained and stored in various ways as part of the data in the predetermined image database. For example, the images of these surgical instruments may be historical images of surgical instruments taken during previous surgeries, and doctors or surgical operators have determined the orientation data of the surgical instruments relative to the imaging plane through various methods; these historical images are The corresponding orientation data can be stored together as image training data. It can be understood that compared to only using historical imaging data as training data, actively adjusting surgical instruments and performing simulated imaging can obtain more comprehensive and accurate training data, so the processing power of the neural network model obtained through its training is more excellent.

In practical applications, the neural network model used in step 208 can be pre-trained. For example, a large number of historical instrument images and simulated images of surgical instruments can be used to train a neural network model; however, since it relies on a large amount of existing image data, this training method takes a long time and is suitable for training before the surgical procedure. . In some cases, real-time imaging of the target instrument image can also be used to filter the image data in the predetermined image database during the operation, and the filtered image data can be used to train the neural network model. For example, the through-hole image of the characteristic through-hole in the target device image may have specific dimensions, so these dimensions of the through-hole image can be used as parameters and criteria for filtering image data. Figures 5a and 5b illustrate a method for filtering image data according to an embodiment of the present application.

As shown in Figure 5a, the length of the through-hole image along the direction of the imaging boundary of the actual target device image (horizontal and vertical directions, which generally depend on the imaging plane of the X-ray imaging device) is expressed as m ₁ and n ₁ respectively. ; And the lengths of the through-hole images in the same direction in the simulated images or historical images are expressed as m ₂ and n ₂ respectively. In some examples, when the ratio of m ₁ to m ₂ is close to 1 (for example, between 0.8 and 1.25), and when the ratio of n ₁ to n ₂ is close to 1 (for example, between 0.8 and 1.25), It can be considered that the imaging conditions/orientations of simulated images or historical images are relatively close to the actual target device images, so simulated images or historical images that meet this image filtering condition can be selected as training data for the neural network model. It can be understood that since some image data with poor correlation are pre-screened, the size of the image training data set is relatively small without losing training accuracy, so the training time of the neural network model is relatively short.

Similarly, as shown in Figure 5b, along the long axis direction and short axis direction of the through hole image in the actual target device image, the length of the through hole image is represented as a ₁ and b ₁ respectively; while the simulated image and/or history The lengths of the through-hole images in the same direction in the image are represented as a ₂ and b ₂ respectively. In some examples, when the ratio of a ₁ to a ₂ is close to 1 (for example, between 0.8 and 1.25), and when the ratio of b ₁ to b ₂ is close to 1 (for example, between 0.8 and 1.25), It can be considered that the imaging conditions/deflection angles of simulated images or historical images are relatively close to actual images, so simulated images and/or historical images that meet this image filtering condition can be selected as training data for the neural network model. It can be understood that for elliptical images, various existing major axis and minor axis identification methods can be used in the processing method described in this application. For example, the line connecting the two longest points in the image can be identified as the long axis, and the direction orthogonal to it is the short axis.

It can be understood that in some embodiments, in order to avoid real-time training of the neural network model during the operation, the length of the through-hole image of the characteristic through-hole in the simulated image and/or historical image (for example, the length m ₂ and n ₂ , and the lengths a ₂ and b ₂ in Figure 5 b ), the image data including simulated images and/or historical images are divided into multiple sub-training data sets, and the neural network model is trained based on these sub-training data sets respectively. In this way, when using the trained neural network model to process actual images, the neural network model trained on the sub-training data set similar to the target device image can be directly selected without additional intraoperative training steps.

It can be understood that although in the above example image filtering is used to determine the training data in the training data set, in actual applications, this image filtering process can also compare simulated device images with through-hole images and/or relative positions in real time Performed during imaging. In other words, before performing the similarity comparison, the characteristic sizes in certain directions of the through-hole image extracted in step 204 can be compared with the characteristic sizes in the corresponding directions in the simulated device image; and only when the characteristic sizes are close, the Continue to compare the similarity between the simulated device image and the through-hole image and relative position image extracted in step 204.

It should be noted that in some examples, the surgical instrument only has one characteristic through hole; accordingly, the above filtering of the image data set only needs to consider one or more lengths of the through hole image of the characteristic through hole. In some other examples, a surgical instrument may have multiple characteristic through-holes; accordingly, the filtering of the image data set may only consider the length of the through-hole image of one of these characteristic through-holes, or may also consider all features The length of the through-hole image, that is, only if the lengths of all through-hole images in a certain historical image and/or simulated image meet the predetermined image filtering conditions, the historical image or simulated image will be selected as the image training data.

In some embodiments, the neural network model can identify the position of the hole center of the characteristic through hole and the direction of the central axis passing through the hole center, and then determine the orientation of the plane where the through hole is located based on the hole center position and the direction of the central axis. Since the position of the characteristic through hole in the surgical instrument is determined, the orientation of the surgical instrument relative to the imaging plane (as a reference plane) can be determined based on the position and orientation of the plane where the through hole is located.

In some embodiments, the neural network model can identify the location of the characteristic through hole and the direction of a central axis passing through the center of the hole, as well as a perpendicular line from the center of the hole to the outer contour of the device. In other embodiments, the surgical instrument may have multiple characteristic through holes, such as the circular through hole and the oblong through hole shown in Figure 4a. In this way, the neural network model can identify two characteristics. The location of the vias and the direction of the central axis passing through the center of the two featured vias. In this manner, the position of one point and the associated orientation of one or more vectors in the target device image can be determined, or the positions of two or more points and the associated orientation of one or more vectors can be determined. , so that the orientation of the surgical instrument relative to the imaging plane can be determined based on the orientation of these points and axes/lines.

After determining the deflection of the surgical instrument relative to the imaging plane, in step 210 , the posture of the surgical instrument in space may be determined based on the orientation of the surgical instrument relative to the imaging plane.

Specifically, the imaging plane depends on the orientation of the X-ray source of the X-ray imaging device during imaging, which is generally the focal plane of the X-ray source during imaging. Furthermore, by means of the equipment parameters of the X-ray imaging device, the orientation of the imaging plane as the focal plane in a certain reference coordinate system (for example, a reference coordinate system related to the operating room or the X-ray imaging device) can be determined. In this way, with the help of the imaging plane, the posture of the surgical instrument in space can be determined. Among them, the posture of the surgical instrument includes orientation and position.

After determining the posture of the surgical instrument, a three-dimensional model of the surgical instrument can be displayed in a graphical user interface to present the position of the surgical instrument within the patient's body to a doctor or other operator. For example, the patient's body shape can be pre-imaged using a CT machine and a three-dimensional model can be generated. In this way, the 3D model of the surgical instrument and the 3D model of the patient's body can be displayed together. In some embodiments, motion trackers or similar sensors may also be mounted on the surgical instrument and/or the patient's body. In this way, after the posture of the surgical instrument in space is determined through method 200, if the posture of the surgical instrument changes during subsequent operations, this change can be tracked by the motion tracker and applied to the determined posture of the surgical instrument. posture, thereby tracking and presenting the three-dimensional model of surgical instruments in real time.

In some optional embodiments, after step 210, step 212 can be continued to correct the posture of the surgical instrument in space determined in step 210.

Specifically, the correction amount can be determined using the target instrument image of the surgical instrument provided in step 202 and the virtual image (simulated image or historical image) of the surgical instrument determined in step 206 . For example, in step 206, the orientation of the surgical instrument relative to the imaging plane may be determined. It can be understood that in the training data set, this orientation corresponds to a simulated image or historical image of the surgical instrument, and the simulated image or historical image reflects the virtual projection of the surgical instrument under the corresponding orientation data; and the target instrument provided in step 202 The image is the actual projection of the surgical instrument. Therefore, it is advantageous to hope that the virtual projection of the surgical instrument can be as close to the actual projection as possible. The correction in step 212 is to adjust/correct the virtual projection to be close to the actual projection.

Figures 7a to 7d illustrate the process of further modifying the posture of the surgical instrument according to an embodiment of the present application. As shown in Figure 7a and Figure 7b (Figure 7b is a partial enlarged view of Figure 7a), in order to determine the aforementioned correction amount, the target device image and the virtual image can be overlapped with each other, and one or more images can be selected on the outer contour image of the target device image. Multiple contour sampling points. Since the target device image and the virtual image are placed substantially overlapping, each selected contour sampling point has a corresponding contour matching point on the device outer contour in the virtual image, which is from the device outer contour in the virtual image to the corresponding contour. The closest point to the sampling point. In this way, one or more pairs of contour sampling points and contour matching points can be determined. For example, in Figure 7b, a total of 4 pairs of contour sampling points and contour matching points are selected, which are located at the upper end of the instrument outer contour image. It can be understood that in practical applications, the number of contour sampling points and contour matching points can be adjusted, for example, more (for a straighter contour) or fewer (for a more tortuous contour) points are selected according to the curve change of the contour.

It can be understood that both the target instrument image and the virtual instrument image are projection images obtained after actual imaging or simulated imaging of the surgical instrument (or its virtual three-dimensional model) from the X-ray source (light source point), but are limited by the orientation of the surgical instrument. Due to the algorithmic accuracy of the determination process, the virtual image and the corresponding three-dimensional model of the surgical instrument are not ideally close to the actual spatial position of the surgical instrument. Correspondingly, as shown in Figure 7c, the X-ray source and each contour matching point can be connected to each other as a matching point connection line, and then the closest point between the three-dimensional model of the surgical instrument and each matching point connection line (called edge point). Then, as shown in Figure 7d, a vertical line is drawn from each edge point to the sampling point connection line connecting the light source point and the corresponding contour sampling point, where the intersection point of the vertical line and the sampling point connection line is the vertical foot. In this way, multiple edge points respectively correspond to multiple vertical feet, and the vectors pointing from the edge points to the vertical feet represent the error between the virtual image and the actual instrument image. This error can be calculated through matrix transformation. Ideally, in order to transform all points (multiple edge points) on one point set (multiple edge points) of two spatial point sets with the same number of points corresponding to the virtual image and the actual device image respectively, into corresponding points (vertical points) on the other point set. (sufficient), a transformation matrix (i.e., orientation correction matrix) for correcting the posture can be designed. Accordingly, in some embodiments, the orientation correction matrix may be calculated through separate sets of edge points and vertical feet. For example, the equation AM=B can be solved, where A represents a point set that can be formed by edge points, B represents a point set that can be formed by vertical feet, and M represents a modification matrix. It can be understood that in practical applications, for example, depending on whether the point set meets the rigid transformation condition, the orientation correction matrix obtained by solving the problem may be an accurate numerical solution, or it may be an optimal approximate solution.

Based on this, the correction matrix can be further applied to the orientation of the surgical instrument relative to the imaging plane to obtain a corrected orientation; or the correction matrix can be applied to the posture of the surgical instrument in space to obtain a corrected spatial posture.

Based on the method for determining the spatial posture of surgical instruments disclosed in this application, during the operation, only a real-time image of the surgical instrument is needed to determine the posture of the surgical instrument in space, which effectively reduces X-ray exposure during the operation. times, thereby reducing radiation exposure to patients and doctors.

In some embodiments, the present application also provides some computer program products including non-transitory computer-readable storage media. The non-transitory computer-readable storage medium includes computer-executable code for performing the steps in the method embodiment shown in FIG. 2 . In some embodiments, the computer program product may be stored in a hardware device.

Embodiments of the present invention may be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or specially designed hardware. Those of ordinary skill in the art will understand that the above-described apparatus and methods may be implemented using computer-executable instructions and/or included in processor control code, for example on a carrier medium such as a disk, CD or DVD-ROM, such as a read-only memory. Such code is provided on a programmable memory (firmware) or on a data carrier such as an optical or electronic signal carrier. The device and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., It can also be implemented by software executed by various types of processors, or by a combination of the above-mentioned hardware circuits and software, such as firmware.

Those of ordinary skill in the art can understand and implement other changes to the disclosed embodiments by studying the specification, disclosure and drawings, and appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude the plural. In the actual application of this application, one part may perform the functions of multiple technical features cited in the claims. Any reference signs in the claims shall not be construed as limiting the scope.

Claims

A method for determining the posture of a surgical instrument in space, characterized in that the surgical instrument is formed of an X-ray imageable material, the surgical instrument has a predetermined instrument outer contour and one or more characteristic through holes, wherein Each characteristic via has a predetermined via profile, and the method includes:

providing a target instrument image of the surgical instrument, wherein the target instrument image includes at least a portion of an outer contour image of the instrument outer contour of the surgical instrument and a through hole image of at least one characteristic through hole;

A relative position image of the at least one characteristic through hole and the through hole image of the at least one characteristic through hole are extracted from the target instrument image, wherein the relative position image reflects the at least one characteristic through hole. The position of the through-hole image relative to the at least part of the outer contour image;

The orientation of the digital model of the surgical instrument relative to the imaging plane is adjusted multiple times, and simulated imaging is performed on the digital model of the surgical instrument after each adjustment to generate multiple images of the surgical instrument corresponding to the orientation. A simulated device image;

The through hole image and the relative position image of the at least one characteristic through hole are processed based on the plurality of simulated instrument images to determine the orientation of the surgical instrument relative to the imaging plane when the target instrument image is imaged. ;as well as

Based on the orientation of the surgical instrument relative to the imaging plane, an attitude of the surgical instrument in space is determined.
The method according to claim 1, characterized in that the through hole image and the relative position image of the at least one characteristic through hole are processed based on the plurality of simulated instrument images to determine the position of the target instrument in the target instrument. The step of orienting the surgical instrument relative to the imaging plane during image imaging further includes:

Comparing similarity between each simulated instrument image of the plurality of simulated instrument images of the surgical instrument and the through hole image of the at least one characteristic through hole and the relative position image; and

Select a simulated instrument image including a through-hole image whose similarity to the through-hole image is higher than a predetermined similarity threshold, and use the orientation corresponding to the simulated instrument image as the orientation of the surgical instrument relative to the target instrument image when imaging Orientation of the imaging plane.
The method according to claim 1, characterized in that the through hole image and the relative position image of the at least one characteristic through hole are processed based on the plurality of simulated instrument images to determine the position of the target instrument in the target instrument. The step of orienting the surgical instrument relative to the imaging plane during image imaging further includes:

Provide neural network models;

The neural network model is trained using a predetermined image database, wherein the image database includes the plurality of simulated instrument images of the surgical instrument, and each simulated instrument image is associated with a number of the surgical instrument when it was simulated to be imaged. Orientation of the model relative to the imaging plane; and

The through hole image and the relative position image of the at least one characteristic through hole are processed by the neural network model to obtain the corresponding orientation of the surgical instrument relative to the imaging plane.
The method according to claim 3, characterized in that the predetermined image database also includes historical image data, and the historical image data is created in the following manner:

providing a plurality of historical instrument images of the surgical instrument, and a calibrated orientation of the surgical instrument relative to the imaging plane corresponding to each historical instrument image;

Historical instrument images of the surgical instrument and the orientation of the surgical instrument relative to the imaging plane are stored correspondingly.
The method according to claim 3 or 4, characterized in that, before using the predetermined image database to train the neural network model, using the target instrument image of the surgical instrument to train the instrument in the predetermined image database. Filter images.
The method of claim 5, wherein the filtering includes filtering out device images whose size of through-hole images in the predetermined image database does not meet predetermined image filtering conditions.
The method according to claim 1, characterized in that, before extracting the relative position image of the at least one characteristic through hole and the through hole image of the at least one characteristic through hole from the target instrument image, the Methods also include:

Image distortion correction is performed on the target device image.
The method according to claim 1, characterized in that, after determining the posture of the surgical instrument in space, the method further includes:

Based on the deviation between the target instrument image and the determined instrument image corresponding to the orientation of the surgical instrument, the posture of the surgical instrument in space is corrected.
The method according to claim 1, characterized in that, after determining the posture of the surgical instrument in space, the method further includes:

Based on the posture of the surgical instrument in space, a three-dimensional model of the surgical instrument is displayed.
A device for determining the posture of a surgical instrument in space, characterized in that the surgical instrument is formed of an X-ray imageable material, the surgical instrument has a predetermined instrument outer contour and one or more characteristic through holes, wherein Each characteristic via has a predetermined via profile, the apparatus includes a non-transitory computer readable storage medium including computer executable code executable to perform the following Described steps:

providing a target instrument image of the surgical instrument, wherein the target instrument image includes at least a portion of an outer contour image of the instrument outer contour of the surgical instrument and a through hole image of at least one characteristic through hole;

A relative position image of the at least one characteristic through hole and the through hole image of the at least one characteristic through hole are extracted from the target instrument image, wherein the relative position image reflects the at least one characteristic through hole. The position of the through-hole image relative to the at least part of the outer contour image;

The orientation of the digital model of the surgical instrument relative to the imaging plane is adjusted multiple times, and simulated imaging is performed on the digital model of the surgical instrument after each adjustment to generate multiple images of the surgical instrument corresponding to the orientation. A simulated device image;

The through hole image and the relative position image of the at least one characteristic through hole are processed based on the plurality of simulated instrument images to determine the orientation of the surgical instrument relative to the imaging plane when the target instrument image is imaged. ;as well as

determining an attitude of the surgical instrument in space based on the orientation of the surgical instrument relative to the imaging plane