US20230252681A1

US20230252681A1 - Method of medical calibration

Info

Publication number: US20230252681A1
Application number: US18/100,826
Authority: US
Inventors: Richard Bäumer; Matthias Karl
Original assignee: Carl Zeiss Meditec AG
Current assignee: Carl Zeiss Meditec AG
Priority date: 2022-02-04
Filing date: 2023-01-24
Publication date: 2023-08-10
Also published as: CN116549110A; US20230248467A1; CN116549109A; DE102022125798A1; DE102023101117A1

Abstract

The present disclosure relates to a method of medical calibration, including the method steps of: capturing an image representation of a medical marker and of a medical instrument connected therewith; determining a spatial pose of the medical marker on the basis of the image representation; determining a first position of a tracking point of the medical instrument on the basis of the determined spatial pose and on the basis of information relating to a spatial pose relationship between medical marker and tracking point, and determining a second position of the tracking point on the basis of the captured image representation; determining a deviation between the first position and the second position; adjusting the information relating to the spatial pose relationship between medical marker and tracking point on the basis of the determined deviation.

Description

SUBJECT MATTER OF THE INVENTION

The present invention relates to a method of medical calibration and a medical calibration system, in particular for calibrating a medical instrument using a medical marker. The present invention further relates to a computer program for carrying out the method according to the invention in the system.

TECHNOLOGICAL BACKGROUND

The use of technological aids is part and parcel of modern medicine. Imaging methods and robotic systems for guiding medical instruments are used equally as a matter of course in both surgery and diagnostics. In this context, the use of imaging methods allows the discrimination of various structures in the patient and the image data obtained in the process can be used advantageously in diagnostics, and also in therapeutic and surgical methods.
By way of example, not only do 3-D image data of a patient allow a surgeon to plan a surgical intervention better, but the 3-D image data can also assist the implementation of the intervention. In particular, image information obtained during the operation can be overlaid on diagnostic 3-D image data obtained in advance in order to indicate poorly visible tissue boundaries to the surgeon. Further, robotic surgical instruments can be controlled on the basis of 3-D image data or are already able to carry out certain operations in partly or fully autonomous fashion.
In this context, correct linking of 3-D image data to a reference coordinate system is of essential importance to the aforementioned applications. In this case, the reference coordinate system can be the coordinate system of the patient during an intervention, the coordinate system of image information captured during the intervention and/or the coordinate system of the robotic system of a surgical microscope. Only such linking allows a subsequent error-free medical navigation of the surgeon on the basis of the image data or of the robotic aids by means of the image data.
Linking 3-D image data to a certain reference coordinate system for the purposes of medical navigation is usually referred to as a “registration”. Such a registration allows unique mapping of coordinates of the patient space to corresponding coordinates of the image space. Once such a mapping is known, structures of the patient situated at defined coordinates of the patient space can be represented at the corresponding coordinates in the image space.
In addition to the capture and representation of image information registered with respect to the reference coordinate system, the capture of the position of medical instruments in the reference coordinate system is also very advantageous. In this case, the capture of the position can serve both for the navigation of the medical instruments in the reference coordinate system and for the capture of the spatial pose of a structure boundary contacted by the medical instrument. Thus, the tracking of medical instruments is also used, inter alia, for capturing patient contours, for example in order to register a patient in the reference coordinate system of the 3-D image data.
As a rule, a registration is carried out using medical markers which allow the spatial position of the markers to be determined in a coordinate system. To this end, the geometry of the markers is predetermined and able to be captured by means of imaging methods. Hence, by evaluating one or more images of such a marker, it is possible to determine the pose (position and orientation) thereof in the coordinate system. When one camera is used, generally at least two marker elements (and optionally the alignment thereof), preferably at least three marker elements (and optionally the alignment thereof), the relative spatial pose of which with respect to one another is known, are captured to determine the pose of a marker.
If, further, the relative spatial pose of the marker with respect to an object is known, the spatial pose of the object is also determinable in the coordinate system on the basis of the marker pose. In this case, initially linking the object pose or the pose of a characteristic point on the object with the marker pose is referred to as “calibrating” the object. In particular, the object can be a medical instrument.
If the data relating to the spatial pose of the object are made available to a medical navigation system, the latter can process the tracking data of the object together with other data determined in the coordinate system or registered therewith. By way of example, this allows the virtual representation of a medical instrument with the correct spatial relationship with respect to an anatomical structure, the capture of contours in the patient space using the medical instrument and/or the automated implementation of surgical interventions using the medical instrument.
Fastening a marker to a medical instrument is known from DE 202015106804 U1, and WO 2016/059250 A1 discloses the use of a marker for medical instrument navigation. The accuracy of the medical calibration of a medical instrument or medical instrument navigation is limited not only by the accuracy with which the pose of the marker fastened thereto is determined but also by tolerances relating to the way the marker is fastened to the medical instrument. Already small deviations of the combined geometry of marker and medical piece of equipment have significant effects on the determined pose of this piece of equipment. Small changes in pose of a registered piece of equipment relative to the marker fastened thereto may already lead to an incorrect medical navigation, especially in the case of a large working distance between the marker and a utilized imaging sensor.
The object of the present invention is to overcome or minimize the disadvantages of the prior art and to provide an improved method of medical calibration, in particular for a medical instrument.

DESCRIPTION OF THE INVENTION

The object according to the invention is achieved by the subjects of the independent patent claims. Preferred developments are the subject matter of the dependent claims.
A first aspect of the present disclosure relates to a medical calibration method, in particular for optimizing the determination of a relative alignment and orientation of a medical instrument or a positioning of a relevant point thereof, including the method steps described below.
In a step of the method according to the present disclosure, an image representation of a medical marker and of a medical instrument connected therewith is initially captured. The medical marker is preferably a medical marker having a plurality of marker elements that are distinguishable from one another, that is to say that are individually detectable, for example in accordance with attached FIG. 6 . Preferably, at least some of the marker elements, preferably all marker elements, are uniquely distinguishable from one another. Within the scope of the present disclosure, the term target may be used synonymously to replace the term marker, with in this case the term marker being able to be used synonymously to replace the term marker element. According to this alternative designation, the target has a plurality of individually detectable (distinguishable) markers. Preferably, at least some of the markers of the target, preferably all markers, are uniquely distinguishable from one another.
By way of example, the medical instrument is a medical probe, a pointer, a sucker, an awl or the like. Preferably, the medical marker and the medical instrument are securely connected to one another or securely connectable to one another, for example by means of a non-detachable connection, an integral connection, a detachable connection, a cohesive connection, a force-fit connection and/or an interlocking connection. In a particularly preferred implementation, the medical marker and the medical instrument are detachably connected to one another, for example by means of a plug-in connection, a clipping connection, a latching connection, a bayonet connection and/or a clamping connection. Below, the combination of medical marker and medical instrument is simply referred to as “combination”.
As a matter of principle, both two-dimensional and three-dimensional image representations of marker and medical instrument can be used within the scope of the method. The image representation of the medical marker can preferably consist only of the image representations of the marker elements of the medical marker, for example as a result of illumination by IR light and IR-reflective marker elements. Equally, the image representation of the medical instrument can consist only of image representations of relevant (tracking) points of the medical instrument. As an alternative or in addition, the captured image representation may however also image the specific spatial shape of the marker and/or of the medical instrument, for example when the image representations thereof are captured in the visible light range.
In the further step of the method according to the present disclosure, a spatial pose of the medical marker is determined on the basis of the captured image representation. In this case, the spatial pose of the medical marker is determined with reference to a reference coordinate system, for example the coordinate system of a patient during an operation, the coordinate system of a surgical microscope and/or the coordinate system of presurgical image data. The spatial pose preferably comprises the pose of the medical marker, that is to say its position and orientation. To this end, the geometry of the medical marker is either fixedly predetermined or calibrated, as will still be explained in detail below. By way of example, the radius and the relative spacing of the marker elements of the marker are known, and so evaluation of the captured image representation of the medical marker renders its position and orientation in space determinable.
In the method according to the present disclosure, a first position of a tracking point of the medical instrument is further determined on the basis of the determined spatial pose of the medical marker and on the basis of information relating to a spatial pose relationship between medical marker and the tracking point. In particular, the tracking point is a characteristic or relevant point of the medical instrument, for example the tip of a medical pointer, sucker or scalpel. The tracking point is a point, the tracking of which is desired. Preferably, a tracking structure may also be used instead of the tracking point. In this case, a tracking structure has a greater spatial extent than a tracking point. A tracking structure preferably has at least two tracking points. Thus, while a tracking point is representable as a geometric point within the scope of the method according to the present disclosure, a tracking structure is representable by more than one geometric point within the scope of the method according to the present disclosure. By way of example, the tracking structure likewise is the tip of a medical marker, but the latter, as a tracking structure, is represented by more than one geometric point, for example as a result of capturing the edges of the tip. For reasons of clarity, the invention is described below with reference to a tracking point. However, wherever it appears appropriate, there is a brief discussion of relevant differences that arise when a tracking structure is used.
The information relating to the spatial pose relationship describes the spatial relationship, that is to say the relative position and alignment of the tracking point relative to the medical marker. In particular, the spatial pose relationship can describe the spatial relationship between a certain point (target point) of the medical marker and the tracking point. In this case, the target point has a fixed spatial relationship with respect to the marker elements of the medical marker. The target point may be predefined or calibrated and is optional in the disclosed method, for example to the extent that it may be contained implicitly in a spatial pose relationship between marker and tracking point. In a preferred implementation, the information relating to the spatial pose relationship describes the relative position and alignment of the tracking structure (containing at least two tracking points) relative to the medical marker.
By way of example, the information relating to the spatial pose relationship is a parameter set of a rigid body transformation. In this case, in the method according to the present disclosure, a parameter set of a rigid body transformation preferably comprises at least three values for the translation along the x-axis, the y-axis and the z-axis (tracking point) and particularly preferably three values for the rotation about the x-axis, the y-axis and the z-axis and also three values for the translation along the x-axis, the y-axis and the z-axis, that is to say a total of six parameters (tracking structure). In the method according to the present disclosure, initial information relating to the spatial pose relationship is preferably available and the first position is first of all determined on the basis of this initial information.
Further, a second position of the tracking point is determined on the basis of the captured image representation in the method according to the present disclosure. In particular, the second position is determined purely on the basis of the image representation and without resorting to the pose of the medical marker or information relating to spatial pose relationships. Consequently, the second position of the tracking point is determined only on the basis of the captured image information in the image representation of the combination of marker and medical instrument, in particular by way of an image analysis with regards to the captured image representation of the medical instrument. To this end, image recognition is preferably implemented in the captured image representation. This is always computationally complex, requires a certain image quality—especially if the tracking point is very small (pointer tip)—and is not always possible, for example if the tracking point is intermittently or partially concealed. In the method according to the present disclosure, the determined second position of the tracking point is preferably considered to be the real position of the tracking point in the captured image representation.
There further is a determination of a deviation between the determined first position of the tracking point and the determined second position of the tracking point within the scope of the method according to the present disclosure. Initially, this first position and the second position generally are not identical to one another, with the determined deviation being based on incorrect or incomplete information relating to the spatial pose relationship between medical marker and tracking point of the medical instrument.
In the method according to the present disclosure, there further is an adjustment of the information relating to the spatial pose relationship between medical marker and tracking point on the basis of the determined deviation between the first position and the second position. In this case, the information relating to the spatial pose relationship is adjusted in such a way that the deviation between the first and the second position of the tracking point is minimized. Consequently, there is an optimization of the information relating to the spatial pose relationship with the constraint of minimizing the deviation between the first position and the second position of the tracking point. Adjusting the information relating to the spatial pose relationship between medical marker and tracking point in this case leads indirectly to an adjustment of the first position of the tracking point. The optimization of the information relating to the spatial pose relationship is implemented by variation, for example. In the case of a parameter set of a rigid body transformation, the optimization is preferably implemented by varying the parameters of the parameter set while simultaneously taking account of the minimization of the deviation between the positions of the tracking point. The minimization of the deviations can be taken into account by a cost function to be minimized.
The method according to the present disclosure consequently advantageously allows an optimization of the determination (estimation) of the position of a tracking point, for example a pointer tip, on the basis of a captured pose of a medical marker. Such inaccuracies, which are traced back to manufacturing tolerances of the medical marker or medical instrument or to tolerances when connecting marker and instrument, are advantageously compensated by adjusting the information relating to the spatial pose relationship. Such inaccuracies could lead to errors in previously used methods of calibration or medical navigation. Such errors are advantageously reduced by the method disclosed herein. Further advantageously, it is no longer necessary to calibrate the spatial pose relationship between tracking point and medical marker, as described for example in DE102018119343A1 on the basis of aligning the tracking point with respect to a calibration point. Moreover, possible variations in the relative pose of medical marker and medical instrument can be taken into account. Consequently, deformations in the detachable or non-detachable connection between the medical marker and the medical instrument (which may have occurred during storage, for example) advantageously do not lead to errors in the navigation of the medical instrument and an incorrect navigation of the medical instrument is thus advantageously avoided.
In a preferred implementation of the method according to the present disclosure, a multiplicity of image representations of the combination are captured. According to this implementation, there is furthermore a determination of a first position of the tracking point as described above and of a second position of the tracking point as described above in each of the captured image representations. Further, a deviation between the determined first position and the determined second position is determined for each of the captured image representations and the information relating to the spatial pose relationship between medical marker and tracking point is adjusted on the basis of the multiplicity of deviations determined thus. This preferred implementation comprises both a one-time adjustment of the information relating to the spatial pose relationship following the determination of a multiplicity of deviations and the successive adjustment of the information relating to the spatial pose relationship on the basis of the deviations.
In both variants of this preferred implementation, a video sequence, that is to say a multiplicity of image representations, of the combination is preferably initially recorded. In this case, the combination is preferably moved and particularly preferably moved and pivoted (rotated) during the recording. A multiplicity of deviations are determined between the first and second positions of the tracking point on the basis of the image representations determined thus. In the case of a one-time adjustment, the multiplicity of deviations are used for a one-time adjustment of the information relating to the spatial pose relationship, with the quality of the adjustment of the information relating to the spatial pose relationship advantageously being increased as a result of the greater number of deviations to be minimized (which, moreover, were preferably determined in different perspectives). In the case of a successive adjustment, there preferably is a first adjustment of the information relating to the spatial pose relationship on the basis of a first deviation determined in a first image representation, and a first position of the tracking point in a second image representation is determined using the information relating to the spatial pose relationship adjusted thus and the marker pose in the second image representation. Then, there is a second adjustment of the information relating to the spatial pose relationship, which has already been adjusted once, on the basis of a second deviation determined in the second image representation. A first position of the tracking point in a third image representation is then determined on the basis of the information relating to the spatial pose relationship, which has been adjusted twice in this way, and the marker pose in the third image representation. By continuing or repeating these steps, a quality of the adjustment of the information relating to the spatial pose relationship can likewise be advantageously increased. Moreover, this implementation advantageously also allows continuous monitoring of the transformation from the medical marker to the tracking point of the medical instrument during the use of the medical instrument.
In a further preferred implementation of the method according to the present disclosure, calibrated information relating to the spatial pose relationship between medical marker and tracking point is obtained if a deviation between the first position and the second position is determined as being below a predetermined threshold value. Expressed differently, the determined deviation of the positions is always compared to a threshold value according to this implementation and dropping below the threshold value preferably represents a termination condition for an optimization of the information relating to the spatial pose relationship. The method according to the present disclosure is particularly preferably carried out at the start of a medical instrument use. Particularly preferably, a user, possibly following a request in this respect, moves the combination while a sequence of image representations of same is captured and the user is informed when an optimization of the information relating to the spatial pose relationship has been completed, that is to say the tracking point has been calibrated with respect to the marker. Likewise preferably, the image representations captured in the process are displayed on a display medium and the first position and the second position, and optionally the determined spacing therebetween, are depicted in the displayed image representations. By way of the detected distances reducing or the displayed positions approaching one another, the user obtains visual feedback relating to the advancing optimization.
In a further preferred implementation of the method according to the present disclosure, there then is a continuous determination of first positions of the tracking point on the basis of continuously captured image representations of the medical marker and of the medical instrument connected therewith and on the basis of the calibrated information relating to the spatial pose relationship between marker and tracking point. Expressed differently, following the completion of the optimization of the information relating to the spatial pose relationship, that is to say following the completion of the calibration of the tracking point relative to the medical marker, a position of the tracking point is determined only on the basis of the marker pose and the transformation conveyed by the information relating to the spatial pose relationship. Hence, it is possible to advantageously dispense with a resource-intensive or computationally complicated permanent determination of the tracking tip on the basis of the image information in the captured image representations (for example, as a result of image recognition). At the same time, a permanently high quality of the determination of the position of the tracking point is ensured.
In a further preferred implementation of the method according to the present disclosure, there is an intermittent determination of two positions of the tracking point in a selection of continuously captured image representations of the combination. Further, first positions of the tracking point are preferably determined in all of the continuously captured image representations of the combination (or at least in more image representations than are contained in the said selection), for example on the basis of the calibrated information relating to the spatial pose relationship. Further, there likewise is an intermittent determination of deviations between the first positions and the second positions of the tracking point in the selection of the continuously captured image representations of the combination, and a comparison with a predetermined limit value. As soon as one of the determined deviations exceeds the predetermined limit value, there further preferably is a (renewed) adjustment of the information relating to the spatial pose relationship between medical marker and tracking point. Consequently, this implementation advantageously realizes continuous monitoring of the information relating to the spatial pose relationship and a renewed adjustment of same as soon as this becomes necessary, for example as a consequence of the marker mounted to the medical instrument slipping. By setting a frequency of the intermittent determination of second positions, deviations and possible limit value overshoots, it is possible to set how often the spatial pose relationship should be monitored, for example following an already implemented calibration. Expressed differently, this implementation represents an option for monitoring the spatial pose relationship at regular intervals after the aforementioned termination condition has been reached or after calibrated information relating to the spatial pose relationship has been obtained.
In a further preferred implementation of the method according to the present disclosure, the medical marker has a predetermined geometry and the spatial pose of the medical marker is determined on the basis of the captured image representation of the medical marker and on the basis of information relating to the geometry of the marker. Expressed differently, the medical marker preferably has a predetermined (nominal) geometry. As a rule, medical markers with a predefined geometry are produced as exactly as possible. Further, the geometry is advantageously optimized in respect of capturing a spatial pose of the marker on the basis of two-dimensional image representations. The spatial pose of the medical marker is consequently determined on the basis of the captured image representation and on the basis of information relating to the geometry of the marker. This information is advantageously stored and able to be stored in the form of a data set.
In a further preferred implementation of the method according to the present disclosure, the image representation of the medical marker contains the image representations of at least two, preferably at least three, marker elements and the information relating to the geometry of the medical marker at least includes information relating to the relative spatial pose of these at least two, preferably at least three, marker elements relative to one another. The presence of at least two, preferably at least three, captured marker elements advantageously enables an accurate determination of the spatial pose of the marker. Particularly preferably, the at least two, preferably at least three, marker elements are at least partially arranged in different planes and further have preferably exactly predetermined absolute dimensions and relative spatial poses. According to this implementation, determining the spatial pose of the marker initially also includes determining the absolute and/or relative spatial poses of the at least two, preferably at least three, marker elements of the marker.
Particularly preferably, the information relating to the spatial pose relationship between medical marker and tracking point defines a relative spatial pose of the tracking point with respect to each one of the at least two, preferably at least three, marker elements. Consequently, the position of the tracking point can be deduced with great reliability on the basis of the captured image representations of the at least two, preferably at least three, marker elements. Further, the information relating to the spatial pose relationship can be optimized on the basis of a few image representations of the medical marker since each of these image representations contains at least two, preferably at least three, mutually different image representations of marker elements, which have different spatial pose relationships with respect to the tracking point. In this respect, the image representations of different marker elements correspond to the image representations of a single marker from different perspectives.
Likewise preferably, the information relating to the spatial pose relationship between medical marker and tracking point defines a relative spatial pose of the tracking point with respect to a target point that has a fixed spatial relationship with the marker elements. By way of example, the target point is the origin of a local coordinate system of the medical marker. The use of a target point advantageously allows the information relating to the spatial pose relationship between medical marker and tracking point to be subdivided into two components, specifically into a first component relating to the spatial pose relationship between the marker elements and the target point and a second component relating to the spatial pose relationship between the target point and the tracking point. Frequently, only one of the first and second components is affected by tolerances in the manufacture of the medical marker or by tolerances due to the medical marker being mounted on a medical instrument. By way of example, a tolerance as a result of mounting leads to a deviation only in the second component while a manufacturing tolerance of the marker leads to a deviation only in the first component. By dividing the information relating to the spatial pose relationship, for example into two mutually different rigid body transformations, the above-described optimization of the information relating to the spatial pose relationship is advantageously applicable in a more targeted manner. Likewise advantageously, a first component of the information relating to the spatial pose relationship can be calibrated by creating a three-dimensional representation of the medical marker, as described below. Likewise advantageously, there can be an optimization of only the second component of the information relating to the spatial pose relationship, as described above.
In a preferred implementation of the method according to the disclosure, the information relating to the geometry of the marker is predetermined. The medical marker consequently has a predetermined (nominal) geometry and is produced as exactly as possible in accordance with this geometry defined in advance. Consequently, the spatial pose of the marker is determined advantageously quickly and resource-sparingly on the basis of the predefined information relating to the nominal geometry of the marker and on the basis of the image representation of the marker. In this process, a separate calibration of the marker can thus advantageously be dispensed with.
In a further preferred implementation of the method according to the present disclosure, the said method further includes a calibration of the marker and the information relating to the geometry of the marker is consequently determined or optimized within the scope of the method itself. Further preferably, a first component of information relating to the spatial pose relationship between medical marker and tracking point is created for the first time during the calibration of the medical marker. According to this preferred implementation, the method according to the present disclosure further includes the method steps of capturing a first image representation of the medical marker in a first perspective, of capturing a second image representation of the medical marker in a second perspective, and of determining the information relating to the geometry of the marker on the basis of the first image representation and the second image representation. Within the scope of the present disclosure, a perspective preferably denotes the relative standpoint, in relation to the marker, from which the image representation of the marker is captured.
Capturing the medical marker from two perspectives advantageously allows the derivation of depth information for the geometry of the marker on the basis of two-dimensional or three-dimensional image representations. The information relating to the geometry of the medical marker preferably comprises a three-dimensional representation of the marker, preferably a three-dimensional model of the shape or geometry of the marker. Particularly preferably, the information relating to the geometry of the medical marker comprises the geometry of the marker elements and their relative spatial pose with respect to one another. Moreover, knowledge of the specific spatial shape of the marker is not necessary, but not damaging either. The determination of a three-dimensional representation of a real object on the basis of two-dimensional or three-dimensional image representations by means of photometric methods is known to a person skilled in the art from the prior art. To this end, many different software solutions are commercially available, and so it is possible to dispense with a detailed description of the 3-D representation of the marker. Preferably, the information relating to the geometry of the medical marker is determined with computer assistance in the method according to the present disclosure, particularly preferably using machine learning (AI) algorithms. The information relating to the geometry of the medical marker is preferably available on a data medium as a computer-readable model of the marker.
This preferred implementation of the method according to the present disclosure advantageously enables an initial registration of the marker and is consequently advantageously largely invariant to variations in the geometry of the medical marker due to production-related reasons or as a result of wear or incorrect handling. Moreover, the initial registration of the medical marker can advantageously be carried out in the same piece of equipment, for example a surgical microscope, as the subsequent calibration of the tracking point. Hence, imaging properties of the piece of equipment used for registration purposes are advantageously considered at least intrinsically.
Further preferably, the predefined information relating to the geometry of the medical marker is adjusted on the basis of the first image representation and the second image representation of the marker (from different perspectives). From the above-described implementation with the sole use of predefined information or the complete calibration of the marker, this implementation differs in terms of the adjustment of the predefined information for the predetermined (nominal) geometry on the basis of the captured image representations. Advantageously, this also takes account of production-related variations, deviations in the geometry on account of ageing or for other reasons, while a three-dimensional representation of the medical marker, in particular in the form of a computer-readable data set of a model, is already present before the start of the method according to the present disclosure.
In the preferred implementations with a calibration of the marker, a relationship is determined between a coordinate system, K_A1, of the first image representation and a coordinate system, K_A2, of the second image representation, preferably by way of a parameter set of a rigid body transformation. Ultimately, the spatial pose of the medical marker should be determined in a reference coordinate system which may differ from the coordinate systems K_A1and K_A2or which may be identical to one of these. If the reference coordinate system differs from the coordinate systems K_A1and K_A2, a relationship between one of the coordinate systems K_A1and K_A2and the reference coordinate system is preferably likewise determined by a parameter set of a rigid body transformation. Should the coordinate systems K_A1and K_A2correspond to a coordinate system of at least one imaging sensor, a parameter set of one rigid body transformation is sufficient to map points in one of the coordinate systems K_A1and K_A2onto points in the other one of the coordinate systems K_A1and K_A2. Otherwise, a relationship between a coordinate system, K_A1, of the first image representation and a coordinate system of an associated imaging sensor is preferably determined by a set of intrinsic parameters. Likewise preferably, a relationship between a coordinate system, K_A2, of the second image representation and a coordinate system of an associated imaging sensor is determined by a set of intrinsic parameters. In this case, the intrinsic parameters determine a relationship between the coordinate system of the image representation and the coordinate system of the associated imaging sensor. In this case, the type of the intrinsic parameters depends, in particular, on the type of imaging sensor utilized, with imaging sensor in this case denoting both the actual sensor and the utilized optics. If intrinsic parameters should be taken into account, a relationship between a coordinate system of the first imaging sensor and a coordinate system of the second imaging sensor is preferably determined by a parameter set of a rigid body transformation.
In a further preferred implementation, the determination of the information relating to the geometry of the medical marker comprises a transformation of the first image representation of the marker into the coordinate system, K_A2, of the second image representation using the parameter set, further preferably using intrinsic parameters and the parameter set. By way of example, the positions, shapes and/or sizes of marker elements and/or characteristic points of the marker in the coordinate system of the first image representation are transferred by calculation into positions, shapes and/or sizes of the marker elements and/or of the characteristic points of the marker in the coordinate system of the second image representation. Subsequently, a deviation is determined between the transformed first image representation of the marker and the second image representation of the marker. Expressed differently, deviations between the transformed positions, shapes and/or sizes of the first image representation and the corresponding positions, shapes and/or sizes of the second image representation are determined in the coordinate system of the second image representation. By way of example, the centre and radius of a circular marker element in the first image representation are transformed by calculation into the coordinate system of the second image representation and deviations from the centre and radius of the corresponding marker element in the second image representation are subsequently determined in the coordinate system of the second image representation. According to this implementation, the information relating to the geometry of the medical marker is finally determined on the basis of the determined deviations. The reversed procedure, that is to say with the transformation of the second image representation into the coordinate system, K₁, of the first image representation and the determination of the deviations in the coordinate system K₁, is also preferred. These implementations advantageously allow an optimization of the information relating to the geometry of the medical marker by minimizing the determined deviations. Provided the determined information maps the geometry of the medical marker and the utilized imaging sensors are each calibrated intrinsically and extrinsically, the captured image representations can be converted into one another by calculation on the basis of parameter sets (a rigid body transformation and/or determined on the basis of the extrinsic and intrinsic calibration parameters) and the corresponding optimization target has been achieved.
Particularly preferably, determining the information relating to the geometry of the medical marker comprises the adjustment, on the basis of the determined deviations, of predefined information relating to the geometry of the marker, with the goal of minimizing these deviations. Moreover, the steps described here can also be used to optimize the utilized parameter sets. To this end, the information relating to the geometry of the medical marker is not varied and there is, instead, varying of the parameter sets for minimizing the determined deviations. Particularly preferably, the optimization of the information relating to the geometry of the medical marker and the optimization of the utilized parameter sets are carried out alternately and/or using different image representations of the marker.
In a likewise preferred implementation of the method, the first image representation of the medical marker is captured by a first camera and the second image representation of the medical marker is captured by a second camera. Expressed differently, two different cameras are used for calibrating the marker. Preferably, this relates to at least one camera of a surgical microscope. By way of example, the first camera is a main observer camera and the second camera is a surround camera of the same surgical microscope. Further preferably, at least one or each of the two cameras can be used to capture a plurality or multiplicity of image representations of the medical marker. In a likewise preferred implementation, the main observer camera captures the first image representation of the medical marker in the visible light range and the surround camera captures the second image representation of the medical marker in the infrared range.
In the aforementioned preferred implementation, a relationship is defined between a coordinate system, K_K1, of the first camera and a coordinate system, K_A1, of the first image representation, preferably by way of first intrinsic parameters of the first camera. The first intrinsic parameters particularly preferably comprise a first effective focal length, the coordinates of a principal image point (centre of the distortion) of the first image representation, a first scaling factor and/or a first radial lens error coefficient (distortion coefficient). Likewise preferably, the second intrinsic parameters of the second camera preferably comprise a second effective focal length, the coordinates of a principal image point (centre of the distortion) of the second image representation, a second scaling factor and/or a second radial lens error coefficient (distortion coefficient). The intrinsic parameters are preferably calibrated by means of Tsai's camera calibration. Alternatively, other intrinsic parameters can also be used, for example according to Zhang's camera calibration (cf., for example, “A practical comparison between Zhang's and Tsai's calibration approaches”, Li et al., Proceedings of the 29th International Conference on Image and Vision Computing New Zealand, November 2014 Pages 166-171, DOI:10.1145/2683405.2683443).
In the aforementioned preferred implementation, a relationship is further preferably determinable between the coordinate system of the first camera, K_K1, and the coordinate system of the second camera, K_K2, on the basis of first and second extrinsic parameters. In this case, the first extrinsic parameters preferably define a relationship between the coordinate system K_K1and a reference coordinate system and the second extrinsic parameters preferably define a relationship between the coordinate system K_K2and the reference coordinate system. Expressed differently, the extrinsic parameters describe the external orientation of the respective camera, that is to say the position and alignment of the respective camera in the reference coordinate system. According to this implementation, a parameter set, which defines the relationship between the coordinate system of the first image representation, K_A1, and the coordinate system of the second image representation, K_A2, is preferably determinable on the basis of the first and second intrinsic parameters and the first and second extrinsic parameters. Such a parameter set preferably contains the first and second intrinsic parameters and a parameter set of a rigid body transformation determined on the basis of the first and second extrinsic parameters.
In a further preferred implementation, the first image representation is captured in a first position of a camera and the second image representation is captured in a second position of the same camera. Particularly preferably, the camera in this case is moved from the first position to the second position by way of a known kinematic system. The known kinematic system permits precise specifications in relation to the rotation of the camera about the x-axis, the y-axis and the z-axis and in relation to the translation of the camera along the aforementioned axes. Consequently, the calibration is advantageously performable using only a single camera. Further preferably, this implementation is also implementable with each camera of the above-described implementation, that is to say with the first and the second camera. The at least one camera is preferably a camera of a surgical microscope, for example a main observer camera or a surround camera. Within the scope of the present disclosure, a known kinematic system is understood to mean a kinematic system that enables a well-defined translation and/or rotation. The geometry of a kinematic system, in particular, must be sufficiently accurate for such a well-defined translation and/or rotation of the kinematic system. This is preferably achieved by a high manufacturing quality and/or by a calibration.
In a likewise preferred implementation, the first image representation, which is of a first pose of the medical marker, is captured by a camera. Expressed differently, an image representation of the marker with the marker in a first position and orientation is captured. The second image representation, which is of a second pose of the medical marker, is then captured by the same camera. Particularly preferably, the marker in this case is moved from the first pose to the second pose by way of a known kinematic system. The known kinematic system permits precise specifications in relation to the rotation of the marker about the x-axis, the y-axis and the z-axis and in relation to the translation of the marker along the aforementioned axes.
Consequently, the calibration of the marker is advantageously performable using only a single camera. Further preferably, this implementation is also implementable with each camera of the above-described implementation, that is to say with the first and the second camera. Likewise preferably, this implementation is also combinable with a movement of the camera, as described above, for example in order to capture image representations of each pose of the marker from different viewing angles (perspectives) by means of the same camera. The at least one camera is preferably a camera of a surgical microscope, for example a main observer camera or a surround camera.
In a further preferred implementation of the method according to the present disclosure, the second position of the tracking point is determined by means of image recognition in the (at least one) captured image representation. This preferably includes detecting the tracking point in the one captured image representation of the combination, continuously tracking the tracking point in a captured sequence of image representations of the combination and/or determining the coordinates of the tracking point in the at least one image representation. In addition to conventional algorithms for image recognition, for example gradient-based algorithms for determining different regions in the image representations, use can also be made here of machine learning algorithms. A person skilled in the art knows of a multiplicity of possible algorithms for detecting the tracking point in the at least one captured image representation, and a detailed description at this juncture is therefore dispensed with. By way of example, known algorithms comprise “SORT—Simple Online And Realtime Tracking”, “DeepSORT”, “FairMOT”, “TransMOT”, “ByteTrack”, “MDNET”, “GOTURN” and “ROLO”.
Likewise preferably, the at least one image representation of the medical marker and of the medical instrument connected therewith, that is to say image representations of both the combination of the aforementioned elements and of the elements on their own, is/are captured by means of a monocular system. Consequently, only a single camera, in principle, is required to carry out the method according to the present disclosure, ensuring the applicability of the method in a multiplicity of systems. Further, a combination of medical marker and tracking point of a medical instrument can be calibrated as soon as the combination is captured by a single camera, for example a surround camera of a surgical microscope. Further preferably, this is a calibrated monocular system, that is to say the extrinsic and/or intrinsic parameters of the monocular system are known. In this case, this relates in particular to intrinsic parameters of Tsai's or Zhang's camera calibration, as described above, and/or extrinsic parameters in relation to a reference coordinate system.
The implementation of the method according to the present disclosure in a monocular system further preferably comprises the capture of the spatial pose of the medical marker on the basis of the captured image representation of same, on the basis of information relating to the geometry of the medical marker and on the basis of an assumed absolute (estimated) size of a dimension of the medical marker, for example the length of the medical marker. As an alternative or in addition, the implementation of the method according to the present disclosure in a monocular system preferably comprises the determination of the first position of the tracking point on the basis of the determined spatial pose of the medical marker, on the basis of information relating to a spatial pose relationship between medical marker and tracking point and an absolute (estimated) size of a dimension of the medical instrument, for example the length of the medical instrument or the like. This advantageously allows, even in a monocular system, the fixation of the otherwise remaining degree of freedom along the optical axis of the monocular system and prevents a scaling error in the medical marker from being transferred as an unknown depth error to the estimation of the position of the tracking point by way of a monocular optimization.
In a further preferred implementation of the method according to the present disclosure, at least one image representation of the medical marker and/or of the medical instrument connected therewith (in particular of the tracking point) is captured by means of a first camera and by means of a second camera. In particular, a multiplicity of image representations of the combination of the medical marker and/or the medical instrument (in particular the tracking point) are captured using only one camera, that is to say using a monocular system, and at least one, preferably exactly one, image representation of the combination of the medical marker and/or the medical instrument (in particular the tracking point) is captured by means of two cameras. This also advantageously allows, in a system used largely in monocular fashion, the fixation of the otherwise remaining degree of freedom along the optical axis of the monocular system and prevents a scaling error in the medical marker from being transferred as an unknown depth error to the estimation of the position of the tracking point by way of a monocular optimization. In particular, when the medical marker is at least intermittently also captured by a second camera, there is an absolute size calibration of the medical marker, preferably by means of the extrinsic calibration of the two cameras to one another. As an alternative or in addition, if the tracking point is at least intermittently also captured by a second camera, for example a second camera of a surgical microscope, the size scale is determined on the basis of at least one absolute position of the tracking point, captured by way of a stereoscopic recording of same.
Likewise preferably, image representations of the medical marker and/or of the medical instrument connected therewith are captured immediately before and after a defined robotic actuation of a capturing camera of a monocular system. Hence, in this way, too, at least one stereoscopic pair of image representations of the medical marker and/or of the medical instrument connected therewith can be obtained, which image representations are extrinsically calibrated to one another as a result of a defined robotic actuation and consequently allow a fixation of the depth information remaining within the scope of the monocular calibration. As an alternative to the robotic actuation of the capturing camera, there is a robotic actuation of the medical marker and/or of the medical instrument connected therewith, between the capture of two image representations of same by means of a capturing camera of a monocular system. Hence, in this way, too, at least one stereoscopic pair of image representations of the medical marker and/or of the medical instrument connected therewith can be obtained.
Further preferably, in the method according to the present disclosure, the image representations of the medical marker and/or of the medical instrument are determined by means of at least one imaging method. Preferably, the same or similar imaging methods are used for all image representations. Preferably, the image representations are captured by means of at least one camera, with a spectral range captured by the image sensor of the camera preferably being dependent upon the utilized illumination source. By way of example, visible light and/or infrared light is used for illumination purposes in a surgical microscope and this is captured by at least one image sensor of a camera. Likewise preferably, x-ray radiation is used for illumination purposes in a computed tomography device and this is captured by at least one sensor of an x-ray detector. Likewise preferably, laser scanning is used to capture the image representations of the medical marker and/or of the medical instrument. Advantageously, the method according to the invention is realizable using a multiplicity of imaging methods and consequently applicable in very different pieces of medical equipment. Further, the same or different (intrinsically and extrinsically calibrated) imaging methods can be used within the scope of the method according to the invention for the purposes of capturing the image representations, as a result of which the method according to the invention further is advantageously versatilely usable and portable.
A further aspect of the present disclosure relates to a medical calibration system comprising an imaging sensor. The imaging sensor preferably is the image sensor of a camera, an x-ray sensor of a computed tomography device or a sensor of a laser scanner. Preferably, the system according to the invention has a multiplicity of imaging sensors, for example image sensors of a first camera and of a second camera. As an alternative or in addition, the system according to the invention comprises a known kinematic system for the defined change in pose of the imaging sensor and/or of the combination of medical marker and instrument. The known kinematic system is preferably the robotic system of a surgical microscope. The known kinematic system is preferably designed to rotate the imaging sensor and/or the combination of medical marker and instrument in a defined manner about the x-axis, the y-axis and the z-axis and to displace these in translational fashion along the x-axis, the y-axis and the z-axis.
The system according to the invention further comprises a medical marker and a medical instrument. By way of example, the medical marker has a geometry as depicted in FIG. 6 . Preferably, the medical marker has a plurality of marker elements of predetermined size, shape and relative pose. The medical marker preferably further comprises a fastening means for establishing a detachable connection to the medical instrument. The marker overall, or at least the marker elements, is/are able to be captured by means of the at least one imaging sensor.
Capturing an image representation of the marker renders the spatial pose of the latter determinable. The medical marker overall, or at least the marker elements, preferably is/are (an) active marker (elements) designed to emit electromagnetic radiation in the infrared, visible and/or ultraviolet spectrum. Likewise preferably, the medical marker overall, or at least the marker elements, is/are (a) passive marker (elements) designed to reflect electromagnetic radiation in the infrared, visible and/or ultraviolet spectrum.
The medical instrument preferably is a medical probe, a pointer, an awl or the like. Preferably, the medical marker and the medical instrument are securely connected to one another or securely connectable to one another, for example by means of a non-detachable connection, an integral connection, a detachable connection, a cohesive connection, a force-fit connection and/or an interlocking connection.
The system according to the invention further comprises a control unit that is connected to the imaging sensor. The system according to the invention further comprises a storage unit connected to the control unit. In this case, the storage unit comprises commands which, upon execution by the control unit, prompt the control unit to carry out the method according to the invention as described above. Expressed differently, the commands are such that the execution thereof prompts the control unit, by means of the imaging sensor, to capture an image representation of a medical marker and of a medical instrument connected therewith, to determine a spatial pose of the medical marker on the basis of its image representation, to determine a first position of a tracking point of the medical instrument on the basis of the determined spatial pose of the medical marker and on the basis of information relating to a spatial pose relationship between medical marker and tracking point, which information has been stored in the storage unit, to determine a second position of the tracking point (only) on the basis of the captured image representation of the medical instrument, to determine a deviation between the first position and the second position and to adjust the stored information relating to the spatial pose relationship between medical marker and tracking point on the basis of the determined deviation (and possibly store or overwrite this information). The storage unit further preferably comprises commands, the execution of which by the control unit prompts the latter to carry out the preferred implementations of the method according to the invention.
The functionalities of the control unit according to the invention can be implemented by electrical or electronic devices or components (hardware), by firmware (ASIC) and/or can be realized by carrying out a suitable program (software). Preferably, the functionalities of the control unit according to the invention are realized or implemented by a combination of hardware, firmware and/or software. By way of example, individual components of the control unit according to the invention for carrying out individual functionalities are in the form of a separately integrated circuit or are arranged on a common integrated circuit.
The individual functionalities of the control unit according to the invention are further preferably in the form of one or more processes which run on one or more processors in one or more electronic computers and which are generated when carrying out one or more computer programs. In this case, the control unit is designed to cooperate with the other components, in particular the imaging sensor, in order to implement the functionalities of the system according to the invention as described herein. It is further evident to a person skilled in the art that the functionalities can be combined by a plurality of computers (data-processing equipment, control units, controllers) or can be combined in a single piece of equipment, or that the functionality of one certain piece of data processing equipment may be available distributed over a plurality of pieces of equipment in order to realize the functionalities of the control unit according to the invention.
In a particularly preferred embodiment of the system according to the invention, the latter is integrated in a surgical microscope. In this case, the surgical microscope preferably comprises at least one imaging sensor and a calibrated kinematic system, with the calibrated kinematic system being designed for the defined rotation (x, y, z) and translation (x, y, z) of a medical instrument and/or at least one of the imaging sensors. According to this embodiment, the reference coordinate system preferably is the coordinate system of the surgical microscope. Further preferably, the imaging sensor is the image sensor of a main observer camera of the surgical microscope. Particularly preferably, a further imaging sensor is the image sensor of a surround camera. Likewise preferably, the known kinematic system is a robotic system of the surgical microscope for guiding a medical instrument, in particular a surgical instrument. The control unit of the surgical microscope is preferably designed as a control unit of the system according to the invention and, in particular, is designed to carry out the method according to the invention, as described above, on the basis of commands stored on a storage unit of the surgical microscope.
Within the scope of the present disclosure, a surgical microscope is understood in the broadest sense to be a microscope suitable for use during an operation. The surgical microscope preferably has a mount which allows imaging of the operating region independently of head movements of the surgeon. Further preferably, the surgical microscope comprises at least one beam splitter and at least two eyepieces. Likewise preferably, the surgical microscope comprises at least one imaging sensor. Further preferably, the surgical microscope comprises a main observer camera and a surround camera. The surgical microscope may comprise kinematic or robotic aids for carrying out surgical interventions. As an alternative, a surgical microscope may be denoted a medical engineering microscope, a medically approved microscope or a medical microscope.
A further aspect of the present disclosure relates to a computer program comprising commands, which when executed by a control unit as described above, preferably a control unit of a surgical microscope or of a system as described above, cause the surgical microscope or the system as described above to carry out the method according to the invention as described above. The computer program preferably comprises commands, which when executed by a control unit as described above, preferably a control unit of a surgical microscope or of a system as described above, cause the surgical microscope or the system as described above to carry out the method according to the invention, in accordance with one of the preferred implementations, as described above. In this case, the computer program according to the invention is preferably stored in a volatile memory, for example a RAM element, or in a non-volatile storage medium, for example a flash memory or the like.
Further preferred embodiments of the invention will become clear from the other features set out in the dependent claims. The various embodiments of the invention that are set forth in this application can advantageously be combined with one another, unless specifically stated otherwise.

DESCRIPTION OF THE FIGURES

The invention is explained below in illustrative embodiments and with reference to the attached drawings, in which:

FIG. 1 shows a schematic representation of a system according to an embodiment;

FIG. 2 shows a schematic representation of a detail of the system from FIGS. 1 and 3 ;

FIG. 3 shows a schematic representation of a system according to a further embodiment;

FIG. 4 shows a schematic representation of a depth calibration of a medical marker and medical instrument;

FIG. 5 shows a schematic representation of a flowchart of a method according to an implementation; and

FIG. 6 shows a schematic representation of a marker for use in the method and the system according to the present disclosure.

FIG. 1 shows a schematic representation of a system 50 according to the present disclosure in one embodiment, the system 50 being part of a surgical microscope 10. Further, the arrangement of FIG. 1 (and FIG. 3 ) has a patient couch 56 with a patient 41 arranged thereon. The patient couch 56 and/or the patient 41 define a coordinate system of the patient 55, which forms a reference coordinate system. The surgical microscope 10 further comprises a piece of medical equipment 42 for holding and guiding a medical instrument 40 by means of a calibrated, more particularly robotic, kinematic system 13. By way of example, the medical instrument 40 is a pointer, an awl, a scalpel, a probe or the like.
A medical marker 20 is arranged on the medical instrument 40. An alignment of the marker 20 defines a coordinate system 53 with respect to the medical marker 20 and/or with respect to the medical instrument 40. The medical marker 20 comprises two marker elements 21, which are able to be captured by means of an imaging sensor and which have a predetermined geometry (in particular size and shape) and relative spatial pose. Further, the marker 20 comprises a fastening clip 22 as fastening means for fastening the marker 20 to the medical instrument 40. Even though the medical marker 20 was produced in accordance with defined specifications in relation to the geometry, the actual geometry of the marker 20 may differ from the specifications on account of production-related variations. In principle, the fastening of the marker 20 to the medical instrument 40 is also well defined but may come loose or slip over time or may deviate from a specification from the outset as a result of inadequate fastening by a user.
The surgical microscope 10 further comprises a first camera 11. The camera 11 is intrinsically and extrinsically calibrated and a coordinate system 51 of the camera 11 is able to be mapped or transformed into the reference coordinate system 55 by means of a parameter set of a rigid body transformation. The camera 11 has a stationary embodiment. Should the camera 11 be designed to be pivotable, the parameter sets thereof are adjusted accordingly.
The surgical microscope 10 further comprises a control apparatus 30 which is connected for data and signal transmission purposes to the camera 11 and the calibrated kinematic system 13, in order to transmit signals to and receive signals from these. The control apparatus 30 comprises a control unit 31, a storage unit 32, a display 33 and a user interface 34 for receiving user inputs. Commands which are carried out when a user selects a corresponding program by way of the user interface 34 are stored in the storage unit 32. The commands subsequently carried out by the control unit 31 are explained in detail below with reference to FIG. 2 .
FIG. 2 shows a schematic representation of a detail of the system 50 from FIG. 1 and FIG. 3 , in particular the portion denoted by “A” in FIGS. 1 and 3 . Moreover, FIG. 2 also illustrates method aspects of the present disclosure.
As already explained, the commands stored in the storage unit 32 cause the control unit 31 to carry out the steps of a method according to the present disclosure. In particular, the commands prompt the control unit 31 to detect the camera 11 for capturing an image representation of the medical marker 20 and of the medical instrument 40 connected therewith.
The control unit 31 is further prompted to determine a spatial pose of the marker 20 on the basis of the captured image representation and preferably on the basis of information, stored in the storage unit 32, relating to a geometry of the marker 20. In this case, this spatial pose for example corresponds to the local coordinate system 53 of the medical marker 20, as depicted in FIGS. 1 and 2 . An origin of the local coordinate system 53 of the medical marker 20 is preferably defined as a characteristic target point 23 of the marker 20.
By way of the commands, the control unit 31 is further caused to determine a first position 61 of a tracking point 43 of the medical instrument 40 on the basis of the determined spatial pose 53 of the medical marker 20 and on the basis of information, stored in the storage unit 32, relating to a spatial pose relationship 64 between medical marker 20 and tracking point 43. According to FIGS. 1 and 2 , the tracking point 43 is a pointer tip of the medical instrument 40 and a first position 61 of the same is given by 3 spatial coordinates (x, y, z). The information relating to the spatial pose relationship 64 between marker 20 and medical instrument 40 is given by a parameter set of a rigid body transformation containing at least 3 parameters regarding translations in the x-, y- and z-direction and preferably 3 parameters regarding rotations about the x-, y- and z-axis. By means of this rigid body transformation, the coordinate system 53 of the medical marker 20 is converted into a coordinate system 54 of the tracking point 43, for example. By way of the commands from the storage unit 32, the control unit 31 is further caused to determine a second position of the tracking point 62 by means of image recognition purely on the basis of the captured image representation of the medical instrument 40. The second position 62 of the tracking point 43 is also given by 3 spatial coordinates (x, y, z). The control unit 31 is further prompted to determine a deviation 63 between the first position 61 and the second position 62. Preferably, an image representation of the medical instrument 40 (optionally including marker 20) is presented on the display 33 by the control unit 31, with further preferably the first position 61, the second position 62 and the deviation 63 or the linear distance between these being presented graphically. This can advantageously inform a user about the required calibration of the pointer tip as tracking point 43.
Finally, the control unit 31 is further caused by way of the commands from the storage unit 32 to adjust the stored information relating to the spatial pose relationship 64 between medical marker 20 and tracking point 43 on the basis of the determined deviation and to store the adjusted information relating to the spatial pose relationship 64 in the storage unit 32. In this case, the information relating to the spatial pose relationship 64 is adjusted in such a way that the deviation 63 between the first position 61 and the second position 62 of the tracking point is minimized.
On the basis of the commands, the control unit 31 will further actuate the camera 11 to capture further image representations of the combination of marker 20 and medical instrument 40 and, using these captured image representations, repeat the steps of determining the spatial pose of the marker 20, the first position 61 of the tracking point 43, the second position 62 of the tracking point 43 and the deviation 63 between the first and second positions 61, 62. In the process, the control unit 31 may use either the respective most up-to-date version of the information relating to the spatial pose relationship 64, that is to say also the information relating to the spatial pose relationship 64 that has already been adjusted on the basis of a previous image representation. However, the control unit 31 can likewise also determine deviations 63 for a plurality of image representations using the same information relating to the spatial pose relationship 64 and carry out a one-time adjustment of this information relating to the spatial pose relationship 64 using the multiplicity of deviations 63 determined in various perspectives. By way of example, image representations from different perspectives totalling n are recorded and spatial nominal transformation from the marker 20 to the tracking point 43 is optimized by virtue of the distance 63 (in this case f) being minimized over the number of available images, for example by means of a cost function cost=Σ_i=1 ⁿf.
Preferably, the control unit 31 presents further image representations of the medical instrument 40 (optionally including the medical marker 20) on the display 33 while the information relating to the spatial pose relationship 64 is adjusted. The presentation can be implemented in discontinuous or continuous fashion in this case and preferably together with the first and second positions 61, 62 and the deviations 63 therebetween. Consequently, the calibration of the tracking point 43 is visualized for a user and the user can recognize that the calibration is complete once the deviation 63 tends to zero and the first and second positions 61, 62 approach one another. By way of example, the user can then terminate the calibration by way of an input by means of the user interface 34. Alternatively, the calibration ends automatically when a termination condition is reached, for example by virtue of a last determined deviation 63 dropping below a predetermined threshold value.
FIG. 3 shows a schematic representation of a system 50 according to the present disclosure in a further embodiment, the system being part of a surgical microscope 10. Inasmuch as the system 50 corresponds to the system 50 described with reference to FIG. 1 , there is no repeated description of the identical components below. Instead, differences in the embodiments are explained.
The system 50 or the surgical microscope 10 of the embodiment in FIG. 3 differs from that in FIG. 1 in that it comprises the camera 11 as a first camera 11 and additionally comprises a second camera 12. The first camera 11 is intrinsically and extrinsically calibrated and a coordinate system 51 of the first camera 11 is able to be mapped or transformed into the reference coordinate system 55 by means of a first parameter set of a rigid body transformation. The second camera 12 is likewise intrinsically and extrinsically calibrated and a coordinate system 52 of the second camera 12 is able to be mapped or transformed into the reference coordinate system 55 by means of a second parameter set of a rigid body transformation. The coordinate system 51 of the first camera 11 is consequently able to be mapped or transformed into the coordinate system 52 of the second camera 12 by means of a third parameter set of a rigid body transformation. The first camera 11 and the second camera 12 have a stationary embodiment. Should the cameras 11, 12 be pivotable, the parameter sets thereof are adjusted accordingly.
In the system 50 of FIG. 3 , an image representation of the medical marker 20 and/or of the medical instrument 40 is consequently able to be captured from a first perspective of the first camera 11 and from a second perspective of the second camera 12. This advantageously allows a depth calibration, as also depicted schematically in FIG. 4 .
As depicted in FIG. 4 , a monocular calibration, as is conceivable in a system of FIG. 1 without absolute length calibration or calibrated spatial pose change of the medical marker (as described above in each case), harbours the risk of a scaling error in the medical marker 20 transferring to the tracking point 42 and hence resulting in an unknown depth error in the estimate of the position of the tracking point 43, that is to say in the z-coordinate of the tracking point 43 in particular. Expressed differently, one degree of freedom remains in the case of a purely monocular calibration, specifically the absolute size scaling of the marker 20, and hence potentially of the tracking point 43 as part of the medical instrument 40. This degree of freedom depicted schematically in FIG. 4 can be fixed in various ways.
In one embodiment, the marker 20 is also observed by a second camera 12, at least intermittently, and there is an absolute size calibration of the marker 20, and hence indirectly also of the tracking point 43 of the medical instrument 40, using the extrinsic calibration of the two cameras 11, 12 with respect to one another. Alternatively, a position of the tracking point 43 itself is captured by a second camera 12, at least intermittently, such that there can be a stereo reconstruction of the tracking point 43. Advantageously, these recordings by the second camera 12 can also be implemented during the use of the medical instrument 40.
A further option for fixing the remaining degree of freedom lies in a defined relative translation of medical instrument 40 relative to the camera 11, for example by way of a positional change of the camera 11 brought about by the calibrated kinematic system 13, and the capture of image representations of the medical instrument 40 before and after the defined translation. Alternatively, the remaining degree of freedom (expressed differently, the scaling) is fixed by estimating a size hypothesis, in particular the length or any other known dimension of the marker or of the medical instrument 40 itself. These two options for size scaling are advantageously also implementable in the system 50 depicted in FIG. 1 .
FIG. 5 shows a schematic flowchart of a method of medical calibration according to the present disclosure in one implementation. In this case, an image representation of a medical marker 20 and of a medical instrument 40 connected therewith is captured in a first step S100. A spatial pose 53 of the medical marker 20 is determined in step S200 on the basis of the captured image representation. A first position 61 of a tracking point 43 of the medical instrument 40 is determined in step S300 on the basis of the determined spatial pose 53 of the medical marker 20 and on the basis of information relating to a spatial pose relationship 64 between medical marker 20 and tracking point 43. A second position 62 of the same tracking point 43 is determined in step S400 on the basis of the captured image representation, preferably only on the basis of the captured image representation and by means of image recognition. Subsequently, a deviation 63 between the first position 61 and the second position 62 is determined in step S500 and finally, in step S600, the information relating to the spatial pose relationship 64 between medical marker 20 and tracking point 43 is adjusted on the basis of the deviation 63 determined in step S500.
FIG. 6 shows a schematic representation of a marker 20 for use in the method and in the system 50. The depicted marker 20 has two circular marker elements 21, which are arranged in two different planes. Each marker element 21 is able to be captured using a camera and has a different colour from the colour of the other marker element 21 for unique recognizability. The medical marker 20 further comprises a fastening clip 22 as a fastening means, for fastening it to a medical instrument 40.

LIST OF REFERENCE SIGNS

10 Surgical microscope
11 First camera
12 Second camera
13 Calibrated kinematic system
20 Medical marker
21 Marker elements
22 Fastening clip
23 Target point
30 Control apparatus
31 Control unit (CPU)
32 Storage unit
33 Display
34 User interface
40 Medical instrument
41 Patient
42 Piece of medical equipment
43 Tracking point
50 System
51 Coordinate system of the first camera
52 Coordinate system of the second camera
53 Coordinate system of the marker
54 Coordinate system of the target point
55 Coordinate system of the patient
56 Patient couch
61 First position of the tracking point
62 Second position of the tracking point
63 Deviation between first and second position
64 Spatial pose relationship between marker and target point
S100 Capturing an image representation
S200 Determining a spatial pose of the marker
S300 Determining a first position of a tracking point
S400 Determining a second position of a tracking point
S500 Determining a deviation between first and second position
S600 Adjusting information relating to the spatial pose relationship between marker and tracking point

Claims

1. A method of medical calibration, comprising the method steps of:

capturing an image representation of a medical marker and of a medical instrument connected therewith;

determining a spatial pose of the medical marker on the basis of the image representation;

determining a first position of a tracking point of the medical instrument on the basis of the determined spatial pose and on the basis of information relating to a spatial pose relationship between medical marker and tracking point, and determining a second position of the tracking point on the basis of the captured image representation;

determining a deviation between the first position and the second position; and

adjusting the information relating to the spatial pose relationship between medical marker and tracking point on the basis of the determined deviation.

2. The method according to claim 1, further comprising the method steps of:

capturing a multiplicity of image representations of the medical marker and of the medical instrument connected therewith;

determining, for each of the captured image representations, the first position and the second position of the tracking point in the captured image representation and a deviation between the first position and the second position; and

adjusting the information relating to the spatial pose relationship between medical marker and tracking point on the basis of the multiplicity of determined deviations.

3. The method according to claim 1, wherein calibrated information relating to the spatial pose relationship between medical marker and tracking point is available if a deviation between the first position and the second position is determined as being below a predetermined threshold value.

4. The method according to claim 3, further comprising a continuous determination of first positions of the tracking point on the basis of continuously captured image representations of the medical marker and of the medical instrument connected therewith and on the basis of the calibrated information relating to the spatial pose relationship between medical marker and tracking point.

5. The method according to claim 1, further comprising:

intermittently determining second positions of the tracking point in a selection of continuously captured image representations of the medical marker and of the medical instrument connected therewith;

determining deviations between the first position and the second position of the tracking point in the selection of continuously captured image representations; and

adjusting the information relating to the spatial pose relationship between medical marker and tracking point if one of the determined deviations exceeds a predetermined limit value.

6. The method according to claim 1, wherein the medical marker has a geometry and the spatial pose of the medical marker is determined on the basis of the captured image representation of the medical marker and on the basis of information relating to the geometry of the medical marker.

7. The method according to claim 6, wherein the image representation of the medical marker contains image representations of at least three marker elements, the information relating to the geometry of the medical marker including information relating to the relative spatial pose of the marker elements with respect to one another and determining the spatial pose of the medical marker including the determination of the spatial poses of the marker elements.

8. The method according to claim 7, wherein the information relating to the spatial pose relationship between medical marker and tracking point defines a relative spatial pose of the tracking point with respect to the marker elements and/or with respect to a target point that has a fixed spatial relationship with respect to the marker elements.

9. The method according to claim 6, wherein the information relating to the geometry of the medical marker is predetermined.

10. The method according to claim 6, further comprising:

capturing a first image representation of the medical marker in a first perspective;

capturing a second image representation of the medical marker in a second perspective; and

determining the information relating to the geometry of the medical marker on the basis of the first image representation and the second image representation.

11. The method according to claim 1, wherein the second position of the tracking point is determined by means of image recognition and/or a machine learning algorithm in the captured image representation.

12. The method according to claim 1, wherein the image representations of the medical marker and of the medical instrument connected therewith are captured by means of a monocular system.

13. The method according to claim 1, wherein at least one image representation of the medical marker and of the medical instrument connected therewith is captured by means of a first camera and a second camera, or image representations of the medical marker and of the medical instrument connected therewith are captured immediately before and after a defined robotic actuation of a capturing camera or of the medical marker and of the medical instrument connected therewith.

14. A medical calibration system comprising:

an imaging sensor;

a medical marker;

a medical instrument;

a control unit connected to the imaging sensor; and

a storage unit connected to the control unit and comprising commands which, upon execution by the control unit, cause the latter to carry out the method according to claim 1.

15. A computer program comprising commands which, upon execution by the control unit of a system according to claim 14, cause the system according to claim 14 to carry out the method according to claim 1.