DE102016117263B4 - Optical observation device system - Google Patents

Abstract

Optical observation device system (68), with:- an optical observation device (2) for generating an image of an observation area (1) with an object of interest (6) and image data representing the image;- an interface arrangement (63) with a data output (63A) for outputting the image data to a registration unit (61), which generates registration data representing a registration of depth image data with the image data, a data input (63B) for receiving the depth image data of the object of interest (6), for receiving topography data associated with the image data from a topography detection unit (60) and for receiving the registration data from the registration unit (61);- an evaluation unit (62) which is designed to use the topography data, the depth image data and the registration data to determine the distances of the depth regions of the object of interest (6) recorded by the depth image data from the surface (12) of the observation area (1) and to output the determined distances,- a data input (63B) connected to the evaluation unit (62) a visualization unit (64) connected to receive the distances, wherein the visualization unit (64) is designed to generate a visualization image in which the received distances are shown and to output the visualization image for display on a display unit (37, 65, 67).

Description

The present invention relates to an optical observation device system, for example a surgical microscope system, and in particular to a surgical microscope system for neurosurgical applications. The invention further relates to a display method for displaying depth information, for example in a surgical microscope, in particular in a surgical microscope for neurosurgical applications.

Tumor resection using a surgical microscope represents a general challenge in surgery. Especially when resectioning in functional areas, the surgeon must know the exact three-dimensional morphology of the tumor in order to be able to choose the most precise cutting boundary between functional tissue and malignant tissue during an intervention.

With currently available navigation solutions, the tumor margin, which is calculated based on pre-operative data, for example from MRI examinations, is only displayed for the current focal plane or focal layer or individual adjacent layers through the eyepiece in the surgical microscope.

For a better representation and recognition of the three-dimensional tumor morphology, the surgeon must either recall the corresponding morphology from memory or give up his usual view through the eyepiece of the surgical microscope during the intervention in order to look at an external visualization unit. Both the previously learned morphology and the brief look at an external visualization unit pose risks for the patient because the flow of the operation is interrupted.

Out of US 2001/0027272 A1 A navigation device for navigating the tip of an endoscope towards a target object is known. The target object can be marked with navigation-related information on a monitor image, with the navigation-related information in the form of a wire model being superimposed on an image of the operating area shown on a display. When the endoscope reaches its target location, the wire model can be replaced by a topographical image based on the three-dimensional data. However, contour lines or the like can also be used instead of the wire model.

Out of WO 03/105709 A1 A navigation system is known which offers the possibility of superimposing preoperative diagnostic data on the optical image of a surgical microscope.

Out of WO95/25979 A1 A surgical microscope is known with which registered image data can be superimposed on an image of the surgical field. It should be possible to select a plane, for example the focal plane, and to visualize the image data within this plane and on the other side of this plane.

Out of EN 10 2005 050 918 A1 A device for monitoring microsurgical interventions using a surgical microscope is known. Using an image display unit, 3D image data and the image obtained with the surgical microscope are visualized in full or in part next to each other or superimposed on each other in the correct position.

Out of EN 10 2011 085 176 A1 A method for generating an angiogram of vessels is known in which a 3D image data set of the vessels is determined, a 2D image data set is determined by means of a camera and the 3D image data set is superimposed on the 2D image data set in the correct position and dimensions.

The EN 10 2014 107 443 A1 describes a microscope system with depth preview.

The EN 10 2012 220 115 A1 and the EN 10 2006 017 003 A1 address the representation of depth data.

This describes DE 2014 210 121 A1 describes a 3D sensor for collecting topography data

It is an object of the present invention to provide an advantageous optical observation device system and an advantageous display method for displaying depth information.

The above-mentioned object of the present invention is achieved by an optical observation device system according to claim 1 and by a display method according to claim 13. The dependent claims contain further advantageous embodiments of the invention.

• - Ein optisches Beobachtungsgerät zum Erzeugen eines Bildes eines Beobachtungsbereiches mit einem interessierenden Objekt und von das Bild repräsentierenden Bilddaten.
• - Eine Schnittstellenanordnung mit einem Datenausgang zum Ausgeben der Bilddaten an eine Registrierungseinheit, welche Registrierungsdaten generiert, die eine Registrierung von Tiefenbilddaten mit den Bilddaten repräsentieren.

An optical observation device system according to the invention comprises:

• - An optical observation device for producing an image of an observation area including an object of interest and image data representing the image.
• - An interface arrangement with a data output for outputting the image data to a registration unit which generates registration data representing a registration of depth image data with the image data.

Depth image data can be, for example, 3D images of the object of interest obtained using magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET) or other tomography methods, or depth data obtained from such 3D images for the object of interest.

The registration unit can be part of the optical observation device system, or it can be an external registration unit, e.g. as part of an external navigation system. It is connected to the data output to receive the image generated with the optical observation device and is designed to register the depth image data and the image generated with the optical observation device with each other and to output corresponding registration data. As part of an image registration, one of the images to be registered is used as a reference image and the other image is adapted to the reference image by means of a transformation in its position and orientation (rigid transformation) and/or in its scaling (non-rigid transformation) in such a way that the best possible match of the image contents is achieved. Even if movements of the object of interest or within the object of interest can occur (such as the so-called brain shift if the object of interest is a section of the brain), it is advantageous to carry out the registration using a non-rigid transformation. For example, to take the brain shift into account, marker elements (so-called fiducial markers) can be applied to the relevant brain section, which are shown in both the depth image data and the image data. As part of a non-rigid registration, the marker elements shown in the depth image data can then be made to coincide with the marker elements shown in the image data.

In addition, the interface arrangement comprises a data input for receiving the depth image data of the object of interest, for receiving topography data associated with the image data from a topography detection unit and for receiving the registration data from the registration unit. The data input can be implemented by a single data input used to receive all of the aforementioned data or by separate data inputs, with each data input only receiving certain of the aforementioned data.

The topography detection unit can be part of the optical observation device system or be designed as an external device. It can, for example, comprise a stereoscopic sensor and/or a laser scanner and/or a sensor for measuring the time of flight of light pulses, such as a time-of-flight camera (TOF camera), and/or a device for structured illumination. If the topography detection unit is part of the optical observation device system, the stereoscopic sensor can, for example, comprise two cameras integrated into the optical observation device. The topography detection unit can, however, also be based on one of the following topography methods: DFD (depth from defocus), in which the distance from the focal plane is determined from the degree of defocus, SFM (structure from motion), SLAM (simultaneous localization and mapping).

• - Eine Auswertungseinheit, die dazu ausgebildet ist, anhand der Topografiedaten, der Tiefenbilddaten und der Registrierungsdaten die Abstände der von den Tiefenbilddaten erfassten Tiefenbereiche des interessierenden Objekts von der Oberfläche des Beobachtungsbereiches zu ermitteln und die ermittelten Abstände auszugeben. Es werden somit relative Abstände ermittelt und ausgegeben. Die Auswertungseinheit kann eine speziell für den beschrieben Zweck konstruierte Einheit sein bspw. in Form einer anwendungsspezifischen integrierten Schaltung (ASIC, Application Specific Integrated Circuit), oder in Form von Software, die auf einem Prozessor des optischen Beobachtungsgerätsystems oder auf einem Computer, bspw. auf einem handelsüblichen PC, läuft.
• - Eine mit der Auswertungseinheit zum Empfang der Abstände verbundene Visualisierungseinheit. Die Visualisierungseinheit ist dazu ausgebildet, ein Visualisierungsbild zu generieren, in dem die empfangenen Abstände dargestellt sind. Außerdem ist die Visualisierungseinheit dazu ausgebildet, das Visualisierungsbild zur Anzeige auf einer Anzeigeeinheit auszugeben. Die Anzeigeeinheit kann dabei Teil des optischen Beobachtungsgerätesystems oder eine externe Anzeigeeinheit sein.

In addition, an optical observation device system according to the invention comprises:

• - An evaluation unit which is designed to use the topography data, the depth image data and the registration data to determine the distances of the depth areas of the object of interest recorded by the depth image data from the surface of the observation area and to output the determined distances. Relative distances are thus determined and output. The evaluation unit can be a unit specially designed for the described purpose, for example in the form of an application-specific integrated circuit (ASIC), or in the form of software which runs on a processor of the optical observation device system or on a computer, for example on a commercially available PC.
• - A visualization unit connected to the evaluation unit for receiving the distances. The visualization unit is designed to generate a visualization image in which the received distances are shown. In addition, the visualization unit is designed to output the visualization image for display on a display unit. The display unit can be part of the optical observation device system or an external display unit.

The visualization unit, like the evaluation unit, can be a unit specially designed for the described purpose, e.g. in the form of an application-specific integrated circuit or in the form of software that runs on a processor of the optical observation device system or on a computer, e.g. on a standard PC. In particular, the software for the evaluation unit and the software for the visualization unit can run on the same processor or computer. In principle, the visualization of the distances can be done either as a transparent or as an opaque overlay. The distances can also be displayed only locally so as not to unnecessarily restrict the field of view of the person examining, e.g. an operator.

The visualization image can be displayed, for example, in the view of the optical observation device. If the view is formed by at least one eyepiece, the display unit can comprise at least one display arranged in the optical observation device, with which an overlay image can be displayed in the optical beam path of the optical observation device. The visualization image then only needs to show the distances and be displayed as an overlay image in the beam path. The distances are then superimposed on the image of the object being observed in the eyepiece. If the display is to take place in an electronic view of the optical observation device or on an external display, the visualization unit can also be connected to the optical observation device to receive the image data representing the image of the observation area. In addition, it can then be designed to generate a visualization image in which both the received image of the observation area and the received distances are displayed. In other words, an electronic overlay of the image data and the received distances can take place. The visualization image, which represents both the image data and the distances in this way, can then be displayed as a display unit in an electronic viewfinder or in an external display of the optical observation device, such as a head-mounted display (HMD).

The optical observation device can be, for example, a surgical microscope or an endoscope, and the observation area can then be tissue, in particular tissue with a tumor as the object of interest. The registration unit is then used to register, for example, depth image data obtained from the tumor preoperatively (e.g. using CT or MRI) and the image created with the surgical microscope or endoscope, and thus to register the depth image data of the tumor to the current observation area or the current surgical scene.

The problem of the inadequate visualization of a tumor during a surgical procedure described above in connection with tumor resection can be solved with the aid of the optical observation device system according to the invention equipped with a surgical microscope or an endoscope as an optical observation device by displaying the distances in the visualization image. The entire morphology of the tumor can thus be displayed, for example, in the form of depth lines (in particular iso-depth lines, so-called isobaths), each of which represents a specific depth below the surface of the observation area and thus a relative distance to the surface. The visualization unit is then designed to display the received distances in the visualization image as depth lines, each of which represents a specific depth below the surface of the observation area. The depth lines can then be displayed directly in the optical observation device, for example in the eyepiece of a surgical microscope, whereby the lines can be displayed as solid, dashed, dotted or in any combination thereof. Additionally or alternatively, the visualization unit can be designed to mark certain depth ranges below the surface of the observation area in color and/or in different shades of gray. For example, individual characteristic depth ranges can be coded in color or with different shades of gray for better differentiation.

When using the invention, the surgeon does not have to give up his usual view through the eyepiece of the surgical microscope in order to experience the depths of the tumor beneath the tissue surface. He also does not have to rely on a previously learned morphology. This improves the flow of the operation and reduces the above-mentioned risks for the patient. Any form of blending of depth lines or opaque representation with the current surgical scene or the observation area is also conceivable in order to enable a realistic "look and feel". Advantageously, different visualization parameters can be selected for visible and non-visible areas of the observation area.

Overall, the optical observation device system according to the invention has the advantage that it enables an improved representation of the overall morphology of an observation object within an observation area, for example an improved representation of the overall morphology of malignant tissue within a surgical scene.

• - mittels des optischen Beobachtungsgeräts, das bspw. ein Operationsmikroskop oder ein Endoskop sein kann, ein Bildes des Beobachtungsbereiches und das Bild repräsentierenden Bilddaten erzeugt werden,
• - mittels einer Topografie-Erfassungseinheit aktuelle, den Bilddaten zugeordnete Topografiedaten des Beobachtungsbereiches erfasst werden,
• - mittels einer Registrierungseinheit, die zur rigiden oder nicht rigiden Registrierung ausgelegt sein kann, die Bilddaten und von der Registrierungseinheit empfangene Tiefenbilddaten des interessierenden Objekts miteinander registriert werden und entsprechende Registrierungsdaten generiert werden,
• - mittels einer Auswertungseinheit anhand der Topografiedaten, der Tiefenbilddaten und der Registrierungsdaten die Abstände der von den Tiefenbilddaten erfassten Tiefenbereiche des interessierenden Objekts von der Oberfläche des Beobachtungsbereiches ermittelt werden,
• - mittels einer Visualisierungseinheit ein Visualisierungsbild generiert wird, in dem die empfangenen Abstände dargestellt sind, und
• - das Visualisierungsbild mittels einer Anzeigeeinheit angezeigt wird.

According to the invention, a display method for displaying depth information to an object of interest in an observation area observed with an optical observation device, e.g. a tissue area with a tumor as the object of interest, in which

• - an image of the observation area and image data representing the image are generated by means of the optical observation device, which can be, for example, a surgical microscope or an endoscope,
• - using a topography acquisition unit, current topography data of the observation area associated with the image data are recorded,
• - by means of a registration unit, which can be designed for rigid or non-rigid registration, the image data and depth image data of the object of interest received by the registration unit are registered with each other and corresponding registration data are generated,
• - using an evaluation unit based on the topography data, the depth image data and the registration data, the distances of the depth areas of the object of interest recorded by the depth image data from the surface of the observation area are determined,
• - a visualization image is generated by means of a visualization unit in which the received distances are shown, and
• - the visualization image is displayed by means of a display unit.

The received distances can be shown in the visualization image as depth lines, for example, each of which represents a certain depth below the surface of the observation area. In addition, certain depth ranges below the surface of the observation area can be marked in color. For example, areas that lie between two neighboring depth lines can be colored.

The visualization image can, for example, be displayed in the beam path of the optical observation device using at least one display, so that the distance information about the distances of different areas of the object of interest from the surface of the observation area is superimposed on the image of the observation area in the eyepiece. The visualization in the eyepiece of the surgical microscope can, for example, be carried out as a perspective-correct overlay or as a picture in a picture (PiP - Picture in Picture) at the edge of the field of view in order to minimize the covering of relevant image information. The visualization unit can also generate a visualization image in which both the image of the observation area and the distances are shown, so that an electronic overlay of the image data and the received distances can be shown in an electronic view of the optical observation device or an external display, e.g. an HMD.

In the display process, the depth image data can be obtained using a tomography process, for example. The depth image data can also be obtained before the display process is carried out. If the display process is used as part of an operation, the depth image data can be obtained preoperatively using the tomography process, for example.

To register the image data with the depth image data, either a registration unit provided specifically for carrying out the process or an external registration unit such as a registration unit of a navigation system can be used within the scope of the display process.

The display method according to the invention can be carried out using the optical observation device system according to the invention described above. It therefore basically has the same advantages as the optical observation device system according to the invention described above. By including current topography information, the depth information and its visualization can be adapted to the current conditions. In addition, if the optical observation device is a surgical microscope or an endoscope and the observation area with the object of interest is a tissue area with a tumor, intraoperative image data can be used to compensate for a possible geographical deviation from pre- and intraoperative data for the visualization of the depth information. Various image processing algorithms and/or different lighting modes and/or markers, e.g. contrast agents, can be used for this purpose.

In principle, the depth image data can be obtained using a navigation device. Specifically, the depth information can be recorded using MRI (magnetic resonance imaging), CT (computer tomography) or similar. The navigation device can segment the data, for example assigning bone or tumor tissue to a tissue area, and can deliver it either as raw data or already mathematically corrected to the optical axis of the optical observation device. In one form of this, the navigation device provides the entire depth image data of the observation area via an interface for the evaluation unit. In another variant, the optical observation device sends "virtual depths" (distance of the adjustable image focus relative to the optical observation device) in a defined range of values to the navigation device based on the current focus plane. This then sends the respective contour of the depth image data to the evaluation unit for the received virtual depths.

• - Registrieren der Bilddaten und der Tiefenbilddaten des interessierenden Objekts und Generieren entsprechender Registrierungsdaten,
• - Ermitteln der Abstände der von den Tiefenbilddaten erfassten Tiefenbereiche des interessierenden Objekts von der Oberfläche des Beobachtungsbereiches anhand der Topografiedaten, der Tiefenbilddaten und der Registrierungsdaten und
• - Generieren und Ausgeben eines Visualisierungsbildes, in dem die Abstände dargestellt sind.

The invention also provides a computer program product for displaying depth information about an object of interest using image data, depth image data and topography data of the object of interest. The computer program product includes stored instructions for performing the following steps:

• - Registering the image data and the depth image data of the object of interest and generating corresponding registration data,
• - Determining the distances of the depth ranges of the object of interest captured by the depth image data from the surface of the observation area using the topography data, the depth image data and the registration data and
• - Generate and output a visualization image showing the distances.

• 1 zeigt schematisch den Aufbau eines Operationsmikroskops.
• 2 zeigt schematisch Beispiel für ein Vario-Objektiv.
• 3 zeigt schematisch eine chirurgische Operationsszene mit Tiefenvorschaukarte.
• 4 zeigt schematisch die chirurgische Operationsszene aus 3 in einer Schnittansicht.
• 5 zeigt schematisch ein Operationsmikroskop-System als ein Ausführungsbeispiel für ein erfindungsgemäßes optisches Beobachtungsgerätsystem.

Further features, properties and advantages of the present invention are described in more detail below using exemplary embodiments with reference to the attached figures. All features described so far and below are advantageous both individually and in any combination with one another. The exemplary embodiments described below serve only to explain the invention and are not intended to limit the subject matter of the invention to the individual exemplary embodiments.

• 1 shows schematically the structure of a surgical microscope.
• 2 shows a schematic example of a varifocal lens.
• 3 shows a schematic of a surgical operation scene with depth preview map.
• 4 shows schematically the surgical operation scene from 3 in a sectional view.
• 5 shows schematically a surgical microscope system as an embodiment of an optical observation device system according to the invention.

The following is based on the 1 and 2 An example of the basic structure of a surgical microscope 2 is explained.

This in 1 The surgical microscope 2 shown comprises as essential components an objective 5 which is to be directed towards an object field 3 and which can be designed in particular as an achromatic or apochromatic objective. In the present exemplary embodiment, the objective 5 consists of two partial lenses which are cemented together and form an achromatic objective. In the case of an apochromatic objective, this consists of three partial lenses. The object field 3 is arranged in the focal plane of the objective 5 so that it is imaged by the objective 5 towards infinity. In other words, a divergent beam 7 emanating from the object field 3 is converted into a parallel beam 9 as it passes through the objective 5.

A magnification changer 11 is arranged on the observer side of the lens 5, which can be designed either as a zoom system for continuously changing the magnification factor, as in the embodiment shown, or as a so-called Galilean changer for gradually changing the magnification factor. In a zoom system that is made up of a lens combination with three lenses, for example, the two lenses on the object side can be moved to vary the magnification factor. In fact, however, the zoom system can also have more than three lenses, for example four or more lenses, in which case the outer lenses can also be fixed. In a Galilean changer, on the other hand, there are several fixed lens combinations that represent different magnification factors and can be alternately introduced into the beam path. Both a zoom system and a Galilean changer convert a parallel beam on the object side into a parallel beam on the observer side with a different beam diameter. In the present embodiment, the magnification changer 11 is already part of the binocular beam path of the surgical microscope 2, i.e. it has its own lens combination for each stereoscopic partial beam path 9A, 9B of the surgical microscope 2. In the present embodiment, the setting of a magnification factor by means of the magnification changer 11 takes place via a motor-driven actuator (not shown), which together with the magnification changer 11 is part of a magnification change unit for setting the magnification factor.

The magnification changer 11 is connected to an interface arrangement 13A, 13B on the observer side, via which external devices can be connected to the Opera tion microscope 2 and which in the present embodiment comprises beam splitter prisms 15A, 15B. In principle, however, other types of beam splitters can also be used, e.g. partially transparent mirrors. In the present embodiment, the interfaces 13A, 13B serve to decouple a beam from the beam path of the surgical microscope 2 (beam splitter prism 15B) or to couple a beam into the beam path of the surgical microscope 2 (beam splitter prism 15A).

In the present exemplary embodiment, the beam splitter prism 15A in the partial beam path 9A serves to reflect information or data for a viewer into the partial beam path 9A of the surgical microscope 1 via the beam splitter prism 15A with the aid of a display 37, for example a digital mirror device (DMD) or an LCD display, and an associated optics 39. In the other partial beam path 9B, a camera adapter 19 with a camera 21 attached to it is arranged at the interface 13B. The camera 21 can be used to record an electronic and in particular a digital image of the tissue region 3. In particular, a hyperspectral sensor can also be used as the image sensor, in which there are not just three spectral channels (for example red, green and blue), but a large number of spectral channels.

In the example shown, a binocular tube 27 is connected to the interface 13 on the observer side. This has two tube lenses 29A, 29B, which focus the respective parallel beam 9A, 9B on an intermediate image plane 31, i.e. image the object being observed 3 on the respective intermediate image plane 31A, 31B. The intermediate images in the intermediate image planes 31A, 31B are finally imaged to infinity by eyepiece lenses 35A, 35B, so that an observer can view the intermediate image with a relaxed eye. In addition, the distance between the two partial beams 9A, 9B is increased in the binocular tube by means of a mirror system or prisms 33A, 33B in order to adapt this to the distance between the observer's eyes. The mirror system or prisms 33A, 33B are also used to erect the image. Instead of viewing the intermediate images using eyepieces, it is also possible to record the intermediate images using image sensors arranged in the intermediate image plane, which can be designed as CCD sensors or CMOS sensors, for example, and to display the recorded images electronically on displays. The electronic display allows the viewer to position himself more freely relative to the object being observed. The displays can be arranged in a viewing area on the surgical microscope, for example, or they can be arranged in a head-mounted display (HMD) that is detached from the surgical microscope, such as data glasses.

The surgical microscope 2 is also equipped with an illumination device with which the object field 3 can be illuminated with mostly broadband illumination light. For this purpose, the illumination device in the present embodiment has a white light source 41, such as a halogen lamp or a gas discharge lamp. However, luminescence emitters such as light-emitting diodes (LEDs) or organic light-emitting diodes (OLEDs) can also be used as light sources. The white light can also be generated by a combination of several colored luminescence emitters. The light emitted by the white light source 41 is directed towards the object field 3 via a deflection mirror 43 or a deflection prism in order to illuminate it. The illumination device also has an illumination optics 45, which ensures uniform illumination of the entire observed object field 3.

It should be noted that the 1 The illumination beam path shown is highly schematic and does not necessarily reflect the actual course of the illumination beam path. In principle, the illumination beam path can be designed as so-called oblique illumination, which corresponds to the schematic representation in 1 In such an oblique illumination, the beam path runs at a relatively large angle (6° or more) to the optical axis of the objective 5 and can, as in 1 shown, run completely outside the lens. Alternatively, however, it is also possible to let the illumination beam path of the oblique illumination run through an edge region of the lens 5. Another possibility for arranging the illumination beam path is the so-called 0° illumination, in which the illumination beam path runs through the lens 5 and is coupled into the lens between the two partial beam paths 9A, 9B along the optical axis of the lens 5 in the direction of the object field 3. Finally, there is also the possibility of designing the illumination beam path as so-called coaxial illumination, in which a first and a second partial illumination beam path are present. The partial beam paths are coupled into the surgical microscope via one or more beam splitters parallel to the optical axes of the partial observation beam paths 9A, 9B, so that the illumination runs coaxially to the two partial observation beam paths.

In the 1 The illumination can be influenced in the surgical microscope shown. For example, a filter 47 can be introduced into the illumination beam path, which allows only a narrow spectral range of the broad spectrum of the white light source 41 to pass through, for example a spectral range with which the fluorescence of a fluorescent dye located in the object field 3 can be excited. To observe the fluorescence, filters 37A, 37B can be introduced into the observation partial beam paths, which filter out the spectral range used for fluorescence excitation in order to be able to observe the fluorescence.

The lighting device can also be equipped with a unit for changing the lighting light source. This is 1 by a system for replacing the white light source 41 with a laser 49. With a laser as a light source, in particular with an infrared laser, in conjunction with a suitable image sensor 23, for example, laser Doppler imaging or laser speckle imaging is made possible. In the present embodiment, the unit for changing the illumination light source is motor-driven and can be controlled by suitable control data from the pathology unit 70

In the 1 In the variant of the surgical microscope 2 shown, the objective 5 consists only of an achromatic lens with a fixed object focal length (distance of the object-side focal plane from the vertex of the first object-side lens surface of the objective 5). However, an objective lens system comprising several lenses can also be used, in particular a so-called varifocal objective, with which the working distance of the surgical microscope 2, ie the object focal length, can be varied. The object field 3 arranged in the focal plane is also imaged to infinity by the varifocal objective 50, so that a parallel beam of rays is present on the observer side.

An example of a varifocal lens is shown schematically in 2 The zoom lens 50 comprises a positive element 51, i.e. an optical element with positive refractive power, which is 2 schematically shown as a convex lens. In addition, the zoom lens 50 comprises a negative element 52, i.e. an optical element with negative refractive power, which is 2 is shown schematically as a concave lens. The negative element 52 is located between the positive element 51 and the object field 3. In the zoom lens 50 shown, the negative element 52 is fixed, whereas the positive element 51 is arranged so as to be displaceable along the optical axis OA, as indicated by the double arrow 53. When the positive element 51 is in the 2 If the position shown in dashed lines is moved, the focal length is extended, so that the working distance of the surgical microscope 2 from the object field 3 changes.

Although in 2 the positive member 51 is designed to be movable, it is also possible in principle to arrange the negative member 52 so that it can move along the optical axis OA instead of the positive member 51. However, the negative member 52 often forms the end lens of the zoom lens 50. A fixed negative member 52 therefore offers the advantage that the interior of the surgical microscope 2 can be sealed more easily against external influences. It should also be noted that, although the positive member 51 and the negative member 52 are in 2 are only shown as individual lenses, each of these elements can also be realized in the form of a lens group or a cemented element instead of in the form of an individual lens, e.g. in order to make the zoom lens achromatic or apochromatic.

The 3 shows a schematic of a surgical, for example a neurosurgical, operation scene 1. The observation area 1 or the operation scene microscopically displayed by the surgical microscope 2 comprises tissue 12 to be removed, for example tumor tissue to be removed as part of a tumor resection. The surgeon's fingers are identified by the reference number 8, the surgical instruments used by the reference number 4. During the operation, a tumor depth preview map 10 is superimposed on the image of the observation area 1 obtained with the surgical microscope 2 using the display method according to the invention. This is done in the present case using iso-depth lines and/or by color coding. The iso-depth lines do not represent a specific depth in relation to a focal plane, but rather a relative distance in relation to the surface of the tissue area 12, as is shown in 4 shown in a schematic side view of the operation scene 1.

The 4 shows the surface of the tissue 12 as well as the tumor 6 and the tumor depth preview map 10. The iso-depth lines shown in the map each represent the depth of a specific area of the tumor 6 below the surface of the tissue 12, as indicated by the arrows in the figure. Also shown in 4 the focal plane 16 of the surgical microscope 2, whereby the position of the focal plane 16 corresponds to the position of the floor 17 of the surgical site. For explanatory purposes, the tumor 6 is in 4 shown symmetrically.

As in 4 can be seen, the tumor depth preview map 10, whose iso-depth lines 18 show the respective distances of the tumor from the surface of the tissue 12, do not show the symmetry of the tumor. It is therefore significantly different from a depth preview map, which shows the respective distances of the tumor from the focal plane 6 with its iso-depth lines. Such a tumor depth preview map would in 4 reflect the symmetry of the tumor. The main difference between the two tumor depth preview maps is not the symmetry consideration, but rather that the iso-depth lines can show the surgeon immediately how deep an incision should be made into the tissue. 4 shown depth preview map, whose iso-depth lines 18 represent the distances from the surface of the tissue 12, the treating surgeon can directly see how deep he has to cut into the tissue 12. If, on the other hand, the distance of the tumor from the focal plane 16 were shown in the depth preview map in the form of iso-depth lines, the surgeon would only be able to read the incision depth directly from the iso-depth lines 18 for those tissue areas that coincide with the focal plane 16 and which, in the present example, are formed by the floor 17 of the operation site. For other tissue areas, the focal plane would have to be shifted in order to then be able to read the incision depth from the newly generated iso-depth lines. The present invention therefore offers the advantage that even without shifting the focal plane 16, all iso-depth lines of the tumor depth preview map 10 represent the required incision depth, so that the surgeon is provided with an immediate aid for deciding on the respective incision depths.

Instead of the iso-depth lines, other techniques can also be used to represent the depth of the tumor below the surface of the tissue 12. In particular, techniques from the field of augmented reality can be used to fuse image data. This enables a seamless and realistic display of distance information in the surgical scene.

5 shows as an embodiment of the optical observation device system according to the invention a surgical microscope system 68 with which the 4 described tumor depth preview map can be generated and displayed on a display unit. Instead of the depth preview map, any other suitable display of the depth information, ie the distances of the tumor from the surface of the tissue 12, can be realized with the device.

The surgical microscope system 68 comprises a surgical microscope 2, with which a visual image of a surgical scene, ie an observation area 1 with a tumor 6, is obtained. The surgical microscope 2 also generates electronic image data representing the image and outputs it. To generate the electronic image data, for example, the camera 21 can be made of 1 find use.

The surgical microscope system 68 also includes an interface arrangement 63 with a data output 63A and a data input 63B. Image data representing the image obtained with the surgical microscope 2 are output to a registration unit 61 via the data output 63A. Current topography data of the observation area 1 from a topography detection unit 60 and depth image data of the tumor 6 in the observation area 1 are received from a tomography system 66 via the data input 63B.

The topography detection unit 60 can be based on a suitable sensor, for example a stereoscopic sensor, a laser scanner, a time-of-flight sensor or structured lighting. A stereoscopic sensor can, for example, be two cameras integrated in the surgical microscope 2, similar to the camera 21 from 1 By including current topography information, the depth information and its visualization can also be adapted to the current conditions. In particular, occurring deformations can be taken into account.

In the present embodiment, the depth image data are 3D data that have been obtained from the position of the tumor 6 below the tissue 12 by means of a tomography method.

The depth image data generated by the tomography system 66 are also present in the registration unit 61. By means of the registration unit 61, the 3D data obtained preoperatively are registered with the image data obtained from the data output 63A of the surgical microscope system 68 with the aid of a suitable rigid or non-rigid transformation, so that there is an optimal match between the 3D data and the image data. The registration unit 61 then outputs registration data, which represent the registration of the 3D data with the image data, which are received by the data input 63B of the interface arrangement 63.

An evaluation unit 62 is connected to the interface arrangement 63, to which the interface unit 63 forwards the received registration data, the received topography data of the observation area 1 and the received depth image data. The depth image data can provide the evaluation unit 62 with complete 3D information. of the tumor 6. Alternatively, it is possible for the surgical microscope system 68 to send “virtual depths” in a defined value range to a navigation device based on the current focal plane, and the navigation device then outputs the respective contour line of the tumor 6 as depth image data to the evaluation unit 62 for the sent “virtual depths”.

The evaluation unit 62 determines the distances of the depth ranges recorded by the 3D data from the tissue surface 12 on the basis of the topography data, the 3D data of the tumor 6 and the registration data provided by the registration unit 61 and outputs the determined distances. Due to the registration, the positions of the tumor 6 in the tissue 12 and the topography of the surface of the tissue 12 are known in the same coordinate system, so that the distances of the tumor from the surface of the tissue can be calculated.

A visualization unit 64 is connected to the evaluation unit 62 to receive the determined distances. The visualization unit 64 serves to generate a visualization image in which the received distances, for example in the form of the 4 shown depth map 10. The visualization image is then sent from the visualization unit 64, for example, to the display 37 of 1 output, where the tumor depth preview map 10 with the iso-depth lines 18 is displayed in the beam path of the surgical microscope 2 and is superimposed on the image of the observation area 1 by means of the beam splitter 13A, so that both the surgical scene 1 and the iso-depth lines 18 of the tumor depth preview map 10 indicating the depth of the tumor 6 below the tissue surface 12 can be seen in the eyepiece.

In the present embodiment, however, the distances are not only visualized in the eyepiece of the surgical microscope 2, with the display 37 serving as a display unit, but also on an external monitor 67 and a head-mounted display (HMD), which in the present embodiment is represented by data glasses 65.

In order to be able to generate a visualization image for the monitor 67 and/or the data glasses 65, the visualization unit 64 in the present embodiment is also connected to the camera 21 from the 1 The visualization unit 64 can therefore receive the image representing the observation area 1 or the image data generated from this image by the camera 21. The image data representing an electronic image of the observation area 1 are then combined by the visualization unit 64 with the distances generated and output by the evaluation unit 62, so that an electronic image is created which contains both the observation area 1 and the iso-depth lines 18 of the tumor depth preview map 10 (or another suitable representation of the distance information). This electronic image is then output to the external monitor 67 and/or the HMD, in the present embodiment, to the data glasses 65.

The depth preview map can be implemented as a linear display of the iso-depth lines, whereby the lines are solid, dashed, dotted or in any combination thereof. The depth preview map can also be displayed using an opaque display. Any form of blending of iso-depth lines or opaque display with the current observation scene or operation scene is also possible in order to enable a realistic "look and feel". Different visualization parameters can be selected for visible or invisible areas of the malignant tissue. In another variant, the depth information can also be displayed locally so as not to unduly restrict the surgeon's field of vision.

The present invention has been described for illustrative purposes using exemplary embodiments. However, a person skilled in the art will understand that deviations from the illustrated exemplary embodiments may occur. For example, in the 5 In the embodiment shown for the optical observation device system, the registration unit, the topography detection unit and the tomography system are not part of the optical observation device system. In other possible embodiments of the invention, the registration unit and/or the topography detection unit and/or the tomography system can be part of the optical observation device system. The invention should therefore not be limited by the embodiments shown, but only by the appended claims.

List of reference symbols

1

Operation scene

2

Surgical microscope

3

Surgical field

4

Surgical instruments

5

lens

6

tumor

7

divergent beam

8th

Surgeon's finger

9

Beam

9A, 9B

stereoscopic partial beam path

10

Tumor depth preview map

11

Magnification changer

12

tissue

13A, 13B

Interface arrangement

15A, 15B

Beam splitter prism

16

Focal plane

17

Floor

18

Iso depth line

19

Camera adapter

21

camera

23

Image sensor

27

Binocular tube

29A, 29B

Tube lens

31A, 31B

Intermediate image plane

33A, 33B

prism

35A, 35B

Eyepiece lens

37

Display

39

optics

40A, 40B

Spectral filter

41

White light source

43

Deflecting mirror

45

Lighting optics

47

Spectral filter

49

Laser

50

Vario lens

51

Positive link

52

Negative link

53

Displacement path

60

Topography acquisition unit

61

Registration unit

62

Evaluation unit

63

Interface arrangement

63A

Data output

63B

Data input

64

Visualization unit

65

Data glasses

66

Tomography system

67

monitor

68

optical observation device system

Claims (19)

Optical observation device system (68), comprising: - an optical observation device (2) for generating an image of an observation area (1) with an object of interest (6) and image data representing the image; - an interface arrangement (63) with a data output (63A) for outputting the image data to a registration unit (61), which generates registration data representing a registration of depth image data with the image data, a data input (63B) for receiving the depth image data of the object of interest (6), for receiving topography data associated with the image data from a topography detection unit (60) and for receiving the registration data from the registration unit (61); - an evaluation unit (62) which is designed to use the topography data, the depth image data and the registration data to determine the distances of the depth regions of the object of interest (6) recorded by the depth image data from the surface (12) of the observation area (1) and to output the determined distances, - a visualization unit (64) connected to the evaluation unit (62) for receiving the distances, wherein the visualization unit (64) is designed to generate a visualization image in which the received distances are shown and to output the visualization image for display on a display unit (37, 65, 67). Optical observation device system (68) according to Claim 1 in which the registration unit (61) is part of the optical observation device system. Optical observation device system (68) according to Claim 1 or Claim 2 in which the topography detection unit (60) is part of the optical observation device system. Optical observation device system (68) according to one of the Claims 1 until 3 in which the display unit (37, 65, 67) is part of the optical observation device system (68). Optical observation device system (68) according to Claim 4 , in which the display unit comprises at least one display (37) arranged in the optical observation device with which an overlay image can be projected into the optical beam path of the optical observation device can be displayed. Optical observation device system (68) according to one of the Claims 1 until 5 in which the display unit comprises a monitor. Optical observation device system (68) according to one of the Claims 1 until 6 , in which the visualization unit (64) for receiving the image data representing the image of the observation area (1) is connected to the optical observation device (2) and is designed to generate a visualization image in which both the received image of the observation area (1) and the received distances are shown. Optical observation device system (68) according to one of the Claims 1 until 7 in which the topography detection unit (60) comprises a stereoscopic sensor and/or a laser scanner and/or a sensor for measuring the time of flight of light pulses and/or a device for structured illumination and/or is based on a DFD method, and/or on an SFM method and/or on a SLAM method. Optical observation device system (68) according to one of the Claims 1 until 8th in which the registration unit (61) is part of a navigation device. Optical observation device system (68) according to one of the Claims 1 until 9 in which the visualization unit (64) is designed to display the received distances in the visualization image as depth lines, each representing a specific depth below the surface of the observation area (1). Optical observation device system (68) according to one of the Claims 1 until 9 in which the visualization unit (64) is designed to mark certain depth ranges below the surface of the observation area (1) in color and/or in different shades of gray. Optical observation device system (68) according to one of the preceding claims, in which the optical observation device is a surgical microscope (2) or an endoscope. Display method for displaying depth information about an object of interest (6) in an observation area (1) observed with an optical observation device (2), in which - an image of the observation area (1) and image data representing the image are generated by means of the optical observation device (2), - current topography data of the observation area (1) associated with the image data are recorded by means of a topography recording unit (60), - the image data and depth image data of the object of interest (6) received by the registration unit (61) in the observation area (1) are registered with one another by means of a registration unit (61) and corresponding registration data are generated, - the distances of the depth areas of the object of interest (6) recorded by the depth image data from the surface of the observation area (1) are determined by means of an evaluation unit (62) based on the topography data, the depth image data and the registration data, - a visualization image is generated by means of a visualization unit (64) in which the received distances are shown, and - the visualization image is displayed by means of a display unit (60). Representation method according to Claim 13 in which the entire depth image data is provided by a navigation device for the evaluation unit (62). Representation method according to Claim 13 in which the optical observation device (2) sends virtual depths in a defined value range to a navigation device based on the current focal plane, and the navigation device sends the respective contours of the depth image data to the evaluation unit (62) for received virtual depths. Representation method according to one of the Claims 13 until 15 in which the depth image data is obtained before the representation process is carried out. Representation method according to one of the Claims 13 until 16 , in which the received distances are displayed in the visualization image as depth lines, each representing a certain depth below the surface of the observation area (1), and/or certain depth ranges below the surface of the observation area (1) are marked in color. Representation method according to one of the Claims 13 until 17 in which the visualization unit (64) generates a visualization image in which both the received image of the observation area (1) and the received distances are shown. Computer program product for displaying depth information about an object of interest (6) using image data, depth image data and topography data of the object of interest (6), which includes instructions for carrying out the following steps: - registering the image data and the depth image data of the object of interest (6) and generating corresponding registration data, - determining the distances of the depth regions of the object of interest (6) captured by the depth image data from the surface of the observation area (1) based on the topography data, the depth image data and the registration data and - generating and outputting a visualization image in which the distances are shown.

Images