EP2914194A1

EP2914194A1 - Imaging system, operating device with the imaging system and method for imaging

Info

Publication number: EP2914194A1
Application number: EP13786464.1A
Authority: EP
Inventors: Christian WINNE; Sebastian Engel; Erwin Keeve; Eckart Uhlmann
Original assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV; Charite Universitaetsmedizin Berlin
Current assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV; Charite Universitaetsmedizin Berlin
Priority date: 2012-11-05
Filing date: 2013-11-04
Publication date: 2015-09-09
Also published as: WO2014068106A1; US20150287236A1; DE102012220115A1

Abstract

The invention relates to an imaging system 200, particularly for an operating device 1000 with a device head 100 that can be adjusted in a mobile manner, comprising: - an image data capturing unit 210 with an image recording unit, particularly an operation camera 211, 212, designed to capture image data 300 in an operating environment OU; - an image data processing unit 220 designed to provide the image data; - an image storage unit 230 designed to hold the image data 300 of the operating environment and volume data of a volume representation 320 assigned to the operating environment. Also provided according to the invention are: - a registration means 260 designed to locate the image recording unit, particularly the operating camera 211, 212, relative to the operating environment; - a virtual pointer means 250 designed to make available, particularly to identify and/or to display, a number of surface points OS of the image data 300 automatically; and - an assignment means, particularly with calculating means 240, designed to assign automatically a volume point VP of the volume representation 320 to at least one of the surface points OS made available.

Description

Imaging System, Surgical System with Imaging System

drive to imaging

The invention relates to an imaging system, in particular for an operating device, comprising: an image data acquisition unit, an image data processing unit, an image storage unit. The invention further relates to an operating device. The invention also relates to a method for imaging, comprising the steps of: acquiring and providing image data and maintaining the image data, in particular in the medical or non-medical field.

Especially in the medical field, approaches to the mobile device of the type mentioned above have developed. At present, in particular, the approach of an endoscopic navigation or instrument navigation is pursued for displaying a guidance device, in which optical or electromagnetic tracking methods are used for navigation; For example, modular systems for an endoscope with expanding system modules such as a tracking camera, a computing unit and a visual display unit for displaying a clinical navigation are known. Under tracking is basically a method for tracking or tracking to understand what the tracking of moving objects - namely in the present case the mobile device head - serves. The goal of this tracking is usually the mapping of the observed actual movement, especially relative to a cartographed environment, for technical use. This may be the merging of the tracked (guided) object - the mobile device head - with another object (e.g., a target point or target trajectory in the environment) or simply the knowledge of the current "pose" - ie position and / or Orientation and / or movement state of the tracked object. So far, absolute data relating to the position and / or orientation (pose) of the object and / or the movement of the object are regularly used for the purpose of tracking, as for example in the abovementioned system. The quality of the particular pose and / or movement information depends first of all on the quality of the observation, the tracking algorithm used and on the modeling that serves to compensate for unavoidable measurement errors. Without modeling, however, the quality of the particular position and movement information is usually relatively poor. Currently, absolute coordinates of a mobile device head-e.g. In the context of a medical application, for example, also from the relative relationship between a patient tracker and a tracker for the device head. Fundamentally problematic in such designated as tracking absolute modules modular systems, the additional effort-spatial and temporal- to represent the required tracker. The space requirement is enormous and proves to be highly problematic in an operating room with a large number of actors.

In addition, adequate navigation information must be available; ie with tracking methods, a signal connection should regularly be established between trackers. B. an explained below Lokalizer- and a Bilddatenerfassungseinheit- z. For example, a locator detection system may be maintained, for example, maintained at a tracking camera or other detection module of a locator detection system. This can be, for example, an optical but also an electromagnetic or similar signal connection. Cancels such a particular optical signal connection abz. B. when an actor in the image pick-up line between tracking camera and a patient tracker device missing the necessary navigation information. In this case, support of the guidance of the device header by navigation information is no longer given. In an exceptional case, the guide of the mobile device head may be interrupted until again Navigation information is available. Particularly in the case of the optical signal connection, this problem is known as the so-called "line of sight" problem.

Although a more stable signal connection can be made available, for example, by means of electromagnetic tracking methods, which is less susceptible than an optical signal connection. On the other hand, such electromagnetic tracking methods are inevitably inaccurate and more sensitive to electrically or ferromagnetically conductive objects in the measuring chamber; This is particularly relevant in the case of medical applications, since the mobile device is to be used regularly to assist in surgical procedures or the like, so that the presence of electrically or ferromagnetically conductive objects in the measuring space, d. H. at the surgical site, which can be a rule.

It is desirable to largely avoid, mitigate and / or circumvent a problem associated with the above-described classical tracking sensor technology for navigation for a mobile device. In particular, this relates to the problems of the aforementioned optical or electromagnetic tracking method. Nevertheless, an accuracy of a guide device for navigation should be as high as possible in order to enable as precise a robotics application as possible to be closer to the mobile device that can be handled, in particular medical application of the mobile device.

However, there is also the problem that the stability of a fixed location of a patient tracker or locator in patient registration is critical to track accuracy; this can not always be guaranteed in the practice of an operating room with a large number of actors. In principle, a mobile device that can be improved in this respect is known with a tracking system from WO 2006/131373 A2, wherein the device is advantageously designed for the contactless determination and measurement of a spatial position and / or spatial orientation of bodies.

New approaches, in particular in the medical field, try to assist the navigation of a mobile device head by means of intraoperative magnetic resonance tomography or computer tomography in general, by coupling these with an imaging unit. The registration of image data obtained, for example, by means of endoscopic video data with a preoperative CT image is described in the article by Mirota et al. "A System for Video-Based Navigation for Endoscopic Endonasal Skull Base Surgery" IEEE Transactions on Medical Imaging, Vol. 31, no. 4, April 2012 or in the article by Burschka et al. "Scale-invariant registration of monocular endoscopic images to CT scans for sinus surgery" in Medical Image Analysis 9 (2005) 413-426. An essential goal of registration of image data obtained by, for example, endoscopic video data is to improve the accuracy of registration.

On the other hand, such approaches are however comparatively inflexible, since always a second image data source has to be prepared, e.g. In a preoperative CT scan. In addition, CT data is associated with high costs and high costs. The acute and flexible availability of such approaches at any desired time, e.g. B. spontaneously during surgery, is therefore not or only limited and possible with preparation.

Recent approaches predict the possibility of using simultaneous localization and mapping techniques "in vivo" for navigation. For example, a general study has been described in the article by Mountney et al. to the 31st Annual International Conference of the IEEE EMBS Minneapolis, Minn., USA, September 2 - 6, 2009 (978-1-4244-3296-7 / 09). In the article by Grasa et al., "EKF monocular SLAM with relocalization for laparoscopic sequences" at 201 IEEE International Conference on Robotics and Automation, Shanghai, May 9-13, 201 1 (978-1-61284-385- 8/1 1) describes a real-time application at 30 Hz for a 3D model in the context of a visual SLAM with an extended Kalman Filter (EKF) The pose (position and / or orientation) of an image data acquisition unit is considered in a three-point algorithm Real-time usability and robustness to a moderate level of object movement was tested.

Similar to the aforementioned article by Mountney et al. is in Totz et al. in Int J CARS (2012) 7: 423-432, "Enhanced visualization for minimally invasive surgery" describes that an area of vision of an endoscope may be augmented with a so-called dynamic view expansion, based on previous observations uses a simultaneous localization and mapping (SLAM) approach.

Such methods are generally promising; However, the presentation does not yet indicate how handling could be done in practice. In particular, the aforementioned article by Mirota et al Registration of image data from the visual recording of an operating environment by means of an operation camera to a preoperatively obtained image material, such as CT data - ie the registration of the current two-dimensional surface data to a preoperative three-dimensional volume representation - in different ways, namely based on a pointer instrument, a Tracker or a visual navigation can be done.

Generally, such and other methods are referred to as so-called registration methods.

Registrar systems based on physical pointers regularly include so-called locators, namely a first locator to be attached to the patient to display the patient's coordinate system and an instrument locator to display the coordinate system of a pointer or instrument. The localizers can be controlled by a 3D measuring camera, eg. B. can be detected with a stereoscopic camera and the two coordinate systems can be associated in the context of Bilddatenverarbei- tion and navigation.

A problem with the aforementioned approaches using physical pointer means is that the use of physical pointer means is comparatively complicated and also prone to failure in the implementation. A problem with solely visually based registration and navigation approaches is their accuracy, which is ultimately determined by the resolution of the surgical camera used. It would be desirable to have an approach that can be implemented comparatively robustly with respect to disturbances with reduced expenditure and yet is available with comparatively high accuracy.

At this point, the invention is based on the object of providing an imaging system and an operating device and a method by means of which a surface model of the operating environment can be registered in an improved manner in a volume representation of the operating environment. In particular, the handling and availability of registered surgical sites should be improved.

The problem concerning the system is solved by an imaging system of claim 1. The imaging system for the surgical device with a mobile handleable device head comprises: an image data acquisition unit having an image acquisition unit, in particular an operation camera, which is designed to capture image data of an operating environment,

an image data processing unit configured to provide the image data,

an image storage unit configured to store the image data of the operation environment and volume data of one of the operation environment. According to the invention are further provided:

Registration means, which are designed to localize the image acquisition unit, in particular the surgical camera, relative to the operating environment,

- Virtual pointer means which are adapted to provide a number of pixels of the image data (300) automatically, in particular to identify and / or display, and

Assignment means, in particular with computing means, which are designed to automatically associate with at least one of the pixels provided a volume location of the volume representation. Particularly preferably, the pixels can be specified as surface locations.

Most preferably, the image data includes surface rendering data and / or the volume data includes volume rendering data. Within the scope of a particularly preferred development, the image data processing unit is designed to create a surface model of the operating environment by means of the image data, and / or an image storage unit is configured to hold the surface model representation data, and / or

the registration means are adapted to locate the operation camera relative to the surface model of the operating environment, and / or

- The virtual pointer means are adapted to provide a number of surface sites in the surface model.

The image acquisition unit may basically comprise any type of imaging device. Thus, the image acquisition unit may preferably be an operation camera directed to the operation environment. Preferably, an image acquisition unit may comprise an optical camera. However, an imaging unit may also include some other form than the optical one in the visible area to image for real or virtual images. For example, the image acquisition unit can operate on the basis of infrared, ultraviolet or X-ray radiation. The image recording unit may also comprise a device which is capable of generating a planar, optionally arbitrarily curved, topography from volume images; insofar a virtual picture. Beispiels- this can also be a sectional plane view of a volume image; z. In a sagittal, frontal or transverse plane of a body.

The object concerning the device is solved by an operation device of claim 18. Said operation device may preferably have in a periphery a device head that can be handled mobile. The mobile device head can in particular comprise a tool, an instrument or sensor or the like device. Preferably, the device head is designed such that it has an image pickup unit, as may be the case with an endoscope, for example. However, the image pickup unit may also be used remotely from the device head; in particular for observing the device head, in particular a distal end thereof in an operating environment.

In particular, the surgical device may be a medical device having a medical mobile device head such as an endoscope, a pointing instrument or a surgical instrument or the like; with a distal end for arrangement relative to a body, in particular body tissue, preferably insertion or attachment to the body, in particular to a body tissue, in particular for processing or observation of a biological body, such as a tissue-like body or the like. In particular, a said surgical device may be a medical device, such as an endoscope, a pointing instrument, or a peripheral surgical instrument, which may be used, for example, in the context of laparoscopy or other medical examination method with the aid of an optical instrument; Such approaches have proven particularly useful in the field of minimally invasive surgery. In particular, the device may be a non-medical device having a non-medical mobile device head such as an endoscope, a pointing instrument or a tool or the like; with a distal end for arrangement relative to a body, in particular a technical object such as a device or a device, preferably attachment or attachment to the body, in particular on a subject, in particular for processing or observation of a technical body, such as an object or Device or the like. Device. The said system can also be used in a non-medical field of application. The said system may be useful in a non-medical field of application, e.g. B. for assistance in the visualization and analysis of non-destructive testing methods in the industry (eg., Material testing) or in everyday life (eg airport controls or bomb disposal). Here, for example, a camera-based visual inspection from a distance (for example, to protect against dangerous content) using the presented invention, the analysis and assessment of, based on previously or simultaneously recorded image data (eg., 3D X-ray image data, ultrasound image data or microwave image data, etc. ), internal views increase safety and / or ease of work. Another exemplary application is the investigation of internal cavities of components or assemblies using the system presented here, for example based on an endoscopic or endoscope-like camera system.

In non-medical applications, where a device head is used meaningfully, the concept of the invention has also proven itself. Especially in assembly or repair, the use of optical vision instruments is helpful. For example, tools, in particular in the field of robotics, can be attached to an operating device that is equipped with an imaging system so that the tools can be navigated by means of the operating device. The system can in particular increase the accuracy in the assembly of industrial robots or make previous assembly activities that are not possible with robots feasible. In addition, a worker / mechanic, by instruction of a data processing based on the above-mentioned imaging system attached to the tool, the assembly activity can be facilitated. For example, by using this navigation option in conjunction with an assembly tool, such as a cordless screwdriver, on a body (eg, a vehicle body), assembly (eg, bolting of spark plugs) of a component (eg, spark plug or screw) with the help of a data processing the amount of work can be reduced by assistance and / or the quality of the executed activity can be increased by checking.

In general, the operating device of the type mentioned can preferably be equipped with a manual and / or automatic guide for guiding the mobile device head, wherein a guide device is designed for navigation in order to enable automatic guidance of the mobile device head. The object concerning the method is solved by a method of claim 19.

The invention is equally applicable in a medical field and in a non-medical field, in particular non-invasive and without physical intervention on a body. The method may preferably be restricted to a non-medical field.

DE 10 2012 21 1 378.9, not published at the time of filing the present application, prior to the filing date of the present application, is a mobile device with a mobile device head, in particular a medically mobile device head with a distal end to Arrangement relative to a body tissue, remove, which is mounted on a guide device with integration of an image data acquisition unit, an image data processing unit and a navigation unit feasible. For this purpose, image data and an image data flow are used to indicate at least one position and orientation of the device head in an operating environment by means of a map.

The invention is based on the consideration that when registering a volume representation on a surface model of the operating environment, the surgical camera, irrespective of the type of registration, has hitherto only been used as image data acquisition means. The invention has recognized that, in addition, when the operation camera is located relative to the surface model, it is particularly registered with respect to the surface model and the volume representation of the operating environment to produce a virtual pointing device. Accordingly, the invention provides:

- registration means adapted to locate the operation camera relative to the surface model;

- Virtual pointer means, which are adapted to automatically provide a number of surface sites in the surface model;

- Calculating means, which are designed to automatically assign at least one of the surface areas provided a volume point of the volume representation. The invention has recognized that the use of a physical pointer means is superfluous in the vast majority of cases via a virtual pointer means produced in this way. Rather, a number of surface sites can be made so automated, that an operator or other user need only be enabled to effectively select the surface site of interest; the selection process is more effective and faster than a cumbersome use of a physical pointer. Rather, a number of surface sites in the surface model can be automatically made available automatically and with reasonable effort with a respective assigned volume location of the volume representation. This leads to an effective registration of the surface location on the site of the volume representation assigned to the surface location.

The concept of the invention is based on the recognition that the registration of the operating camera relative to the surface model also allows the registration with respect to the volume representation of the operating environment and thus a surface location can be unambiguously assigned to a volume representation. With reasonable computational effort this can be done for a number of sites and these can be effectively provided to the surgeon or other users as a selection. This opens up the possibility for the surgeon or other user of any objects imaged in the image of the operating environment, ie. H. To view objects at specific but arbitrary locations of the operation environment in the surface model and the volume rendering. This also makes access to places in the operating environment that would not be accessible with a physical pointer instrument. This possibility opens up independently of the registration means for the surgical camera; this may include registration by means of an external localization of the surgical camera (eg by tracking and / or pointer / pointer) and / or an internal localization by evaluation of the camera image data (visual method by the camera itself). Advantageous developments of the invention can be found in the dependent claims and specify in detail advantageous ways to implement the described concept within the scope of the task, as well as with regard to further advantages.

In a first variant, the registration means preferably comprises a physical patient locator, a physical camera locator and an external optical locator detection system. A particularly preferred embodiment is illustrated in Fig. 3 and Fig. 4 of the drawings. Said further developing first variant considerably increases the accuracy of a registration through the use of physical localizers, ie a registration between image and volume representation or between image data and volume data, in particular between surface model and volume representation of the operating environment. However, for example, physical registration means may also undergo a change in position relative to the identified body, for example, by slipping or releasing relative to the body during surgery. This can be counteracted because the operation camera is also registered with a locator. In particular, in the case of a failure of the physical registration means or an interruption of the line of sight connection between an optical position measuring system and a locator, the image or the image data, in particular the surface model, can advantageously be used to determine the pose of the operating camera relative to the operating environment. Ie. even if a camera localizer is no longer detected by the external, optical localizer detection system for a short time, the pose of the surgical camera to the operating environment can be recalculated from the surface model for the interruption. Thus, a fundamental weakness of a physical registration agent is effectively balanced.

In particular, a combination of several registrant z. Example, an optical position measuring system from the first variant and a visual navigation from the second variant explained below, are used, such that a detection of errors, such as the slipping of a localizer, is possible. Thus, transformation errors can be corrected and corrected quickly by comparing between a desired value of one registration means and an actual value of the other registration means. Nevertheless, the variants which are fundamentally independent of the concept of the invention can, in a particularly preferred manner, be used redundantly, in particular for the aforementioned recognition of errors.

In a second refinement variant, the registration means can be designed essentially for virtual localization of the camera. The registration means referred to here as virtual comprise, in particular, the image data acquisition unit, the image data processing unit and a navigation unit. According to the second variant of a further development, it is provided in particular that

- The image data acquisition unit is designed to image data of an environment of Vorrich- tion head, in particular continuously, to detect and provide and

- The image data processing unit is adapted to create a map of the environment by means of the image data and

- The navigation unit is formed, by means of the image data and an image data flow to indicate at least one position of the device head in a vicinity of the operating environment using the map, such that the mobile device head is feasible based on the map.

Under navigation is basically any type of map creation and arrival of a position in the map and / or the indication of a destination point in the map to understand advantageous in relation to the position; In the following, therefore, the determination of a position with respect to a coordinate system and / or the specification of a destination point, in particular the indication of a route advantageously shown on the map between position and destination point.

The development is based on a substantially image data-based mapping and navigation in a map for the environment of the device head in the broader sense; that is, an environment that is not bound to a vicinity of the distal end of the device head, such as the visually detectable proximity at the distal end of an endoscope - the latter visually detectable proximity is referred to herein as the device head's operating environment. Particularly advantageously, a guide means with position reference to the device head this be assigned. The guide means is preferably designed to make information on the position of the device head with respect to the environment in the map, the environment going beyond the proximity environment.

The position reference of the guide means to the device head may advantageously be rigid. However, the position reference need not be rigid as long as the position reference is determinate variable or movable or at least calibrated. This may for example be the case when the device head at the distal end of a robot arm as part of a handling apparatus and the guide means is attached to the robot arm, such. B. caused by errors or strains variants are calibrated in the non-rigid but basically deterministic position reference between the guide means and device head is in this case. An image data flow is understood to be the flow of image data points in time change, which occurs when one considers a number of image data points at a first and a second time while changing the position, the direction and / or speed thereof for a defined passage area. Preferably, but not necessarily, the guide means comprises the image data acquisition.

Within the scope of a preferred development, the surface location in the surface model is assigned a surface coordinate of the surface model representation data. Preferably, the volume location of the volume representation has a volume coordinate associated with volume rendering data. The data may be stored on the image storage unit in a suitable format, such as a data file or stream, or the like.

The surface location is preferably fixable as the intersection of a virtual viewing beam emanating from the surgical camera with the surface model. In particular, a surface coordinate can be specified as a point of the operating camera assigned to the point of intersection. Such a 2D pixel can be registered after registration of the surgical camera relative to the surface model and registered volume rendering of 3D image data to the patient or the surface model and it can also locate the camera itself in the volume image data or localize relative to them.

Preferred further developments also provide advantageous possibilities for making a selection or definition of a surface location relative to a volume location available.

Preferably, a selection and / or monitor means is provided, which is configured to group the freely selectable and automatically provided and fixed number of surface locations into a selection and to visualize the selection in a selection representation. The selection representation may be an image, but also a selection menu or a list or other representation. The selection representation may also be a linguistic representation or a sensor feature. In particular, it has proven to be preferred that the number of surface sites in the surface model is freely selectable, in particular free of physical Display, ie provided without a physical pointer. The number of surface locations in the surface model is particularly preferably available only by virtual pointing means. However, according to the aforementioned development, the system is also suitable for allowing physical pointing means and for enabling them to locate a surface location relative to a volume representation.

As part of a further development, the number of surface locations in the surface model can be automatically determined. In particular, no further automatic assessment or interaction between the surgeon or other user and selection is required in order to register and display a surface location in a volume location. However, it has proven to be advantageous that the selection comprises at least one automatic pre-selection and one automatic final selection. Advantageously, the at least one automatic preselection can comprise a number of cascaded automatic preselections so that, with appropriate interaction between selection and operator or other users, finally, a desired final selection of a registered surface location to a volume location is available.

Within the scope of a further particularly preferred development, it has proved to be advantageous that a selection and / or monitor means is designed to group the automatic selection on the basis of the image data and / or the surface model. This concerns in particular the preselection. Additionally or alternatively, however, this may also concern the final selection, in particular an evaluation procedure for the final selection.

Grouping may be performed on the basis of first grouping parameters, which include a distance measure, in particular a distance of the operation camera to structures represented in the image data. The grouping parameters preferably also include a 2D and / or 3D topography, in particular a 3D topography of represented structures based on the created surface model; this may include a shape or a depth gradient of a structure. Preferably, the grouping parameters also include a color, in particular a color or color change of the structure shown in the image data.

Such automatic selection grouped essentially on the basis of the image data and / or the surface model can be supplemented by an automatic selection independent of the image data and / or independently of the surface model. For this purpose, in particular second Gruppier parameters, which include a Geometry default or a grid preset. For example, a geometric distribution of pixels for selecting surface locations registered with volume locations and / or a rectangular or circular grid may be specified. In this way, it is possible to select locations which correspond to a specific geometric distribution and / or follow a certain shape or lie in a specific grid, for example a specific quadrant or in a specific area.

Particularly preferably, an automatic final selection can be implemented from the locations provided in a preselection by means of evaluation methods; In particular, as part of the automatic final selection, selected positions can be grouped by means of evaluation procedures. The evaluation methods include, for example, methods for statistical evaluation in connection with other image locations or image positions. Mathematical filtering and / or logic methods are suitable here, such as, for example, a Kalman filter, a fuzzy logic and / or a neural network.

Particularly preferred is an interaction with the selection and / or monitor means, d. H. in particular, implement a manual interaction between an operator or other user and the selection and / or monitor means in the context of a manual selection support by one or more input features. An input feature may be, for example, a keyboard means for the hand or foot of the surgeon or other user; for example, this may be a computer mouse, a button, a pointer or the like. It can be used as input means and a gesture sensor that responds to a specific gesture. Also, a voice sensor or a touch-sensitive sensor such as an input pad is possible. In addition, other mechanical input devices such as keyboards, buttons or push buttons are suitable.

The concept or one of the further developments proves to be advantageous in a large number of technical fields of application such as, for example, robotics; especially in medical technology or in a non-medical field. In particular, the subject matter of the claims comprises a mobile manageable medical device and a particularly non-invasive method for processing or observation of a biological body such as a tissue or the like. An image acquisition unit may in particular be an endoscope or an ultrasound imaging unit or another imaging unit. in particular a previously mentioned unit z. B. based on an IR, X or UV radiation. Thus, for example, in the case of a tracked and calibrated ultrasound probe, 2-D slice images or 3-D volume images can also be registered for the operating environment. For example, by segmenting significant and / or characteristic changes in the gray value, surface models can also be calculated from the image data, which can serve as a basis for the virtual pointer means or the selection and / or monitor means. The use of ultrasound imaging or other radiation-based imaging as an image acquisition unit is particularly advantageous. For example, a device head may also be a pointer instrument or a surgical instrument or the like. Medical device for processing or observing a body or for detecting the own position, or the instrument position, relative to the environment.

Thus, the subject matter of the claims comprises in particular a mobile manageable non-medical device and a particularly non-invasive method for processing or observing a technical body such as an object or a device or the like. For example, the concept may be used in industrial processing, Positioning or monitoring processes are successfully applied. However, for other applications in which a claimed mobile handleable device - such as an instrument, tool or sensor-like system - are used according to the described principle, the described essentially image-based concept is advantageous.

Embodiments of the invention will now be described below with reference to the drawing in comparison with the prior art, which is also partly shown, this in the medical application frame, in which the concept is implemented with respect to a biological body. however, the embodiments also apply to a non-medical application framework in which the concept is implemented with respect to a technical body.

The drawing is not necessarily to scale the embodiments, but the drawing, where appropriate for explanation, executed in a schematized and / or slightly distorted form. With regard to additions to the teachings directly recognizable from the drawing reference is made to the relevant prior art. It should be noted that various modifications and changes may be made in the form and detail of an embodiment without departing from the general idea of the invention. The features disclosed in the description, in the drawing and in the claims features of the invention can be essential both individually and in any combination for the development of the invention. In addition, all combinations of at least two of the features disclosed in the description, the drawings and / or the claims fall within the scope of the invention. The general idea of the invention is not limited to the exact form or detail of the preferred embodiment shown and described below or limited to an article that would be limited in comparison with the subject matter claimed in the claims. For the given design ranges, values within the stated limits should also be disclosed as limit values and arbitrarily usable and claimable. Further advantages, features and details of the invention will become apparent from the following description of the preferred embodiments and from the drawing; this shows in:

Fig. 1 shows a basic scheme of a method and an apparatus for

Imaging registering a surface model to a volume rendering considering the operation camera, optionally based on a physical pointing device, a tracker, or based on a visual navigation method;

2 shows an exemplary embodiment of an operating device with an imaging system, in which the registration of the surgical camera is essentially based on a visual navigation method, in particular as described in the aforementioned DE 10 2012 21 1 378.9, the disclosure of which hereby by citation completely in the disclosure this application is accepted;

3 shows an alternative embodiment of an operating device with an imaging system in which localizers for localizing not only the patient but also the surgical camera via an endoscope locator, so that a virtual pointing device can be formed, automatically make available a number of surface locations in the surface model For example, an endoscope is shown as a special camera system;

4 shows a modification of the embodiment of an operating device shown in FIG. 3, in which a second locator is mounted directly on the surgical camera instead of on the endoscope - the transformation TKL between camera localizer and camera origin (lens) to be calibrated is shown, wherein the camera is represented by a camera icon, which is generally for any advantageous usable camera; 5 shows a detail X of FIG. 3 or FIG. 4 for illustrating a preferred procedure for determining a surface location as the intersection of a virtual viewing beam emanating from the surgical camera with the surface model;

Fig. 6 is a flow chart illustrating a preferred method of implementing automatic pre-selection and automatic final selection by means of a selection and / or monitor means utilizing first and second grouping parameters to provide at least one surface location with associated volume location;

7 shows a flow chart for a particularly preferred embodiment of a procedure for navigating any pixels in medical camera image data of an operation camera.

With reference to the corresponding parts of the description, the same reference numbers are used throughout the description of the figures for identical or similar features or features of identical or similar function. In the following, an apparatus and a method in various embodiments are presented, which are particularly preferred for, but not limited to, clinical navigation.

In a clinical navigation, it is possible in the context of image-based interventions, such. For example, endoscopic interventions or other laparoscopic interventions, at tissue structures G at arbitrary pixels in the camera image to calculate the corresponding position in 3D image data of a patient. In the following, various possibilities for determining 3D positions, the objects represented in the camera image data and their use for clinical navigation are described in detail. 1 shows a schematic diagram of the general structure of a method and a device for clinical navigation.

1 shows an operating device 1000 with a mobile device head 100 and an imaging system 200. In the present case, the device head 100 is in the form of an endoscope 110. An image data acquisition unit 210 of the imaging system 200 has at the endoscope an operation camera 21 1, which is designed to continuously capture and provide image data 300 of an operating environment OU of the device head 100, ie in the field of view with a near environment NU of the operation camera 21 1. The image data 300 are processed according to an image data processing unit 220 provided. The basis of the method and device presented here is now the use of the intraoperative camera systems used, as exemplified here by the surgical camera 21 1 as part of an endoscope 1 10, for obtaining navigation information from objects imaged in the camera image, ie the image data 300, throughout Image area, ie in the entire mapped area of the operating environment OU.

Objects here are a first, more roundish object OU1 and a second, rather elongated OU2. Moreover, while the image data processing unit 220 is designed to create a surface model 310 of the operating environment OU by means of the image data 300, above all pre-operatively obtained volume representation data of a volume representation 320 of the operating environment can be present. The surface model 310 as well as the volume representation 320 can be stored in suitable storage areas 231, 232 of an image storage unit 230. For this purpose, corresponding representation data of the surface model 310 or representation data of the volume representation 320 are stored in the image memory unit 230. The goal is now to determine certain surface locations OS1, OS2 in the view of the two-dimensional camera image, i. H. in the surface model 310, with a corresponding position of a volume location VP1, VP2 in the 3D image data of a patient, i. H. in volume rendering 320, to bring in correspondence; In general, the goal is to register the surface model 310 to the volume rendering 320.

Up to now, however, first or appropriate surface areas OS1, OS2 or volume locations VP1, VP2 must be separately shown or identified by an operator or other user - subsequently it must be costly to check whether the volume location VP1 actually has the surface location OS1 or the volume location VP2 corresponds to the surface location OS2. In the specific case, it may be, for example, the registration of video data, namely the image data 300 and the surface model derived therefrom 310 preoperatively obtained 3D data such. B. CT data, so generally the volume representation act. So far, three key approaches to registration have proven useful, some of which are described in detail below. A first approach uses a pointer or pointer, either as a physical hardware instrument (pointer) or, for example, as a laser pointer to identify and locate certain surface locations OS. A second approach uses the identification and visualization of surface place in a video or the like image data 300 and registers it, for example on a CT data set by means of a tracking system. A third approach identifies and visualizes surface locations in a video or the like image data 300 and registers them for volume rendering, such as a CT dataset, by reconstructing and registering the surface model 310 with the volume rendering 320 by suitable computing means. Regardless of the type of approach for matching the surface model 310 and the volume rendering 320 in image data processing, an operation camera 21 1 has hitherto been used to merely monitor or visualize, for example, a pointer instrument or a pointer or other form of manual display of a surface location , After that, computing means 240 are provided which are designed to match (register) the surface location OS1, OS2 identified by manual display to a volume location VP1, VP2 and thus to correctly associate the surface model 310 with the volume rendering 320. Moreover, the method and apparatus described in the present embodiment provides virtual pointing means 250 within the scope of the image data processing unit 220, which are designed to automatically provide a number of surface locations in the surface model 310, in order to eliminate the difficulties associated with the manual interventions ; So not only a single one is displayed manually but any number in the entire OU is displayed. The number of surface locations OS, in particular in the surface model 310, can be freely selectable, in particular be free from a physical display. Additionally or alternatively, the number of surface locations OS in the surface model 310 may also be automatically determinable. In particular, the selection may at least partially include an automatic pre-selection and / or an automatic final selection. A selection and / or monitor means 500 is designed to group an automatic selection, in particular in a preselection and / or final selection, on the basis of the image data 300 and / or the surface model 310; z. Based on first grouping parameters: a distance measure, a 2-D or 3D topography, a color. Also, a selection and / or monitor means 500 may be configured to group an automatic selection independently of the image data 300 and / or the surface model 310, in particular based on second grouping parameters comprising: a geometry specification, a raster specification. The selection and / or monitor means 500 comprises a MAN machine interface MMI, which is actuatable for manual selection support. In addition, registration means 260 -as hardware and / or software implementation, e.g. B. a number of modules, which are designed to locate the opera- tion camera 21 1 relative to the surface model 310. In particular, the measurement of the position shown by way of example in FIG. 5 KP2 and position KP1 (pose KP) of the surgical camera 21 1 used in particular enters into the concept exemplified here as an embodiment.

First of all, the imaging properties of the image recording unit can be defined in fundamentally different ways, and these and other properties of the image recording can preferably be used to determine position KP2 and position KP1 (pose KP). Preferably, however, for a pose KP of an image acquisition unit, a combination of spatial coordinates and directional coordinates of the imaging system can be used. As a coordinate location, for example, a coordinate of a characterizing location of the imaging system, such as a focal point KP1 of an imaging unit, z. B. lens, the image pickup unit. As a direction coordinate, for example, a coordinate of a direction vector of a visual beam, so z. For example, referring to FIG. 5, an orientation KP2 of the sight beam 330 is shown.

This can be done with reference to FIG. 2 with a calculation method or within the context of a position measuring system explained with reference to FIG. B. can work optically, electromagnetically or mechanically. In both cases, the knowledge of the imaging properties of the camera system and, in particular, the surgical camera 21 1 enter into the analysis. This approach will be further explained with reference to FIG. 4 with an example.

Referring first to Fig. 2, this illustrates an example of a video-based third approach mentioned above for registering a volume rendering 320 to a surface model 310. For like or similar parts or parts of the same or similar function, the same reference numerals will be used herein.

FIG. 2 shows a tissue structure G with objects 01, 02 which are recognizable in a volume representation 320 as a volume location VP or in a surface model as a surface location OS and should be assigned to one another. An advantage of the system 1001 of an operating device illustrated in FIG. 2 is the use of a navigation method, referred to as visual navigation, without additional external position measuring systems. Here, based on image data 300, ie camera image data of an operation camera 21 1 z. B. in an endoscope 1 10 and / or image data 300 of an external camera 212 from an environment U of the device head 100 a card created. This may be a map of the environment U and / or map comprising the surface model 310 of the operation area OU. For a detailed description, reference is made to DE 10 2012 21 1 378.9, which was not published at the time of deposit of the present application and whose seniority is prior to the filing date of the present application. The content of the application is hereby incorporated by reference in full in the disclosure of the present application.

In particular, in the context of the concept of visual navigation illustrated by way of example in FIG. 2, the implementation of a SLAM (simultaneous localization and mapping) is described as an option which exclusively uses sensor signals, in particular image data 300 for orientation in an extended area, namely the operating environment OU using the map of the environment U and / or the map of the operating environment OU. On the basis of the sensor data, a separate movement of the device head 100 is estimated as well as a map of the area covered by the internal camera 21 1 or external camera 212 is continuously created. In addition to map generation and motion detection, the currently acquired sensor information is also checked for matches with the previously stored image map data. If a match is found, the system knows its own current position and orientation (pose) within the map. As described in the aforementioned application, a monocular SLAM method is already suitable as an information source in which feature points in the video image are continuously recorded and their movement in the image is evaluated. If the surface map 310 explained with reference to FIG. 1 can now be registered to a volume data record 320 of the patient, the visualization of positions of objects OS1 in the video image within the 3D image data VP is possible. Likewise, the common use of visual navigation with classical tracking methods is possible to determine the absolute position of the created surface map.

A disadvantage of this approach is the accuracy of navigation in the OU operational environment. The navigation surface map to be created should range from the region of image data registration (eg, face on paranasal sinuses) to the surgical environment (eg, ethmoidal cells). Due to the piecewise construction of the map and the addition of new data on the basis of the existing map material, errors in the map structure can accumulate. Problems can also arise if it is not possible to generate video image data with distinctive and traceable image content for certain areas of the surgical area. A safe creation of a precise surface map with the help of the example monocular SLAM Procedure is thus a prerequisite to provide sufficient accuracy of surgical procedures.

FIG. 3 shows a modified system 1002 of an operating device 1000 with a device head 100 and an imaging system 200 applied to a tissue structure G in a patient as a particularly preferred embodiment. To achieve the accuracy requirements discussed above, OU pointer instruments have heretofore been used to acquire a 3D position of objects in OS in an operational environment, to which locators are attached for position sensing; The localizers can be detected with optical or electromagnetic position measuring systems. The background to this is that in surgical interventions with intraoperative real-time imaging-like the endoscopy exemplified in FIG. 3 with an endoscope as the device head 100 or another laparoscopic application-it is often necessary to determine position information for structures shown in the camera image and to the surgeon or to a surgeon or other user. The special navigation instruments (pointers) referred to as pointers with locator are guided by the surgeon by hand or with a robot arm to touch a tissue structure G in the operating environment OU. From the visualization of the pointer position in the image data 300, the surgeon or other user can conclude a tissue position 02 of the surface location OS. However, the use of special navigation instruments requires additional instrument change during the procedure and thus complicates the operation and an exact sampling of the tissue G by the surgeon.

As an example in FIG. 3, a new solution for the determination of pixels in current image data 300 of an intra-operative real-time imaging is proposed, which makes it possible to register a surface location OS to a volume location VP or, concretely, a 3D position in the reference coordinate system of the 3D image data (FIG. Volume rendering 320) of the patient, e.g. B. CT image data to bring in correspondence. 3, the position of the surgical camera 21 1 as well as of the patient and thus of the tissue structure G in the operating environment OU is detected by means of a position measuring system 400. For this purpose, the position measuring system has a position measuring unit 410, which can be formed, for example, as an optical, electromagnetic or optical measuring unit; as well as a first locator 420 and a second locator 430. The first locator 420 may be attached to the endoscope 110, as shown in FIG. 3, thereby representing a rigid, at least determinate, connection 422 to the surgical camera 21 1, or preferably directly, as in FIG Fig. 4 shown attached to an externally provided by the endoscope 1 10 provided surgical camera 212 or the like camera system. The localizers 420 or 410 and 430 are designed as so-called optical trackers with localizer balls and can be attached to the object on display of the endoscope 110 or the external surgical camera 212 or to the patient (and thus to the tissue structure G).

Possible camera systems are conventional cameras (eg endoscopes), but also 3D time-of-flight cameras or stereoscopic camera systems for executing the surgical camera 21 1, 212.

Time-of-flight cameras also provide a picture with depth information in addition to a color or gray value image of the OU operating environment. Thus, a surface model 310 can already be generated with a single image such that the calculation of the 3D position relative to the endoscope optics of the surgical camera 21 1 is possible for each 2D position in the image. With the aid of a suitable calibration, the transformation of the 3D coordinates supplied by the camera (surface coordinate of the surface model representation data) to the reference coordinate system R421 or R420 of the camera localizer or instrument localizer 421, 420 can be calculated.

Stereoscopic camera systems provide camera images of the operating environment simultaneously from two slightly different positions and thus allow a reconstruction of a 3D surface as a surface model 310, the objects shown in the camera images. After calibration of the camera system, the surface model 310, which is reconstructed on the basis of one or more pairs of images, may be z. B. can be realized as a point cloud and referenced to the localizer 420, 421; as it does about suitable transformations TLL1, TKL.

Conventional camera systems make it possible, on the basis of image sequences of a calibrated camera tracked by the position measuring system, to reconstruct a surface model 310 of the objects OS represented in the image data 300 with sufficient movement of the camera and sufficiently prominent image contents. These methods are similar to the monocular SLAM method discussed above. In the present case, however, the position, in particular pose of the camera need not be estimated but can be determined by the position measuring system 400. In this respect, the embodiment of FIGS. 3 and 4 represents a modification of the embodiment of FIG. 2. By means of the surgical camera 21 1, 212 thus based on a sequence of one or more consecutive camera images, the representation of a surface model of the operating environment OU, which here is visualized by the detection area KOL) of the camera system, takes place as essentially within the outer limits of the field of vision Operation camera 21 1, 212 is located. On the basis of this surface model 310, the calculation of 3D positions 01 for any 2D image positions 02 in the image data 300 takes place. After then successful registration of the 3D patient image data to the patient locator 430, 3D coordinates can be determined using the position measuring system 400 or the position measuring unit 410 into the reference coordinate system of the 3D image data 320. In the following case, the reference coordinate system R430 of the object localizer 430, that is, the 3D image data of the patient 320, is designated. In the present case, the reference coordinate system R420 or R421 of the camera localizer 420, 421 as well as the reference coordinate system R212 of the operation camera 212 are referred to, which merge into one another by simple transformation TKL.

The principle of the navigation method illustrated in FIG. 3 and FIG. 4 is that the pixels in the camera image, ie the image data 300 or the associated surface model, have a corresponding 3D position, namely a volume coordinate of the volume representation data, in the pre- or intraoperative To determine volume image data of the patient. The localization of the endoscope relative to the patient takes place with the aid of an external position measuring system 400 using the described optical measuring unit 410 with the associated reference coordinate system R410. As has been explained above, the optical measuring unit can each be realized as required with a different camera technology for generating a surface model 310. The 3D position of a pixel determined from the camera image data is determined using transformations TKL, TLL1 (transformation between the camera locator reference coordinate systems and the position measuring system) transmitted by measurement, registration and calibration, TLL2 (transformation between the reference coordinate systems of the object locator and of the position measuring system) and TLL3 (transformation between the reference coordinate systems of the 3D image data and the object locator) and can then be used for visualization. G designates the object of a tissue structure shown in the camera image data. The localizers 420, 421, 430 have optical trackers 420T, 421T, 430T in the form of spheres, which together with the associated object (endoscope) Skop 1 10, camera 212, tissue structure G) can be detected by the position measuring system 400.

FIG. 3 and FIG. 4 show the volume representations 320 with their associated reference coordinate system R320, which results from the reference coordinate system R430 of the tissue structure G due to the indicated transformation TLL3. Finally, the reference coordinate system R212 of the image data 300 can be brought into correspondence with the reference coordinate system of the volume representation 320. Volume rendering can be thought of as a collection of volume co-ordinates of volume rendering data, e.g. B. for a CT, DVT or MRI image realize; so far, the volume representation 320, shown as a cube, is an exemplary representation of patient SD image data.

5 illustrates the objects used by the exemplified concept as well as the principle of operation in connection with an optical position measuring system. The main component of the present concept is, as explained, the operating camera 21 1, 212 with its associated reference coordinate system R 420 or R 421, R 2 12 (FIG. via transformation TKL). As is explained with reference to FIG. 2, the position of the surgical camera can be performed purely computationally via a SLAM method or-as explained with reference to FIG. 3 and FIG. 4 -with the aid of a localizer 420, 421 within the position measuring system 400 become. The position of the camera jump (for example, the main lens) and the orientation or viewing direction of the camera thereby represent the camera coordinate system R212; this results relative to the reference coordinate system R421, R420 of the camera localizer by calibration, measurement or from the construction of the arrangement (dimensions of the camera, imaging geometries and dimensions of the camera, dimension of the endoscope 1 10). The surgical camera 21 1, 212 is aligned by the surgeon or another user on a tissue structure G, the position of which can also be determined with the help of the localizer 430 fixedly connected thereto. The camera image data, ie image data 300, which images the tissue structure G, are evaluated and used to represent the surface model 310 of the surgical area OU shown in FIG. With the aid of the surface model 310, the 3D position can then be determined as a surface coordinate of the surface model representation data for a certain pixel in the reference coordinate system of the camera R212 or R421, R420. For this purpose, for example, based on the results of the camera calibration, a shown in Fig. 5 Viewing beam 330 are calculated for a desired pixel 301 of the image data 300 of the camera image. The desired 3D position as the surface coordinate of the surface model representation data can thus be determined as the intersection of the viewing beam 330 with the surface model 310 as the point 31 1. The image data 300 represents the camera image positioned in the focal plane of the surgical camera 21 1, 212.

The embodiments described in FIGS. 3 to 5 have considerable advantages over the embodiment shown in FIG. 2 and can continue to be operated even in the event of a short-term failure of the position measuring system 400. A failure of the position measuring system 400 may occur, for example, if there is an interruption of the line of sight of the optical measuring unit 410 to the localizers 420, 430, 421. Then, however, the previously created surface model 310 can be used to adjust the camera position. H. comprising determining pose KP from focal point coordinates KP1 and orientation KP2 of the viewing beam 330 relative to the patient or the tissue structure G. For this purpose, based on a camera image, i. H. The image data 300 or an image sequence thereof, the current topology of the operating environment OU relative to the operation camera 21 1, 212 are determined. This data, z. B. surface coordinates of a point cloud of the surface model 310, can then be registered to the existing surface model 310. Due to the known transformation of patient localizer 430 to the 3D image data 320 as well as to the existing surface model 310, the 3D positions of arbitrary pixels OS1, OS2 as well as the camera 21 1, 212 itself can be calculated in the volume image data 320 and thus Ensure continuous navigation support for the surgeon or other user.

The aim of these methods is to automatically identify one or more image positions of interest, which are then used for a subsequent manual or automatic final selection of the image position. In the following, an exemplary selection of steps is explained with reference to FIG. 5, which enables a selection, in particular preselection and final selection, of the navigated image positions. The presented method allows the calculation and visualization of the 3D position in the patient for different 2D positions in the camera image. In this application, it is necessary to select a point for which the navigation information is calculated and displayed. The following automatic and manual methods are suitable for this purpose; The following criteria can be taken into account as a basis for automatic preselection of image positions:

- Removal of the camera to the structures shown in the picture - 3D topography of the displayed structures based on the created surface model (shape or depth gradient)

- Color or color change of the structures shown in the video image. Alternatively, a set of image positions can be used as pre-selection, which is specified by the system independently of the image content or the surface model. This is z. B. a geometric distribution of the pixels in a rectangular or circular grid conceivable.

For the automatic final selection of the image position for calculating the 3D position in the volume image data, the following evaluation methods can be used: Evaluation of the image positions based on the criteria specified in the automatic pre-selection

- Evaluation based on statistical evaluation of previous selected image positions. Here are u for the use. a. the following mathematical methods are conceivable:

- Kalman filters or fuzzy methods

- Neural Networks.

The manual methods are characterized by the involvement of the user. The following methods are suitable for the manual selection of a picture position, if necessary using a previous automatic preselection of picture positions:

- Mouse interaction on the screen of the navigation device: The surgeon or assistant clicks on the desired position in the displayed video image. For this selected

Image position, the corresponding 3D position is subsequently determined and displayed.

- Gesture control: With the help of suitable sensors (eg PMD camera or Microsoft Kinect) the movement of the user can be recorded and evaluated. Thus, for example, the hand movement can be tracked and interpreted as a gesture which allows the selection of the desired 2D image point. So z. B. a hand movement to the left to move the image position to be controlled also to the left. Alternatively, in a previous automatic preselection of image positions with the gesture control, the final selection of the given image position could be controlled.

- Foot pedal control: With the aid of a foot pedal, the user can select the final selection of the given position between image positions of a previous automatic preselection

Image position are controlled. If the surgeon presses on a foot pedal, the output changes. Selects the selected image position, which is used to calculate and visualize an SD position on the patient.

- Voice control: Similar to the gesture control, the selected picture position can be moved directly by voice commands. So z. For example, the voice command "left" will shift the 2D image position to the left by a predetermined offset.

- Touchscreen: If the camera image is displayed on a touch screen or multi-touch screen, the operator can directly select the 2D image position by touching it at the screen.

- Mechanical input devices: If mechanical input devices are connected to the navigation system (eg keyboard, control buttons), the user can control the 2D image position via these input devices. Here either a shift of the current image position or a change of the selection in a set of preselected image positions can be triggered. The novelty of this concept therefore basically lies in the possibility of calculating the corresponding 3D position in the volume image data for any 2D positions in image data of an intraoperatively used and tracked camera. Also new are the methods described for selecting or defining the 2D position in the camera image for which the 3D information is to be calculated and displayed. A navigation system visualizes position information in 3D image data of the patient for any image positions of the camera image data. As a technical application areas include in particular medical technology, but also all other applications in which an instrument-like system according to the principle described is used.

In detail, FIG. 6 shows an operating device 1000 with a device head 100, as already illustrated with reference to FIG. 1, and an exemplary image representation of a monitor module in versions (A), (B), (C), and in detail a representation either from image data 300, surface model 310, or volume rendering 320, or a combination thereof.

For the version (A), the surface model 310 is combined with the volume rendering 320 via a computing means 240. In the present case, the referenced representation of surface model 310 with volume rendering 320 as well as pose of operation camera 21 1 is achieved because the selection and / or monitor means provides image data 300 with an automatic selection of surface locations OS1, OS2. which can be selected via a mouse pointer 501 of the monitor module 500. For this purpose, the option of a mouse pointer 501 'is selected in the selection menu 520 of the monitor module 500. A predefinition of the surface locations OS1, OS2 can take place according to the specifications of the monitor module via a distance measure or a topography or a color representation in the image data.

In version (B), a default geometry 521 can be created via the selection menu 510, in this case a circle specification, so that only the surface location OS1 is displayed insofar as the final selection determines. In a third modification, a raster selection 531 can be selected in a selection menu 530, for example to display all structures in the second quadrant x-this only leads to the display of the second surface location OS2.

7 shows a preferred sequence of steps for carrying out a method for medical navigation of arbitrary pixels in medical camera image data 300. It is to be understood that each of the method steps explained below can also be implemented as an action unit in the context of a computer program product that is designed to be executed of the explained method step. Each of the action units recognizable from FIG. 7 can be realized within the scope of a previously explained image data processing unit, image storage unit and / or navigation unit. In particular, the action units are suitable for implementation in a registering means, a virtual pointer means and a correspondingly formed computing means; d. H. Registration means adapted to locate the operation camera relative to the surface model; virtual pointing means adapted to automatically provide a number of surface locations in the surface model; Calculating means which are designed to automatically associate with at least one of the surface points provided a volume location of the volume representation.

Starting from a node K1, in a first step VS1 of the method, a preoperative volume rendering 320 with volume representation data is provided, here for example in a memory unit 232 of an image storage unit 230. In a second method step VS2 starting from the node K2, image data 300 are taken as camera image data of an operation camera 21 1, 212 are provided, from which a surface model 310 can be created via the image data processing unit 220 shown in FIG. 1 and can be stored in a memory unit 231 of the image data storage unit 230 in a method step VS3. In a proceeding from the node K3 fourth step VS4 is a camera registration by means of registration, in particular in the context of a position measuring system 400 and / or a visual navigation, for example using a SLAM method. For this purpose, in particular the pose KP of the operation camera 21 1, 212- namely in a method step VS4.1 the focal point coordinate KP1 and in a method steps VS4.2 the orientation KP2- recorded. In a fifth method step VS5, a virtual pointing means, such as for example a sight ray 330, can be made available as a virtual pointing means. In general, a number of surface locations in the surface model can be automatically made available with the virtual pointer means. For this purpose, use is made of the known imaging properties of the camera and of the shape and relative position of the surface model for determining a selection of surface locations. In the specific case, the surface location 31 1 may be fixable as an intersection of a virtual viewing beam 330 emanating from the surgical camera 21 1, 212 with the surface model 310, in particular a surface coordinate indicating a pixel 301 of the surgical camera 21 1, 212 assigned to the intersection point. In a sixth method step VS6, using suitable transformations; previously TKL, TLL1, TLL2, TLL3- via a computing module 240, a referencing of the volume representation 320 and the surface model 310 take place. Alternatively or in combination, in a seventh method step VS7.1, VS7.2 a preliminary and / or final selection of all possible objects - in particular surface locations and / or volume locations US, VP by means of a selection and / or monitor means available become. In an eighth method step VS8, the selected locations can be displayed by referencing the camera 21 1 and the volume and surface representations 320, 310 to an output module, as has been explained with reference to FIG. 6, for example. At the node K3, a loop can be made, for example, to the previously explained nodes K1 and / or K2 in order to allow the method to proceed completely from the beginning. It may also be a loop to only one of the nodes K1, K2 formed. Also, only part of the process may be repeatable, e.g. B. when a camera position changes, so that, for example, a loop to the node K3 and the method step VS4 can be implemented. Also, only the selection sequence can be repeatable, so that a loop for method step VS7 is feasible. LIST OF REFERENCES

100 device head

1 10 endoscope

200 imaging system 210 image data acquisition unit

21 1, 212 operation camera

220 image data processing unit

230 image storage unit

231, 232 Memory area

240 computing resources

250 pointers

260 registration means

300 image data

301 pixel

310 surface model

31 1 intersection

320 volume rendering

330 sight beam

400 position measuring system

410 position measuring unit

420, 421, 430 locator

422 determinate connection

420T, 421T, 430T tracker

500 selection and monitor media

501, 501 'mouse pointer 510, 520, 530 Selection menu 521 Geometry default 531 Grid default 1000 Operating device 1001 System with an operating device 1002 Applied system with an operating device

G Tissue structure KP Camera position, pose KP1 Position, focal point coordinates KP2 Alignment 02 Tissue position

OS, OS1, OS2 Surface area OU Operating environment OU1 Roundish object OU2 Oblong object

R212, R320, R410, R420, R421, R430 reference coordinate system

TKL transformation

TLL1 Transformation between the camera localizer reference coordinate systems and the position encoder

TLL2 Transformation between the reference locator systems of the object locator and the position measuring system TLL3 Transformation between the reference co-ordinate systems of the 3D image data and the object locator) U Environment

VP, VP1, VP2 volume points

Claims

claims

An imaging system (200), in particular for an operating device (1000) having a mobile device head (100), comprising:

an image data acquisition unit (210) having an image acquisition unit, in particular an operation camera (21 1, 212), which is designed to acquire image data (300) of an operating environment (OU),

an image data processing unit (220) configured to provide the image data,

an image storage unit (230) configured to hold the image data (300) of the operating environment and volume data of a volume representation (320) associated with the operating environment, further characterized by

Registration means (260), which are designed to localize the image acquisition unit, in particular the surgical camera (21 1, 212), relative to the operating environment,

- Virtual pointer means (250) adapted to automatically provide a number of pixels of the image data (300), and

- Mapping means, in particular with computing means (240), which are adapted to automatically assign at least one of the pixels provided a volume point (VP) of the volume representation (320).

2. System according to claim 1, characterized in that the image recording unit is adapted to continuously acquire image data (300) of the operating environment (OU) and / or the image data (300) of the operating environment can be recorded on a device head.

3. System according to claim 1 or 2, characterized in that the image recording unit as an operation camera (21 1, 212), in particular an optical surgical camera (21 1, 212) is formed.

4. System according to any one of the preceding claims, characterized in that the pixels are formed as surface locations (OS) from the image data (300) and - The image data processing unit (220) is adapted to create a surface model (310) of the operating environment by means of the image data, and / or

- An image memory unit (230) is designed to hold the surface model representation data, and / or

- The registration means (260) are adapted to locate the operation camera (21 1, 212) relative to the surface model (310) of the operating environment, and / or

- The virtual pointer means (250) are adapted to provide a number of surface sites (OS) in the surface model.

5. System according to any one of the preceding claims, characterized in that the image data surface representation data, in particular a surface model, preferably for determining surface locations, and / or the volume data volume representation data.

6. System according to any one of the preceding claims, characterized in that the virtual pointer means (250) are adapted to automatically provide a number of pixels of the image data (300), in particular surface locations (OS), in particular of the surface model, by identifying them and / or displayed.

7. System according to one of the preceding claims, characterized in that the registration means comprises a position measuring system (400), in particular an optical or electro-magical localizer detection system comprising:

a physical patient locator (430), a physical camera locator (421), in particular a physical instrument locator (420).

8. System according to one of the preceding claims, characterized in that the registration means (260) comprises a number of registration modules, in particular as part of the image data acquisition unit (210), the image data processing unit (220) and / or a navigation unit, wherein:

a first registration module, in particular the image data acquisition unit (210), is designed to acquire and provide image data (300) of an environment (U) of the device head, in particular continuously, and

a second registration module, in particular the image data processing unit (220), is designed to create a map of the environment (U) by means of the image data (300), and a third registration module, in particular a navigation unit, is configured to use the image data (300) and an image flow to indicate at least one position of the device head (100) in a proximity environment (NU) of the operating environment using the map such that the mobile device head (100 ) is feasible based on the map.

9. System according to any one of the preceding claims, characterized in that a position reference to the device head (100), associated therewith guide means is formed to make information on the position of the device head (100) with respect to the environment (U) in the map, the environment (U) goes beyond the near environment (NU).

10. System according to one of the preceding claims, characterized in that the surface location (OS) in the surface model (310) is associated with a surface coordinate in surface model representation data and the volume point (VP) of the volume representation (320) a volume coordinate in volume representation - Data is assigned.

1 1. System according to one of the preceding claims, characterized in that the surface location (OS) as an intersection (311) of an operating camera (21 1, 212) outgoing assignment means with the image data (300) can be fixed, in particular as an intersection (31 1 ) of a virtual viewing beam (330) emanating from the surgical camera (21 1, 212) can be fixed with the surface model (310), in particular a surface coordinate a pixel (301) assigned to the intersection point of the image data (300) of the surgical camera (21 1, 212).

12. System according to any one of the preceding claims further characterized by a selection and / or monitor means (500), which is formed a freely selectable and automatically provided and specified number of pixels, in particular surface locations (OS) to group in a selection and to visualize the selection in a selection representation.

13. System according to one of the preceding claims, characterized in that

- The number of pixels, in particular surface locations (OS) in the surface model (310), freely selectable, in particular free of a physical display, are available and / or

- The number of pixels, in particular surface locations (OS) in the surface model (310), are automatically fixed, and / or the selection comprises, at least in part, an automatic pre-selection and / or an automatic final selection.

14. System according to claim 12 or 13, characterized in that a selection and / or monitor means (500) is formed, the automatic selection, in particular in a preselection and / or final selection, based on the image data (300) and / or the surface model (310), in particular based on first grouping parameters comprising: a distance measure, a 2-D or 3D topography, a color.

15. System according to one of claims 12 to 14, characterized in that a selection and / or monitor means (500) is adapted to group an automatic selection independently of the image data (300) and / or the surface model (310), in particular Base of second grouping parameters comprising: a geometry preset, a grid preset.

16. System according to one of claims 12 to 15, characterized in that a selection and / or monitor means (500) is adapted to group an automatic selection, in particular final selection, by means of evaluation methods, in particular comprising: evaluation method with a statistical evaluation and / or Filtering the preselected surface sites (OS), in particular by means of a Kalmann filter, a fuzzy logic and / or a neural network.

17. System according to one of claims 12 to 16, characterized in that the selection and / or monitor means (500) comprises a MAN-machine interface (MMI), which can be actuated for manual selection support, in particular by one or more of the Input features selected from the group comprising: hand or foot keyboard means such as a mouse, key, pointer or the like, a gesture sensor, a voice sensor, a touch-sensitive sensor such as an input pad, a touch screen, a mechanical input means.

18. Operating device (1000) with an imaging system (200) according to any one of the preceding claims, in particular comprising a mobile hand-held device head (100).

19. The method (FIG. 7) for imaging, in particular for an operating device (1000) according to claim 18, comprising the steps:

Detecting and providing image data (300) of an operating environment (OU), in particular by means of creating a surface model (310) of the operating environment (OU) the image data (300),

- maintaining the image data (300) of the operating environment (OU) and volume data of a volume representation (320) associated with the operating environment,

further characterized by the steps:

Localizing an image acquisition unit, in particular the operation camera (21 1, 212) relative to the operating environment,

virtual automatic display of a number of pixels, in particular surface locations (OS), of the image data (300),

- Automatic assignment of at least one of the provided pixels, in particular surface locations, to a volume location of the volume representation.

20. The method according to claim 19, characterized in that the surface location (OS) as an intersection (31 1) of the operation camera (21 1, 212) outgoing virtual sight beam (330) with the surface model (310) is set, in particular a surface Coordinate one the intersection (31 1) associated pixel (301) of the image data (300) of the operation camera (21 1, 212) indicates.

21. The method of claim 19 or 20 further characterized by grouping a freely selectable and automatically provided and fixed number of pixels, in particular surface locations (OS), in a selection and visualization of the selection in a selection representation.

22. The method according to any one of claims 19 to 21, characterized in that the number of pixels, in particular surface locations (OS) in the surface model (310), are freely selected, in particular free of a physical display, are provided.

23. The method according to any one of claims 19 to 22, characterized in that the number of pixels, in particular surface locations (OS) in the surface model (310), is automatically determined.