DE102009019019A1 - Video-based mono camera navigation system - Google Patents

Abstract

The invention relates to a method for producing a video-based mono-camera navigation system. The system is able to determine the location information of an object using a single camera and markers with different geometries and colors. The system includes a video camera (5), an electronic data processing system (8), a first marker (6) of a specific pattern and color, located in a known fixed position with respect to a stationary or mobile object (10) , and a second marker (7) provided with another particular pattern and color and associated with a position-variable object (2) whose respective position is to be detected in a measuring area. This system has the advantage over known solutions that only one video camera (5) is required and only such technical demands are placed on this camera (5) that are met by virtually any commercially available video camera.

Description

The The invention relates to a video-based camera navigation system, consisting of a device and a method which the Detection of tagged objects in the room and their Location with only a video camera allows.

It is known, the positional relationship of objects to each other with optical To determine navigation systems. For example, in the fields warehousing, transportation, metalworking and the surgery. In surgery, these systems are in the operating room applied to the position of instruments relative to the patient to determine and thus the chances of success of an operation too improve.

In "Khan M, corner U, man WJ:" The application of an optical navigation system in endonasal sinus surgery ", ENT, 2003 ; Stange T, Schultz-Coulon, Hans-Jürgen: "Clinical experience with an optical navigation system in clinical routine surgery", 77th Annual Meeting of the German Society of Oto-Rhino Laryngology, Head and Neck Surgery, 2006 ; Nobuhiko Sugano: "Computer Assisted Navigation in Hip Arthroplasty", Springer Berlin, 2001 " Optical stereo camera navigation systems are described. At least two or more cameras circularly arranged diodes emit infrared light. This is reflected by markers located on the patient and surgical instruments. The reflected infrared light is detected by at least two cameras. Software-supported triangulation determines the spatial coordinates of the markers in the room.

The Coordinate systems of the camera, of the patient (CT sectional images) and the surgical instrument are registered by registration in a uniform coordinate system.

Thereby it is possible in surgical CT scans Instrument or a so-called pointer. The surgeon is thus able to display in CT scans on a monitor to determine the current position of the instrument on or in the patient. Recognized by the graphical representation of the instrument in the CT slice images the surgeon's distance to the target area and the one to be protected Risk structures (eg nerves or vessels).

From the publication US 2004/0002642 A1 is known a stereo optical camera navigation system which operates in the visible wavelength range. The localization is realized here by the recognition of black and white contrast markers with two cameras.

The Disadvantages of these known solutions are that at least two cameras needed in defined distance to each other be used to capture objects in space. It is problematic, the associated large housing or equipment cart in an operating room.

One Another aspect is the high costs due to the special stereo cameras used.

Of the Invention is based on the object, a video-based camera navigation system to develop that realized with the use of a single video camera can be and to such a video camera no special requirements be put. This makes it possible to be a commercial available video camera to use.

These Task is solved by an optical measuring system, which from a device according to claim 1 and a method according to claim 5.

• • eine Videokamera,
• • eine elektronische Datenverarbeitungsanlage,
• • ein mit einem bestimmten Muster und einer bestimmten Farbe versehener erster Marker, der sich in einer bekannten festen Position zu einem ortsfesten oder ortsveränderlichen Objekt befindet, und
• • ein zweiter Marker, der mit einem anderen bestimmten Muster und einer anderen bestimmten Farbe versehenen ist und mit einem lageveränderlichen Gegenstand, dessen jeweils eingenommene Lage in einem Messbereich erfasst werden soll, verbunden ist.

Belong to the device

• • a video camera,
• • an electronic data processing system,
• • a first marker provided with a specific pattern and color, which is in a known fixed position with respect to a stationary or mobile object, and
• • a second marker, which is provided with a different pattern and a different color, and which is connected to a variable object whose position is to be recorded in a measuring area.

The Both markers together form a unit for recording the spatial position and rotation in environments of the virtual Reality of a first, spatially fixed or movable object with respect to a second, fixed or movable object and will as well the video camera connected to the electronic data processing system.

At the video camera to monitor the measuring range No special requirements. For this one can Commercially available video camera can be used over a USB or other data interface with the electronic Data processing system is connected.

On the objects, such as persons, instruments or tools whose location parameters are to be determined, a characteristic feature (marker) is attached. The marker is provided with various markings or geometries, which are provided by the fully automatic image processing be recognized by the video camera and can be assigned by the color-shape coding. The size, color and shape of the marker is arbitrary. The base color should differ only with the colored markings applied to it. The color tones of the markers can theoretically be chosen arbitrarily. For optimal image processing, however, strong color contrasts are preferable. For the position calculation, characteristic points of the geometry such as corner points, centers or centers of gravity are used. The shape, the color and the distance or the arrangement of the geometries on the respective marker is known to the mono-camera navigation system.

A Specially developed image processing software attacks the recorded Pictures of the video camera on and detected in these fully automatic the colored markers. By the subsequent assignment the characteristic heavy or corner points of the markings in the recorded camera image to the attached to the marker geometries the consideration of the camera characteristics is the Position calculation of the object in the room. The image processing software does not need special hardware. It can be on a commercial PC installed and running.

• a) Anbringen eines ersten Markers in einer festen Position am ersten feststehenden oder beweglichen Objekt,
• b) Anbringen eines zweiten Markers am zweiten feststehenden oder beweglichen Objekt,
• c) Ausrichten der Videokamera auf beide Marker unter der Berücksichtigung ihres möglichen Bewegungsbereiches,
• d) Eingabe der Position des feststehenden Zielgebiets,
• e) Bildakquirierung über die Videokamera,
• f) Bildvorverarbeitung,
• g) Objekterkennung,
• h) Zuordnung der Objekte zu den bekannten Geometrien – Klassifizierung,
• i) Farbbestimmung der Geometrien der Markierungen,
• j) Zuordnung der aufgenommenen charakteristischen Bildpunkte (beispielsweise Eckpunkte) einer Geometrie zu den am realen Objekt angebrachten Markierungen,
• k) Berechnen der Transformation T über den zugeordneten Punkten der Marker und den Kameraparametern (Brennweite, Bildverzerrung),
• l) Anzeigen des Bildes auf dem Bildschirm,
• m) Erfassung der neuen Positionen der beiden Marker durch die Videokamera,
• n) Senden der neuen Kameraparameter an die elektronische Datenverarbeitungsanlage,
• o) Wiederholung der Schritte e) bis n).

The navigation system works according to the following process steps:

• a) attaching a first marker in a fixed position on the first fixed or movable object,
• b) attaching a second marker to the second fixed or movable object,
• c) Aligning the video camera on both markers taking into account their possible range of motion,
• d) entering the position of the fixed target area,
• e) image acquisition via the video camera,
• f) image preprocessing,
• g) object recognition,
• h) assignment of the objects to the known geometries - classification,
• i) color determination of the geometries of the markings,
• j) assignment of the recorded characteristic pixels (for example corner points) of a geometry to the markings attached to the real object,
• k) calculating the transformation T over the associated points of the markers and the camera parameters (focal length, image distortion),
• l) displaying the image on the screen,
• m) recording of the new positions of the two markers by the video camera,
• n) sending the new camera parameters to the electronic data processing system,
• o) repetition of steps e) to n).

by virtue of the use of a commercially available video camera, the simple Structure of the markers and the use of a standard PC's is the Users offered a very economical navigation solution. The cost-effective hardware allows the To establish technology in new market fields and make them a bigger one To provide quantity to potential customers.

• – Der Aufbau des Systems wird sehr klein gehalten. Es müssen keine Kameras entfernt im Raum aufgestellt werden, was die Flexibilität und Nutzbarkeit des Systems einschränkt.
• – Es können beliebige Kamerasysteme eingesetzt werden. Somit ist eine Anpassung an die Messanforderung, den Stand der modernsten Kameratechnik und wirtschaftlicher Systeme ist jederzeit ohne größere Aufwendungen möglich.
• – Die Sichtbarkeit der Marker (Line of Sight-Problem) ist besser gewährleistet, da sich die Videokamera im Gegensatz zu herkömmlichen Systemen nicht zwei bis drei Meter vom zu messenden Gebiet entfernt befindet. Am bevorzugten Beispiel eines medizinischen Einsatzes wird sie entweder am OP-Tisch oder direkt am Patienten bzw. chirurgischen Instrument angebracht.
• – Die Kameraposition kann flexibler an die Dimensionen des zu vermessenen Gebietes angepasst werden. Bei einem kleinen Areal kann die einzelne Videokamera problemlos näher an das zu vermessende Gebiet bewegt werden. Muss ein größeres Gebiet überwacht werden, wird die Kamera in einem größeren Abstand aufgestellt. Die Auflösung der Videokamera und die damit korrelierende Messgenauigkeit kann dadurch optimal an die jeweilige Messsituation angepasst werden. Eine höhere Messgenauigkeit wird bei kleineren und eine gröbere Messgenauigkeit bei größeren Gebieten erzielt.

Another advantage to previous optical navigation systems is that instead of two or more, only one camera is used for position determination. This novel implementation solves various problems:

• - The structure of the system is kept very small. No need to place cameras in the room, which limits the flexibility and usability of the system.
• - Any camera system can be used. Thus, an adaptation to the measurement requirement, state of the art camera technology and economic systems is possible at any time without major expenses.
• - The visibility of the markers (line of sight problem) is better ensured, since the video camera, in contrast to conventional systems is not two to three meters from the area to be measured. The preferred example of a medical insert is either attached to the operating table or directly to the patient or surgical instrument.
• - The camera position can be adapted more flexibly to the dimensions of the area to be measured. In a small area, the single video camera can be easily moved closer to the area to be surveyed. If a larger area needs to be monitored, the camera will be set up at a greater distance. The resolution of the video camera and the correlating measuring accuracy can thus be optimally adapted to the respective measurement situation. Higher measurement accuracy is achieved with smaller and coarser measurement accuracy in larger areas.

embodiments

The Invention will be described below with reference to 3 embodiments and associated drawings explained in more detail, all the application of the mono camera navigation system in the operation of a patient.

It demonstrate:

1 the construction of a mono-camera navigation system in a schematic representation,

2 Possibilities for the construction of markers in a schematic representation and

3 the sequence of process steps for the image processing and tracking process.

This in 1 The mono-camera navigation system shown is housed in an operating room for assistance and monitoring during an operation for the operating surgeon so that he is on the screen 1 the location of his surgical instrument 2 to the place or target area 3 surgery with healthy and diseased tissues and risk structures. The screen 1 belongs together with the processor 4 and the drawing keyboard and computer mouse not shown to the electronic data processing system 8th , For optically detecting the respective situation is a video camera 5 provided also with the electronic data processing system 8th connected is.

Essential to the invention is the use of two markers 6 . 7 , which together form a unit for detecting the spatial position and rotation in virtual reality environments of a first, spatially movable object with respect to a second stationary object. The one marker 6 is in a fixed position on the patient 10 attached while the second marker 7 in a certain position on the surgical instrument 2 is attached. Both markers 6 . 7 are after 2 with markings 9 provided after 2 have a certain color and a certain shape and are arranged in different fixed positions. The letters entered in the outlines are to be given an example of certain colors such as g for yellow, R for red, O for orange and so on. Due to the different colors and contours as well as the different arrangement of the markings can with only one video camera 5 the respective position of the two markers 6 . 7 to each other and the position and position of the surgical instrument 2 to the place or target area 3 of the surgical procedure and on the screen 2 being represented.

1st embodiment

This 1st embodiment involves the use of the camera navigation system in the operation of a patient, wherein the operative target area occupies the same position and position on the operating table during the entire surgical procedure. It is of great importance to protect the operative target area while at the same time sparing the risk structures such as B. Nerves or vessels to reach. The assistance of the surgeon in the operation by the mono-camera navigation system consists in the optical navigation. The surgeon can if necessary on the screen at any time 1 see exactly where he is with his surgical instrument 2 exactly located.

If the on the instrument 2 attached markers 7 opposite to the patient 10 fixed markers 6 moved, this is due to the different patterns and colors through the video camera 5 recognized and on the screen 1 so the surgeon can see the location of his instrument even in difficult situations 2 to the operative target area 3 can recognize.

By a point-based registration, the coordinate systems of the patient (CT / MRI slice images), video camera 5 and instrument 2 transferred into a coordinate system.

While the surgeon the instrument 2 at the same time he can determine the actual position of the instrument he controls 2 on a screen 1 recognize in the CT slice images.

• 1. Anbringen eines ersten Markers 6 ist in einer festen Position am Patienten 10,
• 2. Anbringen eines zweiten Markers 7 am Instrument 2 des Chirurgen,
• 3. Eingabe der Position des operativen Zielgebiets 3,
• 4. Eingabe der Position des Angriffspunktes des chirurgischen Instruments 2,
• 5. Bildakquirierung über die Videokamera 5,
• 6. Bildvorverarbeitung,
• 7. Objekterkennung,
• 8. Zuordnung der Objekte zu den bekannten Geometrien-Klassifizierung,
• 9. Farbbestimmung der Geometrien der Markierungen,
• 10. Zuordnung der aufgenommenen charakteristischen Bildpunkte (beispielsweise Eckpunkte) einer Geometrie zu den am realen Objekt angebrachten Markierungen,
• 11. Berechnen der Transformation T über den zugeordneten Punkten der Marker und den Kameraparametern (Brennweite, Bildverzerrung),
• 12. Anzeigen des Bildes auf dem Bildschirm 1,
• 13. Erfassung der neuen Positionen der beiden Marker 6, 7 durch die Videokamera 5,
• 14. Senden der neuen Kameraparameter an die elektronische Datenverarbeitungsanlage 8,
• 15. Wiederholung der Schritte 5 bis 14.

The procedure for using the video-based mono-camera navigation system in the operation is as follows:

• 1. Attaching a first marker 6 is in a fixed position on the patient 10 .
• 2. Attach a second marker 7 on the instrument 2 of the surgeon,
• 3. Enter the position of the operative target area 3 .
• 4. Entry of the position of the point of application of the surgical instrument 2 .
• 5. Image acquisition via the video camera 5 .
• 6. image preprocessing,
• 7. object recognition,
• 8. Assignment of the objects to the known geometry classification,
• 9. color determination of the geometries of the markings,
• 10. Assignment of the recorded characteristic pixels (for example corner points) of a geometry to the markings attached to the real object,
• 11. Calculate the transformation T over the assigned points of the markers and the camera parameters (focal length, image distortion),
• 12. Display the image on the screen 1 .
• 13. Detecting the new positions of the two markers 6 . 7 through the video camera 5 .
• 14. Send the new camera parameters to the electronic data processing system 8th .
• 15. Repeat steps 5 through 14.

2nd embodiment

In operations on the brain, its position is changed after cranial opening (brain shift). Since the MRI data were taken preoperatively the actual position of the brain tumor as the operative target area 3 no longer exactly determinable.

A 2D ultrasound head scans the patient's brain intraoperatively. The described invention is combined with the 2D ultrasound system. Both on the patient 10 as well as on the 2D ultrasound head are markers 6 . 7 different pattern and color. By a point-based registration, the coordinate systems of the patient (MRI slice images), video camera 5 and 2D ultrasound head converted into a coordinate system.

Of the Location of the 2D ultrasound head is recorded at the time of taking the 2D ultrasound image. Thus, it is possible from the 2D a 3D ultrasound image to investigate. The intraoperative 3D ultrasound image is combined with the superimposed preoperatively acquired MRI slice images.

Consequently the surgeon receives the information about the current actual position of the brain tumor.

3. Application example

The mono camera system can be used as a direct 3D input device for various program and operating system surfaces. For this, the position information of the tracker is converted into actions and positions of the computer mouse. The marker 6 can be attached to the hand on the head or to different objects, z. B. take over the control of the computer mouse. In 3D scenes or in 3D games is therefore also a control of the video camera 5 possible with, for example, the rotation or movement of the head. By the direct connection between real movement of the patient's head 10 and changing the virtual camera position, the user gains a vivid and realistic impression of the 3D scene.

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• US 2004/0002642 A1 [0006]

Cited non-patent literature

• - "Khan M, corner U, man WJ .:" The application of an optical navigation system in endonasal sinus surgery ", ENT, 2003 [0003]
• - Stange T, Schultz-Coulon, Hans-Jürgen: "Clinical Experience with an Optical Navigation System in Clinical Routine Operation", 77th Annual Meeting of the German Society of Oto-Rhino Laryngology, Head and Neck Surgery, 2006 [0003]
• - Nobuhiko Sugano: "Computer Assisted Navigation in Hip Arthroplasty", Springer Berlin, 2001 " [0003]

Claims (5)

Device for a video-based camera navigation system, consisting of • a video camera ( 5 ), • one, from a keyboard, a computer mouse, a processor ( 4 ) and a screen ( 1 ) existing electronic data processing system ( 8th ), • one with two, from the video camera ( 5 ) optically distinguishable identification types provided first marker ( 6 ), which is in a known fixed position, and • a second, with two, of the video camera ( 5 ) markers provided with optically distinguishable types of markings ( 7 ) which is rigidly connected to a position-variable object, each of which is the first marker ( 6 ) is to be recorded in a measuring range, Device for a video-based camera navigation system according to claim 1, characterized in that the two on the markers ( 6 . 7 ) and from the video camera ( 5 ) optically distinguishable types of identification consist of a two-dimensional characteristic with sharp contours that are filled with a particular color. Device for a video-based camera navigation system according to claims 1 and 2, characterized in that the markers ( 6 . 7 ) with different markings ( 9 ), consisting of several characters, are provided. Device for a video-based camera navigation system according to claim 1, characterized in that it as an input device suitable for various program and operating system surfaces is. Method for using the device for a video-based camera navigation system according to claim 1, characterized by the following method steps: a) attaching a first marker ( 6 ) in a fixed position on the first object ( 10 b) attaching a second marker ( 7 ) on the second object ( 2 ), c) Aligning the video camera ( 5 ) on both markers ( 6 . 7 ), d) input of the position of the fixed target area, e) image acquisition via the video camera ( 5 f) image preprocessing, g) object recognition, h) assignment of the objects to the known geometry classification, i) color determination of the geometries of the markings, j) assignment of the recorded characteristic pixels (for example corner points) of a geometry to the markings applied to the real object , k) calculating the transformation T over the associated points of the markers and the camera parameters (focal length, image distortion), l) displaying the image on the screen ( 1 ), m) recording the new positions of the two markers ( 6 . 7 ) through the video camera ( 5 ), n) sending the new camera parameters to the electronic data processing system ( 8th ), o) repetition of points e) to n).