DE102022126824A1 - Method for superimposing overlay recording information with a live image and a corresponding device - Google Patents

Abstract

The invention relates to a method for superimposing superimposed recording information (140) with a live image (135), in particular with an endoscopic and/or an exoscopic and/or a laryngoscopic and/or a microscopic live image (135), comprising at least the following steps: capturing at least one recorded image (134) of a moving object (112) by means of a moving image capturing device (104); determining the superimposed recording information (140) in the at least one captured recorded image (134) at least with regard to the object (112); capturing at least one live image (135) of the moving object (112) following the recorded image (134) by means of the moving image capturing device (104); Calculating a transformation rule at least between the recorded image (134) and the live image (135) in order to track a change in position and/or location and/or depth of the moving object (112) between the recorded image (134) and the live image (135); and overlaying the previously determined overlay recording information (140) with the live image (135) at least as a function of the calculated transformation rule.

Description

The present invention relates to a method for superimposing overlay recording information with a live image, in particular with an endoscopic and/or an exoscopic and/or a laryngoscopic and/or a microscopic live image, according to independent claim 1. The invention further relates to a device with a movable image capture device with an image sensor and at least one image capture unit, a computing device and a display device according to independent claim 15.

Optical visualization systems, such as microscopes, exoscopes and endoscopes, enable a representation of a scene and/or a work area in which fine motor work and/or visual inspections are performed. In medical procedures, for example, the work area is an operating field in an internal region, for example within a thorax or a head, of the human body.

Exoscopy describes the observation and, if necessary, illumination of an operating field on a patient and/or an object field on any object from a location away from, i.e. outside, the patient's body or away from the object.

Endoscopy describes an imaging technique in which an endoscope is inserted into a cavity. The specialist who carries out such an intervention with an endoscope looks at the image captured by the endoscope on a screen and can direct his actions on the basis of this image. In medical procedures, for example minimally invasive procedures, an endoscope is inserted into the body in order to capture an internal image of the body and display it on the screen. Because the specialist often needs to work precisely, it is desirable to provide as accurate, high-resolution an image as possible of the cavity and/or the work area where the examination and/or operation is to be carried out.

The images captured by the endoscope and displayed on a display device are usually two-dimensional, so that even the specialist personnel is unable to accurately determine the dimensions and/or measurements of an object being viewed in the imaged scene due to the lack of depth information.

In order to solve this problem and/or to enable depth measurements in a measurement image, three-dimensional stereo endoscopes and/or stereo exoscopes and/or microscopes are also known, in which a scene is recorded from two different angles by two image capture units. The two image capture units record one image of the scene to be viewed, preferably synchronized per unit of time and angle of view. This results in so-called stereo measurement image pairs with a common time stamp. Depth information can be evaluated from such a stereo measurement image pair using stereo reconstruction methods.

During this stereo reconstruction, a disparity, i.e. a horizontal pixel offset, is determined in the respective stereo image pairs by an algorithm pixel by pixel, i.e. per pixel. Objects at a great distance have a small disparity, i.e. a small pixel offset, within a respective image pair in a stereo capture. Objects in the foreground, on the other hand, have a large disparity. From the disparity, with knowledge of further optical parameters of the image capture device(s), a type of depth map can be calculated for the stereo-reconstructed stereo measurement image, which, for example, includes depth information of the object to be viewed pixel by pixel. Using the depth information, it is possible, for example, to determine a distance between any two image points in Euclidean space.

The use of fluorescence endoscopy and/or fluorescence exoscopy and/or fluorescence laryngoscopy in the surgical environment is also known from the state of the art. The use of multispectral imaging (MSI) and/or hyperspectral imaging (HSI) is also known, through which individual tissue areas can be highlighted by displaying multi- or hyperspectral tissue parameters.

Fluorescence endoscopy is an endoscopic examination procedure in which substances (e.g. monoclonal antibodies) are used that are marked with fluorescent dyes, for example with indocyanine green (ICG). After the fluorescent dye is injected into the relevant tissue area and/or the patient's bloodstream, the dye and its carrier material accumulate in dysplasias or tumors, for example, due to its specific properties. This is known as the dye's flooding behavior. By illuminating the tissue area with an excitation light source, the introduced dye is stimulated to fluoresce, so that the altered tissue is clearly visible during the flooding phase. This gives the surgeon a visual indication before and/or during the operation. on the damaged tissue or on the tissue to be removed. In other words, in fluorescence imaging, the flooding behavior of the fluorescent dye provides crucial information for diagnosing the tissue in question. The fluorescence triggered by the illumination increases in intensity during the flooding phase and reaches its maximum intensity after a certain period of time. At this point, the organ areas and/or objects in question are most prominent. After this, the intensity decreases again. With ICG, the excitation and fluorescence are in the near infrared. Fluorescence endoscopy usually requires a cold light source, at least one optical filter and/or at least one optical blocking filter and a commercially available endoscope. A large number of systems from the applicant are available on the market, for example the Tipcam Rubina with ICG imaging, the TH 121 for ICG imaging or the TH 112 for PDD imaging.

Since the contrast agent is usually only injected once, the user can only observe the flooding behavior once. This applies in a similar way to multispectral and/or hyperspectral imaging, for example when observing irreversible tissue changes and the resulting changes in the tissue properties recorded by the multispectral or hyperspectral imaging, such as oxygen saturation. It is therefore desirable for a surgeon and/or a doctor commissioned to carry out an examination to be able to observe and/or understand the flooding behavior again after the flooding phase, namely after the fluorescent dye has broken down again.

To partially solve this problem, it is known from the prior art that a relevant tissue area is recorded using a camera during the flooding phase and the resulting video sequence is saved. If the intensity of the fluorescence decreases later on, the previously recorded and saved video sequence of the fluorescence can be called up and, for example, overlaid with a current white light image of the relevant tissue area. The problem here is that the current view of the tissue may have changed from the view at the time of the flooding phase, for example due to movement of the tissue or movement of the camera.

Overlaying fluorescence image data with 3D images is, for example, possible in US10,849,710 B2 for preoperatively recorded 3D images. These image data are recorded, for example, by a C-arm, which is called registration. The recorded image data are adjusted to the 3D images. Saving and repeatedly playing back fluorescence images for overlay with a white light video is in the EP 359 048 A1 and the JP 2019 098 008 A described. From the JP 5160 276 B2 It is also known that a vessel image from fluorescence images is combined with a white light image. Furthermore, the US8,767,057 B2 a capsule endoscope in which the movement of a camera capsule is determined from previously recorded image data and taken into account when creating a 3D model. From the scientific publication Selka, F., Agnus, V., Nicolau, S., Bessaid, A., Soler, L., Marescaux, J., & Diana, M. (2014, July). Fluorescence-based enhanced reality for colorectal endoscopic surgery. In International Workshop on Biomedical Image Registration (pp. 114-123) . Springer, Cham also knows how to overlay a white light video sequence with an "ischemic map" generated from a fluorescence sequence. ICG is used to check the blood flow of tissue after an anastomosis, i.e. a connection of tissue. Ischemia would be present if the blood flow in question was insufficient. The overlay image generated according to the publication is adjusted to the running white light video and overlaid with it. The "ischemic map" used is created by compressing a dynamic in the recording into a static image, i.e. freezing an image sequence in a single image for overlay.

However, the above-mentioned prior art has the disadvantage that a change in position and/or location of a tissue area in question, for example an organ to be examined, and/or the image capture device in use, leads to errors in the representation of the superimposed fluorescence information. For example, the superimposition of fluorescence information with a white light video recording known in the prior art already reaches its limits when an object in question moves rhythmically and/or at a frequency during the recording. The same applies if the user moves the image capture device back and forth during the recording of the object, for example in order to record the object from different angles and/or under different lighting situations and/or shadow directions. With such movements of the object and/or the image capture device, the fluorescence information cannot be correctly superimposed on the virtual image of the object, so that, for example, the fluorescence information is displaced and/or offset within the image, which the user perceives as blurred images. and/or blurred overlay is perceived. This is disadvantageous because it may lead to an incorrect assessment of the position of the fluorescence information superimposed on the (white light) image of the object. In the worst case, this can lead to, for example, tissue areas that are incorrectly shown as fluorescence areas in the overlay representation being perceived as diseased tissue and surgically removed in response. This must be avoided.

The object of the present invention is therefore to overcome the above-mentioned disadvantages of the prior art. In particular, it is an object to provide a method and a device by means of which a superimposition of overlay recording information with a live image of an object is possible even with a moving object and/or with a moving image capture device and at the same time a superimposition error is minimized.

The above statements apply in a similar form to known multispectral and/or hyperspectral imaging techniques.

The above-mentioned object is achieved by a method having the features of independent claim 1. Furthermore, the object is achieved by a device according to the invention having the features of independent claim 15.

Advantageous developments of the invention are specified in the dependent claims. The scope of the invention includes all combinations of at least two features disclosed in the description, the claims and/or the figures. It is understood that embodiments and forms of embodiment described in relation to the method according to claim 1 can refer to the device according to claim 15 in an equivalent, if not word-for-word, form without being explicitly mentioned for this. It is also understood that customary linguistic transformations and/or a corresponding replacement of respective terms within the scope of usual linguistic practice, in particular the use of synonyms supported by generally accepted language literature, are also included in the present disclosure content without being explicitly mentioned in their respective formulation.

According to a first aspect of the invention, a method is proposed for overlaying overlay recording information with a live image, in particular with an endoscopic and/or an exoscopic and/or a laryngoscopic and/or a microscopic live image. The method comprises at least the steps: capturing at least one recorded image of a moving object using a moving image capturing device; determining the overlay recording information in the at least one captured recorded image at least with regard to the object; capturing at least one live image of the moving object that follows the recorded image in time using the moving image capturing device; calculating a transformation rule at least between the recorded image and the live image in order to track a change in position and/or location and/or depth of the moving object between the recorded image and the live image; and overlaying the determined overlay recording information with the live image at least depending on the calculated transformation rule.

According to a second aspect of the invention, a device is proposed which has the following: a moving image capture device with an image sensor (e.g. a photo chip) and at least one image capture unit, which preferably comprises at least one optical device, e.g. a lens, which is set up to capture at least one recorded image of a moving object and to capture at least one live image of the moving object following the recorded image in time; a computing device which is set up to determine overlay recording information in the at least one captured recorded image relating to the object, to calculate a transformation rule at least between the recorded image and the live image in order to track a position and/or location and/or depth change of the moving object between the recorded image and the live image, and to overlay the previously determined overlay recording information with the live image at least depending on the calculated transformation rule; and a display device, in particular a display, which is set up to display at least the live image together with the overlay recording information. The recorded image is preferably an endoscopic and/or an exoscopic and/or a laparoscopic and/or a microscopic recorded image. The live image is preferably an endoscopic and/or an exoscopic and/or a laparoscopic and/or a microscopic live image.

In contrast to the prior art, the invention proposes to take into account a changing position and/or location of the image capture device and/or the tissue when superimposing the overlay recording information with the at least one live image. The consideration is preferably made by applying the calculated transformation rule to the overlay recording information included in the at least one recorded image. In other words, the present invention particularly preferably involves overlaying a live video of an operation site in which an object to be observed is located with a previously recorded video in which the overlay recording information is included, taking into account a change in position and/or location of the object and/or the image capture device. This makes it possible, for example, to repeatedly display the overlay recording information with the live image during an operation. In contrast to the prior art, the inventors therefore propose adapting an overlay of the overlay recording information with the at least one live image, which is preferably a single image of a live video sequence, to a current viewing direction and/or position of the image capture device, e.g. the endoscope, and/or to a current movement of the tissue in question. For this purpose, the position and/or location of the object or tissue in question is recorded and tracked. The overlay of the overlay recording information with the live image is then adapted to the respective current perspective of the image capture device and/or to the respective current location and/or position of the object on the basis of the transformation rule and overlaid with the live image. In this way, the overlay recording information is still available as error-free as possible even with a moving object and/or a moving image capture device and is used, for example, to mark moving tumor tissue.

The overlay recording information generated during the recording of the at least one recorded image and preferably the white light recording information are preferably stored. The overlay recording information is particularly preferably recorded and stored during the flood phase. According to the invention, it is possible to take into account that the user generally changes a perspective of the image capture device, in particular of the endoscope, by moving during the recording. Alternatively or additionally, a position and/or location of the object to be recorded can also change due to a movement of the object during the recording phase. These movements can be compensated according to the invention, since a transformation rule is preferably calculated based on a current (white light) recorded image. This transformation rule preferably depicts how the (white light) recorded image can be transformed during the live recording in a way that is as suitable as possible to the at least one live image. The transformation rule can comprise a transformation matrix. It is preferred if, during the recording of the at least one recorded image and the at least one live image, a 3D structure of the object or of the tissue region is calculated by a stereo reconstruction of a preferably recorded stereo recorded image and/or stereo live image and/or by an estimate using artificial intelligence algorithms from a 2D recorded image and/or a 2D live image or by other known methods. The transformation rule can preferably be calculated between a 3D point cloud generated on the basis of the recorded image and a 3D point cloud generated on the basis of the live image in order to subsequently enable the overlay recording information to be superimposed on the live image. In particular, a transformation rule can be calculated between the live image and parts of the recorded image without the overlay recording information and this transformation rule can then be applied to the overlay recording information in order to adapt it to the current live image.

The recorded image preferably comprises white light information, in particular of the object within a recorded scene, thus image information that is displayed in a wavelength range visible to the human eye. The live image preferably comprises white light information, in particular of the object within a recorded scene, thus image information that is displayed in a wavelength range visible to the human eye.

It is understood that the at least one recorded image and the at least one live image can in principle also be captured by different devices. For example, the at least one recorded image can be captured and/or recorded by a first image capture device that is included in a first device. The at least one live image can, however, be captured and/or recorded by a second image capture device that is included in a second device. If different image capture devices are involved, it is preferred if, for example, different reference systems and/or reference coordinate systems of the different image capture devices are taken into account when calculating the transformation rule. For example, the first image capture device can be referenced and/or calibrated with respect to a first coordinate system having a first coordinate origin. For example, the second image capture device can be referenced and/or calibrated with respect to a second coordinate system having a second coordinate origin. When calculating the transformation rule, these two mutually different coordinate systems can preferably be taken into account, for example by transforming image information on the relevant coordinate systems to a common coordinate system.

The calculation of the transformation rule preferably comprises determining position and/or location and/or depth information of the moving object in the recorded image and/or determining position and/or location and/or depth information of the moving object in the live image and/or preferably a resulting change in position and/or location and/or depth information. Tracking the change in position and/or location and/or depth information between the at least one recorded image and the at least one live image is preferably carried out using a deterministic algorithm and/or by using and/or training an artificial neural network, thus using a mathematical-statistical estimate based on artificial intelligence, which is preferably trained to recognize such changes in position and/or location and/or depth information using known machine learning approaches. The calculation of the transformation rule and/or the overlaying of the previously determined overlay recording information with the at least one live image preferably takes place in real time or at least in quasi-real time, i.e. with a time offset that cannot be tracked by the human eye, for example < 70 milliseconds. The overlaying of the previously determined overlay recording information with the at least one live image preferably takes place depending on the calculated transformation rule, supplemented by previously determined position and/or location and/or depth information of the at least one recorded image.

• - Ermitteln der Überlagerungsaufnahmeinformationen in den Aufnahmebildern der Sequenz zumindest bezüglich des Objektes;
• - Fortlaufendes Berechnen einer bildweisen Transformationsvorschrift zwischen einzelnen Aufnahmebildern der Sequenz von Aufnahmebildern und einzelnen Livebildern der Sequenz von Livebildern, um derart eine Positions- und/oder Lage- und/oder Tiefenänderung des beweglichen Objektes zwischen den Aufnahmebildern und den Livebildern nachzuverfolgen;
• - Fortlaufendes bildweises Überlagern der zuvor ermittelten Überlagerungsaufnahmeinformationen mit der aktuellen Sequenz von Livebildern jeweils zumindest in Abhängigkeit der bildweise berechneten Transformationsvorschrift.

Furthermore, the recorded image can be part of a sequence of at least two recorded images that were captured by means of the moving image capture device, and the live image can be part of a sequence of at least two live images of the moving object that are captured by the moving image capture device, wherein the following steps are carried out:

• - Determining the overlay recording information in the recording images of the sequence at least with respect to the object;
• - Continuously calculating an image-by-image transformation rule between individual recorded images of the sequence of recorded images and individual live images of the sequence of live images in order to track a change in position and/or location and/or depth of the moving object between the recorded images and the live images;
• - Continuous image-by-image overlaying of the previously determined overlay recording information with the current sequence of live images, each at least depending on the transformation rule calculated image-by-image.

It is understood that the method steps according to the invention do not necessarily have to be carried out in the order listed, but this order can also be changed. It is also possible that one or more intermediate steps can be carried out between one or more method steps.

In a preferred embodiment, the change in position and/or location and/or depth of the object between the at least one recorded image and the at least one live image is based on a movement of the moving object and/or on a movement of the image capture device. The change in position and/or location and/or depth of the object can, for example, be based on a rhythmic and/or periodic and/or frequency-based movement of the moving object. The object is preferably an organ and/or a tissue region and/or a tissue section of a human and/or animal body. Alternatively or additionally, the change in position and/or location and/or depth of the object between the at least one recorded image and the at least one live image can also be based on a movement of the image capture device. According to the invention, this movement is compensated when the overlay recording information is superimposed on the at least one live image. For this purpose, the movement of the object is preferably tracked at least between the at least one recorded image and the at least one live image using optical tracking methods. Preferably, the image capture device additionally has at least one position or attitude sensor, for example a gyro sensor, an acceleration sensor, a GPS sensor and/or an optical sensor, by means of which a change in position of the image capture device can be detected in the form of a sensor signal. Such a sensor signal is preferably included in the calculation of the transformation rule.

In a further preferred embodiment, the transformation rule is calculated on the basis of an optical flow algorithm and/or an elastic image registration algorithm and/or a point cloud registration algorithm and/or landmark-based tracking, preferably applied to image points of the recorded image and/or the live image assigned to the object. The transformation rule can preferably be calculated using optical tracking models between the recorded image and the live image, which are generally known from the prior art. The calculation of the transformation rule is particularly preferably supplemented by the inclusion of motion sensor data obtained by at least one motion sensor (e.g. an inertial sensor) arranged on the image capture device in order to further improve or refine it using physical reference variables. The calculation of image transformations is known from the prior art, but from other technical fields, see e.g. for optical flow: S. Shah et al. Traditional and modern strategies for optical flow: an investigation. 2021; for landmark-based methods: M. Webster et al. A practical introduction to landmark based geometric morphometrics. 2010; and for point cloud registration: X. Hang et al. A comprehensive survey on point cloud registration. 2021). According to the invention, the calculation of a transformation rule preferably takes place between a time of recording the at least one recorded image and a time of repeated display of the overlay recording information with the at least one live image with the aim of transforming the overlay recording information to the live image in accordance with the calculated transformation rule.

The transformation rule is preferably calculated frame by frame, i.e. between each successive image sequence, particularly preferably in real time. This makes it possible to take movements during the recording phase and/or during the live phase or during the playback of the overlay into account.

In a further preferred embodiment, the overlay recording information comprises fluorescence information, in particular indocyanine green fluorescence information and/or fluorescein fluorescence information, and/or intraoperative hyperspectral recording information and/or multispectral recording information. The fluorescence information results, for example, from an examination method in which substances (e.g. monoclonal antibodies) are used that are marked with fluorescent dyes, for example with indocyanine green (ICG) or fluorescein. After injecting the fluorescent dye into the relevant tissue area and/or the patient's bloodstream, the dye and carrier material accumulate, due to its specific properties, preferably in dysplasias or tumors. This is preferably referred to as the dye's flooding behavior. The introduced dye is preferably excited to fluoresce by illuminating the tissue area with an excitation light source, so that the altered tissue is clearly visible during the flooding phase. This gives the surgeon a visual indication of the damaged tissue or the tissue to be removed. The fluorescence triggered by the illumination increases in intensity during the initial phase and reaches its maximum intensity after a certain waiting period. At this point, the organ areas and/or objects in question are most prominent. After this, the intensity decreases again.

Intraoperative hyperspectral recording information preferably results from an intraoperative hyperspectral imaging method that is used to display and/or highlight predetermined object or tissue properties. Such intraoperative hyperspectral imaging methods are known from the medical-technical field and are referred to as intraoperative hyperspectral imaging (HSI). The multispectral recording information preferably results from a multispectral imaging method that is used to display and/or highlight predetermined object or tissue properties. Such multispectral imaging methods are known from the medical-technical field and are referred to as multispectral imaging (MSI).

In a further preferred embodiment, the captured image comprises the overlay recording information and/or the white light image information. The captured image, which is also referred to as a white light image, and the overlay recording information, in particular fluorescence overlay recording information and/or multispectral overlay recording information and/or hyperspectral overlay recording information, are preferably captured using the same image capture device. Preferably, each captured image comprises at least one overlay recording information item for each pixel in the imaging area assigned to the object. Particularly preferably, the image capture device is set up to capture the white light image information and the overlay recording information simultaneously. and then to carry out a spectral separation. The term "white light recording image" and the associated white light information are understood here to mean a recording image in which an image scene is recorded with the image capture device in the wavelength range visible to the human eye. Particularly preferably, the live image is also recorded as a white light live image and therefore includes white light information of the object recorded in real time.

In a further preferred embodiment, the step of determining the overlay recording information comprises a spectral separation between the overlay recording information and the white light image information. In other words, both pieces of information are preferably recorded simultaneously by the image capture device. Subsequently, the overlay recording information can preferably be separated or differentiated from the white light image information by means of a signal evaluation, which is preferably carried out using one or more optical filters, in order to obtain the overlay recording information, for example, on a pixel-by-pixel basis, so that a later overlay of this information with a (new) live image is possible. Particularly preferably, the at least one recording image per pixel comprises respective overlay recording information or a statement (e.g. a zero value) that no overlay recording information is assigned to a respective pixel. When calculating the transformation rule, the overlay recording information assigned pixel by pixel can then be transformed to the live image. In this case, a change in the location and/or position of the respective pixel between the at least one recorded image and the at least one live image is taken into account by the transformation rule, so that in the captured live image, even with a changing position and/or location, a respective overlay recording information can be correctly assigned to a pixel that is assigned to an original real point of the recording scene, even if the object and/or the image capture device has moved.

The overlay imaging information can also be captured separately, for example by additional cameras or image sensors or by rapidly switching between white light and fluorescence imaging, for example alternating frame by frame.

In a preferred embodiment, the at least one recorded image and/or the at least one live image is/are each an individual image of a video sequence from a large number of chronologically successive recorded images and/or live images. In other words, both the recorded image and the live image are preferably each individual images of a large number of chronologically successive recorded images or live images, which when placed one after the other result in a video recording of the at least one object. A time interval between the individual images is preferably determined by a predetermined frame rate. It is understood that both the recorded image and the live image can in principle be any individual image of such a video recording and/or video sequence and of course do not have to be an initially recorded image. Particularly preferably, both a recorded video or a recorded video sequence and a live video or a live video sequence of the object to be observed are recorded by the first and/or the second image capture device. In other words, it is preferred according to the invention if the overlay recording information is included in the recording video and/or the recording video sequence as an overlay recording video sequence and can be overlaid with the live video or the real-time video. The overlay recording information, preferably provided as an overlay recording video sequence, can preferably be overlaid frame by frame with individual time periods of the live video. According to the invention, it is preferred if the overlay of the overlay recording information with the live image sequence is perspectively corrected on the basis of the transformation rule. A display duration of the overlay recording information with the live image sequence can preferably be shortened (i.e. accelerated) or extended (i.e. slowed down) compared to an original recording duration.

For an accelerated display of the overlay recording information, for example, every nth recording image of the sequence of recording images can be overlaid with the live images of the current sequence of live images, with n > 1, wherein at least the overlay information of two recording images is overlaid so that the overlay recording information is overlaid on the live images in fast motion. Instead of using every nth recording image, another rule can also be applied to select a subset of recording images for the overlay from the sequence of recording images.

In a preferred embodiment, the step of determining the overlay recording information is preceded by exposing the object by means of a light source and detecting a exposed recording image by the image capture device. Such exposure of the object with a light source or with an excitation light source preferably takes place at a different time to a recording of the white light image or the white light recording information. For example, the image capture device can be set up to produce a white light recording of the object over a predetermined period of time. After illuminating the object with the excitation light source, the image capture device is preferably set up to capture the overlay recording information, in particular the fluorescence information about the object. Thus, the white light recording information does not have to be captured at the same time as the overlay recording information, but can also be carried out at a different time. For example, the image capture device can switch back and forth between the two recording modes by switching on or off an optical filter.

In a preferred embodiment, the recorded image is a stereo recorded image and/or the live image is a stereo live image, which is/are processed by stereo reconstruction of a respective stereo image pair on the basis of optical and/or dimensional parameters of the image capture device. The recorded image and/or the live image is preferably in each case an already stereo reconstructed stereo measurement image, in which depth information is preferably made available on the basis of a performed stereo reconstruction at least for each pixel in relation to the at least one moving object. In order to capture the stereo recorded image and/or the stereo live image, a stereo image pair of the moving object is preferably captured from two different viewing angles (predetermined by a stereo basis) synchronized with respect to a predetermined time unit. The two images captured synchronously in time preferably form the stereo measurement image pair with a common time stamp. Depth information can be evaluated pixel by pixel for each time stamp from each stereo measurement image pair using stereo reconstruction methods. In the course of the stereo reconstruction, a so-called disparity, i.e. a horizontal pixel offset, between the respective image of the respective stereo image pair is determined by an algorithm pixel by pixel, i.e. per pixel. Objects at a great distance preferably have a small disparity, i.e. a small pixel offset, within a respective stereo image pair during stereo capture. In contrast, objects in the foreground have a large disparity. From the disparity determined pixel by pixel, with knowledge of further optical parameters of the image capture unit(s), a depth map can preferably be calculated for the stereo-reconstructed first and/or second stereo measurement image, which preferably includes pixel by pixel depth information of the object to be viewed.

In a preferred embodiment, the method according to the invention further comprises the steps of: recording the recorded image and the overlay recording information by the image capture device; and storing the recorded image and the overlay recording information. This makes it possible to generate a data set. The overlay recording information can in particular be stored separately from the recorded image. For example, a recorded image can contain white light and fluorescence information. The fluorescence image can include or display the overlay recording information. This can be stored separately from the white light image information by spectral separation, for example during the recording by spectral separation directly in the instrument, or by separate camera systems. The same camera can also display white light and fluorescence images almost simultaneously by quickly changing the recording modes of white light and fluorescence, e.g. frame by frame, but the fluorescence image information and thus the overlay recording information are still available separately. This preferably allows the overlay recording information to be overlaid again with a newly recorded live image of the object, without necessarily having to provide another recording image including overlay recording information during a further recording phase. For documentation and/or further analysis, it is preferred that the recording image and/or the live image overlaid with the overlay recording information is/are saved. In this way, a follow-up examination based on this saved information can preferably be carried out following an examination and/or operation. In addition, saving serves to make it possible to compare saved overlay recordings with new live recordings in order to historically trace, for example, a development and/or a change in the object based on the saved information. This also enables the possibility of multiple overlay recording information with different time stamps, which can be displayed in different colors, for example. The overlay recording information and/or the white light recording information is preferably saved automatically during the recording phase and/or during the overlay phase (during the live display) with the aim of to overlay the overlay recording information several times with a newly captured live image or live video.

In a preferred embodiment, the method according to the invention further comprises the steps of: checking the calculated transformation rule and/or a live image superimposed on the basis thereof with regard to the ability of the overlay recording information to be superimposed on the live image; and preferably issuing a warning if an overlay of the overlay recording information with the live image is only possible if a permissible error tolerance limit is exceeded. The focus of the invention is therefore also on displaying the reliability of the generated overlay. In other words, if a transformation based on the calculated transformation rule between the at least one recorded image and the at least one live image is only possible with an intolerable error or is not possible at all, this is displayed according to the invention or issued as a warning message. Furthermore, the overlay of the overlay recording information with the at least one live image is preferably not displayed or output. Instead, the user is preferably given a warning that the representation of the overlay is not possible while maintaining a predetermined error tolerance and therefore cannot be displayed. Alternatively or additionally, the user can be given a reason for the inability to overlay if, for example, the image capture device was moved too quickly. The user can also preferably be given an indication of a disruptive change between the recorded image and the live image if, for example, the object depicted in the recorded image is only partially present or no longer present, for example if it has been surgically removed. The image capture device can preferably be set up to compare the live image with the recorded image in such a way that the two images are checked for the presence or absence of the at least one object. This makes it possible to effectively avoid errors in the representation of the overlay and consequently errors based on such an incorrect overlay. It is also preferred according to the invention if the user is given the option of manually or partially manually adjusting a representation of the overlay and is informed of this option if necessary. Alternatively or additionally, it is preferred if, in addition to the overlay recording information being overlaid with the at least one live image, a message is also displayed for the user that the overlay recording information is not being overlaid based on live overlay recording information, but that this was recorded in advance. Particularly preferably, according to the invention, a check is carried out to determine whether the transformation rule was correctly calculated. If the transformation rule could not be correctly calculated, an output can be given to the user, for example, that the overlay recording information being overlaid with the at least one live image must be stopped due to an incorrectly calculated transformation rule. The check can preferably be carried out via a plausibility check. Particularly preferably, the transformation rule can be checked based on manual input or automatically. It is also possible according to the invention for the user to make manual adjustments to the transformation rule.

In a preferred embodiment, the method according to the invention further comprises the steps of: displaying the live image together with the superimposed overlay recording information; and highlighting an image area in a capture scene in the live image in which the overlay recording information is displayed. When displaying the overlay of the overlay recording information in the live image, areas for which the overlay is possible on the basis of the transformation rule and is therefore also carried out can preferably be highlighted in the at least one live image by area indicators, for example colored borders or the like. This preferably shows the user information about the imaging area in which the overlay is possible and takes place. For example, the live image can include a larger section of an imaging scene in which, for example, other objects, for example other organs, are also imaged. The overlay only takes place with respect to one of these objects, which is shown in the imaging scene, for example highlighted in color, for example with a red border, for better recognition. The display device is preferably designed to highlight areas in which the overlay recording information is superimposed on the live image.

In addition, while the live image superimposed with the overlay recording information is displayed, a notice can be displayed to the user on a monitor that the overlay recording information is not a live image.

A flooding behavior of an indicator substance is preferred for the generation of fluorescence information and/or multispectral information and/or hyperspectral information relating to a real image and/or a live image is displayed at an accelerated rate in order to enable the user to more effectively assess the recorded overlay recording information. For example, a recording video sequence is shortened or accelerated from an example of 2 minutes to an example of 10 seconds by using a time lapse. For example, the overlay recording information contained in a recording video sequence per unit of time can be overlaid on a live video sequence at an x-fold acceleration compared to this. The recording video sequence is preferably on the order of 1 to 5 minutes, but can in principle also be shorter or longer. The acceleration of the time lapse is preferably 2-fold or more than 2-fold. When evaluating multispectral or hyperspectral information, which is used, for example, to detect oxygen saturation, the recording video sequence can be, for example, 10 to 20 seconds and the overlay recording information can be overlaid on a live video sequence at a 1.5 to 2-fold acceleration.

In a preferred embodiment, the device is comprised in an endoscope or a stereoendoscope and/or in an exoscope or a stereoexoscope and/or in a microscope or a stereomicroscope and/or in a laryngoscope or a stereolaryngoscope.

In a preferred embodiment, the device comprises a user input device and/or a user output device. The user input device can comprise a keyboard and/or a mouse and/or a joystick and/or a touchscreen and/or a touchpad and/or another manual input device. The user output device is preferably designed to enable the user of the device to interact and/or react and/or at least partially intervene manually, in particular in the display process of overlaying overlay recording information with the live image. The user output device can comprise a screen and/or a touchscreen and/or glasses and/or 3D glasses and/or augmented reality glasses.

Further advantages and details of the invention will become apparent from the following description of preferred embodiments of the invention and from purely schematic drawings.

• 1: eine schematische Ansicht eines Ausführungsbeispiels einer erfindungsgemäßen Vorrichtung;
• 2: eine schematische Ansicht eines weiteren Ausführungsbeispiels einer erfindungsgemäßen Vorrichtung;
• 3: eine schematische Darstellung einer zeitlichen Abfolge von Aufnahmebildern und Livebildern; und
• 4: eine schematische Darstellung einer Überlagerung von in zeitlicher Abfolge erfassten Überlagerungsaufnahmeinformationen mit einer zeitlichen Abfolge von Livebildern.

Show it:

• 1 : a schematic view of an embodiment of a device according to the invention;
• 2 : a schematic view of a further embodiment of a device according to the invention;
• 3 : a schematic representation of a temporal sequence of recorded images and live images; and
• 4 : a schematic representation of an overlay of overlay recording information acquired in chronological order with a chronological sequence of live images.

Identical elements or elements with the same function are provided with the same reference numbers in the figures.

1 shows in a schematically simplified block diagram an exemplary embodiment of a device 100 according to the invention. The device 100 is according to 1 for example in a stereoendoscope 102. In principle, the device 100 can also be included in a stereoexoscope and/or a stereomicroscope and/or a stereolaryngoscope.

Furthermore, the device 100 can also be included in an endoscope and/or an exoscope and/or a microscope and/or a laryngoscope. An exemplary embodiment of the device 100 included in an endoscope 103 (without stereo vision) is shown in 2 shown.

In the following, the basic operation of an embodiment of the device 100, included in the stereoendoscope 102, according to 1 described. The basic mode of operation of an embodiment of the device 100 in an endoscope 103 (without stereo vision) is then described. The method according to the invention, which can be carried out by the device 100, is also discussed.

The device 100 according to 1 has an image capture device 104 with an image sensor 106. Due to the design as a stereo endoscope 102, the image capture device 104 comprises a first image capture unit 108 and a second image capture unit 110. When designed as an endoscope 103, the image capture device 104 can comprise only one of the image capture units 108, 110 (cf. 2 ). It is understood that the stereo endoscope 102 can be movable and can be moved relative to the object 112, for example by a hand movement of a user or by a robot-controlled movement.

The first image capture unit 108 is at a predetermined distance from the second image capture unit 110, which defines a stereo base of the stereo endoscope 102. The first and second image capture units 108, 110 are preferably each a camera. The image sensor 106 is preferably set up to receive image data in the form of recorded images and/or live images from the first and second image capture units 108, 110 and to evaluate them. The image sensor 106 preferably interacts with an evaluation and computing unit, which enables processing of the captured image data. The computing device 109 can, for example, comprise a camera control unit (CCU).

A lens assembly 114 is arranged in front of the first and second image capture units 108, 110. The lens assembly 114 comprises, for example, a cover glass and optical units 116, 118 with apertures that are assigned to the image capture units 108, 110. The optical units 116, 118 define the respective field of view of the image capture units 108, 110. Each of the two image capture units 108, 110 is assigned to an observation channel 120, 122. The observation channels 120, 122 are each designed to transmit the recorded images and/or the live images in the form of signal-like image information to the image sensor 106. To provide the image information, each of the image capture units 108, 110 is assigned a signal converter 124, 126. The signal converters 124, 126 are each designed to convert the optically captured images and/or live images into image information. For example, the signal converters 124, 126 are photo chips.

The first image capture unit 108 is configured to capture at least one recorded image 134 and/or a live image 135 of at least one moving object 112 (see 3 ). Object 112 is here in 1 represented by way of example as a letter P. However, the object 112 is normally preferably a human or animal organ or an organ region or several organs or another part of a human or animal body or a component. The first image capture unit 108 captures the at least one recorded image 134 and/or the at least one live image 135 of the object 112 preferably from a first viewing angle.

For stereo recording, the second image recording unit 110 is set up to also record at least one recorded image 134 and/or at least one live image 135 of the at least one moving object 112 from a second viewing angle. The first and second image recording units 108, 110 are each set up to record the at least one recorded image 134 and/or the at least one live image 135 preferably in a time-synchronized manner. Recorded images 134 recorded in such a time-synchronized manner preferably form a stereo recorded image pair. Live images 135 recorded in such a time-synchronized manner preferably form a stereo live image pair.

The image sensor 106 is designed to determine stereo image information from the stereo recording image pair and/or the stereo live image pair or from the signal-based image information using known methods of stereo reconstruction. In the stereo image information, depth information is available for each captured pixel of the at least one object 112, which can be used, for example, to calculate a distance between two measurement points on the object 112 in Euclidean space.

The stereo image information can preferably be transmitted to a user output device 132 and/or to a data storage device via a first and/or a second output channel 128, 130. At least one stereo live image 135 acquired in this way can preferably be output and/or displayed to a user on the user output device 132. The user output device 132 can be a display and/or a touch display, for example. The at least one object 112 is displayed on the user output device 132 in the form of an observation object 136. A user can make one or more manual or partially manual inputs using a user input device (not shown). The user input device can be included in the user output device 132.

It is understood that the device 100 is designed to basically capture a plurality of recorded images 134 and/or stereo recorded image pairs of the object 112. It is particularly preferably possible with the device 100 to capture a recorded video and/or a stereo recorded video of the at least one object 112, which is composed of a plurality of temporally successive recorded images 134 and/or stereo recorded images (each resulting from a stereo recorded image pair) that are captured one after the other at a predefined time interval (determined by the frame rate). The same applies to capturing the live images 135 and/or the stereo live images, which are preferably captured as live video and/or stereo live video.

In 2 a second embodiment of the device 100 according to the invention is shown. An exemplary system architecture is shown. The device 100 is contained in the endoscope 103, which is not designed for stereo vision. Accordingly, the image capture device 104 also comprises only one image capture unit 108, 110. The device 100 together with the endoscope 103 is shown here as a schematic block without further detailing. The image capture device 104 can be understood as a camera head, for example. The device 100 can also comprise at least one observation filter 138, for example an optical filter, through which predetermined wavelengths and/or wavelength ranges can be filtered, for example.

The image capture unit 108, 110, which can be designed as a camera, for example, is set up to capture at least one recorded image 134 and/or at least one live image 135 of the at least one object 112. It is understood that the endoscope 103 can be movable and can be moved relative to the object 112, for example, by a hand movement of a user or by a robot-controlled movement. The at least one live image 135 is preferably captured chronologically following the at least one recorded image 134.

The at least one recorded image 134 is preferably captured during a recording phase during which the at least one object 112 is captured by the device 100 or by the endoscope 103 and/or by the stereoendoscope 102. The at least one recorded image 134 is preferably at least temporarily stored or saved in order to continue to be available. According to the invention, it is desirable that the at least one recorded image 134 includes overlay recording information 140. This overlay recording information 140 results, for example, from a fluorescence excitation of the at least one object 112 by means of a fluorescent dye, such as indocyanine green or fluorescein. Alternatively, the overlay recording information 140 results from known intraoperative hyperspectral methods and/or from multispectral methods. In 2 By way of example, a fluorescence excitation of at least one object 112 is shown as dark or black areas.

To generate the overlay recording information 140, for example, a fluorescent dye is injected and/or introduced into the relevant object 112 or into a relevant tissue region 142 in which the object 112 is contained. The fluorescent dye is then illuminated, for example, by an excitation light source 144 and thus excited to fluoresce. The excitation light source 144 can preferably be connected in a light-conducting manner to the endoscope 103 (and of course in the same way to the stereoendoscope 102) via a light guide 146 in order to illuminate the object 112 with the light from the excitation light source 144 directly via the endoscope 103 or the stereoendoscope 102. Particularly preferably, the light guide 146 can be coupled to the endoscope 103 or the stereo endoscope 102 in such a way that light can be coupled into the light guide 146 of the endoscope 103 or the stereo endoscope 102 with as little loss as possible. The light guide 146 preferably comprises a plurality of glass fibers.

The excitation of fluorescence or the illumination of the object 112 by the excitation light source 144 preferably takes place immediately before or during the recording phase. During the recording phase, the at least one recorded image 134 is then captured, in which the overlay recording information 140, in particular the fluorescence recording information, is included. In addition to the overlay recording information 140, the image capture unit 108, 110 also preferably captures white light image information. In other words, the tissue or the object 112 can preferably be excited to fluoresce using a fluorescent dye and examined by the endoscope 103 and/or the stereoendoscope 102. For example, white light or excitation light from the excitation light source 144 falls on the tissue through the endoscope 103, is reflected there or emitted by the fluorescence process and can be detected by the image capture device 104. The computing device 109 or the camera control unit (CCU) then preferably processes the captured images 134.

Preferably, the computing device 109, preferably designed as a CCU, is configured to provide the overlay recording information 140 by means of a spectral separation between the acquired overlay recording information 140 and the acquired white light image information. Alternatively or additionally, it is possible that the determination of the overlay recording information 140 is preceded by an exposure of the object 112 by means of the excitation light source 144 and a capture of a recording image 134 exposed in this way.

In order to ensure that the overlay recording information 140 is superimposed with the at least one live image 135 as error-free as possible even with a moving object 112 and/or a moving image capture device 104, In order to enable this, it is provided according to the invention that a transformation rule is calculated at least between the recorded image 134 and the live image 135 using the computing device 109 in order to track a position and/or location and/or depth change of the moving object 112 between the recorded image 134 and the live image 135.

In 3 On the left side, a plurality of temporally consecutive (t, t-1, t-2, t-3, t-4) recorded images 134 are shown, each of which includes overlay recording information 140 and white light image information. Several objects 112 are shown in the recorded images 134. On the right side, several live images 135, in particular white light live images, are shown, which are also recorded in temporally consecutive (t, t-1, t-2) of the object 112. No overlay recording information 140 is included in the live images 135, since the overlay recording information 140 was only recorded during the recording phase preceding the live phase. As can be seen from a summary of the right and left side images in 3 As can be seen, the object 112 has changed its position between the recording phase (shown in the recording images 134) and the live phase (shown in the live images 135). Furthermore, only one of the objects 112 is shown in the live images 135 compared to the recording images 134.

According to the invention, the overlay recording information 140 is continuously overlaid with the live images 135, in particular image by image, wherein a movement of the object 112 and/or the image capture device 104 is taken into account by calculating the transformation rule. This overlay is in 4 shown as an example.

Between the (white light) recording images 134 and the live images 135, as can be seen from 4 , a transformation rule is calculated in which the movement of the at least one object 112 within a detection range of the image capture device 104 is included. In this case, the recorded images 134 captured during the recording phase can be accelerated compared to the live phase, for example accelerated twice, and taken into account in the calculation of the transformation rule, so that, for example, only every second recorded image 134 is taken into account for calculating the transformation rule (indicated by t, t-2, t-4, where t-1 and t-3 were skipped due to the 2-fold acceleration). The calculation of the transformation rule is preferably repeated in predetermined time periods in order to track an object movement as continuously as possible. The calculated transformation rule makes it possible to overlay the overlay recording information 140 included in the recorded images 134, which was preferably extracted from them in isolated form or recorded separately, with the live images 135, as shown on the right-hand side in 4 is shown. Due to the transformation rule or due to the position and/or location change of the at least one object 112 that can be included thereby, the overlay recording information 140 can be overlaid with the respective live images 135 as error-free as possible.

The overlay recording information 140 can include fluorescence information, such as ICG, PDD, etc. Overlay recording information from a hyperspectral or multispectral image series is also conceivable. The decisive step according to the invention is the correct calculation of the transformation rule. There are basically several possible approaches here. On the one hand, a most probable pixel shift can be calculated from two white light images (a recorded image and a live image) using dense optical flow algorithms. On the other hand, there are landmark-based approaches by which prominent points in the recorded image and the live image are detected. From these prominent points, at least one pair of points can preferably be identified, which can be used as a starting point for further interpolation to calculate the transformation rule. In addition, a 3D point cloud can be calculated from two 3D image pairs (each consisting of a 3D recorded image and a 3D live image) (e.g. with stereo reconstruction algorithms, see e.g. H. Hirschmüller et al. 2005). Using point cloud registration algorithms, a transformation rule can be calculated by which both 3D point clouds are superimposed as precisely as possible. Inertial sensor data can also be included here, for example. As a result, for example, information about an angle of attack of the device 100 with respect to the at least one object 112 can reduce a number of possible degrees of freedom of movement when estimating the movement.

Robotic systems can also reduce the number of possible degrees of freedom by using information about the position and/or orientation of, for example, an endoscope tip.

From a back projection of a current live image 135, which is superimposed with overlay recording information 140 based on the calculated transformation rule, it can preferably be estimated by a comparison with the previous image whether the calculated transformation rule is correct. Given If the transformation rule is incorrectly calculated, the user may be informed that the superimposition is only possible with the risk of a large error or not at all, because, for example, there was a strong tissue movement during the acquisition phase and/or during the live phase.

List of reference symbols

100

contraption

102

Stereoendoscope

103

endoscope

104

Image capture device

106

Image sensor

108

first image capture unit

109

Computing device

110

second image acquisition unit

112

object

114

Lens assembly

116

optical unit

118

optical unit

120

Observation channel

122

Observation channel

124

Signal converter

126

Signal converter

128

first output channel

130

second output channel

132

User output device, display device

134

Recording image, stereo recording image

135

Live image, stereo live image

136

Object of observation

138

Observation filter

140

Overlay recording information

142

Tissue area

144

Excitation light source

146

Light guide

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• US 10849710 B2 [0012]
• EP 359048 A1 [0012]
• JP 2019098008 A [0012]
• JP 5160276 B2 [0012]
• US 8767057 B2 [0012]

Cited non-patent literature

• Selka, F., Agnus, V., Nicolau, S., Bessaid, A., Soler, L., Marescaux, J., & Diana, M. (2014, July). Fluorescence-based enhanced reality for colorectal endoscopic surgery. In International Workshop on Biomedical Image Registration (pp. 114-123) [0012]

Claims (17)

Method for superimposing superimposed recording information (140) with a live image (135), in particular with an endoscopic and/or an exoscopic and/or a laryngoscopic and/or a microscopic live image (135), comprising at least the following steps: - capturing at least one recorded image (134) of a moving object (112) by means of a moving image capture device (104); - determining the superimposed recording information (140) in the at least one captured recorded image (134) at least with respect to the object (112); - capturing at least one live image (135) of the moving object (112) following the recorded image (134) by means of the moving image capture device (104); - Calculating a transformation rule at least between the recorded image (134) and the live image (135) in order to track a change in position and/or location and/or depth of the moving object (112) between the recorded image (134) and the live image (135); and - Overlaying the previously determined overlay recording information (140) with the live image (135) at least as a function of the calculated transformation rule. Procedure according to Claim 1 , characterized in that the recorded image (134) is part of a sequence of at least two recorded images that were captured by means of the moving image capture device (104) and that the live image (135) is part of a sequence of at least two live images of the moving object (112) that are captured by the moving image capture device (104); further characterized by the steps - determining the overlay recording information (140) in the recorded images of the sequence at least with regard to the object (112); - continuously calculating an image-by-image transformation rule between individual recorded images (134) of the sequence of recorded images and individual live images (135) of the sequence of live images in order to track a change in position and/or orientation and/or depth of the moving object (112) between the recorded images and the live images; - Continuous image-by-image superimposition of the previously determined overlay recording information (140) with the current sequence of live images, each at least as a function of the transformation rule calculated image-by-image. Procedure according to Claim 2 , characterized in that only the overlay recording information (140) of every n-th recorded image (134) of the sequence of recorded images is overlaid with the live images (135) of the current sequence of live images, with n > 1, wherein at least the overlay information (140) of two recorded images is overlaid so that the overlay recording information (140) is overlaid on the live images in time lapse. Method according to one of the preceding claims, characterized in that the change in position and/or location and/or depth of the object (112) between the at least one recorded image (134) and the at least one live image (135) is based on a movement of the movable object (112) and/or on a movement of the image capture device (104). Method according to one of the preceding claims, characterized in that the calculation of the transformation rule is carried out on the basis of an optical flow algorithm and/or an elastic image registration algorithm and/or a point cloud registration algorithm and/or a landmark-based tracking, preferably in each case applied to image points of the recorded image and/or the live image assigned to the object. Method according to one of the preceding claims, characterized in that the overlay recording information (140) comprises fluorescence information, in particular indocyanine green fluorescence information and/or fluorescein fluorescence information, and/or intraoperative hyperspectral recording information and/or multispectral recording information. Method according to one of the preceding claims, characterized in that the at least one captured image (134) comprises the overlay image information (140) and white light image information. Procedure according to Claim 7 , characterized in that the step of determining the overlay image information (140) comprises a spectral separation between the overlay image information (140) and the white light image information. Method according to one of the preceding claims, characterized in that the at least one recorded image (134) and/or the at least one live image (135) is/are each an individual image of a video sequence from a plurality of temporally successive recorded images and/or live images. Method according to one of the preceding claims, characterized in that the step of determining the overlay recording information (140) is preceded by an exposure of the object (112) by means of a light source and a capture of an exposed recording image by the image capture device (104). Method according to one of the preceding claims, characterized in that the recorded image (134) is a stereo recorded image and/or the live image (135) is a stereo live image which is/are processed by stereo reconstruction of a respective stereo image pair on the basis of optical and/or dimensional parameters of the image capture device (104). Method according to one of the preceding claims, characterized in that it further comprises the following steps: - capturing the captured image and the overlay capture information (140) by the image capture device (104); and - storing the captured image and the overlay capture information (140). Method according to one of the preceding claims, characterized in that it further comprises the following steps: - checking the calculated transformation rule and/or a live image (135) superimposed on the basis thereof with regard to the ability of the overlay recording information (140) to be superimposed on the live image (135); and preferably - issuing a warning if the overlay recording information (140) to be superimposed on the live image (135) is only possible if a permissible error tolerance limit is exceeded. Method according to one of the preceding claims, characterized in that it further comprises the following steps: - displaying the live image (135) together with the superimposed overlay recording information (140); and - highlighting an image area in a capture scene in the live image (135) in which the overlay recording information (140) is displayed. Device (100) comprising: a moving image capture device (104) with an image sensor (106) and at least one image capture unit (108, 110) which is configured to capture at least one recorded image (134) of a moving object (112) and to capture at least one live image (135) of the moving object (112) following the recorded image (134) in time; a computing device (109) which is set up to determine overlay recording information (140) in the at least one captured recording image (134) relating to the object (112), to calculate a transformation rule at least between the recording image (134) and the live image (135) in order to track a position and/or location and/or depth change of the moving object (112) between the recording image (134) and the live image (135), and to overlay the previously determined overlay recording information (140) with the live image (135) at least depending on the calculated transformation rule; and a display device (132), in particular a display, which is set up to display at least the live image (135) together with the overlay recording information (140). Device according to Claim 15 , characterized in that the display device (132) is designed to highlight areas in which the overlay recording information (140) is superimposed on the live image (135). Device according to Claim 15 or 16 , characterized in that the image capture device (104) is comprised in an endoscope (103) and/or a stereoendoscope (102) and/or an exoscope and/or a stereoexoscope and/or a microscope and/or a stereomicroscope and/or a laryngoscope and/or a stereolaryngoscope.

Images