DE102011053250B4 - A method of capturing images of an observation object with an electronic image sensor and an optical observation device - Google Patents

Abstract

Method for capturing images of an observation object (17) with an electronic image sensor (5), in which: a full image is imaged onto an exposed sensor surface (19) of the image sensor (5), wherein an image region of interest of the frame is projected onto a partial surface (21 ) of the image sensor (5) is imaged, and the images are taken by exposing the image sensor (5) for the respective image and reading it out after each exposure, whereby exposing the image sensor (5) and reading all of the image sensor (5) a frame exposed sensor surface (19) at least once an exposure of the image sensor (5) and a readout only the partial surface (21) of the image sensor (5) with the image region of interest takes place before again exposing the image sensor (5) and a reading of the entire exposed sensor surface (19), characterized in that as image sensor (5) a monochrome image sensor is used, for illuminating a lighting of the Observation object (17) takes place in lighting cycles, in which the observation object (17) is time-sequentially illuminated with N different wavelength distributions (L1-LN), and the readout of the image sensor (5) takes place in exposure and readout cycles, wherein the illumination cycles with the exposure and reading-out cycles are synchronized in such a way that, per exposure and readout cycle a) the image sensor (5) is illuminated when the observation object is illuminated with a first of the N wavelength distributions of a lighting cycle and then the entire exposed sensor surface (19) is read out, b) then at least Once an exposure and read sequence of the image sensor takes place, in which the observation object (17) is illuminated successively with different wavelength distributions of a lighting cycle, with each illumination of the observation object (17) with one of the wavelength distributions an exposure takes place and after each exposure n only a readout of the partial surface (21) of the image sensor (5) with the image region of interest takes place, and c) the steps a) and b) are performed N times, wherein in step a) a different wavelength distribution is used each time.

Description

The present invention relates to a method of capturing images of an observation object with an electronic image sensor. In addition, the invention relates to an optical observation device for taking pictures of an observation object with at least one electronic image sensor.

For example, to provide good visibility to a surgeon in a surgical procedure using a surgical microscope, the surgical field is illuminated. Typically, a surgical microscope is an optic comprising a so-called varioscope for focusing the surgical field, a zoom unit (called pancake for stereoscopic imaging), and an eyepiece as an interface to the eye. The lighting is usually realized by thermal radiators, such as a halogen or xenon lamp, with a downstream optics to focus the light on the surgical field. Frequently, the illumination beam path and the observation beam path are partially guided by a common optical system, for example in the varioscope.

If one goes on to digitize the image recording by means of camera chips and to carry out the image reproduction via displays, in particular stereo displays or digital eyepieces, not only the image recording and the image reproduction can be spatially decoupled. In addition, it is possible to reduce the burden on the patient by illuminating the surgical field, since the sensitivity of the camera is superior to that of the eye, at least for the adaptation of the eye to the typical brightness in the operating theater. Studies have confirmed that the light intensity in the surgical field can be reduced if an operation is not performed with a purely optical microscope, but a camera is used for image acquisition.

When capturing color images, for example during a surgical procedure, color cameras are typically used. These are either expensive cameras with three chips, one each for the red, the green and the blue spectral range, or cameras with a single chip, with a so-called Bayer filter in front of the chip, in which each pixel of the sensor chip a red, green or blue filter element is upstream. If appropriate, spectral filters can also be present for other spectral ranges. A camera chip with upstream Bayer filter is for example in the German patent application DE 10 2008 017 390 A1 described. Since using a Bayer filter requires at least three pixels in order to obtain the full color information, this solution leads to a lower spatial resolution than would be possible with the chip in principle. In addition, the upstream of the Bayer filter requires an additional fabrication step.

Alternatively, it is possible to record a color image with a monochrome and thus less expensive camera time sequential. The observation object is illuminated in succession with different primary colors and the images are taken during the lighting in the respective primary colors one after the other with a single camera chip. The images in the primary colors are then superimposed electronically to obtain an image that corresponds to a white light illumination. Such methods for time sequential recording are, for example, in EP 0 601 179 or DE 10 2008 017 390 A1 described. In the method described in DE 10 2008 017 390 A1, LDI signals (LDI: laser Doppler imaging) can also be recorded between the images for the color images. The time-sequential recording of color images, however, requires a camera with a very high frame rate, which results in the cameras that can be used being very expensive.

Also in other situations, such as recording stereoscopic images with a single camera chip, wherein the stereoscopic partial images are then taken in succession with the same camera chip, the image refresh rate of the camera chip plays a role. High frame rates are also desirable here to avoid flickering the stereo image.

Andrew Adams et al. "The Frankencamera ..." describe a camera for taking pictures of an observation object with an electronic image sensor, wherein the image sensor is used to take a sequence of pictures, in which the entire scene and a section of the scene are taken entire scene and the shots of the clipping alternate. Both the entire scene as well as the section of the scene are shot with 640 × 480 pixels. For this purpose, when capturing the full screen, a combination of pixels to pixel blocks.

Aptina Imaging Coorp .: MT9P031: 1 / 2.5-inch 5Mp CMOS Digital Image Sensor, Data Sheet F describes a CMOS image sensor that can read the entire sensor field, including dark pixels. In addition, a pixel window can be defined that defines the output image. It is also possible to combine pixels or to use only a part of the pixels in the window.

CYPRESS Semiconductor Corp .: LUPA-4000 - 4M Pixels CMOS Image Sensor, Data Sheet describes a CMOS image sensor that can increase the read speed by either sub-sampling or just a window with a region of interest (Region Of interest).

It is an object of the present invention to provide a method and an optical observation apparatus with which sufficiently high frame rates can be realized even when using a low-cost camera.

This object is achieved by a method for capturing images of an observation object with an electronic image sensor according to claim 1 or a method for capturing images of an observation object with an electronic image sensor according to claim 2 and by an optical observation apparatus according to claim 20. The dependent claims contain advantageous embodiments of the invention.

In the method according to the invention for capturing images of an observation object with an electronic image sensor, a full image is imaged on the sensor surface, preferably on the entire sensor surface of the image sensor. An interesting region of interest (ROI) of the frame is imaged on a partial surface of the image sensor. The images are captured by exposing the image sensor for each image and reading it out after each exposure. In this case, an exposure of the image sensor and a readout of the entire illuminated for a frame sensor surface at least once exposing the image sensor and reading out only the partial surface of the sensor with the image region of interest, before again exposing the image sensor and reading the entire exposed sensor surface , A readout of the entire exposed sensor area is to be understood here as a read-out related to the area of the sensor and not as a read-out related to the total number of pixels in this area. Reading out the entire exposed sensor area may mean reading out all pixels of this area, but also reading out only a part of the pixels, with the pixels being distributed over the entire area, for example if only every n-th pixel row and / or every m-th pixel column be read in the case of subsampling. Likewise, a so-called pixel-binning is possible in which pixels (picture elements) of the sensor are combined into blocks of pixels and the pixels of a block of pixels are read together in each case.

The invention is based on the following consideration:
For example, in a surgical procedure performed using a surgical microscope, the operator is presented with the full image at a viewing angle of about 30 ° to 40 ° (field of view, FoV). The macula, which is the area in the eye that allows for sharp vision, limits the viewing angle, where sharp vision is possible, to about 5 °. The invention is based on the realization that it is therefore often not necessary to maintain a high image quality in the entire viewing angle. Rather, it is sufficient to maintain a high image quality in the field of sharp vision.

If one thus omits to read out the entire exposed sensor area each time, but concentrates on the region of interest (ROI), which corresponds to only a fraction of the sensor area, then the time required for reading this section is also reduced will, on just that fraction. If, for example, the region of interest in both image directions corresponds in each case to only one third of the total extent of the exposed sensor surface in the two image dimensions, only one ninth of the pixels of the exposed sensor surface are located in the region of interest. If, for example, the full image is read out with a low-cost 60 Hz camera in 16.67 milliseconds (one-sixtieth second), then the read-out time of the region of interest is only one-ninth of the total read-out duration, ie only 1.85 milliseconds. Because of the readout time for readout of the partial area of the exposed sensor area compared to readout of the entire exposed sensor area, it is possible to increase the image refresh rate of the camera used compared to the prior art, by only exposing the entire image at every k th readout of the image sensor Sensor surface is read out. In the case of the other exposures, on the other hand, only that partial area of the sensor which corresponds to the image region of interest is read out.

If, for example, a readout of the entire exposed sensor surface occurs three times the partial area corresponding to the region of interest before the entire exposed sensor surface is read out, the time required to read the entire exposed sensor surface is denoted by T and the area of the region of interest is the same as previously example is a ninth of the total area, so for the reading of the subarea a readout time of 1 / 9T is needed. For the three partial images, a total of three times 1/9 T is needed, which corresponds to 1/3 T. The time that elapses between reading out two frames has increased by one third reading out the partial images. During this time, however, a four-time reading of the interest takes place Area-representative field, which is indeed read together with the frame, so that a reading of this field on average every ((1 + 1/3) / 4) T takes place. For the area of interest, therefore, the duration between the read-out has been reduced to one-third compared with the prior art in which one frame is always read, so that the refresh rate has tripled for the area of interest. Outside the area of interest, the duration between reading two frames has increased to 4 / 3T, but this is still ¾ of the frame rate that would be achieved if only frames were read.

In fact, in many applications the area of interest is considerably smaller than the overall picture. For example, in the case of the surgical microscope already mentioned, the region of interest has a visual angle of about 5 ° with respect to the angle of vision of 30 ° to 40 ° of the full image. In this case, for example, if a reduction in the frame rate of the frames is accepted, then a multiplicity of partial pictures can be taken in comparison to the actually possible rate between two frames, so that the smaller the area of the frame, the more the frame rate for the corresponding image segment can be increased on the area of the full screen.

It is also important for a surgeon, in particular, that there is a slight delay between image acquisition and image playback (delay). Only then can a good hand-eye coordination, as it is of great importance for an operation, be achieved. In comparison with the prior art, this delay for the partial image representing the region of interest can be reduced by the method according to the invention in comparison to the prior art. The slightly increased delay in the area outside the area of interest is of less importance since it only takes place in the peripheral area of the observer, which is of minor importance for carrying out the operation.

In addition to increasing the refresh rate in the area of interest, the inventive method also offers other applications. In systems where captured images are transmitted over a limited bandwidth data link, for example, the method of the present invention allows the image region of interest to be transmitted at a high frame rate while the remaining regions are transmitted at a low frame rate to save bandwidth. Furthermore, the inventive method can be used to increase the depth of field of an optical observation device.

In the case of electronic image sensors, for example in the case of CCD sensors or CMOS sensors, exposure of the image sensor for the i-th image takes place during the readout of the (i-1) -th image. Reading out the entire exposed sensor surface, however, requires more time than reading out the partial area representing the region of interest. The theoretically possible exposure time of an image which is exposed during the readout of the entire exposed sensor area is therefore greater than the exposure time of an image which is exposed during the readout of the partial area of the sensor. In order to achieve a balanced modulation of the sensor, it is therefore advantageous if the intensity of the illumination for the i-th illumination is reduced compared to the illumination for the (i-1) -th exposure when reading (i-1) -th image the entire exposed sensor surface is read out to compensate for the longer exposure time. Alternatively, it is also possible to reduce the exposure time for the i-th exposure relative to the exposure time for the (i-1) th exposure when the entire exposed sensor area is read out for reading out the (i-1) -th image. In other words, only part of the basically possible exposure time is used. Of course, it is also possible to combine a shortened exposure time and a reduced illumination intensity. The combination should be such that both effects combined result in an exposure that would take place during the reading out of only the partial area of the sensor, possibly taking into account spectral sensitivities of the sensor.

An increase in the reduced frame rate for the frame as well as an increase in the frame rate for the region of interest are possible if when reading the entire sensor surface, ie when reading a frame, only every jth sensor row and / or only every k th sensor column with j > 1, k> 1 is read out. Here, the parameters j and k can be the same or different. The more resolution of the frame is reduced by subsampling, the higher frame rates can be achieved. Since only the full-screen is read with reduced resolution, but the sub-images with the full resolution, this leads to a total reduction only in the peripheral field of view of the viewer to a resolution of the image. Since in the area of interest the majority of the images, namely all partial images, are still recorded with the full resolution, the reduction of the resolution of the full image in the area of interest is practically not noticeable.

• a) ein Belichten des Bildsensors bei Beleuchtung des Beobachtungsobjekts mit einer ersten der N Wellenlängenverteilung eines Beleuchtungszyklus und ein anschließendes Auslesen der gesamten belichteten Sensorfläche erfolgt,
• b) danach wenigstens einmal eine Belichtungs- und Auslesesequenz des Bildsensors erfolgt, in der das Beobachtungsobjekt nacheinander mit verschiedenen Wellenlängenverteilungen eines Beleuchtungszyklus beleuchtet wird, bei jedem Beleuchten des Beobachtungsobjekts mit einer der Wellenlängenverteilungen eine Belichtung erfolgt und nach jedem Belichten jeweils nur ein Auslesen der Teilfläche des Sensors mit der interessierenden Bildregion erfolgt, und
• c) die Schritte a) und b) N mal durchgeführt werden, wobei in Schritt a) jedes mal eine andere Wellenlängenverteilung Verwendung findet.

Within the scope of a first embodiment of the method according to the invention, the image sensor used is a monochrome image sensor. This may be a CCD sensor or in particular a CMOS sensor. For illuminating the monochrome image sensor, the observation object is illuminated in illumination cycles in which the observation object is illuminated time-sequentially with N different wavelength distributions. The readout of the image sensor takes place in exposure and read-out cycles, wherein the illumination cycles are synchronized with the exposure and readout cycles such that per exposure and read-out cycle

• a) exposing the image sensor when the observation object is illuminated with a first of the N wavelength distribution of an illumination cycle and then reading out the entire exposed sensor surface,
• b) thereafter at least once an exposure and read sequence of the image sensor takes place, in which the observation object is illuminated successively with different wavelength distributions of a lighting cycle, with each illumination of the observation object with one of the wavelength distributions an exposure takes place and after each exposure only one readout of the partial surface of the Sensor with the image region of interest takes place, and
• c) the steps a) and b) are performed N times, wherein in step a) each time a different wavelength distribution is used.

Here, the N different wavelength distributions may comprise at least the basic colors of an RGB color space or a CMY color space. In addition to the basic colors of the RGB color space or of the CMY color space, optionally at least one further wavelength distribution may be present, for example for fluorescence measurements, laser Doppler imaging (LDI), functional measurements on function-bearing brain areas, etc. The basic colors of the RGB color space or of the CMY color space can be used in particular for generating a color image.

In the described embodiment of the method according to the invention, therefore, an exposure and readout cycle begins with the acquisition of a full image at a specific wavelength distribution of the illumination cycle. After the frame has been taken, an exposure and read-out sequence follows, in which, for example, partial images are taken successively for all wavelength distributions contained in the illumination cycle. Then, if step a) is performed again as part of the exposure and read cycle, one frame is taken for the next of the N wavelength ranges. Here too, for example, the inclusion of partial images for all wavelength distributions of the illumination cycle follows again. The exposure and readout cycle is terminated when step a) has been performed once for all wavelength distributions contained in the illumination cycle and also for the last step a) step b) is completed. Therefore, if in each exposure and read sequence the observation object is illuminated at least once in succession with all N wavelength distributions, a total exposure and read cycle will comprise at least xN + 1 illumination cycle, where x indicates how many frames are taken successively with all N wavelength distributions before one resume a full screen. For example, if frames are captured exactly once for all N wavelength distributions after capturing a frame, x = 1, so that one entire exposure and read cycle will include N + 1 lighting cycles. In order to be able to estimate the time savings made possible by the further development of the method described, and the resulting image refresh rates, the following calculation can be made:
If T is the time required to read one frame and y represents the area ratio of the field to the frame, the read-out time for one field T is _T = y * T. If N is the number of partial images necessary for a full color image (for example three partial images in the RGB color space), then the time for the readout of a full color image in the described development of the method N · T (for the N during an exposure and Read-out cycle) + (x * N * Y * T) (for the fields captured between two frames).

For the case x = 1 and N = 3 (for three primary colors of the RGB color space), the time duration for taking a color image of 3 * T + N * yT is thus obtained. In a prior art method of taking only frames, the time taken to take a color image would be 3 * T. Thus, with the method according to the invention, the time duration is lengthened by N · y · T. It is readily apparent that the smaller the factor y, the smaller is the sensor area required for one sub-image with respect to the entire exposed sensor area, the less important is the extension of the time duration compared with the prior art. If, for example, the partial area corresponds to one-ninth of the total exposed sensor area, a read-out duration T is reached 3T + 3 · 1 / 9T = 4 / 3T, thus corresponds to four thirds of the readout time according to the prior art. This corresponds to a refresh rate of three quarters of the frame rate according to the prior art.

In the area of interest, in addition to the full screen, which is also the area of interest, x × N field images recorded. This means that at x = 1 in the time in which a frame is recorded, a total of four images of the area of interest are recorded. The refresh rate for the region of interest is therefore a factor of 4 higher than for the frame.

• a) ein Belichten des Bildsensors mit dem weißen Licht eines Beleuchtungszyklus und ein anschließendes Auslesen der gesamten Sensorfläche erfolgt,
• b) danach wenigstens einmal eine Belichtungs- und Auslesesequenz des Bildsensors erfolgt, in der das Beobachtungsobjekt nacheinander mit verschiedenen Wellenlängenverteilungen eines Beleuchtungszyklus beleuchtet wird, bei jedem Beleuchten des Beobachtungsobjekts mit einer der Wellenlängenverteilungen eine Belichtung erfolgt und nach jedem Belichten jeweils nur ein Auslesen der Teilfläche des Sensors mit der interessierenden Bildregion erfolgt.

In an alternative embodiment of the method according to the invention, in which a monochrome image sensor is used as the image sensor and illumination of the observation object likewise takes place in illumination cycles, the observation object is illuminated time-sequentially with white light and N different wavelength distributions. The white light can be realized, for example, in that all N different wavelength distributions simultaneously illuminate the observation object. Also in this embodiment of the method, the reading of the image sensor in exposure and readout cycles takes place. The lighting cycles are synchronized with the exposure and readout cycles such that per exposure and readout cycle

• a) exposing the image sensor to the white light of a lighting cycle and then reading out the entire sensor surface,
• b) thereafter at least once an exposure and read sequence of the image sensor takes place, in which the observation object is illuminated successively with different wavelength distributions of a lighting cycle, with each illumination of the observation object with one of the wavelength distributions an exposure takes place and after each exposure only one readout of the partial surface of the Sensor takes place with the image region of interest.

Thus, in the alternative embodiment of the method according to the invention, an illumination cycle comprises one illumination with white light and xN time-sequential illumination with the different wavelength distributions, if every N wavelength distributions are contained in the exposure and read sequence. In the simplest case, x = 1, so that one illumination cycle comprises N + 1 illuminations, namely the illumination with white light and, subsequently, the time-sequential illumination with the N different wavelength distributions. As in the previously described embodiment of the method according to the invention, the N wavelength distributions may comprise at least the basic colors of an RGB color space. Optionally, in addition to the primary colors of the RGB color space and the white light, at least one further wavelength distribution may be present, for example for fluorescence measurements, laser Doppler imaging, functional measurements on functional brain areas, etc. A color image can be generated from the N primary colors of the RGB color space become. In the described development of the method according to the invention, in which white light is used to take the frames, because of the monochrome image sensor, only the region of interest is shown in color, while the periphery is shown only in black and white. However, for many applications where often only the area of interest matters, such a representation is quite sufficient.

In many applications, the image region of interest of an image will not always occupy the same position in the frame. For example, in a surgical microscope, the image region of interest depends on the viewing direction of the observer. It is therefore advantageous if the position of the image region of interest in the frame is determined by determining the viewing direction of an observer viewing the frame. In an optical observation system such as a surgical microscope, this can be done by means of a so-called eye-tracking system, which detects the viewing direction. If the viewing direction is determined repeatedly, the position of the partial area of the image sensor representing the region of interest of the image sensor on the sensor surface can be updated on the basis of the currently determined viewing direction. In this case, the viewing direction is preferably determined at a rate of less than 20 Hz, preferably less than 10 Hz and in particular at a rate in the range of 5 to 10 Hz. The eye moves very quickly in so-called saccades, without the information obtained thereby processed by the brain. Saccades are usually completed after 50 milliseconds. A frequency in determining the viewing direction of more than 20 Hz could therefore lead to the position of the partial surface of the image sensor is tracked by means of saccades, but this does not correspond to the line of sight, which is also processed by the brain. Although a rate of less than 20 Hz should in principle be sufficient to avoid tracking the patch using saccades, it is advantageous if the rate at which the gaze direction is updated is less than 10 Hz. A range of 5 to 10 Hz is particularly advantageous since, in the case of an update at a rate of less than 5 Hz, the synchronization of the partial area of the sensor representing the region of interest on the sensor surface with the viewing direction could have disturbing delays, for example during operations can not be accepted.

Since the above-described estimates with respect to the read-out time for the image region of interest do not depend on the position of the corresponding subarea of the image sensor on the exposed sensor surface, the estimates apply regardless of the location of this patch on the image sensor.

As an alternative to tracking the partial area of the image sensor representing the region of interest of the image frame on the basis of the viewing direction, it is also possible to use a moving object present in full image for tracking purposes. For this purpose, a moving object is detected in full screen by means of an image recognition software and positioned the subarea of the image sensor on the sensor surface and tracked the image of the moving object on the sensor surface that the image of the moving object remains in this subarea. This embodiment of the method according to the invention is particularly suitable for automated observation operations, such as in the context of monitoring systems. For example, as soon as a movement is detected in the area to be monitored, the partial area of the image sensor can be positioned such that the partial area detects the movement as long as it can be observed by the surveillance camera. If the movement of the object in the image is due to a movement of the observation device due to vibrations, the tracking can be used to compensate for the vibrations. This is advantageous, for example, in surgical microscopes.

In all described embodiments of the method according to the invention, three-dimensional images can be recorded as images of the observation object, which are composed of at least two perspective images recorded at different viewing angles. In particular, as three-dimensional images, stereoscopic images composed of stereoscopic partial images can be recorded. All perspective images (or stereoscopic partial images when taking a stereoscopic image) are recorded with an electronic image sensor, whereby the method according to the invention is applied to taking the perspective images. If a monochrome image sensor is used for this purpose, for which illumination the observation object is illuminated in illumination cycles in which the observation object is time-sequentially illuminated with N different wavelength distributions, and the image sensor is read in exposure and read-out cycles, the illumination cycles being determined by the illumination exposure. and read-out cycles according to the above-described steps a) to c) are synchronized, for taking a three-dimensional image, steps a), b) and c) of a perspective image exposure and readout cycle may first be carried out in sequence, before steps a), b) and c) are executed for the next perspective image. Alternatively, for taking a three-dimensional image, it is possible to perform the exposure and read-out cycle as many times as the three-dimensional image has perspective image and make a change between taking a perspective image and taking the next perspective image whenever exposing and reading the image Image sensor with all N wavelength distributions of the lighting cycle is done. Each perspective image of the three-dimensional image is then composed of N sub-images representing the region of interest (hereinafter referred to as clipping images to distinguish them from the stereoscopic sub-images) or from N-1 clipping images and a full image. It may be advantageous if the Beilichtungs- and Auslesezyklen each alternately begin reading the first frame at different wavelength distributions of the lighting cycle, so that for the exposure and readout cycles, for example, alternately N, N + 1, N + 2 and N + 1, N + 2, N for the order of illuminations at which the frames are taken.

If an illumination cycle includes N wavelength distributions and white light, the exposure and read-out cycle may also be performed as often as the three-dimensional image has perspective images to capture a stereoscopic image. A change between the taking of a perspective image and the taking of the next perspective image takes place after an exposure and readout of the image sensor with the white light and all N wavelength distributions of the illumination cycle has taken place.

The method according to the invention can be realized as a computer program with instructions for carrying out the method.

An optical observation device according to the invention for recording images of an observation object comprises at least one electronic image sensor and observation optics which images an object field of the observation object as a full image onto the sensor surface of the image sensor. In addition, the optical observation apparatus according to the present invention comprises selecting means for selecting a partial area of the image sensor corresponding to an image region of interest of the frame, and a sensor control unit configured to control the exposure and readout of the image sensor according to the method of the present invention. The selection device can be a manual selection device or an automatic selection device, for example a device based on an eye-traker.

With the optical observation device according to the invention, the inventive Perform the procedure. The properties and advantages described with reference to the method can therefore be realized with the optical observation device according to the invention. The optical observation device can be designed, for example, as a medical-optical observation device such as a surgical microscope or an endoscope, as a surveillance camera, as a scientific microscope, as a microscope for material examinations, etc.

In addition, the optical observation device comprises a lighting device with pulsed light sources that can be pulsed in different wavelength distributions. In addition, it has an illumination optics for supplying the light of the illumination light sources to the observation object and a lighting control unit. The illumination control unit acts on the illumination light sources in such a way that they are operated pulsed alternately, in particular in cyclical change. Furthermore, a synchronization unit connected to the sensor control unit and the lighting control unit for synchronizing the pulsed operation of the illumination light sources with the exposure and readout of the image sensor according to the embodiment variants of the inventive method described above is provided. At least three illumination light sources can be present in the illumination device, each of which has a wavelength distribution with a maximum for a base color of an RGB color space or CMY color space. In addition, optionally at least one illumination light source may be present whose wavelength distribution has at least a maximum outside the range from 430 to 750 nm, in particular outside the range from 400 nm to 800 nm. Such an illumination light source can be used, for example, for fluorescence measurements, laser Doppler imaging, functional measurements on functional brain areas, etc.

If an illumination light source having a wavelength distribution having a maximum outside the range of 430-750 nm, especially outside the range of 400-800 nm, is present, the optical observation apparatus may further comprise a display unit composed of those exposures of the image sensor, the when illuminating the observation object with an illumination light source having a maximum in a basic color of an RGB color space, a color image synthesized and from that exposure of the image sensor, which in the illumination of the observation object with the illumination light source, in the wavelength distribution at least a maximum outside the range of 430 to 750 nm, in particular outside the range of 400 to 800 nm, is present, an independent of the color image generated image.

The image, which is independent of the color image and is formed by the visual display unit, can be superimposed on the color image in an overlay unit. In this case, for example, an overlay in the form of false colors, contours, changes in brightness, etc. take place. In particular, the at least one further illumination light source in whose wavelength distribution is at least a maximum outside the range of 430-750 nm, in particular outside the range of 400 to 800 nm, have a maximum wavelength that excites fluorescence in the observed object. In the observation beam path of the optical observation device may then be present or einbringbar a filter which blocks the wavelengths of this wavelength maximum. This can also be introduced into the beam path, for example, pivoted, only for the purpose of recording the image independent of the color image.

If the optical observation device according to the invention has an illumination device with illumination light sources emitting in different wavelength distributions, the illumination optics can have an association device for merging the illumination beam paths of the individual illumination light sources. As a result, the number of optical components can be kept low, since no optical components for different illumination beam paths must be present multiple times, so that a compact design is possible. In addition, by combining the illumination beam paths, it can be avoided that the illumination light of the different illumination light sources is supplied to the observation object at different illumination angles. Both are important, for example, in surgical microscopes.

The optical observation device according to the invention may, in particular, comprise observation optics which allow the taking of perspective images from at least two different viewing angles. In particular, the observation optics can be designed as stereoscopic observation optics. In a further development of this observation optics permitting the taking of perspective images, the latter can be embodied such that the ray bundles of the perspective images, ie the stereo channels in the case of stereoscopic observation optics, are time-sequentially imaged onto the same image sensor. Such observation optics are for example in or German patent application DE 10 2011 010 262 described in terms of possible Embodiments of the observation optics, such that a time-sequential mapping of the stereo channels can be made on a single image sensor is referenced.

Further features, properties and advantages of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying figures.

1 shows a first embodiment of an inventive optical observation system in a highly schematic representation.

2 shows the exposed sensor surface of an image sensor with a partial area of the sensor representing the region of interest of an image.

3 shows a timing diagram for a time-sequential color image recording with a monochrome image sensor and a time-sequential illumination in the primary colors of an RGB color space.

4 shows a timing diagram for a time-sequential image acquisition with a monochrome image sensor and a time-sequential illumination with white light and the basic colors of an RGB color space.

5 shows a second embodiment of the optical observation device according to the invention.

6 shows a timing diagram for a time-sequential image acquisition with a monochrome image sensor and a time-sequential illumination with a fluorescence-generating radiation source and the basic colors of an RGB color space.

7 shows a timing diagram for a time-sequential image acquisition with a monochrome image sensor and a time-sequential illumination with white light, fluorescence exciting radiation and the basic colors of an RGB color space.

8th shows an adaptation of the time scheme 3 to a stereoscopic optical observation device.

9 shows a variant of the readout scheme 8th , which can shorten the read-out time for a full screen.

10 shows an alternative variant of the adaptation of the timing scheme 3 to a stereoscopic optical observation device.

11 shows an adaptation of the time scheme 4 to a stereoscopic optical observation device.

12 shows an adaptation of the time scheme 6 to a stereoscopic optical observation device.

13 shows an alternative variant of the adaptation of the timing scheme 6 to a stereoscopic optical observation device.

14 shows an adaptation of the time scheme 7 to a stereoscopic optical observation device.

An exemplary embodiment of an optical observation device according to the invention is highly schematized in FIG 1 shown. The illustrated optical observation device comprises an observation optics 1 , a camera 3 with a monochrome electronic image sensor 5 , a presentation unit 7 (Arithmetic unit), which comes from the with the image sensor 5 Recorded data images are then generated on a display unit 9 , For example, a monitor, a data glasses, an electronic eyepiece, etc. are shown.

Furthermore, the optical observation device comprises at least one light source L1 to LN, wherein in the present embodiment, at least three light sources are present, the wavelength distributions, which correspond to different basic colors of an RGB color space. The number of light sources and their wavelength distributions may depend on the intended field of use of the observation device. In addition, if at least two light sources are present, the optical observation device advantageously comprises an assembly device 11 in which the illuminating beam paths of the individual light sources L1-LN are in a lighting optics common to all the light sources 13 be introduced.

The optical observation device is controlled by a control unit 15 controlled, at least with the camera 3 connected is. If there are multiple light sources L1 to IN that are pulsed, the control unit is 15 also connected to the light sources. With the help of the control unit 15 Then there is a synchronization between the lighting on the one hand and the exposure and reading of the image sensor 5 on the other hand. If the observation optics is designed as a varioscope with which the focus position of the optical observation device in the observation object 17 can vary, can also be a control line to the observation optics, in particular to the zoom lens to the focus position with the exposure and the reading of the image sensor 5 to synchronize. Finally, can the control unit 15 also with the presentation unit 7 be connected to control the image representation.

The principle of taking pictures with the help of in 1 is based on the optical observation apparatus shown below, with reference to 2 explained. 2 shows a plan view of the electronic image sensor 5 in which the sensor surface 19 , which is exposed for a full screen, can be seen. In addition, a partial area 21 of the image sensor, in which a region of interest (ROI) of interest in the following called ROI, on the sensor surface 19 is shown.

When taking pictures with the optical observation device, the image sensor becomes 5 repeatedly exposed and read out, wherein the reading of the exposed with the current exposure sensor during the subsequent exposure takes place. The control unit controls 15 the exposure and reading of the electronic image sensor 15 such that on a readout of the entire sensor surface 19 at least once reading out only the sensor area representing the ROI 21 takes place before the entire sensor surface is read again. Because the sensor surface 21 , which represents the ROI, significantly smaller than the entire sensor surface 19 is, it can be read at a much higher speed. This makes it possible by reading only the sensor surface representing the ROI 21 reach a higher frame rate than if every time the entire sensor surface 19 is read out. Although this somewhat reduces the frame rate outside of the ROI; however, this is not usually annoying because the essential image information is in the ROI.

The taking of color images is done by the observation object 17 time sequentially illuminated with the primary colors of an RGB color space and the sensor 5 is illuminated and read at each illumination of the observation object. For this purpose, the optical observation device comprises 1 at least three light sources L1 to L3 emitting, for example, red (L1), green (L2) and blue (L3) light. Correspondingly, the monochrome image sensor records red, green and blue images which are sent to the presentation unit 7 (Arithmetic unit) passed and combined there to form a synthetic color image, which then on the display unit 9 is pictured.

A typical consumer camera has a refresh rate of 60 Hz, which means that it takes around 16.67 milliseconds to read an image. If color images are to be synthesized with such a consumer camera according to the method described above, they can only be displayed with a refresh rate of 20 Hz, since three monochrome images in red, green and blue must be recorded for each color image. However, if you omit it, always the entire sensor surface 19 but focuses on the ROI 21 , which is usually only a fraction of the total sensor area 19 is the reading of the ROI corresponding sensor surface 21 required time is reduced in the ratio in which the partial area 21 to the entire sensor surface 19 stands. If, as in 2 schematically illustrated, the ROI corresponding sensor surface 21 in both image dimensions has an extent that is one third of the total exposed sensor area 19 corresponds, this sub-area contains one-ninth of the pixels of the sensor surface 19 , Accordingly, this partial area can be 21 in a ninth of the time, to read the entire sensor surface 19 would be necessary to read. In the example with the consumer camera, which has a refresh rate of 60 Hz, the readout duration is the entire sensor area 19 about 16.67 milliseconds. The described subarea 21 can be read in about 1.85 milliseconds.

3 shows an example of a timing scheme how color images can be taken according to the described principle. The image acquisition can be thoughtfully divided into exposure and readout cycles. In 3 Three such exposure and readout cycles are shown. In addition, the illumination of the observation object takes place in lighting cycles, wherein in each lighting cycle the light is illuminated once with the light of the light source L1 (red), the light of the light source L2 (green) and the light of the light source L3 (blue).

In the in 3 illustrated embodiment, an exposure and Auslesezyklus each four lighting cycles. An exposure and readout cycle begins with the first illumination of a lighting cycle, that is, in the illustrated example, with the illumination of the observation object 17 with the red light of the light source L1. During this illumination, the image sensor is exposed 5 , This is followed by a lighting of the observation object with the light of the light source L2, that is, in the present embodiment with green light. At the same time, the image sensor is exposed 5 , In addition, the image taken during the previous exposure is read out (the reading of the nth image takes place in the case of a camera with global shutter during the exposure for the subsequent image n + 1). When reading out the image recorded during the illumination with L1, the entire sensor surface becomes 19 , so a full screen, read.

After the illumination of the observation object 17 with the light from the light source L2 and the corresponding exposures of the image sensor 5 a lighting of the observation object takes place 17 with the light of the light source L3, so in the present embodiment with blue light, and re-exposing the image sensor 5 , At the same time, the image taken with the green illumination of the light source L2 is read out. In this reading, however, only a readout of the partial area 21 on which the ROI is mapped so that only one ROI image is read. After lighting the observation object 17 and exposing the image sensor 5 with the light of the light source L3, the first lighting cycle is completed.

In the context of the second illumination cycle, a re-illumination with the light of the light source L1 takes place, wherein an exposure of the image sensor 5 and a reading of the image obtained with the light of the light source L3 takes place. Again, only the partial area 21 of the image sensor 5 read out, so that in turn a ROI image is created. When re-illuminating the observation object 17 with the light from the light source L2 and the exposure image sensor 5 Now, the image, which has been obtained with the light source L1, read out, in turn, only the partial area 21 , So a ROI image is read out. After this ROI image is extracted, ROI images are in green, blue, and red, so that a color ROI image can be synthesized. Next is the observation object 17 again illuminated with the light of the light source L3, in turn, an exposure of the image sensor 5 takes place. At the same time, the image taken with the green light during the previous illumination becomes the entire sensor area 19 read out, so that a green full screen is obtained. Thereafter, the second lighting cycle is completed.

In the third illumination cycle, in turn, a lighting of the observation object takes place in succession 17 and a corresponding exposure of the image sensor 5 with red, green and blue light of the light sources L1, L2 and L3, obtaining a red ROI image, a green ROI image and a blue frame. After the blue frame is recorded, a full color picture can be synchronized.

In the fourth and final exposure cycle of the exposure and readout cycle, only ROI images are captured in all colors. Thereafter, the first exposure and readout cycle is completed, and the subsequent exposure and readout cycle may begin. This subsequent cycle corresponds to the first cycle. In each exposure and read-out cycle, a full color image and additionally three color ROI images are recorded.

The table in 3 contains not only the information about the respective exposure and readout cycle (1st column), and the lighting cycles (2nd column), but also whether a full image or a ROI image is taken (3rd column). In addition, the table contains the example of the consumer camera with 60 Hz refresh rate information about the beginning and end of the respective exposure (4th and 5th column), measured from the beginning of the first lighting cycle on. In addition, it describes the length of time until a full color image is read out and can be synthesized (6th column), the time between recording a monochrome frame (7th-9th column), and the color frame representation, the time duration until a color ROI image is read out and can be synthesized (10th column) and, for ROI images, the time between taking a monochrome ROI image and displaying the corresponding color image (11th-13th column). Note in the table that all numbers are rounded to the first decimal place.

As it turned out 3 The typical length of time between two output color frames is about 66.7 milliseconds (column 6), which corresponds to an image refresh rate of 15 Hz. The frame rate for frames therefore only equates to three quarters of the frame rate achievable with the 60 Hz consumer camera when reading only frames (at 60 Hz frame rate, the frame rate for color frames and time-sequential color shots is 20 Hz). In contrast, the average time between two color ROI images is only 16.7 milliseconds (see column 10), which corresponds to a refresh rate for the colored ROI images of 60 Hz. The region of interest of the image is therefore represented at a refresh rate of 60 Hz, which is three times that achievable with the exclusive reading of frames with the consumer camera.

It should be noted once again that, as previously mentioned, in a camera with global shutter reading of an image takes place during the exposure of the next image. The readout time required to read out the nth image therefore determines the minimum time available for exposing the image n + 1. In the in 3 The timing shown in columns 4 and 5, respectively, the beginning and the end of the exposure is indicated. As can be seen from these columns, is in each case when a full screen, so the entire sensor surface 19 , is read out, the maximum time available. If only the partial area 21 is read, only a fraction of this maximum time is available for exposure. For a balanced modulation of the image sensor 5 to allow either the maximum Lighting time or the maximum exposure time to the shorter period of reading a ROI image are limited. Alternatively, the longer illumination or exposure time can be maintained and the illumination intensity can be regulated down accordingly. A combination of both, ie from the reduction of the illumination intensity and the reduction of the illumination and exposure time is possible.

With the described method can be realized with a low-priced camera, which allows only a limited refresh rate, in the area of the ROI flicker-free, color sequential image acquisition. It makes use of the fact that an observer usually only needs the full picture quality in the ROI. Reducing the refresh rate outside the ROI can be tolerated.

In the general case that n _color monochrome frames are taken for the capture of a full _color image and monochrome ROI images are taken between two monochrome frames n _ROI , the refresh rate for the colored frames decreases compared to capturing frames only by n _color / ( n _color + y * n _ROI ), where y represents the area ratio of an ROI image to a frame. In contrast, the refresh rate of the colored ROI images increases significantly by the factor (n _color + n _ROI ) / (n _color + y · n _ROI ).

If it is not necessary to obtain color information about the observation object outside the ROI, an alternative timing scheme may be used. This is in 4 shown. While after the in 3 The time diagram shown in the context of an exposure and readout cycle after reading the entire sensor surface at least once followed by an exposure and read sequence of the image sensor, in which the observation object is illuminated sequentially with all wavelength distributions of a lighting cycle and each ROI image is taken with each illumination, so an exposure and read-out cycle comprises at least four lighting cycles corresponds to the timing scheme 4 an exposure and readout cycle largely a lighting cycle.

In the in 4 As shown at the beginning of the exposure and read-out cycle, a lighting of the observation object with white light takes place. This can be done, for example, in that the observation object is illuminated simultaneously with the light of all three illumination light sources L1 to L3, ie simultaneously with red, green and blue light, or by the fact that at least one fourth light source L4 is present which emits white light. After the exposure of the image sensor during the illumination of the observation object 17 is done with the white light, there is a lighting of the observation object with the light of the light source L1, that is, with red light, and an exposure of the image sensor 5 , At the same time, the image taken during exposure to white light is read out. In this reading, the entire sensor surface 19 read, so that a full screen is created. Since the image sensor used is monochrome, so can not record color information, a full screen is created in grayscale.

After illuminating the observation object with the light of the light source L1 and the corresponding exposure of the image sensor 5 a lighting of the observation object with the light of the light source L2, ie with green light, and a re-exposure of the image sensor 5 , At the same time, the image previously taken in the red illumination is removed from the sensor surface 21 which matches the ROI, so that a red ROI image is created.

After lighting the observation object 17 and exposing to the light of the light source L2, the observation object is illuminated with the light of the light source L3, ie with blue light, and a corresponding exposure of the image sensor 5 , During this exposure, the image previously taken in the green illumination is read out, with only the partial area 21 the sensor surface 19 is read, creating a green ROI image. After illuminating the observation object with the blue light and corresponding exposure of the image sensor 5 the lighting cycle is complete and the next lighting cycle can begin with white light illumination. During this time, the previously exposed during illumination with the blue light image sensor is read again, in turn, only the partial area 21 the sensor surface 19 is read, creating a blue ROI image. Then, from the three ROI images obtained, a color image can be synthesized for the ROI. In the subsequent exposure and read cycle, this entire process is repeated.

As already mentioned, if the image is taken according to this timing scheme, only the ROI image is displayed in color. However, the refresh rates for the full screen and the colored ROI image are identical. If again it is assumed that the subarea 21 a ninth of the sensor surface 19 With this timing scheme, every (1 + 3 × 1/9) × 16.67 milliseconds, that is, every 22.2 milliseconds, can be represented as both a grayscale frame and a color ROI image. This corresponds to a refresh rate of 45 Hz (to remind you, when only frames are taken with a 60 Hz camera, you can use color images only shown with a frame rate of 20 Hz).

A second exemplary embodiment of the optical observation device according to the invention is shown in FIG 5 shown. Elements similar to those of 1 correspond to the first embodiment shown are in 5 with the same reference numerals as in 1 and will not be explained again to avoid repetition.

This in 5 illustrated embodiment is designed except for the recording of color images for receiving further information. In the present exemplary embodiment, these further information are fluorescence images of the observation object 17 , But also images outside the visible spectral range can be taken instead of the fluorescence images or in addition to the fluorescence images. To record the fluorescence images, the optical observation device comprises one in the observation beam path, ie between the observation object 17 and the image sensor 5 , arranged spectral filters 23 , This is located in the present embodiment between the observation optics 1 and the camera 3 , In addition, one of the light sources L1 to IN has a spectrum that allows the excitation of fluorescence in the observation object. The fluorescence-exciting part of the spectrum of this light source is from the spectral filter 23 filtered out of the observation beam path. It is possible here to use excitation radiation outside the wavelength range of 437-750 nm, in particular outside the wavelength range of 400-800 nm, if the fluorescence to be excited permits this.

This embodiment of the optical observation device is advantageous, for example, if a surgeon is to be provided with medically relevant information that lies outside the visible spectral range or overlaps with it. Exemplary here is called fluorescence imaging, for which there are currently three approved in the field of medical fluorescent dyes, namely the compound 5-aminolevulinic acid (short 5-ALA), indocyanine green (short ICG) and fluorescein.

Compound 5-ALA is a compound used as a dye in, for example, brain surgery. Upon administration of 5-ALA, accumulations of fluorescent compounds take place in the cells with increased metabolism, typically tumor cells. The fluorescence is typically excited at a wavelength <430 nm, while the emission takes place in the red spectral region. In purely optical surgical microscopes, the fluorescence is visualized after treatment with 5-ALA by providing the illumination with an edge filter, so that only light ≤ 430 nm reaches the surgical field. In turn, a filter is introduced into the observation beam path, which only passes light with wavelengths ≥ 430 nm to the eye of the surgeon. The filters are designed so that there is a very narrow wavelength range at 430 nm, in which excitation radiation can pass through the filter, so that a part of the excitation light is available in the observation, as in WO 1997/011636 is described. The blue light component in the image is used by surgeons to orient themselves in the surgical field. The fluorescence takes place in the red; the fluorescent light enters the observer's eye unfiltered. The distinction between tumor tissue and healthy tissue is made by the differentiation of blue shades (nonspecific reflection on the tissue) and red shades (fluorescence of tumor tissue). In this method, the ratio between "blue" nonspecific tissue and red tumor can not be changed because of the defined - and therefore unchangeable - filter edges in the illumination and observation beam path. This prevents weak fluorescence from being perceived by the surgeon. The possibilities of tissue differentiation are thus limited in principle.

ICG is a dye used to visualize blood vessels (angiography). For example, ICG can be excited at wavelengths <780 nm, while fluorescence above 780 nm is detected - typically with a camera. In brain surgery, it is important to clamp Gefäßaussackungen, so-called aneurysms, for example, with a clip, as the Aussackungen tear easily and can lead to brain bleeding. After attaching the clip, ICG is typically injected into the bloodstream. A camera is then used to record how the fluorescent dye flows into the blood vessels. To determine if the clip has been set correctly, the surgeon must be able to determine if the fluorescence in the clamped aneurysm is flowing in through the large blood vessel or through capillary. This means that a fast sequence of camera images is to be evaluated. Often the frame rate is not sufficient for typical 60 Hz cameras to evaluate the flow behavior. Since both the influx must be observed at the same time - typically on a monitor that displays the infrared fluorescence image as a grayscale image - as well as the patient is monitored, several people are required to do so. Here, a solution is sought, which can represent both a fluorescence image with high frame rate (> 60 Hz), and at the same time allows monitoring of the patient, for example via an image in the VIS area.

Fluorescein is a fluorescent dye used in the operating room for both angiography and tissue differentiation. Excitation is typically done at wavelengths of about 480 nm to observe fluorescence in the green. This means that both excitation and fluorescence are in the middle of the visible spectral range.

Regardless of the three fluorescent dyes mentioned above, the challenge for dyes that will receive medical approval in the future is to simultaneously visualize both a nonspecific overview image and the specific fluorescence in the case of tissue differentiation or fast fluorescence imaging in angiography as well as to allow monitoring of the patient.

In the 5 The arrangement shown makes it possible to simultaneously display both a visual image and fluorescence information. Tumor recognition (5-ALA, fluorescein) or angiography (ICG, fluorescein) can thus be carried out, for example, by means of the known fluorescent dyes, while a surgeon can simultaneously orientate himself or monitor the patient on the visual image of the surgical field. By means of time-sequential illumination, a temporal separation of the visual image from the fluorescence image can take place, with no moving parts, such as a filter wheel, being necessary. In addition, the visual image of the arithmetic unit 7 be synthesized separately from the fluorescence image, so that both images can be displayed on different monitors. Alternatively, there is also the possibility that the arithmetic unit 7 a superimposition of the fluorescence image with the visual image is calculated, for example in such a way that the fluorescence image of the visual image in the form of contours, which indicate, for example, a tumor, in the form of false color information, in the form of "Highlighting" by lightening or darkening of certain image areas, etc . he follows.

The recording of the fluorescence image can be readily in the with reference to 3 time schedule are integrated. A corresponding timing scheme supplemented for the excitation radiation L4 for the fluorescence is shown in FIG 6 shown. In accordance with the above-mentioned formula, a frame rate of 10.4 Hz (corresponding to a time interval between two frames of 96.3 milliseconds) and 51.9 Hz in the case of ROI images (corresponding to a time period between two ROI images) can be set for the frames. Images of 19.3 milliseconds).

Except in the timetable off 3 can also take up fluorescence images in the time scheme 4 be readily integrated. A corresponding timing scheme supplemented with the fluorescence-exciting light source L4 is shown in FIG 7 shown. For this time scheme, image refresh rates of 41.5 Hz can be achieved for both the greyscale and the colored ROI images, which corresponds to an average time between two images of 24.1 milliseconds.

The time schemes described so far for taking color images or for taking color images and fluorescence images can also be transferred to stereoscopic optical observation devices such as surgical microscopes. A first variant according to which the in 3 shown in a stereoscopic optical observation device is used, is in the timing of 8th shown. From 3 known exposure and readout cycles are carried out alternately for the stereoscopic observation beam paths. For example, after the first cycle for the right stereo channel has been performed, the first cycle for the left stereo channel is performed before the second cycle for the right stereo channel is performed, etc. Since each exposure and read cycle must be performed twice (once for the right and once for the left stereo channel), the frame rates achievable according to this timing scheme are halved, and the time intervals between two frames or between two ROI images are doubled in comparison to that described with reference to FIG 3 described timing. But synonymous with a 60-Hz consumer camera in which only frames are recorded, would be synonymous for the right and the left stereo channel sequentially frames are recorded, so that even with this camera, the refresh rate for colored images from 20 Hz to 10 Would reduce Hz. The realizable with the timing advantage, as he with respect to 3 has been described, therefore, remains.

If the color frames do not need to have the same spatial resolution as the colored ROI images, increasing the frame rate for the frames and the ROI images can be achieved by reading the entire sensor surface 19 (Full screen) subsampling is used. An appropriate time scheme is in 9 shown. For example, instead of each pixel, the sensor area 19 only every second pixel per column and row can be read. As a result, the readout duration for reading out the sensor surface 19 necessary to be reduced by a factor of 4. Although the subsampling is not limited to stereoscopic optical observation devices, it is particularly advantageous when taking stereoscopic images, since then even with a 60 Hz consumer camera for frames a frame rate of 17.1 Hz and in the case of ROI images a refresh rate of 69 Hz can be realized. Compared to that with respect to 8th Thus, the frame rate can be more than doubled when recording stereoscopic images. The refresh rate is even higher than that with the time scheme 3 realized refresh rate (where no subsampling was done).

In general, subsampling, in which only every nth pixel in a line required for a frame and only every mth line is read out, can shorten the read duration by the factor n × m. Since the ROI usually corresponds to the viewing direction of the eye and the eye has a smaller angular resolution outside the viewing direction than in the viewing direction, the recording of the frames with subsampling is often readily possible.

A shortening of the readout duration can also be achieved by a so-called pixel binning. In pixel binning, adjacent pixels are grouped into blocks of pixels, and all the pixels in a block are read together. If a block has n × m pixels, it is possible to shorten the read duration by the same factor.

An alternative variant, like the time scheme 3 can be transferred to the recording of color images with a stereoscopic viewing device is in 10 shown. According to this timing scheme, switching from one to the other stereo channel does not occur after an exposure and readout cycle, as in the timing diagram 8th the case is, but after each lighting cycle. In addition, with a change between two exposure and read-out cycles, a change of the illumination takes place in which the first monochrome frame is recorded. In 10 1, the first exposure and readout cycle is completed after the fourth illumination cycle. The second exposure and readout cycle begins with the fifth illumination cycle. If this were designed according to the first exposure and read-out cycle, the full image would be picked up in the red illumination in the first illumination cycle of the second exposure and read-out cycle. Instead, the second exposure and readout cycle is modified such that the first frame is captured in the green illumination. The second frame is then taken in the subsequent illumination cycle with the blue illumination. In the next illumination cycle, no full image is then recorded at all, while in the last illumination cycle of the second exposure and readout cycle the red frame is then recorded. The third exposure and readout cycle then again corresponds to the first exposure and readout cycle, the fourth exposure and readout cycle again to the second exposure and readout cycle, etc. In other words, in the present embodiment, the first frame is included in all odd-numbered exposure and readout cycles the red illumination, however, in all even-numbered exposure and read-out cycle, the first frame is taken in green illumination. As a result, when changing between an odd-numbered exposure and read-out cycle, four ROI images are successively taken, but when changing from an even-numbered exposure and readout cycle to an odd-numbered exposure and readout cycle, only two monochrome ROI images are taken. The change described when recording the first frame is to ensure that for each stereo channel actually frames are recorded in red, green and blue. In the present embodiment, if all the exposure and read-out cycles were identical, only red and blue frames would always be recorded for the right stereo channel and only green frames for the left stereo channel.

The with the in 10 Real-time frame rates for stereoscopic color frames and stereoscopic ROI images, respectively, realize average frame rates corresponding to the frame rates associated with the in 8th shown time scheme are feasible. However, since there is a more frequent change between the stereo channels, the variations in time between two consecutive colored ROI images are less than in the time scheme according to FIG 8th ,

As it turned out 8th The last colored ROI image of the right observation channel is read out after 68.5 milliseconds in the first exposure and readout cycle. The first color ROI image of the second exposure and readout cycle of the right stereo channel is read out after 155.6 milliseconds. Thus, between the last colored ROI image of the first cycle for the right stereo channel and the first colored ROI image of the second cycle of the right stereo channel, 87.1 milliseconds pass. However, within a right stereo channel exposure and read cycle, the time between two color ROI images is not more than 20.4 milliseconds. The minimum amount of time between two colored ROI images of the right stereo channel within an exposure and readout cycle is only 5.5 milliseconds (between the last and last but one ROI image of the right stereo channel in the first exposure and readout cycle). There are therefore quite large fluctuations in the time duration between two colored ROI images in the time scheme 8th in front.

In the in 10 On the other hand, the time interval between two ROI images is not more than 40.8 milliseconds since the stereo channel is changed after each ROI image. The minimum amount of time between two ROI images is in the in 10 25.9 milliseconds (between the third ROI image and the second ROI image of the right stereo channel). The fluctuations between two colored ROI images are therefore significantly lower than in the 8th illustrated time schedule. However, the time schedule increases according to 10 the maximum time between taking a monochrome frame and synthesizing the color frame to a maximum of 107.4 milliseconds (for the red frame), whereas the maximum time between taking a monochrome frame and synthesizing the color frame in the timing scheme 8th only at 61.11 milliseconds. Depending on the application or the speed of the movement to be expected in the object under observation, it must be weighed whether the time schedule is off 8th or the time schedule 10 should be used.

Also the timetable off 4 in which the frames are captured when the observation object is illuminated with white light and the ROI images with time-sequential colored illumination can be transferred to a stereoscopic optical observation device. An appropriate time scheme is in 11 shown. The image refresh rates that can be achieved for a stereoscopic color image or a stereoscopic ROI image in this time scheme are 22.4 Hz, which corresponds to a time interval between two frames or between two ROI images of 44.4 milliseconds.

In addition to taking color images, the taking of color images and, for example, fluorescence images or images outside the visual spectral range can also be transferred to the recording of stereoscopic images. The possible time schedules are in the 12 to 14 shown. They essentially correspond to the processes from the 8th . 10 and 11 However, the timing is supplemented by a fourth spectral range of the lamp L4, with which, for example, a fluorescence can be excited.

In the in 12 displayed time schedule that the process off 8th Thus, a refresh rate for a full color stereo image of 5.2 Hz can be achieved, a refresh rate for a color stereo ROI image of 26 Hz. This corresponds to a time between two color stereo frames of 192.6 milliseconds or between two colored stereo ROI images of 38.5 milliseconds. However, applying subsampling in case of capturing the frames may increase the refresh rates.

In the case of in 13 displayed time schedules, the expiry of 10 The same average frame rates for the color stereo frames and the color stereo ROI images are the same as for the timing scheme 12 , With reference to the time schedules from the 8th and 10 Explanations regarding the maximum time duration between two color stereoscopic frames and between the recording of a monochrome frame and the reproduction of the colored frame apply correspondingly also to the timing diagrams of FIGS 12 and 13 , In contrast to the timing scheme 10 needs at the in 13 shown time scheme between two consecutive exposure and readout cycles to take place no change in the illumination color for the first frame.

Also in 7 shown variant of a time scheme for recording black and white frames, colored ROI images and fluorescent images can be transferred to the recording of stereoscopic images. A corresponding time schedule transmitted to the recording of stereoscopic images is in 14 shown. After each illumination cycle, there is a change between the stereo channels, so that the refresh rates compared to the in 7 Halve the time scheme shown or double the time between two consecutive stereo images. This applies to both frames and ROI images.

In all the methods described, the positioning of the ROI representing partial surface 21 on the sensor surface 19 For example, done with the help of an eye tracker. By means of the eye tracker, a determination of the viewing direction of the observer and a tracking of the sub-area 21 on the sensor surface based on the determined viewing direction. Determining the viewing direction and updating the position of the subarea 21 This is done at a rate of less than 20 Hz and preferably at a rate of less than 10 Hz. It is particularly advantageous when determining the viewing direction and updating the position of the partial area 21 on the sensor surface 19 at a rate in the range between 5 and 10 Hz. At a rate in this range, on the one hand, the tracking of the partial surface on the basis of saccades can be avoided. On the other hand, a rate of 5 to 10 Hz is high enough to cause no disturbing delay between a change in the viewing direction and the tracking of the ROI.

In the context of the method according to the invention, moreover, movements of the optical observation system can be compensated.

For example, if a movement is perpendicular to the optical axis (for example due to Vibrations) so it is possible to detect them for example by means of acceleration sensors or a registration of temporally successive images and to perform a shift of the ROI on the image sensor, which compensates for the movement of the optical observation system.

In addition, the inventive method can also be used advantageously in surveillance cameras. When monitoring by means of a camera (for example in public buildings, for border surveillance or in military intelligence), the problem frequently arises that only a limited bandwidth is available for transmitting the acquired image information to a monitoring station. The principle according to the invention can also be used for such applications, but the aim is not to achieve a sufficiently high refresh rate but to reduce the transmission rate from the camera used for monitoring to the monitoring point. The method according to the invention makes it possible to transmit frames at a low frame rate and ROI images at a normal frame rate. Since a lower bandwidth is required for an ROI image, overall the bandwidth required to transmit the surveillance images can be reduced. If, moreover, an evaluation unit is present at the location of the camera, which can determine the ROI, for example based on movements in the image, such a surveillance camera can act largely autonomously.

The light sources used in the optical observation device according to the invention may in particular be LEDs which emit in the red, green or blue wavelength range. But also other wavelength ranges that span an RGB color space or CMY color space can be used. In addition, it is possible to provide additional light sources that emit, for example, in the infrared or ultraviolet spectral range.

For optical observation devices intended for use on the eye, it is advantageous to keep the burden of the patient on light low so as not to damage the visual cells. The use of infrared lighting makes it possible to reduce the load since it has a spectral dependence. For example, the IR image can be used to guide the treating surgeon and the color image to detect different types of tissue in the eye.

UV illumination can be used to stimulate fluorescence in the examination subject. In this case, either an autofluorescence of the tissue or fluorescence of an injected dye can be excited.

In the context of the method according to the invention, moreover, the illumination times with the individual light sources can be controlled so that the individual images are optimally controlled, that is to say have an optimum signal-to-noise ratio. In addition, it is possible to choose the illumination times with regard to the lowest possible tissue load. Furthermore, the provision of separate light sources with different spectral ranges makes it possible to optimize the white point of the illumination and at the same time to optimize the amount of light to be applied.

If the image acquisition is also extended by another image in which all the lights are turned off, this "dark image" can be used to subtract the ambient light from the "color images", to influence the color representation both in the visual and in the external area to diminish by ambient light. It is also possible to use more than three colors in the visual area to improve the differentiation of colors, that is, to better recognize color differences of objects. The evaluation of color differences can be automated. For example, it is also possible to show only the color differences for illustration. When using four-color displays, it is also possible to dispense with the use of four spectrally matched light sources on a billing of color images. Rather, the recorded and reproduced color space are adapted to each other.

Although the recording of three-dimensional images has been described using the example of stereoscopic images in the exemplary embodiments, the method described can also be used in other types of recording of three-dimensional images, for example when taking pictures with plenoptic observation optics or observation optics, such as in DE 10 2011 010 262 is described.

With the method according to the invention and the optical observation device according to the invention, the image repetition rate in the image area of interest (ROI) can be significantly increased compared with the prior art, whereas the image refresh rate for frames is only slightly reduced. In this way it is possible, even with slow consumer cameras, which can realize only a refresh rate of 60 Hz, to realize a color image with time-sequential color illumination, wherein in the ROI a refresh rate of more than 50 Hz can be achieved. Even for a stereo recording with only one image sensor can achieve refresh rates that are above the cinema standard of 24 Hz. Furthermore, the method according to the invention and the optical observation device according to the invention have the advantage that the perception of color fringes by moving objects is reduced. In addition, the inventive method and the optical observation device according to the invention offer the advantage that the delay between the image acquisition and the image representation for the ROI is lower than in the prior art.

LIST OF REFERENCE NUMBERS

1

observation optics

3

camera

5

electronic image sensor

7

computer unit

L1 ... LN

light sources

9

display unit

11

Combining device

13

illumination optics

15

control unit

17

observation object

19

sensor surface

21

ROI (region of intrest)

23

spectral

Claims (28)

Method for taking pictures of an observation object ( 17 ) with an electronic image sensor ( 5 ), in which: - a full image on an exposed sensor surface ( 19 ) of the image sensor ( 5 ), wherein an image region of the frame of interest on a partial surface ( 21 ) of the image sensor ( 5 ), and - the images are taken by the image sensor ( 5 ) is exposed for the respective image and read out after each exposure, wherein an exposure of the image sensor ( 5 ) and a readout of the entire image area exposed to a full image ( 19 ) at least once exposing the image sensor ( 5 ) and reading out only the partial area ( 21 ) of the image sensor ( 5 ) with the image region of interest before re-exposure of the image sensor ( 5 ) and a reading of the entire exposed sensor surface ( 19 ), characterized in that as an image sensor ( 5 ) a monochrome image sensor is used, for illuminating a lighting of the observation object ( 17 ) takes place in lighting cycles in which the observation object ( 17 ) is time-sequentially illuminated with N different wavelength distributions (L1-LN), and the readout of the image sensor ( 5 ) in exposure and read-out cycles, wherein the illumination cycles are synchronized with the exposure and read-out cycles such that, per exposure and read-out cycle, a) exposure of the image sensor (FIG. 5 when the observation object is illuminated with a first of the N wavelength distributions of a lighting cycle and then reading out the entire exposed sensor area (FIG. 19 ), b) at least one time then an exposure and read-out sequence of the image sensor takes place, in which the observation object ( 17 ) is successively illuminated with different wavelength distributions of a lighting cycle, each time the object to be illuminated is illuminated ( 17 ) with one of the wavelength distributions an exposure takes place and after each exposure only one readout of the partial area ( 21 ) of the image sensor ( 5 ) with the image region of interest, and c) steps a) and b) are performed N times, wherein in step a) a different wavelength distribution is used each time. Method for taking pictures of an observation object ( 17 ) with an electronic image sensor ( 5 ), in which: - a full image on an exposed sensor surface ( 19 ) of the image sensor ( 5 ), wherein an image region of the frame of interest on a partial surface ( 21 ) of the image sensor ( 5 ), and - the images are taken by the image sensor ( 5 ) is exposed for the respective image and read out after each exposure, wherein an exposure of the image sensor ( 5 ) and a readout of the entire image area exposed to a full image ( 19 ) at least once exposing the image sensor ( 5 ) and reading out only the partial area ( 21 ) of the image sensor ( 5 ) with the image region of interest before re-exposure of the image sensor ( 5 ) and a reading of the entire exposed sensor surface ( 19 ), characterized in that as an image sensor ( 5 ) a monochrome image sensor is used, for illuminating a lighting of the observation object ( 17 ) takes place in lighting cycles in which the observation object ( 17 ) is time-sequentially illuminated with white light and with N different wavelength distributions (L1-LN), and the readout of the image sensor ( 5 ) in exposure and read-out cycles, wherein the illumination cycles and the exposure and read-out cycles are synchronized in such a way that, per exposure and read-out cycle, a) exposure of the image sensor (FIG. 5 ) with the white light of a lighting cycle and then reading the entire sensor surface ( 19 ), b) at least one time then an exposure and read-out sequence of the image sensor ( 5 ), in which the observation object ( 17 ) is successively illuminated with different wavelength distributions of a lighting cycle, each time the object to be illuminated is illuminated ( 17 ) with one of the wavelength distributions an exposure takes place and after each exposure only one readout of the partial area ( 21 ) of the image sensor ( 5 ) takes place with the image region of interest. Method according to Claim 1 or Claim 2, in which an exposure of the image sensor ( 5 ) for the i-th image while reading the i-1th image. Method according to Claim 3, in which the intensity of the illumination for the i-th illumination relative to the illumination for the i-th first exposure is reduced if, for reading out the i-1-th image, the entire exposed sensor surface ( 19 ) is read out. Method according to Claim 3, in which the exposure time for the i-th exposure is reduced relative to the exposure time for the i-th first exposure if, for reading out the i-1-th image, the entire exposed sensor area ( 19 ) is read out. Method according to one of Claims 1 to 5, in which, when the entire sensor surface is read ( 19 ) only every jth sensor line and / or every k-th sensor column with j> 1, k> 1 is read out or are combined in the pixel of the image sensor into pixel blocks and the pixels of a pixel block are read together in each case. Method according to one of the preceding claims, in which the N different wavelength distributions (L1-LN) comprise at least the basic colors of an RGB color space or of a CMY color space. Method according to Claim 7, in which the N different wavelength distributions (L1-LN) comprise at least one further wavelength distribution in addition to the basic colors of an RGB color space or of a CMY color space. Method according to Claim 7 or Claim 8, in which a color image is generated from the primary colors of an RGB color space or of a CMY color space. Method according to one of the preceding claims, in which, in addition to the exposures in the N wavelength distributions, at least one exposure takes place without illumination. Method according to one of the preceding claims, in which the position of the image region of interest in the frame is determined by determining the viewing direction of an observer viewing the frame. Method according to Claim 11, in which the viewing direction is determined repeatedly and the position of the subarea representing the region of interest ( 21 ) of the image sensor ( 5 ) on the sensor surface ( 19 ) is updated on the basis of the currently determined viewing direction. The method of claim 11, wherein determining the line of sight is at a rate of less than 20 Hz. Method according to one of the preceding claims, in which a moving object is recognized in full image by means of an image recognition software and the partial area representing the region of interest ( 21 ) of the image sensor ( 5 ) so on the exposed sensor surface ( 19 ) and the image of the moving object on the exposed sensor surface ( 19 ), that the image of the moving object in the subarea ( 21 ) remains. Method according to one of the preceding claims, in which as images of the observation object ( 17 ) three-dimensional images, which are composed of at least two perspective images taken at different viewing angles, wherein all perspective images are time-sequential with an electronic image sensor ( 5 ) and the method of any one of the preceding claims is applied to taking the perspective images. A method according to claim 15 and claim 1, wherein, for taking a three-dimensional image, steps a), b) and c) of a perspective image exposure and readout cycle are first performed before the steps a), b) and c) for the next perspective image to be executed. A method according to claim 15 and claim 1, wherein for capturing a three-dimensional image The exposure and readout cycle is carried out successively as the three-dimensional image has perspective images, A change takes place between the taking of a perspective image and the taking of the next perspective image whenever a lighting cycle has ended. A method according to claim 15 and claim 2, wherein for taking a three-dimensional image, the exposure and readout cycle is performed as many times as the three-dimensional image comprises perspective images, and there is a change between taking a perspective image and taking the next perspective image after exposing and reading the image sensor ( 5 ) with the white light and all N wavelength distributions of the illumination cycle. Computer program with instructions for carrying out the method according to one of the preceding claims. Optical observation device for taking pictures of an observation object with - at least one electronic image sensor ( 5 ), - an observation optics ( 1 ), which is an object field of the observation object ( 17 ) in full screen on the exposed sensor surface ( 19 ) of the image sensor ( 5 ), - a selection device for selecting a partial area ( 21 ) of the image sensor ( 5 ), which corresponds to an image region of interest of the frame, and a sensor control unit adapted to control the exposure and readout of the image sensor according to the method of any of claims 1 to 18, characterized in that it further comprises: - a lighting device with in pulsed illumination light sources (L1-LN) emitting pulsed light which can be operated in different wavelength distributions, 13 ) for supplying the light of the illumination light sources (L1-LN) to the observation object ( 17 ) and - a lighting control unit which acts on the illumination light sources (L1-LN) in such a way that they are pulsed alternately, - one with the sensor control unit and the lighting control unit ( 1 ) connected synchronization unit ( 15 ) for synchronizing the pulsed operation of the illumination light sources (L1-LN) with the exposure and readout of the image sensor ( 5 ) according to the method of any one of claims 1 to 18. An optical observation apparatus according to claim 20, wherein there are at least three illumination light sources (L1-LN) each of which has a wavelength distribution having a maximum in a basic color of an RGB color space or a CMY color space. An optical observation apparatus according to claim 20 or claim 21, wherein there is at least one illumination light source (L1-LN) whose wavelength distribution has at least a maximum outside the range of 430-750 nm. An optical observation apparatus according to claim 21 and claim 22, further comprising a display unit (16). 7 ), which - from those exposures of the image sensor ( 5 ), which are illuminated when the observation object ( 17 ) with an illumination light source (L1-LN) with a maximum in a base color of an RGB color space or CMY color space, a color image is synthesized and - from that exposure of the image sensor ( 5 ), which are illuminated when the observation object ( 17 ) with the illumination light source (L1-LN), in whose wavelength distribution at least a maximum outside the range of 430-750 nm is present, an image independent of the color image is generated. An optical observation apparatus according to claim 23, further comprising an overlay unit overlaying the image independent of the color image onto the color image. Optical observation device according to one of claims 22 to 24, in which - the at least one further illumination light source (L1-LN), whose wavelength distribution is at least a maximum outside the range of 430-750 nm, has a wavelength maximum which is a fluorescence in the observed object ( 17 ), and - in the observation beam path a filter ( 23 ) is present or can be introduced, blocking the wavelengths of this wavelength maximum. Optical observation device according to one of Claims 20 to 25, in which the illumination optics ( 13 ) an assembly device ( 11 ) for merging the illumination beam paths of the individual illumination light sources (L1-LN). Optical observation device according to one of Claims 20 to 26, in which the observation optics ( 1 ) is an observation optics which allows the taking of perspective images from at least two different angles of view. An optical observation device according to claim 27, in which the observation optics are designed in such a way that the beams of the perspective images are time-sequentially directed to the same image sensor ( 5 ).

Images