EP4327548A1

EP4327548A1 - Imaging device, and corresponding method and microscope

Info

Publication number: EP4327548A1
Application number: EP22722747.7A
Authority: EP
Inventors: Gao Yang
Original assignee: Leica Microsystems CMS GmbH; Leica Instruments Singapore Pte Ltd
Current assignee: Leica Microsystems CMS GmbH; Leica Instruments Singapore Pte Ltd
Priority date: 2021-04-20
Filing date: 2022-04-13
Publication date: 2024-02-28
Also published as: WO2022223400A1

Abstract

Examples relate to an optical imaging device, to a corresponding method for an optical imaging device, and to a microscope, such as a surgical microscope, comprising such an optical imaging device. The optical imaging device comprises an optical imaging sensor. The optical imaging sensor is configured to generate frames of imaging sensor data, and to provide a latest generated frame of imaging sensor data to a frame buffer and to a switching circuitry of the optical imaging device. The optical imaging device further comprises the frame buffer that is configured to store the latest generated frame. The optical imaging device further comprises the switching circuitry. The switching circuitry is configured to output, via an output circuitry of the optical imaging device during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer. The switching circuitry is configured to output, via the output circuitry of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer.

Description

Imaging Device, and Corresponding Method and Microscope

Technical field

Examples relate to an optical imaging device, to a corresponding method for an optical imag ing device, to a microscope, such as a surgical microscope, comprising such an optical imag ing device, and more specifically, to a concept for improving a latency between generation and output of a frame in an optical imaging device.

Background

A major trend in surgical microscopy is the use of head -up displays. When a head-up display is used, instead of using the oculars of the microscope, the surgeon or surgeons observe the surgical site via a display that is arranged at eye-level of the surgeons, enabling them to col laborate in the visual analysis of the surgical site. To show the surgical site on the head-up display, the surgical site is observed via one (or more) video cameras (i.e., optical imaging devices) of the microscope. The video camera(s) generates a sequence of frames, which are subsequently displayed on the head-up display (and other displays, such as digital oculars of the microscope). Care may be taken to reduce the latency between the generation of the frames and the subsequent display of the frames on the display.

The amount of light gathered by the video camera(s) may determine the quality of the image frames (at a desired brightness of the image frame) - the more light can be gathered, the lower the amplification gain of the optical imaging sensor can be, and the lower the gain of the optical imaging sensor, the less noise the image frames contain. For example, the duration of the exposure of the sensor to the light may be increased to also increase the amount of light that is gathered. However, a video camera’s exposure duration for each frame is limited by the frame duration, which corresponds to the framerate selected for capturing the sequence of frames.

A video camera may operate at a fixed output framerate while having a different and variable framerate at the optical imaging sensor. Alternatively, the output framerate may follow the optical imaging sensor’s framerate. If the video camera operates at a fixed framerate that is different from the optical imaging sensor’s framerate, a frame buffer is used to resolve the asynchrony. However, such a frame buffer usually introduces significant delays to the video signals.

There may be a desire for an improved concept for an optical imaging device, which supports an image generation framerate that can be different from an output framerate, and that has a reduced latency.

Summary

This desire is addressed by the subject-matter of the independent claims.

Various examples of the present disclosure are based on the finding, that, while a frame buffer may be used to support an image generation framerate that is different from an image output framerate, this frame buffer may be bypassed when a latest image frame is available from the optical imaging sensor. When the optical imaging sensor outputs a latest generated frame, this frame is output to both a frame buffer, which is used to store the latest generated frame, and to a switching circuitry, which may output the latest frame directly, introducing less latency (e.g., only a few clock cycles) than the frame buffer, thereby reducing the latency relative to an approach where all of the frames are output via a frame buffer.

Various examples of the present disclosure relate to an optical imaging device. The optical imaging device comprises an optical imaging sensor. The optical imaging sensor is configured to generate frames of imaging sensor data, and to provide a latest generated frame of imaging sensor data to a frame buffer and to a switching circuitry of the optical imaging device. The optical imaging device further comprises the frame buffer that is configured to store the latest generated frame. The optical imaging device further comprises the switching circuitry. The switching circuitry is configured to output, via an output circuitry of the optical imaging de vice during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer. The switching circuitry is configured to output, via the output circuitry of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer. By bypassing the frame buffer for frames that can be output directly, the latency can be reduced relative to an approach where all of the frames are delayed by a frame buffer.

In various examples, the switching circuitry is configured to give preference to the frame provided by the optical imaging sensor bypassing the frame buffer over the stored latest gen erated frame stored in the frame buffer. In other words, when a latest generated frame is di rectly provided by the optical imaging sensor, the frame buffer may be bypassed, and the additional latency of the frame buffer may be avoided.

In general, the frame buffer may be configured to output the stored latest generated frame until the stored latest generated frame is replaced by the latest generated frame generated by the optical imaging sensor. Thus, the frame stored in the frame-buffer may be used to “fill” the frames of the output while the optical imaging sensor is in the process of exposing for a latest frame.

As alluded earlier, the optical imaging sensor may be configured to generate the frames of imaging sensor data at an image generation framerate. The switching circuitry may be con figured to output the respective latest generated frames at an image output framerate. For example, the image generation framerate and the image output framerate may be set inde pendent from another. In particular, in various examples, the image output framerate may be fixed, and the image generation framerate may be variable. In other words, the image output framerate may be fixed to a pre-defined value. The optical imaging device may further com prise input circuitry for obtaining an image generation framerate control signal. The optical imaging sensor may be configured to control the image generation framerate based on the image generation framerate control signal. For example, the image generation framerate may be controlled based on the amount of available light.

In various examples, the optical imaging device is suitable for operating with an image gen eration framerate that is an integer fraction of the image output framerate. For example, the image output framerate may be set to 60 Hz, and the image generation framerate may be set to 30 Hz, 20 Hz or 15 Hz, or the image output framerate may be set to 120 Hz, which further enables an image generation framerate of 60 Hz or 24 Hz. For example, the optical imaging device may be further suitable for operating with an image generation framerate that corresponds to the image output framerate, e.g., at 60 Hz when the image output framerate is set to 60 Hz, or 120 Hz if the image output framerate is set to 120 Hz.

The switching circuitry may be configured to, if the optical imaging device operates with an image generation framerate that corresponds to the image output framerate, output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer without switching to outputting the stored latest generated frame. In other words, if the image gener ation framerate corresponds to the image output framerate, the frame buffer may be bypassed entirely.

To ensure a smooth switch, the timing of the framerates may be time-synchronized, so that the latest generated frame that is output by the optical imaging sensor can be output directly by the switching circuitry, without requiring additional buffering (apart from line buffering that is inherent to the circuitry). For example, the image output framerate may be time-syn chronized with the image generation framerate, such that a timing of frames provided by the optical imaging sensor bypassing the frame buffer is suitable for a timing of the switching circuitry.

Accordingly, the stored latest generated frame may be provided to the switching circuitry in a manner that enables the switching circuitry to output the stored latest generated frame re quiring additional buffering (apart from line buffering that is inherent to the circuitry). The optical imaging device may comprise a timing generation circuitry that is configured to pro vide the stored latest generated frame to the switching circuitry at the image output framerate, such that a timing of the frames provided by the timing generator circuitry is suitable for the timing of the switching circuitry.

The switching circuitry may switch automatically between outputting the latest generated frame that is provided bypassing the frame buffer and the stored latest generated frame, e.g., by detecting whether the latest generated frame provided by the optical imaging sensor by passing the frame buffer is provided in time for the next frame of the image output framerate. For example, the switching circuitry may be configured to detect, at the image output fram erate, whether the latest generated frame is provided by the optical imaging sensor bypassing the frame buffer, and to output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer upon detection. Optical imaging sensors are often read-out in small increments. For example, the optical im aging sensor may comprise a two-dimensional array of pixels, comprising a plurality of lines of pixels and a plurality of columns of pixels. The optical imaging sensor is often read-out line-by-line (or blocks of lines at a time), e.g., as the bandwidth of the output of the optical imaging sensor is limited and as crucial circuitry can be re-used by reading out the optical imaging sensor in increments. To avoid introducing additional delays, the frames may also be transmitted as subsequent fractions of the frame, e.g., line-by-line (or blocks of lines at a time), without buffering the entire frame at the switching circuitry. In other words, the switch ing circuitry may be configured to receive the respective latest generated frames successively in data units representing a fraction of the frame, and to output the data units representing a fraction of the latest generated frame without buffering the entire frame.

In some examples, the logic components of the optical imaging device may be implemented by a Digital Signal Processor (DSP). Alternatively, e.g., to allow for limited-run series and/or to for additional functionality being added over time, the logic components may be imple mented using a re-programmable hardware device. For example, the switching circuitry, a logic portion of the frame buffer and an interconnect circuitry between the optical imaging sensor, the frame buffer, the switching circuitry and the output circuitry may be implemented on a field-programmable gate array.

Various examples of the present disclosure relate to a microscope, such as a surgical micro scope, comprising at least one optical imaging device as introduced above.

Various examples of the present disclosure relate to a corresponding method for an optical imaging device. The method comprises providing, by an optical imaging sensor, a latest gen erated frame of imaging sensor data. The method comprises storing the latest generated frame in a frame buffer. The method comprises outputting, during a first time period, the latest gen erated frame provided by the optical imaging sensor bypassing the frame buffer. The method comprises outputting, during a second time period, the stored latest generated frame stored in the frame buffer.

Short description of the Figures Some examples of apparatuses and/or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which:

Figs la and lb show schematic diagrams of examples of an optical imaging device;

Figs. 2a and 2b show schematic diagrams of a timing of frames according to an example;

Fig. 3 shows a schematic diagram of an example of a surgical microscope system with a microscope comprising an optical imaging system according to an example; and

Fig. 4 shows a flow chart of an example of a method for an optical imaging device.

Detailed Description

Various examples will now be described more fully with reference to the accompanying draw ings in which some examples are illustrated. In the figures, the thicknesses of lines, layers and/or regions may be exaggerated for clarity.

Various examples of the present disclosure relate to means for variable optical imaging sensor framerate (i.e., a variable framerate of the optical imaging sensor) in a low-latency video sys tem. For example, the proposed concept may be used in head-up surgery.

In general, a video camera’s exposure duration for each frame is limited by the frame duration, which is based on the selected framerate. A video camera may operate at a fixed output fram erate while having a different and variable framerate at the optical imaging sensor. Alterna tively, the output framerate may need to follow the optical imaging sensor’s framerate. If the video camera operates at a fixed framerate that is different from the optical imaging sensor’s framerate, a frame buffer may be introduced between the optical imaging sensor and the out put of the video camera, to resolve the asynchrony, thus introducing significant delays to the video signals. On the other hand, if the video camera’s output follows the optical imaging sensor’s framerate, the sink device (e.g., a video capture device or display) may lose video synchronisation every time the framerate changes, resulting in a blank screen or flickering.

Various examples of the present disclosure address the above-referenced shortcomings. Figs la and lb show schematic diagrams of examples of an optical imaging device 100. The optical imaging device 100 comprises an optical imaging sensor 110 that is configured to generate frames of imaging sensor data, and to provide a latest generated frame of imaging sensor data to a frame buffer 120 and to a switching circuitry 130 of the optical imaging device. The optical imaging device 100 further comprises the frame buffer 120, which is con figured to store the latest generated frame. The optical imaging device 100 further comprises the switching circuitry 130. The switching circuitry is configured to output, via an output circuitry 140 of the optical imaging device during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer, and to output, via the output circuitry 140 of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer. Accordingly, the optical imaging device further comprises the output circuitry 140.

The optical imaging device 100 shown in Figs la and lb includes a frame buffer - however, this frame buffer 120 is operated in synchronicity with the optical imaging device. Using the frame buffer, the video output of the optical imaging device may be locked at a certain fram- erate and the optical imaging sensor can operate at an integer fraction of this framerate. When the optical imaging sensor and the video output operate at the same framerate, this frame buffer may remain unused. When the optical imaging sensor is operating at an integer fraction 1/N (e.g., 1/2, 1/3) of the video output framerate, the frame buffer may be used to record and repeat the latest frame from the optical imaging sensor. For example, a video timing generator 150, as shown in Fig. lb, may be used so the read out from the frame buffer closely emulates the timing pattern of the optical imaging sensor. With this design, a blank screen or flickering may be avoided since the output framerate is locked. No or only a few clock cycles of addi tional latency may be introduced because the latest generated frames are output without going through the frame buffer.

In Figs la and lb, arrows are shown - arrows with a dotted line pattern, which show the path of the latest generated frame bypassing the frame buffer, and arrows with a dashed line pat tern, which show the path of the latest generated frame through the frame buffer. When the optical imaging sensor and video output operate at the same framerate, or when they are dif ferent, but the optical imaging sensor is producing data, the generated frames of the video signal may follow the arrows with the dotted line pattern: the latest generated frame is transmitted from the optical imaging sensor 110 through the switching circuitry 130 to the video output 140. The data is also written into the frame buffer 120. When the optical imaging sensor is not producing data, the generated frames of the video signal may follow the arrows with the dashed line pattern: the stored latest frames are obtained from the frame buffer 120, optionally throttled by the video timing generator 150, pass through the switching circuitry 130, and are provided to the output 140.

The proposed concept may be used with many types of optical imaging devices 100. In par ticular, however, the optical imaging device 100 may be a video camera or a video sensor arrangement, i.e., a device being sued to provide (low-latency) video data. Consequently, the optical imaging device 100 may be configured to provide video data comprising the generated frames via the output circuitry. For example, the optical imaging device 100 may be an optical imaging device or a video camera for use in a digital microscope. In other words, the optical imaging device 100 may be used to generate video data based on light reflected or emitted by a sample being observed through the microscope. Accordingly, a microscope may comprise at least one optical imaging device 100, e.g., two optical imaging devices, in case of a stere oscopic microscope. An example of a surgical microscope system comprising a microscope with at least one optical imaging system is shown in Fig. 3.

The optical imaging sensor 110 is used to generate the frames of the imaging sensor data. For example, the optical imaging sensor may comprise or be an APS (Active Pixel Sensor) - or a CCD (Charge-Coupled-Device)-based imaging sensor. For example, in APS-based imaging sensors, light is recorded at each pixel using a photo-detector and an active amplifier of the pixel. APS-based imaging sensors are often based on CMOS (Complementary Metal-Oxide- Semiconductor) or S-CMOS (Scientific CMOS) technology. In CCD-based imaging sensors, incoming photons are converted into electron charges at a semiconductor-oxide interface, which are subsequently moved between capacitive bins in the imaging sensors by a control circuitry of the imaging sensors to perform the imaging. The optical imaging sensor thus measures the incoming light, and generates the frames of imaging sensor data based on the measured incoming light.

The optical imaging sensor further provides the latest generated frame of imaging sensor data to the frame buffer and to the switching circuitry of the optical imaging device. In various examples, the optical imaging sensor is connected to the frame buffer and to the switching circuitry via interconnect circuitry. The optical imaging sensor may transmit the latest gener ated frame to the frame buffer and to the switching circuitry, e.g., by generating a correspond ing signal and outputting the signal to the interconnect circuitry. At the frame buffer and the switching circuitry, the respective signals may be received, e.g., via a (small) input buffer.

In various examples, optical imaging sensors, and in particular optical imaging sensors being used to generate video data, often apply a so-called “rolling shutter”. In general, optical im aging sensors may comprise a two-dimensional grid of pixels, comprising a plurality of lines of pixels and a plurality of columns of pixels. The content of the two-dimensional grid of pixels is read out to generate a frame of imaging sensor data. In many cases, only a subset of pixels of the two-dimensional grid of pixels is read out at once, e.g., line-by-line or block-by block. To ensure that all of the pixels are exposed to the same (or at least similar) amounts of light, the duration of exposure to light of the pixels may be (substantially) the same for all of the pixels. As some pixels are read out earlier than others, these pixels are also exposed to the light earlier, to maintain the same exposure duration. The later a pixel is read out, the later the exposure of the pixel begins. To further reduce the latency, the optical imaging sensors may start providing the frame while the frame is still being generated - as soon as the first line or block is generated, the frame may be provided to the frame buffer and to the switching cir cuitry. In more general terms, frames of imaging sensor data may comprise a plurality of data units each representing a fraction of the frame. For example, each data unit may represent a line of pixels or a block of pixels. The optical imaging sensor may be configured to provide the frame successively in data units representing a fraction of the frame, e.g., by outputting the plurality of data units successively until the entire frame has been provided.

The frames of imaging sensor data are provided to the frame buffer and to the switching cir cuitry (thereby bypassing the frame buffer). The frame buffer is configured to store the latest frame provided by the optical imaging sensor. For example, the frame buffer may comprise a (volatile) memory, such as a dynamic random-access memory, for storing the latest generated frame. For example, the frame buffer may further comprise logic circuitry configured to re ceive the latest generated frame from the optical imaging sensor, e.g., via the interconnect circuitry, to store the received frame (e.g., data unit by data unit) in the memory, and to pro vide the stored frame to the switching circuitry, e.g., via another portion of the interconnect circuitry. In various examples, the frame buffer may keep outputting the same stored frame until the stored frame is replaced by a subsequent frame provided by the optical imaging sensor. In other words, the frame buffer may be configured to output the stored latest gener ated frame until the stored latest generated frame is replaced by the latest generated frame generated by the optical imaging sensor. Again, the stored frame may be replaced data unit by data unit as the latest generated frame is provided by the optical imaging sensor.

The switching circuitry 130 is used to switch between outputting, with minimal latency, the frame that is provided directly by the optical imaging sensor (thereby bypassing the frame buffer), and outputting the frame that is stored in the frame buffer. In general, to reduce or minimize the latency, the switching circuitry outputs the latest frame just as it is provided by the optical imaging sensor, bypassing the frame buffer. Only when no latest frame is available from the optical imaging sensor, e.g., as a framerate at which frames are output by the switch ing circuitry is higher than a framerate at which frames are generated by the optical imaging sensor, the stored latest generated frame might by output instead. Therefore, the switching circuitry may be configured to give preference to the frame provided by the optical imaging sensor bypassing the frame buffer over the stored latest generated frame stored in the frame buffer.

As is evident, the ratio between the framerate at which the images are output by the switching circuitry and the framerate, at which the optical imaging sensor generates the frames of im aging sensor data, affects the operation of the optical imaging device. In formal terms, the optical imaging sensor may be configured to generate the frames of imaging sensor data at an image generation framerate, and the switching circuitry may be configured to output the re spective latest generated frames at an image output framerate. In general, a framerate is a rate, at which frames are generated or provided. Framerates are usually defined in terms of frames per unit of time, e.g., frames per second. For example, the optical imaging device may have an image output framerate of 24 frames per second, of 30 frames per second, of 60 frames per second, of 90 frames per second, of 120 frames per second etc. Similarly, the optical imaging sensor may have an image generation framerate of 12 frames per second, of 15 frames per second, of 24 frames per second, of 30 frames per second, of 60 frames per second, of 90 frames per second, of 120 frames per second as well.

In various examples of the present disclosure, the image output framerate may be fixed to a pre-defined value (e.g., one of 24/30/60/90/120 frames per second), while the image genera tion framerate may be variable. In particular, the image generation framerate may be kept variable as the framerate also influences the amount of light the optical imaging sensor is exposed to while generating the respective frame. The higher the framerate, the shorter the optical imaging sensor is exposed to the light, the less light is being used to expose the optical imaging sensor. While the amplification gain of the sensor can be changed to compensate for the shorter exposure time, higher gains tend to lead to higher levels of noise, as the sensor is not only more sensitive to light, but also more sensitive to noise. Therefore, if little light is available, the image generation framerate may be reduced to allow for longer exposure times. For example, the image generation framerate may be set by an external entity, e.g., a control system of the microscope, to adapt the exposure time of the optical imaging sensor. Accord ingly, the optical imaging device may comprise input circuitry for obtaining an image gener ation framerate control signal, e.g., from a control system of a microscope comprising the optical imaging device. The optical imaging sensor may be configured to control (e.g., set) the image generation framerate based on the image generation framerate control signal.

As is evident, two cases may be distinguished - a first case, where the image generation fram erate is lower than the image output framerate, and a second case, where the image generation framerate is the same as the image output framerate. Therefore, the optical imaging device may support two modes - a first mode, where the optical imaging device operates with an image generation framerate that is lower than the image output framerate, and a second case, where the optical imaging device operates with an image generation framerate that is the same as the image output framerate. However, to output the latest generated frame as soon as it is generated, i.e., bypassing the frame buffer, the choice of image generation framerate and im age output framerate may be limited. In particular, the image generation framerate may be set at an integer fraction of the image output framerate. In other words, the optical imaging device may be suitable for operating with an image generation framerate that is an integer fraction of the image output framerate, and the optical imaging device may be further suitable for operating with an image generation framerate that corresponds to the image output framerate. For example, the image output framerate may correspond to the image generation framerate, may be twice the image generation framerate, three times the image generation framerate, four times the image generation framerate or five times the image generation framerate. In case the image generation framerate is a fraction of the image output framerate, the latest generated frame may (always) be output bypassing the framerate, and the subsequent frame (to fill up the integer fraction), e.g., one subsequent frame if the image output framerate is twice the image generation framerate, two subsequent frames if the image output framerate is three times the image generation framerate etc., may be filled with the stored latest generated frame. Consequently, the proposed concept may be used to “fill” frames that are not provided in time by the optical imaging sensor, as the image generation framerate is lower than the image output framerate.

Consequently, in some cases, when the image generation framerate corresponds to the image output framerate, the frame buffer might remain unused, or frames that are stored in the frame buffer might not be output by the switching circuitry. In particular, the switching circuitry may be configured to, if the optical imaging device operates with an image generation fram erate that corresponds to the image output framerate, output the latest generated frame pro vided by the optical imaging sensor bypassing the frame buffer without switching to output ting the stored latest generated frame.

The difference between the two modes is illustrated in Figs. 2a and 2b. Figs. 2a and 2b show schematic diagrams of a timing of frames (i.e., video frame timing) according to an example. In Fig. 2a, the framerate of the optical imaging sensor (i.e., the image generation framerate) is the same as the image output framerate. As can be seen in Fig. 2a, a first frame 201 of the optical imaging sensor is provided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 211 is (directly) provided at the video output, i.e., by the switching circuitry via the output circuitry. This is repeated for subsequent frames 202- 204 of the optical imaging sensor and corresponding video output frames 212 to 214. As is evident from Fig. 2a, each of the frames being output is a frame that is provided directly by the optical imaging sensor, bypassing the frame buffer.

In Fig. 2b, the image generation framerate is half the image output framerate. In Fig. 2b, a first frame 221 of the optical imaging is provided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 231 is (directly) provided at the video output, i.e., by the switching circuitry via the output circuitry. Next, a stored version 222 of the first frame, as stored in the frame buffer, is provided by the frame buffer and a correspond ing video output frame 232 is provided at the video output. Next, a second frame 223 is pro vided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 233 is (directly) provided at the video output, followed by a stored version 224 of the second frame and a corresponding video output frame 234. In the example of Fig. 2b, the switching circuitry switches between outputting, during the first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer and outputting, during the second time period, the stored latest generated frame stored in the frame buffer. As the image generation framerate is half the image output fram- erate, the first time period has the same duration as the second time period, as both alternatives apply half of the total time. If the image generation framerate is a third of the image output framerate, the first time period may have half the duration of the second time period, as the first alternative (bypassing the frame buffer) takes up a third of the total time, and the second alternative (frame stored in the frame buffer) takes up the remaining two thirds of the total time. If, however, the image generation framerate corresponds to the image output framerate, all of the frames that are output by the switching circuitry may bypass the frame buffer, and thus the first time period may take up the entire time, with no second time period being re quired.

As is further evident from Figs. 2a and 2b, the switching circuitry may begin outputting the frames while they are still being provided to the switching circuitry. For example, as alluded earlier, the switching circuitry may be configured to receive the respective latest generated frames successively in data units representing a fraction of the frame, and to output the data units representing a fraction of the latest generated frame without buffering the entire frame. In other words, the switching circuitry may be configured to begin outputting the respective frame before having received the entire frame. However, some buffering may occur, as the switching circuitry may require one or more clock cycles to output the received frames.

In Figs. 2a and 2b, the frames being output by the switching circuitry are output according to the image output framerate. To avoid buffering at the switching circuitry, the respective frames may be received at the same framerate by the switching circuitry, either (directly) from the optical imaging sensor bypassing the framerate, or from the frame-buffer. Consequently, the switching circuitry may be configured to receive at least one of a) the frame provided by the optical imaging sensor bypassing the frame buffer and b) the frame provided by the frame buffer at the image output framerate. In other words, at a time slot of each frame of the image output framerate, the switching circuitry may be configured to receive at least one of a) the frame provided by the optical imaging sensor bypassing the frame buffer and b) the frame provided by the frame buffer. To fill the time slot, the frame provided by the optical imaging sensor bypassing the frame buffer and the frame stored in the frame buffer may be received by the switching circuitry just in time so the respective frame can be output according to the image output framerate. This may require synchronization between the image generation framerate and the image output framerate. For example, the image output framerate may be time-synchronized with the image generation framerate, such that a timing of frames provided by the optical imaging sensor bypassing the frame buffer is suitable for a timing of the switching circuitry, e.g., such that the latest generated frame provided by the optical imaging sensor bypassing the frame buffer is provided in time to be output by the switching circuitry according to the video output fram erate. Similarly, the stored latest generated frame may be provided in time to be output by the switching circuitry according to the video output framerate. The optical imaging device may comprise a timing generation circuitry 150 to provide the stored latest generated frame in time to the switching circuitry. For example, the timing generation circuitry may be configured to provide the stored latest generated frame to the switching circuitry at the image output fram erate, such that a timing of the frames provided by the timing generator circuitry is suitable for the timing of the switching circuitry, e.g., such that the stored latest generated frame is provided in time to be output by the switching circuitry according to the video output fram erate.

In some examples, the ratio between image generation framerate and image output framerate may be known by the logic circuitry of the optical imaging device, and also the timing at which the imaging frames are generated by the optical imaging sensor may be known. From this knowledge, the switching circuitry may be configured to automatically switch between outputting the frames received from the respective source.

Alternatively, the switching circuitry may comprise a detector for data from the optical imag ing sensor. When new data is available from the optical imaging sensor, the switching cir cuitry accepts the data. In other words, the switching circuitry may be configured to detect, at the image output framerate, whether the latest generated frame is provided by the optical im aging sensor bypassing the frame buffer, and to output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer upon detection. When new data from the optical imaging sensor is due (e.g., according to the video timing generator) but does not arrive, the switching circuitry may be configured to toggle to accept data from the frame buffer. The switching circuitry may therefore be configured to automatically switch based on the availability of imaging sensor data without external intervention.

In general, the circuitry of the optical imaging device, e.g., the switching circuitry, a logic portion of the frame buffer, the timing generator circuitry, input/output circuitry and/or the interconnect circuitry, may be implemented as application-specific circuitry, e.g., as part of an application-specific integrated circuit or as part of a digital signal processor (DSP). Alter natively, reconfigurable hardware may be used to implement the logic circuitry and the inter connect circuitry. For example, the switching circuitry, a logic portion of the frame buffer and an interconnect circuitry between the optical imaging sensor, the frame buffer, the switching circuitry and the output circuitry is implemented on a field-programmable gate array (FPGA). In this case, the input/output circuitry may be implemented as fixed -function input/output blocks of the FPGA, and the memory of the frame buffer may be implemented as static ran dom-access memory that is connected to the FPGA.

More details and aspects of the optical imaging device are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., Figs. 3 to 4). The optical imaging device may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

As mentioned above, the optical imaging device may be used to provide low-latency video data in a digital microscope, e.g., in order to improve the video latency in a head -up display or digital oculars of the microscope or of a microscope system comprising the microscope. Therefore, as shown in Fig. 3, various examples of the present disclosure provide a micro scope 310 comprising at least one optical imaging device as introduced in connection with Figs la to 2. In general, a microscope is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone). For example, a micro scope may provide an optical magnification of a sample. In modern microscopes, the optical magnification is often provided for a camera or an imaging sensor, such as an optical imaging device introduced in connection with Figs la to 2. The microscope 310 may further comprise one or more optical magnification components that are used to magnify a view on the sample, such as an objective (i.e., lens). There are a variety of different types of microscopes. If the microscope is used in the medical or biological fields, the object being viewed through the microscope may be a sample of or ganic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient. For example, the microscope may be a microscope system for use in a laboratory, e.g., a micro scope that may be used to examine the sample of organic tissue in a petri dish. Alternatively, the microscope 310 may be part of a surgical microscope system 300, e.g., a microscope to be used during a surgical procedure. Such a system is shown in Fig. 3, for example. The microscope 310 is a surgical microscope for use in ophthalmology (eye surgery). Therefore, various examples provide a surgical microscope 310 comprising at least one optical imaging device as introduced in connection with Figs la to 2. Accordingly, an object being viewed through the microscope, and shown in the frames of imaging sensor data, may be a sample of organic tissue of a patient. Although embodiments are described in connection with a micro scope, they may also be applied, in a more general manner, to any optical device.

Fig. 3 shows a schematic diagram of an example of a surgical microscope system 300 with a microscope 310 comprising an optical imaging system according to an example. Examples thus further provide the surgical microscope system 300 comprising the surgical microscope 310 with the at least one optical imaging device as introduced in connection with Figs la to 2. The surgical microscope system 300 shown in Fig. 3 comprises a number of optional com ponents, such as a base unit 320 with a (rolling) stand, a head-up display 330, and a (robotic or manual) arm 340 which holds the microscope 310 in place, and which is coupled to the base unit 320 and to the microscope 310. In the context of this application, the term “(surgical) microscope system” is used, in order to cover the portions of the system that are not part of the actual microscope (which comprises the optical components and is thus also denoted “op tics carrier”), but which are used in conjunction with the microscope, such as the head-up display or an illumination system.

The surgical microscope system comprises the head-up display 330. Additionally or alterna tively, the microscope 310 may comprise ocular displays 311. For example, the frames of imaging sensor data being output by the at least one optical imaging device may be processed by a control system of the surgical microscope system or of the microscope, and a display signal may be provided to the head-up display 330 or to the ocular displays 311 based on the frames of imaging sensor data being output by the at least one optical imaging device. More details and aspects of the microscope and of the surgical microscope system are men tioned in connection with the proposed concept or one or more examples described above or below (e.g., Figs la to 2, 4). The microscope and/or the surgical microscope system may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

Fig. 4 shows a flow chart of an example of a corresponding method for an optical imaging device, e.g., for the optical imaging device 100 introduced in connection with Figs la to 3. The method comprises providing 410, by an optical imaging sensor, a latest generated frame of imaging sensor data. The method comprises storing 420 the latest generated frame in a frame buffer. The method comprises outputting 430, e.g., by switching circuitry and via out put circuitry, during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer. The method comprises outputting 440, e.g., by switching circuitry and via output circuitry, during a second time period, the stored latest generated frame stored in the frame buffer.

As indicated above, features described in connection with the optical imaging device of Figs la to 2 and of the microscope or surgical microscope system of Fig. 3 may be likewise applied to the method of Fig. 4.

As used herein the term “and/or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “/”.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or de vice corresponds to a method step or a feature of a method step. Analogously, aspects de scribed in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. List of reference Signs

100 Optical imaging device

110 Optical imaging sensor 120 Frame buffer 130 Switching circuitry

140 Output circuitry 150 Timing generation circuitry

201-204 Optical imaging sensor output 211-214 Video output 221, 223 Optical imaging sensor output

222, 224 Frame buffer output 231-234 Video output 300 Microscope system 310 Microscope 311 Ocular displays

320 Base unit 330 Head-up display 340 Arm 410 Providing a latest generated frame of imaging sensor data 420 Storing the latest generated frame in a frame buffer

430 Outputting the latest generated frame bypassing the frame buffer

440 Outputting the stored latest generated frame stored in the frame buffer

Claims

1. An optical imaging device (100), comprising: an optical imaging sensor (110), configured to: generate frames of imaging sensor data; and provide a latest generated frame of imaging sensor data to a frame buffer (120) and to a switching circuitry (130) of the optical imaging device; the frame buffer (120) configured to store the latest generated frame; and the switching circuitry (130) configured to: output, via an output circuitry (140) of the optical imaging device during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer; and output, via the output circuitry (140) of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer.

2. The optical imaging device according to claim 1, wherein the switching circuitry is configured to give preference to the frame provided by the optical imaging sensor bypassing the frame buffer over the stored latest generated frame stored in the frame buffer.

3. The optical imaging device according to one of the claims 1 or 2, wherein the frame buffer is configured to output the stored latest generated frame until the stored latest generated frame is replaced by the latest generated frame generated by the optical imaging sensor.

4. The optical imaging device according to one of the claims 1 to 3, wherein the optical imaging sensor is configured to generate the frames of imaging sensor data at an image generation framerate, and wherein the switching circuitry is configured to output the respective latest generated frames at an image output framerate.

5. The optical imaging device according to claim 4, comprising input circuitry for ob taining an image generation framerate control signal, wherein the optical imaging sen sor is configured to control the image generation framerate based on the image gener ation framerate control signal, and/or wherein the image output framerate is fixed to a pre -defined value.

6. The optical imaging device according to one of the claims 4 or 5, wherein the optical imaging device is suitable for operating with an image generation framerate that is an integer fraction of the image output framerate.

7. The optical imaging device according to claim 6, wherein the optical imaging device is further suitable for operating with an image generation framerate that corresponds to the image output framerate.

8. The optical imaging device according to claim 7, wherein the switching circuitry is configured to, if the optical imaging device operates with an image generation fram erate that corresponds to the image output framerate, output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer without switching to outputting the stored latest generated frame.

9. The optical imaging device according to one of the claims 4 to 8, wherein the image output framerate is time-synchronized with the image generation framerate, such that a timing of frames provided by the optical imaging sensor bypassing the frame buffer is suitable for a timing of the switching circuitry.

10. The optical imaging device according to claim 9, comprising a timing generation cir cuitry (150), configured to provide the stored latest generated frame to the switching circuitry at the image output framerate, such that a timing of the frames provided by the timing generator circuitry is suitable for the timing of the switching circuitry.

11. The optical imaging device according to one of the claims 4 to 10, wherein the switch ing circuitry is configured to detect, at the image output framerate, whether the latest generated frame is provided by the optical imaging sensor bypassing the frame buffer, and to output the latest generated frame provided by the optical imaging sensor by passing the frame buffer upon detection.

12. The optical imaging device according to one of the claims 1 to 11, wherein the switch ing circuitry is configured to receive the respective latest generated frames succes sively in data units representing a fraction of the frame, and to output the data units representing a fraction of the latest generated frame without buffering the entire frame.

13. The optical imaging device according to one of the claims 1 to 12, wherein the switch ing circuitry, a logic portion of the frame buffer and an interconnect circuitry between the optical imaging sensor, the frame buffer, the switching circuitry and the output circuitry is implemented on a field-programmable gate array.

14. A microscope, such as a surgical microscope, comprising at least one optical imaging device according to one of the claims 1 to 13.

15. A method for an optical imaging device, the method comprising: providing (410), by an optical imaging sensor, a latest generated frame of imaging sensor data; storing (420) the latest generated frame in a frame buffer; outputting (430), during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer; and outputting (440), during a second time period, the stored latest generated frame stored in the frame buffer.