WO2022223400A1 - Dispositif d'imagerie, et procédé et microscope correspondants - Google Patents

Dispositif d'imagerie, et procédé et microscope correspondants Download PDF

Info

Publication number
WO2022223400A1
WO2022223400A1 PCT/EP2022/059888 EP2022059888W WO2022223400A1 WO 2022223400 A1 WO2022223400 A1 WO 2022223400A1 EP 2022059888 W EP2022059888 W EP 2022059888W WO 2022223400 A1 WO2022223400 A1 WO 2022223400A1
Authority
WO
WIPO (PCT)
Prior art keywords
optical imaging
frame
framerate
imaging device
output
Prior art date
Application number
PCT/EP2022/059888
Other languages
English (en)
Inventor
Gao Yang
Original Assignee
Leica Instruments (Singapore) Pte. Ltd.
Leica Microsystems Cms Gmbh
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Leica Instruments (Singapore) Pte. Ltd., Leica Microsystems Cms Gmbh filed Critical Leica Instruments (Singapore) Pte. Ltd.
Priority to EP22722747.7A priority Critical patent/EP4327548A1/fr
Publication of WO2022223400A1 publication Critical patent/WO2022223400A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Definitions

  • Examples relate to an optical imaging device, to a corresponding method for an optical imag ing device, to a microscope, such as a surgical microscope, comprising such an optical imag ing device, and more specifically, to a concept for improving a latency between generation and output of a frame in an optical imaging device.
  • a major trend in surgical microscopy is the use of head -up displays.
  • a head-up display instead of using the oculars of the microscope, the surgeon or surgeons observe the surgical site via a display that is arranged at eye-level of the surgeons, enabling them to col laborate in the visual analysis of the surgical site.
  • the surgical site is observed via one (or more) video cameras (i.e., optical imaging devices) of the microscope.
  • the video camera(s) generates a sequence of frames, which are subsequently displayed on the head-up display (and other displays, such as digital oculars of the microscope). Care may be taken to reduce the latency between the generation of the frames and the subsequent display of the frames on the display.
  • the amount of light gathered by the video camera(s) may determine the quality of the image frames (at a desired brightness of the image frame) - the more light can be gathered, the lower the amplification gain of the optical imaging sensor can be, and the lower the gain of the optical imaging sensor, the less noise the image frames contain.
  • the duration of the exposure of the sensor to the light may be increased to also increase the amount of light that is gathered.
  • a video camera’s exposure duration for each frame is limited by the frame duration, which corresponds to the framerate selected for capturing the sequence of frames.
  • a video camera may operate at a fixed output framerate while having a different and variable framerate at the optical imaging sensor.
  • the output framerate may follow the optical imaging sensor’s framerate.
  • a frame buffer is used to resolve the asynchrony.
  • such a frame buffer usually introduces significant delays to the video signals.
  • a frame buffer may be used to support an image generation framerate that is different from an image output framerate
  • this frame buffer may be bypassed when a latest image frame is available from the optical imaging sensor.
  • the optical imaging sensor outputs a latest generated frame
  • this frame is output to both a frame buffer, which is used to store the latest generated frame, and to a switching circuitry, which may output the latest frame directly, introducing less latency (e.g., only a few clock cycles) than the frame buffer, thereby reducing the latency relative to an approach where all of the frames are output via a frame buffer.
  • the optical imaging device comprises an optical imaging sensor.
  • the optical imaging sensor is configured to generate frames of imaging sensor data, and to provide a latest generated frame of imaging sensor data to a frame buffer and to a switching circuitry of the optical imaging device.
  • the optical imaging device further comprises the frame buffer that is configured to store the latest generated frame.
  • the optical imaging device further comprises the switching circuitry.
  • the switching circuitry is configured to output, via an output circuitry of the optical imaging de vice during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer.
  • the switching circuitry is configured to output, via the output circuitry of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer.
  • the switching circuitry is configured to give preference to the frame provided by the optical imaging sensor bypassing the frame buffer over the stored latest gen erated frame stored in the frame buffer. In other words, when a latest generated frame is di rectly provided by the optical imaging sensor, the frame buffer may be bypassed, and the additional latency of the frame buffer may be avoided.
  • the frame buffer may be configured to output the stored latest generated frame until the stored latest generated frame is replaced by the latest generated frame generated by the optical imaging sensor.
  • the frame stored in the frame-buffer may be used to “fill” the frames of the output while the optical imaging sensor is in the process of exposing for a latest frame.
  • the optical imaging sensor may be configured to generate the frames of imaging sensor data at an image generation framerate.
  • the switching circuitry may be con figured to output the respective latest generated frames at an image output framerate.
  • the image generation framerate and the image output framerate may be set inde pendent from another.
  • the image output framerate may be fixed, and the image generation framerate may be variable. In other words, the image output framerate may be fixed to a pre-defined value.
  • the optical imaging device may further com prise input circuitry for obtaining an image generation framerate control signal.
  • the optical imaging sensor may be configured to control the image generation framerate based on the image generation framerate control signal. For example, the image generation framerate may be controlled based on the amount of available light.
  • the optical imaging device is suitable for operating with an image gen eration framerate that is an integer fraction of the image output framerate.
  • the image output framerate may be set to 60 Hz
  • the image generation framerate may be set to 30 Hz, 20 Hz or 15 Hz, or the image output framerate may be set to 120 Hz, which further enables an image generation framerate of 60 Hz or 24 Hz.
  • the optical imaging device may be further suitable for operating with an image generation framerate that corresponds to the image output framerate, e.g., at 60 Hz when the image output framerate is set to 60 Hz, or 120 Hz if the image output framerate is set to 120 Hz.
  • the switching circuitry may be configured to, if the optical imaging device operates with an image generation framerate that corresponds to the image output framerate, output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer without switching to outputting the stored latest generated frame. In other words, if the image gener ation framerate corresponds to the image output framerate, the frame buffer may be bypassed entirely.
  • the timing of the framerates may be time-synchronized, so that the latest generated frame that is output by the optical imaging sensor can be output directly by the switching circuitry, without requiring additional buffering (apart from line buffering that is inherent to the circuitry).
  • the image output framerate may be time-syn chronized with the image generation framerate, such that a timing of frames provided by the optical imaging sensor bypassing the frame buffer is suitable for a timing of the switching circuitry.
  • the stored latest generated frame may be provided to the switching circuitry in a manner that enables the switching circuitry to output the stored latest generated frame re quiring additional buffering (apart from line buffering that is inherent to the circuitry).
  • the optical imaging device may comprise a timing generation circuitry that is configured to pro vide the stored latest generated frame to the switching circuitry at the image output framerate, such that a timing of the frames provided by the timing generator circuitry is suitable for the timing of the switching circuitry.
  • the switching circuitry may switch automatically between outputting the latest generated frame that is provided bypassing the frame buffer and the stored latest generated frame, e.g., by detecting whether the latest generated frame provided by the optical imaging sensor by passing the frame buffer is provided in time for the next frame of the image output framerate.
  • the switching circuitry may be configured to detect, at the image output fram erate, whether the latest generated frame is provided by the optical imaging sensor bypassing the frame buffer, and to output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer upon detection.
  • Optical imaging sensors are often read-out in small increments.
  • the optical im aging sensor may comprise a two-dimensional array of pixels, comprising a plurality of lines of pixels and a plurality of columns of pixels.
  • the optical imaging sensor is often read-out line-by-line (or blocks of lines at a time), e.g., as the bandwidth of the output of the optical imaging sensor is limited and as crucial circuitry can be re-used by reading out the optical imaging sensor in increments.
  • the frames may also be transmitted as subsequent fractions of the frame, e.g., line-by-line (or blocks of lines at a time), without buffering the entire frame at the switching circuitry.
  • the switch ing circuitry may be configured to receive the respective latest generated frames successively in data units representing a fraction of the frame, and to output the data units representing a fraction of the latest generated frame without buffering the entire frame.
  • the logic components of the optical imaging device may be implemented by a Digital Signal Processor (DSP).
  • DSP Digital Signal Processor
  • the logic components may be imple mented using a re-programmable hardware device.
  • the switching circuitry, a logic portion of the frame buffer and an interconnect circuitry between the optical imaging sensor, the frame buffer, the switching circuitry and the output circuitry may be implemented on a field-programmable gate array.
  • a microscope such as a surgical micro scope, comprising at least one optical imaging device as introduced above.
  • the method comprises providing, by an optical imaging sensor, a latest gen erated frame of imaging sensor data.
  • the method comprises storing the latest generated frame in a frame buffer.
  • the method comprises outputting, during a first time period, the latest gen erated frame provided by the optical imaging sensor bypassing the frame buffer.
  • the method comprises outputting, during a second time period, the stored latest generated frame stored in the frame buffer.
  • Figs la and lb show schematic diagrams of examples of an optical imaging device
  • Figs. 2a and 2b show schematic diagrams of a timing of frames according to an example
  • Fig. 3 shows a schematic diagram of an example of a surgical microscope system with a microscope comprising an optical imaging system according to an example
  • Fig. 4 shows a flow chart of an example of a method for an optical imaging device.
  • variable optical imaging sensor framerate i.e., a variable framerate of the optical imaging sensor
  • the proposed concept may be used in head-up surgery.
  • a video camera s exposure duration for each frame is limited by the frame duration, which is based on the selected framerate.
  • a video camera may operate at a fixed output fram erate while having a different and variable framerate at the optical imaging sensor. Alterna tively, the output framerate may need to follow the optical imaging sensor’s framerate. If the video camera operates at a fixed framerate that is different from the optical imaging sensor’s framerate, a frame buffer may be introduced between the optical imaging sensor and the out put of the video camera, to resolve the asynchrony, thus introducing significant delays to the video signals.
  • the sink device e.g., a video capture device or display
  • the sink device may lose video synchronisation every time the framerate changes, resulting in a blank screen or flickering.
  • Figs la and lb show schematic diagrams of examples of an optical imaging device 100.
  • the optical imaging device 100 comprises an optical imaging sensor 110 that is configured to generate frames of imaging sensor data, and to provide a latest generated frame of imaging sensor data to a frame buffer 120 and to a switching circuitry 130 of the optical imaging device.
  • the optical imaging device 100 further comprises the frame buffer 120, which is con figured to store the latest generated frame.
  • the optical imaging device 100 further comprises the switching circuitry 130.
  • the switching circuitry is configured to output, via an output circuitry 140 of the optical imaging device during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer, and to output, via the output circuitry 140 of the optical imaging device during a second time period, the stored latest generated frame stored in the frame buffer. Accordingly, the optical imaging device further comprises the output circuitry 140.
  • the optical imaging device 100 shown in Figs la and lb includes a frame buffer - however, this frame buffer 120 is operated in synchronicity with the optical imaging device.
  • the video output of the optical imaging device may be locked at a certain fram- erate and the optical imaging sensor can operate at an integer fraction of this framerate.
  • this frame buffer may remain unused.
  • the frame buffer may be used to record and repeat the latest frame from the optical imaging sensor.
  • a video timing generator 150 as shown in Fig.
  • lb may be used so the read out from the frame buffer closely emulates the timing pattern of the optical imaging sensor. With this design, a blank screen or flickering may be avoided since the output framerate is locked. No or only a few clock cycles of addi tional latency may be introduced because the latest generated frames are output without going through the frame buffer.
  • arrows are shown - arrows with a dotted line pattern, which show the path of the latest generated frame bypassing the frame buffer, and arrows with a dashed line pat tern, which show the path of the latest generated frame through the frame buffer.
  • the generated frames of the video signal may follow the arrows with the dotted line pattern: the latest generated frame is transmitted from the optical imaging sensor 110 through the switching circuitry 130 to the video output 140. The data is also written into the frame buffer 120.
  • the generated frames of the video signal may follow the arrows with the dashed line pattern: the stored latest frames are obtained from the frame buffer 120, optionally throttled by the video timing generator 150, pass through the switching circuitry 130, and are provided to the output 140.
  • the optical imaging device 100 may be a video camera or a video sensor arrangement, i.e., a device being sued to provide (low-latency) video data. Consequently, the optical imaging device 100 may be configured to provide video data comprising the generated frames via the output circuitry.
  • the optical imaging device 100 may be an optical imaging device or a video camera for use in a digital microscope.
  • the optical imaging device 100 may be used to generate video data based on light reflected or emitted by a sample being observed through the microscope.
  • a microscope may comprise at least one optical imaging device 100, e.g., two optical imaging devices, in case of a stere oscopic microscope.
  • An example of a surgical microscope system comprising a microscope with at least one optical imaging system is shown in Fig. 3.
  • the optical imaging sensor 110 is used to generate the frames of the imaging sensor data.
  • the optical imaging sensor may comprise or be an APS (Active Pixel Sensor) - or a CCD (Charge-Coupled-Device)-based imaging sensor.
  • APS-based imaging sensors light is recorded at each pixel using a photo-detector and an active amplifier of the pixel.
  • CMOS Complementary Metal-Oxide- Semiconductor
  • S-CMOS Stientific CMOS
  • incoming photons are converted into electron charges at a semiconductor-oxide interface, which are subsequently moved between capacitive bins in the imaging sensors by a control circuitry of the imaging sensors to perform the imaging.
  • the optical imaging sensor thus measures the incoming light, and generates the frames of imaging sensor data based on the measured incoming light.
  • the optical imaging sensor further provides the latest generated frame of imaging sensor data to the frame buffer and to the switching circuitry of the optical imaging device.
  • the optical imaging sensor is connected to the frame buffer and to the switching circuitry via interconnect circuitry.
  • the optical imaging sensor may transmit the latest gener ated frame to the frame buffer and to the switching circuitry, e.g., by generating a correspond ing signal and outputting the signal to the interconnect circuitry.
  • the respective signals may be received, e.g., via a (small) input buffer.
  • optical imaging sensors and in particular optical imaging sensors being used to generate video data, often apply a so-called “rolling shutter”.
  • optical im aging sensors may comprise a two-dimensional grid of pixels, comprising a plurality of lines of pixels and a plurality of columns of pixels. The content of the two-dimensional grid of pixels is read out to generate a frame of imaging sensor data. In many cases, only a subset of pixels of the two-dimensional grid of pixels is read out at once, e.g., line-by-line or block-by block. To ensure that all of the pixels are exposed to the same (or at least similar) amounts of light, the duration of exposure to light of the pixels may be (substantially) the same for all of the pixels.
  • frames of imaging sensor data may comprise a plurality of data units each representing a fraction of the frame.
  • each data unit may represent a line of pixels or a block of pixels.
  • the optical imaging sensor may be configured to provide the frame successively in data units representing a fraction of the frame, e.g., by outputting the plurality of data units successively until the entire frame has been provided.
  • the frames of imaging sensor data are provided to the frame buffer and to the switching cir cuitry (thereby bypassing the frame buffer).
  • the frame buffer is configured to store the latest frame provided by the optical imaging sensor.
  • the frame buffer may comprise a (volatile) memory, such as a dynamic random-access memory, for storing the latest generated frame.
  • the frame buffer may further comprise logic circuitry configured to re ceive the latest generated frame from the optical imaging sensor, e.g., via the interconnect circuitry, to store the received frame (e.g., data unit by data unit) in the memory, and to pro vide the stored frame to the switching circuitry, e.g., via another portion of the interconnect circuitry.
  • the frame buffer may keep outputting the same stored frame until the stored frame is replaced by a subsequent frame provided by the optical imaging sensor.
  • the frame buffer may be configured to output the stored latest gener ated frame until the stored latest generated frame is replaced by the latest generated frame generated by the optical imaging sensor.
  • the stored frame may be replaced data unit by data unit as the latest generated frame is provided by the optical imaging sensor.
  • the switching circuitry 130 is used to switch between outputting, with minimal latency, the frame that is provided directly by the optical imaging sensor (thereby bypassing the frame buffer), and outputting the frame that is stored in the frame buffer.
  • the switching circuitry outputs the latest frame just as it is provided by the optical imaging sensor, bypassing the frame buffer. Only when no latest frame is available from the optical imaging sensor, e.g., as a framerate at which frames are output by the switch ing circuitry is higher than a framerate at which frames are generated by the optical imaging sensor, the stored latest generated frame might by output instead. Therefore, the switching circuitry may be configured to give preference to the frame provided by the optical imaging sensor bypassing the frame buffer over the stored latest generated frame stored in the frame buffer.
  • the ratio between the framerate at which the images are output by the switching circuitry and the framerate, at which the optical imaging sensor generates the frames of im aging sensor data affects the operation of the optical imaging device.
  • the optical imaging sensor may be configured to generate the frames of imaging sensor data at an image generation framerate
  • the switching circuitry may be configured to output the re spective latest generated frames at an image output framerate.
  • a framerate is a rate, at which frames are generated or provided. Framerates are usually defined in terms of frames per unit of time, e.g., frames per second.
  • the optical imaging device may have an image output framerate of 24 frames per second, of 30 frames per second, of 60 frames per second, of 90 frames per second, of 120 frames per second etc.
  • the optical imaging sensor may have an image generation framerate of 12 frames per second, of 15 frames per second, of 24 frames per second, of 30 frames per second, of 60 frames per second, of 90 frames per second, of 120 frames per second as well.
  • the image output framerate may be fixed to a pre-defined value (e.g., one of 24/30/60/90/120 frames per second), while the image genera tion framerate may be variable.
  • the image generation framerate may be kept variable as the framerate also influences the amount of light the optical imaging sensor is exposed to while generating the respective frame. The higher the framerate, the shorter the optical imaging sensor is exposed to the light, the less light is being used to expose the optical imaging sensor. While the amplification gain of the sensor can be changed to compensate for the shorter exposure time, higher gains tend to lead to higher levels of noise, as the sensor is not only more sensitive to light, but also more sensitive to noise.
  • the image generation framerate may be reduced to allow for longer exposure times.
  • the image generation framerate may be set by an external entity, e.g., a control system of the microscope, to adapt the exposure time of the optical imaging sensor.
  • the optical imaging device may comprise input circuitry for obtaining an image gener ation framerate control signal, e.g., from a control system of a microscope comprising the optical imaging device.
  • the optical imaging sensor may be configured to control (e.g., set) the image generation framerate based on the image generation framerate control signal.
  • the optical imaging device may support two modes - a first mode, where the optical imaging device operates with an image generation framerate that is lower than the image output framerate, and a second case, where the optical imaging device operates with an image generation framerate that is the same as the image output framerate.
  • the choice of image generation framerate and im age output framerate may be limited.
  • the image generation framerate may be set at an integer fraction of the image output framerate.
  • the optical imaging device may be suitable for operating with an image generation framerate that is an integer fraction of the image output framerate, and the optical imaging device may be further suitable for operating with an image generation framerate that corresponds to the image output framerate.
  • the image output framerate may correspond to the image generation framerate, may be twice the image generation framerate, three times the image generation framerate, four times the image generation framerate or five times the image generation framerate.
  • the latest generated frame may (always) be output bypassing the framerate, and the subsequent frame (to fill up the integer fraction), e.g., one subsequent frame if the image output framerate is twice the image generation framerate, two subsequent frames if the image output framerate is three times the image generation framerate etc., may be filled with the stored latest generated frame. Consequently, the proposed concept may be used to “fill” frames that are not provided in time by the optical imaging sensor, as the image generation framerate is lower than the image output framerate.
  • the switching circuitry may be configured to, if the optical imaging device operates with an image generation fram erate that corresponds to the image output framerate, output the latest generated frame pro vided by the optical imaging sensor bypassing the frame buffer without switching to output ting the stored latest generated frame.
  • Figs. 2a and 2b show schematic diagrams of a timing of frames (i.e., video frame timing) according to an example.
  • the framerate of the optical imaging sensor i.e., the image generation framerate
  • a first frame 201 of the optical imaging sensor is provided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 211 is (directly) provided at the video output, i.e., by the switching circuitry via the output circuitry.
  • This is repeated for subsequent frames 202- 204 of the optical imaging sensor and corresponding video output frames 212 to 214.
  • each of the frames being output is a frame that is provided directly by the optical imaging sensor, bypassing the frame buffer.
  • the image generation framerate is half the image output framerate.
  • a first frame 221 of the optical imaging is provided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 231 is (directly) provided at the video output, i.e., by the switching circuitry via the output circuitry.
  • a stored version 222 of the first frame, as stored in the frame buffer is provided by the frame buffer and a correspond ing video output frame 232 is provided at the video output.
  • a second frame 223 is pro vided by the optical imaging sensor bypassing the frame buffer and a corresponding video output frame 233 is (directly) provided at the video output, followed by a stored version 224 of the second frame and a corresponding video output frame 234.
  • the switching circuitry switches between outputting, during the first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer and outputting, during the second time period, the stored latest generated frame stored in the frame buffer.
  • the image generation framerate is half the image output fram- erate
  • the first time period has the same duration as the second time period, as both alternatives apply half of the total time.
  • the first time period may have half the duration of the second time period, as the first alternative (bypassing the frame buffer) takes up a third of the total time, and the second alternative (frame stored in the frame buffer) takes up the remaining two thirds of the total time. If, however, the image generation framerate corresponds to the image output framerate, all of the frames that are output by the switching circuitry may bypass the frame buffer, and thus the first time period may take up the entire time, with no second time period being re quired.
  • the switching circuitry may begin outputting the frames while they are still being provided to the switching circuitry.
  • the switching circuitry may be configured to receive the respective latest generated frames successively in data units representing a fraction of the frame, and to output the data units representing a fraction of the latest generated frame without buffering the entire frame.
  • the switching circuitry may be configured to begin outputting the respective frame before having received the entire frame.
  • some buffering may occur, as the switching circuitry may require one or more clock cycles to output the received frames.
  • the frames being output by the switching circuitry are output according to the image output framerate.
  • the respective frames may be received at the same framerate by the switching circuitry, either (directly) from the optical imaging sensor bypassing the framerate, or from the frame-buffer. Consequently, the switching circuitry may be configured to receive at least one of a) the frame provided by the optical imaging sensor bypassing the frame buffer and b) the frame provided by the frame buffer at the image output framerate. In other words, at a time slot of each frame of the image output framerate, the switching circuitry may be configured to receive at least one of a) the frame provided by the optical imaging sensor bypassing the frame buffer and b) the frame provided by the frame buffer.
  • the frame provided by the optical imaging sensor bypassing the frame buffer and the frame stored in the frame buffer may be received by the switching circuitry just in time so the respective frame can be output according to the image output framerate.
  • This may require synchronization between the image generation framerate and the image output framerate.
  • the image output framerate may be time-synchronized with the image generation framerate, such that a timing of frames provided by the optical imaging sensor bypassing the frame buffer is suitable for a timing of the switching circuitry, e.g., such that the latest generated frame provided by the optical imaging sensor bypassing the frame buffer is provided in time to be output by the switching circuitry according to the video output fram erate.
  • the stored latest generated frame may be provided in time to be output by the switching circuitry according to the video output framerate.
  • the optical imaging device may comprise a timing generation circuitry 150 to provide the stored latest generated frame in time to the switching circuitry.
  • the timing generation circuitry may be configured to provide the stored latest generated frame to the switching circuitry at the image output fram erate, such that a timing of the frames provided by the timing generator circuitry is suitable for the timing of the switching circuitry, e.g., such that the stored latest generated frame is provided in time to be output by the switching circuitry according to the video output fram erate.
  • the ratio between image generation framerate and image output framerate may be known by the logic circuitry of the optical imaging device, and also the timing at which the imaging frames are generated by the optical imaging sensor may be known. From this knowledge, the switching circuitry may be configured to automatically switch between outputting the frames received from the respective source.
  • the switching circuitry may comprise a detector for data from the optical imag ing sensor. When new data is available from the optical imaging sensor, the switching cir cuitry accepts the data.
  • the switching circuitry may be configured to detect, at the image output framerate, whether the latest generated frame is provided by the optical im aging sensor bypassing the frame buffer, and to output the latest generated frame provided by the optical imaging sensor bypassing the frame buffer upon detection.
  • the switching circuitry may be configured to toggle to accept data from the frame buffer. The switching circuitry may therefore be configured to automatically switch based on the availability of imaging sensor data without external intervention.
  • the circuitry of the optical imaging device e.g., the switching circuitry, a logic portion of the frame buffer, the timing generator circuitry, input/output circuitry and/or the interconnect circuitry
  • the circuitry of the optical imaging device may be implemented as application-specific circuitry, e.g., as part of an application-specific integrated circuit or as part of a digital signal processor (DSP).
  • DSP digital signal processor
  • reconfigurable hardware may be used to implement the logic circuitry and the inter connect circuitry.
  • the switching circuitry, a logic portion of the frame buffer and an interconnect circuitry between the optical imaging sensor, the frame buffer, the switching circuitry and the output circuitry is implemented on a field-programmable gate array (FPGA).
  • the input/output circuitry may be implemented as fixed -function input/output blocks of the FPGA
  • the memory of the frame buffer may be implemented as static ran dom-access memory that is connected to the FPGA.
  • optical imaging device More details and aspects of the optical imaging device are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., Figs. 3 to 4).
  • the optical imaging device may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.
  • the optical imaging device may be used to provide low-latency video data in a digital microscope, e.g., in order to improve the video latency in a head -up display or digital oculars of the microscope or of a microscope system comprising the microscope. Therefore, as shown in Fig. 3, various examples of the present disclosure provide a micro scope 310 comprising at least one optical imaging device as introduced in connection with Figs la to 2.
  • a microscope is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone).
  • a micro scope may provide an optical magnification of a sample.
  • the microscope 310 may further comprise one or more optical magnification components that are used to magnify a view on the sample, such as an objective (i.e., lens).
  • an objective i.e., lens
  • the object being viewed through the microscope may be a sample of or ganic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient.
  • the microscope may be a microscope system for use in a laboratory, e.g., a micro scope that may be used to examine the sample of organic tissue in a petri dish.
  • the microscope 310 may be part of a surgical microscope system 300, e.g., a microscope to be used during a surgical procedure. Such a system is shown in Fig. 3, for example.
  • the microscope 310 is a surgical microscope for use in ophthalmology (eye surgery). Therefore, various examples provide a surgical microscope 310 comprising at least one optical imaging device as introduced in connection with Figs la to 2. Accordingly, an object being viewed through the microscope, and shown in the frames of imaging sensor data, may be a sample of organic tissue of a patient.
  • embodiments are described in connection with a micro scope, they may also be applied, in a more general manner, to any optical device.
  • Fig. 3 shows a schematic diagram of an example of a surgical microscope system 300 with a microscope 310 comprising an optical imaging system according to an example. Examples thus further provide the surgical microscope system 300 comprising the surgical microscope 310 with the at least one optical imaging device as introduced in connection with Figs la to 2.
  • the surgical microscope system 300 shown in Fig. 3 comprises a number of optional com ponents, such as a base unit 320 with a (rolling) stand, a head-up display 330, and a (robotic or manual) arm 340 which holds the microscope 310 in place, and which is coupled to the base unit 320 and to the microscope 310.
  • optional com ponents such as a base unit 320 with a (rolling) stand, a head-up display 330, and a (robotic or manual) arm 340 which holds the microscope 310 in place, and which is coupled to the base unit 320 and to the microscope 310.
  • the term “(surgical) microscope system” is used, in order to cover the portions of the system that are not part of the actual microscope (which comprises the optical components and is thus also denoted “op tics carrier”), but which are used in conjunction with the microscope, such as the head-up display or an illumination system.
  • the surgical microscope system comprises the head-up display 330. Additionally or alterna tively, the microscope 310 may comprise ocular displays 311. For example, the frames of imaging sensor data being output by the at least one optical imaging device may be processed by a control system of the surgical microscope system or of the microscope, and a display signal may be provided to the head-up display 330 or to the ocular displays 311 based on the frames of imaging sensor data being output by the at least one optical imaging device. More details and aspects of the microscope and of the surgical microscope system are men tioned in connection with the proposed concept or one or more examples described above or below (e.g., Figs la to 2, 4). The microscope and/or the surgical microscope system may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.
  • Fig. 4 shows a flow chart of an example of a corresponding method for an optical imaging device, e.g., for the optical imaging device 100 introduced in connection with Figs la to 3.
  • the method comprises providing 410, by an optical imaging sensor, a latest generated frame of imaging sensor data.
  • the method comprises storing 420 the latest generated frame in a frame buffer.
  • the method comprises outputting 430, e.g., by switching circuitry and via out put circuitry, during a first time period, the latest generated frame provided by the optical imaging sensor bypassing the frame buffer.
  • the method comprises outputting 440, e.g., by switching circuitry and via output circuitry, during a second time period, the stored latest generated frame stored in the frame buffer.
  • Base unit 330 Head-up display 340 Arm 410 Providing a latest generated frame of imaging sensor data 420 Storing the latest generated frame in a frame buffer

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Microscoopes, Condenser (AREA)

Abstract

Les exemples concernent un dispositif d'imagerie optique, un procédé correspondant pour un dispositif d'imagerie optique, et un microscope, tel qu'un microscope chirurgical, comprenant un tel dispositif d'imagerie optique. Le dispositif d'imagerie optique comprend un capteur d'imagerie optique. Le capteur d'imagerie optique est configuré pour générer des trames de données de capteur d'imagerie, et pour fournir une dernière trame générée de données de capteur d'imagerie à un tampon de trame et à un circuit de commutation du dispositif d'imagerie optique. Le dispositif d'imagerie optique comprend en outre le tampon de trame qui est configuré pour stocker le dernier cadre généré. Le dispositif d'imagerie optique comprend en outre le circuit de commutation. Le circuit de commutation est configuré pour délivrer, par l'intermédiaire d'un circuit de sortie du dispositif d'imagerie optique pendant une première période de temps, la dernière trame générée fournie par le capteur d'imagerie optique contournant le tampon de trame. Le circuit de commutation est configuré pour délivrer, via le circuit de sortie du dispositif d'imagerie optique pendant une seconde période de temps, la dernière image générée stockée dans le tampon de trame.
PCT/EP2022/059888 2021-04-20 2022-04-13 Dispositif d'imagerie, et procédé et microscope correspondants WO2022223400A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP22722747.7A EP4327548A1 (fr) 2021-04-20 2022-04-13 Dispositif d'imagerie, et procédé et microscope correspondants

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102021109994 2021-04-20
DE102021109994.3 2021-04-20

Publications (1)

Publication Number Publication Date
WO2022223400A1 true WO2022223400A1 (fr) 2022-10-27

Family

ID=81603771

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2022/059888 WO2022223400A1 (fr) 2021-04-20 2022-04-13 Dispositif d'imagerie, et procédé et microscope correspondants

Country Status (2)

Country Link
EP (1) EP4327548A1 (fr)
WO (1) WO2022223400A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080151109A1 (en) * 2006-12-26 2008-06-26 Advanced Micro Devices, Inc. Low latency cadence detection for frame rate conversion
US20110298814A1 (en) * 2010-06-07 2011-12-08 Apple Inc. Switching video streams for a display without a visible interruption

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080151109A1 (en) * 2006-12-26 2008-06-26 Advanced Micro Devices, Inc. Low latency cadence detection for frame rate conversion
US20110298814A1 (en) * 2010-06-07 2011-12-08 Apple Inc. Switching video streams for a display without a visible interruption

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
BRANUM STEVEN: "buffer bypass via software control?: Nikon FX SLR (DF, D1-D5, D600-D850) Talk Forum: Digital Photography Review", 17 June 2002 (2002-06-17), XP055945234, Retrieved from the Internet <URL:https://www.dpreview.com/forums/thread/265775> [retrieved on 20220722] *

Also Published As

Publication number Publication date
EP4327548A1 (fr) 2024-02-28

Similar Documents

Publication Publication Date Title
WO2018016345A1 (fr) Dispositif de détection et appareil électronique
JP5948512B2 (ja) 処理装置、撮像装置および内視鏡システム
KR20140034066A (ko) 화상처리장치, 화상처리방법 및 기억매체, 및 화상처리장치를 갖는 촬상장치
JP6136085B2 (ja) 画像取得装置、画像取得方法、およびコンピュータプログラム
JP2021145873A (ja) 医療用画像処理装置及び医療用観察システム
US11272092B2 (en) Imaging device, endoscope apparatus, and operating method of imaging device
JP2001189891A (ja) センサのセンサ要素を読み出す方法及びセンサ
CZ38099A3 (cs) Televizní systém s vysokým rozlišením
JP2009240531A (ja) 撮影装置
WO2022223400A1 (fr) Dispositif d&#39;imagerie, et procédé et microscope correspondants
JP2006053096A (ja) レーザー顕微鏡
JP3560663B2 (ja) 走査型光学顕微鏡
US7076027B2 (en) Fluoroscopic apparatus and method
US4905264A (en) Method and system for displaying flickerless X-ray dynamic images
US5268757A (en) High speed imaging apparatus without lowering of dynamic range
US10416321B2 (en) X-ray diagnostic apparatus
JP2008509383A (ja) 画像化システム
JP2002359786A (ja) 撮像装置および撮像制御方法
US7432971B2 (en) In-situ storage image sensor and in-situ storage image pickup apparatus
US20240053594A1 (en) Method and Device for Microscopy
JP2023061590A (ja) 撮像処理装置、撮像装置および撮像処理方法
JP2010118067A (ja) 画像処理装置、コンピュータ読み取り可能な記憶媒体及びデジタル撮影装置
US20210231938A1 (en) Microscope
JP6412361B2 (ja) 内視鏡用撮像装置
JP2001091838A (ja) 走査型レーザ顕微鏡

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22722747

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022722747

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022722747

Country of ref document: EP

Effective date: 20231120