EP4734811A1 - System and method for operating a light source for an endoscope - Google Patents

Abstract

The invention provides a system (100) for operating a light source for an endoscope, a method for operating such a light source, and a method for training an artificial neural network. The system (100) comprises: a light source (10); an analysis module (31) configured to receive at least one image from a camera unit (21) of the endoscope (20) and to make a determination, based thereon, whether the endoscope (20) is currently inserted into a patient's body (1) or not; and a lighting control module (32) configured to control the light source (10) based on the determination of the analysis module (32).

Description

System and Method for operating a light source for an endoscope

Technical field of the invention

The present invention relates to endoscopy and provides a system and a method for operating a light source for an endoscope, wherein the light source is put into different light emission modes depending on whether or not the endoscope is currently positioned within a patient’s body or outside of it.

Background of the invention

Endoscopes are coupled with light sources to be able to emit light when introduced into a patient’s body, for example during a laparoscopy or thoracoscopy. The output power of the light sources is conventionally adaptively controlled so that the amount of light is enough for the physician to evaluate the scenery via an external display.

Typically, light incident onto the endoscope is detected by a sensor, and based thereon, the light output power is maintained, increased, or decreased. This may lead to an issue when the endoscope is retrieved from the patient’s body: at the instant when said sensor exits the body, the amount of incident light detected by the sensor sharply drops. In the conventional systems, this leads to a sudden increase in the output power of the light, which may be irritating or in some instances even harmful to the patient or medical personal surrounding the patient. Summary of the invention

The above-described problems are solved by the subject-matter of the independent claims of the present invention.

According to a first aspect, the invention provides a system for operating a light source for an endoscope, comprising: a light source; an analysis module configured to receive at least one image from a camera unit of the endoscope and to make a determination, based thereon, whether the endoscope is currently inserted into a patient’s body or not; and a lighting control module configured to control the light source based on the determination of the analysis module.

One main idea of the present invention is that, depending on whether the endoscope is inside or outside of a patient’s body, different modes of controlling the light source are advantageous, as opposed to the light source being always in the same mode according to the conventional wisdom in the prior art. Moreover, it has been found by the inventors that the images from the camera unit of the endoscope itself are well suited to be the basis for this determination of the best mode for each situation.

The system may also include the endoscope itself, the endoscope comprising the camera unit and a light emitter for emitting light generated by the light source. The system may further comprise additional elements such as a display for displaying the images acquired by the camera unit and the like.

The determination about whether the endoscope is currently inserted into a patient’s body or not will sometimes also be designated as the “IN/OIIT determination” herein for the sake of brevity.

Although here, in the foregoing and in the following, some functions are described as being performed by modules, it shall be understood that this does not necessarily mean that such modules are provided as entities separate from one another. In cases where one or more modules are provided as software, the modules may be implemented by program code sections or program code snippets, which may be distinct from one another but which, may also be interwoven. Similarly, in case where one or more modules are provided as hardware, the functions of one or more modules may be provided by one and the same hardware component, or the functions of one module or the functions of several modules may be distributed over several hardware components which need not necessarily correspond to the modules one-to-one. Thus, any apparatus, system, method and so on which exhibits all of the features and functions ascribed to a specific module shall be understood to comprise, or implement, said module.

In particular, it is a possibility that all modules are implemented by program code executed by a computing device (or: computer), e.g. a server or a cloud computing platform.

The computing device may be realized as any device, or any means, for computing, in particular for executing a software, an app, or an algorithm. For example, the computing device may comprise at least one processing unit such as at least one central processing unit, CPU, and/or at least one graphics processing unit, GPU, and/or at least one field-programmable gate array, FPGA, and/or at least one application-specific integrated circuit, ASIC and/or any combination of the foregoing. The computing device may further comprise a working memory operatively connected to the at least one processing unit and/or a non-transitory memory operatively connected to the at least one processing unit and/or the working memory. The computing device may be implemented partially and/or completely in a local apparatus and/or partially and/or completely in a remote system such as by a cloud computing platform.

The present invention has particular advantages when the invention, or at least the modules, are implemented locally within one or more devices commonly used in endoscopy, for example, within a camera control unit, a light source, or even the endoscope itself. In this way, no internet connection is necessary and the signal transport and processing times are reduced.

In some advantageous embodiments, refinements, or variants of embodiments, the lighting control module is configured to control the light source to be in a first, inside mode when the determination indicates that the endoscope is currently inserted into the patient’s body, and to be in a second, outside mode when the determination indicates that the endoscope is currently not inserted into the patient’s body.

In some advantageous embodiments, refinements, or variants of embodiments, the light source is configured to emit, in the inside mode, light according to an automatic control algorithm, according to which a output power of the emitted light is automatically adapted based on light incident on the camera unit. In some advantageous embodiments, refinements, or variants of embodiments, the light source is configured to emit, in the outside mode, light with a constant output power, in particular an output power smaller than a maximum output power of the light source. Preferably, in the outside mode, the light source emits light with the minimum output power available to the light source apart from being switched off and/or with a minimum output power that is required to illuminate the inside of a patient’s body sufficiently for the camera unit to acquire images. In this way, it may be ensured that the determination can be made at all times.

In some advantageous embodiments, refinements, or variants of embodiments, the analysis module is configured to implement an artificial neural network, in particular a convolutional neural network, configured and trained to differentiate between sceneries inside and outside of the human body, and to make the determination based thereon. The artificial neural network may, for example, have a VGG-16 architecture or a MobileNet architecture.

When the system is intended for a particular endoscopic procedure, such as laparoscopy or thoracoscopy, the artificial neural network may specifically be configured and trained to differentiate sceneries occuring inside and outside of the human body during the corresponding procedure. The system may provide a user interface in which a user may indicate the intended use or medical procedure. As a result, the analysis module may choose a corresponding variant of the trained artificial neural network for increased accuracy.

In some advantageous embodiments, refinements, or variants of embodiments, the analysis module is configured to receive a video stream from the camera unit, and the IN/OIIT determination at any time point is made based on one or more of the N most recent frames of the video stream, N < 100, preferably N<10, most preferably N=1. Either all of the N most recent frames may be used, or a selection thereof may be made, for example every second, third, fifth, or the like of the images.

In some advantageous embodiments, refinements, or variants of embodiments, an image output of the camera unit has a first resolution, and the analysis module is configured to make the determination based on scaled-down images having a second resolution lower than the first resolution.

The first resolution may be an HD resolution, e.g., a 4K HD resolution, and may generally be used for the images to be displayed, e.g. by a display of the system, to a user of the endoscope and/or a surgeon for use during an endoscopic operation. Using lower-scale images for the determination may reduce the necessary bandwidth, computing power, and data storage capacity. Moreover, in variants where the analysis module employs an artificial neural network, reducing the resolution of images input into the artificial neural network greatly reduces the effort involved in training the artificial neural network. It has been found by the inventors that down-scaled images are sufficient for the purposes of the present invention. Performing the IN/OIIT determination on down-scaled images may aid in implemetn the invention locally, i.e. , in a camera control unit or light source or the like.

The images acquired by the camera unit, and in particular the images used for the determination, advantageously may be RGB images.

In some advantageous embodiments, refinements, or variants of embodiments, the system comprises a camera control unit configured to control and/or process an image acquisition by the camera unit. The analysis module may be integrated into the camera control unit. In some other variants, the analysis module is integrated into the light source.

According to a second aspect of the present invention, a computer-implemented method of training an artificial neural network, in particular for use in the system according to any embodiment of the present invention. The method comprises at least the steps of: providing training images comprising images showing sceneries within a human body and images showing sceneries outside of a human body, wherein the training images are labelled accordingly; and training an artificial neural network with the training images using supervised learning.

According to a third aspect of the present invention, a computer-implemented method for operating a light source for an endoscope, comprising: acquiring at least one image using a camera unit of an endoscope; making a determination, based on the at least one image, whether the endoscope is currently inserted into a patient’s body or not; and controlling the light source of the endoscope based on the made determination.

According to a fourth aspect, the invention provides a computer program product comprising executable program code configured to, when executed, perform the method according to any embodiment of the second or third aspect of the present invention.

According to fifth aspect, the invention provides a non-transient computer-readable data storage medium comprising executable program code configured to, when executed, perform the method according to any embodiment of the second or third aspect of the present invention. The non-transient computer-readable data storage medium may comprise, or consist of, any type of computer memory, in particular semiconductor memory such as a solid-state memory. The data storage medium may also comprise, or consist of, a CD, a DVD, a Blu-Ray-Disc, an USB memory stick or the like.

According to a sixth aspect, the invention provides a data stream comprising, or configured to generate, executable program code configured to, when executed, perform the method according to any embodiment of the second or third aspect of the present invention.

Further advantageous variants, options, embodiments and modifications will described with respect to the description and the corresponding drawings as well as in the dependent claims.

Further applicability of the present invention will become apparent from the following figures, detailed description, and claims. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art.

Brief description of the figures

Aspects of the present disclosure will be better understood with reference to the following figures. The components in the drawings are not necessarily to scale, emphasis being placed instead upon clearly illustrating the principles of the present disclosure. Parts in the different figures that correspond to the same elements have been indicated with the same reference numerals in the figures, in which:

Fig. 1 shows a schematic overview of a system according to an embodiment of the present invention in a first situation;

Fig. 2 shows a schematic overview of the system according of Fig. 1 in a second situation;

Fig. 3 shows a possible architecture for an artificial neural network for use with the system of Fig. 1 and Fig. 2;

Fig. 4 shows a schematic flow diagram illustrating a method according to another embodiment of the present invention; Fig. 5 shows a schematic flow diagram illustrating a method according to yet another embodiment of the present invention;

Fig. 6 shows a schematic block diagram illustrating a computer program product according to still another embodiment of the present invention; and

Fig. 7 shows a schematic block diagram illustrating a data storage medium according to yet another embodiment of the present invention.

The figures are not necessarily to scale, and certain components can be shown in generalized or schematic form in the interest of clarity and conciseness. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the present invention.

Detailed description of the figures

Fig. 1 shows a schematic overview of a system 100 according to an embodiment of the present invention, i.e. of a system 100 for operating a light source 10 for an endoscope 20. The endoscope 20 may be part of the system 100 or may be separate from it. In Fig. 1 , the endoscope 20 is currently inserted into the patient’s body 1, indicated by a schematic cavity.

The endoscope 20 includes a camera unit 21 configured for acquiring images, and a light emitter 22 configured to emit light 2 provided by the light source 10 to illuminate the scenery from which the images are to be taken. In the situation of Fig. 1 , the scenery will be from inside the patient’s body 1.

The system 100 comprises an analysis module 31 configured to receive at least one image from the camera unit 21 of the endoscope 20, and to make a determination (the “IN/OLIT determination”), based on the received at least one image, whether the endoscope 20 is currently inserted into the patient’s body 1 or not.

In Fig. 1, the analysis module 31 is shown to be integrated into a camera control unit, CCU 30, which may be a part of the system 100 as well. The camera control unit, CCU 30 is configured to control a display 40, which may also be part of the system 100, to display images of (or based on) the images acquired by the camera unit 21. In other variants, the analysis module 31 may be integrated into the light source 10 itself, into the endoscope 20, or may even be provided as a cloud service implemented by a cloud computing platform.

The system 100 further comprises a lighting control module 32 configured to control the light source 10 based on the determination of the analysis module 31, i.e. based on whether the endoscope 20 has been determined to be inside the patient’s body 1 or not. As shown in Fig. 1, the lighting control module 32 may also be integrated into the camera control unit, CCU 30.

Specifically, the lighting control module 32 may be configured to control the light source 10 to be in a first, inside mode when the determination indicates that the endoscope 20 is currently inserted into the patient’s body 1, and to be in a second, outside mode when the determination indicates that the endoscope 20 is currently not inserted into the patient’s body 1.

The light source 10 may be configured to emit, in the inside mode (shown in Fig. 1, marked with “IM”), light according to an automatic control algorithm, according to which a output power of the emitted light 2 is automatically adapted based on light incident on the camera unit 21. In other words, the camera unit 21 may act as a light sensor, and images acquired by the camera unit 21 may be seen as raw sensor signals, which are then further processed. In Fig. 1 , a schematic output power bar is shown within the light source 10 that is 90% shaded to indicate that currently, for example, light 2 with an output power of 90% of the maximum output power available to the light source 10 is emitted in accordance with the automatic adaption in the inside mode, IM.

In general, the light incident on the camera unit 21 will be a result of the emitted light 2, be it reflected, re-emitted and/or the like. The automatic control algorithm may be configured to maintain a pre-defined or user-definable level of brightness or visibility, or to maintain a constant value of another kind of metric for measuring the light incident on the camera unit 21, for example regarding the color spectrum of the pixels of the acquired images or the like.

Advantageously, the light source 10 is further configured to emit, in the outside mode (shown in Fig. 2, marked with “OM”), light 2 with a constant output power, in particular an output power smaller than a maximum output power of the light source 10. For example, the output power may be a minimum output power settable with the light source 10 apart from disabling light emission altogether. This has the advantage that there is, in any conceivable scenario, always sufficient light for the camera unit 21 to capture images which are then usable by the analysis module 31 to make its determination. In Fig. 2, a schematic output power bar is shown within the light source 10 that is 5% shaded to indicate that currently, for example, light 2 with an output power of 5% of the maximum output power available to the light source 10 is emitted as the minimum output power in accordance with the constant output power emission in the outside mode, OM.

The light source 10 may also be configured such that, apart from being switched off, it always emits light 2 with a specific (e.g., minimum) amount of output power, as part of all the modes into which it might be set. In other words, all of the modes of which the light source 10 is capable (except a switched-off mode) may be designed such that they comprise at least a minimum amount of output power at all times. In this way, it guaranteed that images can always be acquired by the camera unit 21. This, in turn, is not only useful for the IN/OLIT determination, but also for the user (or: operator) of the endoscope 20 to be able to see, via the display 40 and the acquired images displayed thereby, the position and/or orientation of the endoscope 20.

The camera unit 21 may provide an image output (of acquired images) having a first resolution, for example a 4K-HD signal, intended for viewing by a user of the endoscope 20 and/or a surgeon. The IN/OLIT determination may, however, be based on scaled-down images having a second resolution lower than the first resolution, for examples 512x512 pixels, or 224x224 pixels or the like. It shall be understood that, whenever it is mentioned that the IN/OLIT determination is based on acquired images, this may mean that the determination is directly made on the raw acquired images, and/or on scaled-down images as just described, and/or on other processed data structures generated on the basis of the raw acquired images.

The analysis module 31 may be configured to receive a video stream from the camera unit 21 , and the IN/OLIT determination at any point in time may be made based on one or more of the N most recent frames of the video stream. N herein is an integer such as N<100, preferably N<10, or even N=1. The selection of an appropriate number N of frames to be considered can depend on the specifics of the intended use, the parameter of the endoscope 20 and other such parameters.

For example, a common frame rate of endoscopes 20 is 30 fps (frames per second). Thus, a choice of N corresponds with a time delay between the time the first image to be taken into account for the IN/OLIT determination is available and the last image to be taken into account is available. For example, when N=1 or N= 10 or N = 30 and the frame rate is 30 fps, this would result in a time delay of (about) 1/30 second or 1/3 second or 1 second, respectively.

Equivalently, a buffer time may be defined, wherein all images acquired within an immediately preceding time span of the length of the buffer time are used for the IN/OLIT determination. All of the N images may be considered equally, or they may be considered to a different degree, e.g., subjected to different weights, wherein in particular later images (i.e. images of situations closer to the time of the IN/OIIT determination) may be given comparatively larger weights than earlier images. For example, a weighted mean for each pixel over the N images may be calculated to form a single image used for the IN/OIIT determination. The weights may also be equal for all of the N images in some variants.

Moreover, the IN/OIIT determination may be made based not on consecutive images (step rate of 1) within the video stream but may be made based on images therein that are separated by a step rate of 2 or more, for example a step rate of 5. This may take into account how fast, especially compared to the frame rate, the scenery has been found to generally change in the intended application. If, for example, it has been found that meaningful changes only happen over at least 5 frames, then a step rate of 5 is useful to reduce data traffic and/or necessary computing power and computing time.

The analysis module 31 may be configured to implement a machine-learning model, preferably an artificial neural network, more preferably a convolutional neural network, which may be configured and trained to differentiate between sceneries inside and outside of the human body, and to make the determination based thereon.

Specifically, the convolutional neural network may be of the VGG-16 framework, or a type of MobileNet.

Fig. 3 shows a potential architecture for a machine-learning model, here an artificial neural network 131, illustrated in the known manner by rectangular blocks, according to a variant of the VGG-16 framework. In Fig. 3, as an example, the processing of an RGB input image 71 with 224 x 224 pixels, each of the three color channels carrying a color intensity value typically between 0 and 255, is illustrated. As is shown in Fig. 3, several convolutional layers convl ...conv5 are applied, each followed by a pooling layer. It shall be understood that also higher or lower numbers of convolutional layers may be provided.

Finally, a fully connected layer structure 133 is applied, for example, a densely-connected multilayer perceptron, followed by a sigmoid function (or any other suitable activation function). In this way, the dimensionality of the original input image 71 of 224 x 224 x 3 is transformed to 224 x 224 x 64, then to 112 x 112 x 128, then to 56 x 56 x 256, then to 28 x 28 x 512, then to 14 x 14 x 512, then to 7 x 7 x 512 and then to the IN/OIIT determination made by the fully connected layer structure 133. Fig. 4 illustrates a schematic flow diagram for illustrating a method according to an embodiment of the present invention, i.e., a method for training an artificial neural network 131.

In a step S10, training images comprising images showing sceneries within a human body and images showing sceneries outside of a human body are provided, wherein the training images are labelled accordingly. It has been found that the transition time for transition of an endoscope 20 between the inside and the outside of a patient’s body is so short that training images showing a scenery exactly at the transition stages are unnecessary. If they are included, they may be labelled according to either one of the two labels corresponding to “inside” or “outside”, respectively.

Preferably, for the training images labelled with the label corresponding to “inside”, comprise an image showing a scenery inside a body of at least one woman, an image showing a scenery inside a body of at least one man, an image showing a scenery inside a body of at least one boy, an image showing a scenery inside a body of at least one girl, and/or an image showing a scenery inside a body of at least one cadaver.

The images (those labelled corresponding to “inside” and/or those corresponding to “outside”) may be taken during, at, or immediately before, or immediately after, a laparoscopy and/or a thoracoscopy, or any other similar procedure using an endoscope.

In a step S20, an artificial neural network 131 (which may be pre-trained or randomly initialized) is with the provided training images, using supervised learning. Any of the known techniques for supervised learning may be applied.

The supervised training may be performed, for example, with the following hyperparameters for a MobileNet architecture:

- Number of epochs" 20,

- Batch size: 100,

- Learning rate: 0.001 ,

- Learning rate decay: true,

- Backbone training: true,

- Number of training samples: 18692.

The cost function may be based, for example, on accuracy, or on cross-binary entropy. Original training images may be subjected to data augmentation or alteration (for example, to reduce overfitting), in any known manner. These may include random rotations, distortions, blurs, up- and/or downscaling and the like.

Fig. 5 illustrates a schematic flow diagram for illustrating a method according to an embodiment of the present invention, i.e. , a computer-implemented method for operating a light source 10 for an endoscope 20. The method of Fig. 5 may be performed with any system according to any embodiment of the present invention, or independently of it. Accordingly, the method may be adapted according to any embodiment, option, variant, or refinement that has been described with respect to the system 100, or vice versa.

In a step S100, at least one image 71 is acquired, using a camera unit 21 of an endoscope 20. As has been described in the foregoing, the camera unit 21 may be configured to acquire a continuous video stream of individual images 71 frames, for example with 30 fps.

In a step S200, a determination is made, based on the at least one image 71 , whether the endoscope 20 is currently inserted into a patient’s body 1 or not.

In a step S300, the light source 10 of the endoscope 20 is controlled based on the made determination. In particular, as has been described in the foregoing with respect to the system 100, the light source 10 may be controlled to be in a first, inside mode when the determination yields that the endoscope 20 (specifically: a light emitter 22 of the endoscope 20) is currently inside of the patient’s body 1, and to be in a second, outside mode when the determination yields that the endoscope 20 (specifically: the light emitter 22 of the endoscope 20) is currently outside of the patient’s body 1.

Fig. 6 shows a schematic block diagram illustrating a computer program product 200 according to an embodiment of the fourth aspect of the present invention. The computer program product 200 comprises executable program code 250 configured to, when executed, perform the method according to any embodiment of the second aspect of the present invention and/or the third aspect of the present invention, in particular as has been described with respect to the preceding figures.

Fig. 7 shows a schematic block diagram illustrating a non-transitory computer-readable data storage medium 300 according to an embodiment of the fifth aspect of the present invention. The data storage medium 300 comprises executable program code 350 configured to, when executed, perform the method according to any embodiment of the second aspect of the present invention and/or the third aspect of the present invention, in particular as has been described with respect to the preceding figures.

The non-transient computer-readable data storage medium may comprise, or consist of, any type of computer memory, in particular semiconductor memory such as a solid-state memory. The data storage medium may also comprise, or consist of, a CD, a DVD, a Blu-Ray-Disc, an USB memory stick or the like.

List of reference signs

1 patient s body

2 light

10 light source

20 endoscope

21 camera unit

22 light emitter

30 camera control unit, CCU

31 analysis module

32 lighting control module

40 display

71 input image

100 system

131 artificial neural network

133 fully connected layer structure

200 computer program product

250 program code

300 data storage medium

350 program code

S10, S20 method steps

S100-S300 method steps

Claims

Claims

1. A system (100) for operating a light source (10) for an endoscope (20), comprising: a light source (10); an analysis module (31) configured to receive at least one image (71) from a camera unit (21) of the endoscope (20) and to make a determination (S200), based thereon, whether the endoscope (20) is currently inserted into a patient’s body (1) or not; and a lighting control module (32) configured to control the light source (10) based on the determination of the analysis module (31).

2. The system (100) of claim 1 , wherein the lighting control module (32) is configured to control the light source (10) to be in a first, inside mode (IM) when the determination (S200) indicates that the endoscope (20) is currently inserted into the patient’s body (1), and to be in a second, outside mode (OM) when the determination (S200) indicates that the endoscope (20) is currently not inserted into the patient’s body (1).

3. The system (100) of claim 2, wherein the light source (10) is configured to emit, in the inside mode (IM), light (2) according to an automatic control algorithm, according to which a output power of the emitted light (2) is automatically adapted based on light incident on the camera unit (21).

4. The system (100) of claim 2 or 3, wherein the light source (10) is configured to emit, in the outside mode (OM), light (2) with a constant output power, in particular an output power smaller than a maximum output power of the light source (10).

5. The system (100) of any of claims 1 to 4, wherein the analysis module (31) is configured to implement an artificial neural network (131), in particular a convolutional neural network, configured and trained to differentiate between sceneries inside and outside of the human body, and to make the determination (S200) based thereon.

6. The system (100) of any of claims 1 to 5, wherein the analysis module (31) is configured to receive a video stream from the camera unit (21), and the determination (S200) at any time point is made based on one or more of the N most recent frames of the video stream, N < 100, preferably N<10, most preferably N=1.

7. The system (100) of any of claims 1 to 6, wherein an image output of the camera unit (21) has a first resolution, and wherein the analysis module (31) is configured to make the determination (S200) based on scaled-down images having a second resolution lower than the first resolution.

8. The system (100) of any of claims 1 to 7, wherein the system (100) comprises a camera control unit (30) configured to control and/or process an image acquisition by the camera unit (21) and the analysis module (31) is integrated into the camera control unit (30), or wherein the analysis module (31) is integrated into the light source (10).

9. A computer-implemented method of training an artificial neural network according to claim 5, comprising: providing (S10) training images comprising images showing sceneries within a human body and images showing sceneries outside of a human body, wherein the training images are labelled accordingly; and training (S20) an artificial neural network (131) with the training images using supervised learning.

10. A computer-implemented method for operating a light source for an endoscope, comprising: acquiring (S100) at least one image (71) using a camera unit (21) of an endoscope (20); making a determination (S200), based on the at least one image (71), whether the endoscope (20) is currently inserted into a patient’s (1) body or not; and controlling (S300) the light source (10) of the endoscope (20) based on the made determination (S200).