KR102697147B1 - Control Method of Enhanced Imaging System for an Operating Endoscopic Microscope - Google Patents

Abstract

A method for controlling an endoscopic device including an endoscopic microscope according to the invention includes an endoscopic microscope, including the steps of obtaining a bright field image through a first light source, obtaining a microscope image through a second light source, bringing a microscope window into contact with a sample to obtain the microscope image, calculating a relative position of the microscope image, generating a composite image by synthesizing the bright field image and the microscope image, and outputting the composite image.

Description

{Control Method of Enhanced Imaging System for an Operating Endoscopic Microscope}

The present invention relates to an improved image acquisition system for a surgical endoscopic microscope and a control method thereof.

In addition, the present invention relates to an endoscope head hardware and a control process technology thereof that can simultaneously display a brightfield image and a laser light source-based microscope image (or confocal fluorescence image) with excellent image quality (such as improved transparency or image resolution when switching screens) so that a surgical surgeon can easily distinguish between a surgical site (cancer tissue) and another site (normal tissue) and more intuitively and clearly understand the device position or movement path during movement of the endoscope.

Cancer patients are treated with chemotherapy, radiation, and surgical treatment. Among them, surgical treatment completely removes cancer tissue while the surgeon confirms it through a surgical microscope or endoscopic microscope. Recently, endoscopic microscopes are often used to confirm and remove cancer tissue inside the body through minimal resection to improve the patient's surgical prognosis. In order to easily distinguish between normal tissue and cancer tissue in endoscopic microscope images, a technology is being studied to inject a fluorescent substance that reacts to cancer, acquire a brightfield image and a fluorescent image using two different filters, and synchronize them to display a composite image. However, due to the limited magnification of the endoscopic microscope lens, it is difficult to confirm microscopic cancer tissue and to distinguish the border between normal and cancer tissue.

The advantage of optical microscopes over electron microscopes is that they can capture changes in microorganisms or tissues while they are still alive. Electron microscopes have the advantage of high magnification and high resolution, but because they require preprocessing such as metal coating to observe the sample, and must be measured under a vacuum, the sample may be damaged after observation. Confocal laser scanning microscopes have a resolution that is about 1.4 times better than conventional optical microscopes, and have the advantage of being able to capture microorganisms or tissues while they are still alive.

In particular, confocal microscopy has a more important meaning in the field of life science. Since confocal microscopy can measure depth, it is possible to measure the inside of cells noninvasively. In other words, it is an important experimental device that reveals changes or lesions in the early stages by measuring structural changes in cells or tissues. Recently, research on dynamic phenomena occurring at the cell or subcellular level is the main focus in the field of life science. Since these biological dynamic phenomena occur at a very fast speed, the demand for equipment that can measure them is rapidly increasing.

The principle of the confocal microscope can be understood by looking at the principle diagram shown in Figure 1. First, a beam from a light source is reflected by a beam splitter and directed toward the objective lens. The beam that passes through the objective lens is focused on the focal plane on the sample, and the beam focused on the focal plane is reflected, transmitted, or refracted depending on the characteristics of the sample. Due to this optical phenomenon, light is emitted from other locations than the focal plane. The light reflected on the sample passes through the objective lens again, the beam splitter, and then goes toward the detector. At this time, only the light reflected from the focal plane reaches the detector due to the spatial filter (pinhole) located at the focus and conjugate point, and the light reflected from other locations is blocked by the spatial filter (pinhole) and is not detected, so it has no effect on the final image. Because it only obtains information on the conjugate point of the visual phenomenon corresponding to the position of the pinhole, it is called confocal.

The greatest advantage of these confocal microscopes is that they eliminate light information from non-focus planes and can acquire three-dimensional information, but on the other hand, they do not provide sufficient resolution for observing subcellular structures.

This is a task that needs to be improved in microscopes and endoscopic microscopes that use microscopic images other than confocal fluorescence images.

Republic of Korea Patent Publication No. 10-2203843

The present invention has been made to solve the problems of the above-described prior art, and one object of the present invention is to provide an endoscopic microscope and a control technology thereof which can improve the precision and efficiency of surgery by synthesizing and displaying together a laser light source-based microscopic image or confocal image (microscopic image of the surgical site) and a wide view image (bright field image) including the surgical range during cancer tissue resection surgery.

In addition, it is intended to provide an endoscopic microscope and its control technology that can more clearly distinguish the boundary between normal tissue and cancer tissue.

An endoscopic microscope image acquisition system according to the present invention comprises: an endoscopic microscope head including a first light source capable of implementing a brightfield image capable of viewing a tissue or surgical site as a whole, a second light source capable of implementing a laser light source-based microscope image capable of viewing details of the tissue or surgical site, and a microscope window that contacts a sample to acquire the microscope image; a calculation unit that calculates a relative position of the microscope image; a composite image generation unit that generates a composite image by synthesizing the brightfield image and the microscope image; and an output unit that outputs the composite image.

In the endoscopic microscope image acquisition system according to the present invention, the microscope window can be configured to contact the sample to acquire an image and then retract into position.

In the endoscopic microscope image acquisition system according to the present invention, the synthetic image generation unit may include a synthetic image control unit that calculates coordinates at which the microscope image is acquired from coordinates of the bright-field image, adjusts transparency of the microscope image on top of the bright-field image, and superimposes the microscope image on the bright-field image and outputs it through the output unit.

In the endoscopic microscope image acquisition system according to the present invention, the synthetic image generation unit may include a synthetic image update unit that calculates the distance movement in the x-axis, y-axis, and z-axis from the initial position at the time point when the microscope image was acquired and periodically updates the changed bright field image to generate a synthetic image.

In the endoscopic microscope image acquisition system according to the present invention, if the location of the first time point at which the microscope image was acquired cannot be found in the bright-field image of the second time point thereafter, the microscope image may be configured to separately output a position deviation from the bright-field image, remove the overlapping microscope images, and output only the bright-field image.

The endoscopic device of the present invention includes an endoscopic microscope image acquisition system as described above and a cable detachably connected to the endoscopic microscope head.

The endoscope system of the present invention includes an endoscope device as described above, and a display that is electrically connected to the endoscope device through the cable and can display a composite image output by the output unit.

In the endoscopic microscope image acquisition system according to the present invention, the microscope window can be configured to contact the tissue while moving so as to protrude forward within a fixed angle.

In the endoscopic microscope image acquisition system according to the present invention, the transparency of the microscope image can be adjusted to a state where the tissue structure and location can be determined through the bright field image.

In the endoscopic microscope image acquisition system according to the present invention, in order to calculate the distance movement in the x-axis, y-axis, and z-axis from the initial position at the time point when the microscope image is acquired, a bright-field image containing tissue structure information is acquired together at the moment when the microscope image is taken and stored as a reference image, and thereafter, a new bright-field image is acquired again at a moved position, and feature points of the reference image and the image acquired at the moved position are extracted to calculate the movement in the x and y directions and the distance movement in the z-axis.

In the endoscopic microscope image acquisition system according to the present invention, only a bright field image may be output when the bright field image saved as the reference image does not pass the reference score, and the synthetic image generation unit may be configured to generate a synthetic image only when the bright field image saved as the reference image passes the reference score.

Another endoscopic microscope image acquisition system of the present invention comprises: an endoscopic microscope head including a first light source capable of implementing a brightfield image capable of viewing a tissue or surgical site as a whole, a second light source capable of implementing a confocal image capable of viewing details of the tissue or surgical site, and a confocal window that contacts a sample to acquire the confocal image; a calculation unit that calculates a relative position of the confocal image; a composite image generation unit that generates a composite image by synthesizing the brightfield image and the confocal image; and an output unit that outputs the composite image.

Another endoscopic device of the present invention comprises another endoscopic microscope image acquisition system as described above, and a cable detachably connected to the endoscopic microscope head.

Another endoscopic device of the present invention may further include a nozzle for applying a fluorescent dye to the sample to obtain the fluorescent image.

In the above endoscopic microscope image acquisition systems of the present invention, the first light source may be configured to include a plurality of LEDs to prevent the occurrence of shadows.

In the endoscopic microscope image acquisition system of the present invention, the composite image generation unit may be configured to reflect a weight (α) to the pixel values of the laser light source-based microscope image and (1-α) to the pixel values of the bright field image so that the structure of a tissue can be confirmed in the bright field image even after synthesizing the two images (i.e., the laser light source-based microscope image and the bright field image).

In another endoscopic microscope image acquisition system of the present invention, the composite image generation unit may be configured to reflect a weight (α) to the pixel values of the confocal image and to reflect (1-α) to the pixel values of the bright field image so that the structure of the tissue can be confirmed in the bright field image even after synthesizing the two images (i.e., the confocal image and the bright field image) when generating a composite image of the confocal image and the bright field image.

In the endoscopic microscope image acquisition system of the present invention, calculating the coordinates at which the microscope image is acquired from the bright field image coordinates can be performed by the following formula.

formula

Here,

x ₀ : x-direction pixel position value in the microscope image coordinate system before 2D transformation

y ₀ : y-direction pixel position value in the microscope image coordinate system before 2D transformation

x ₁ : x-direction pixel position value in the bright-field image coordinate system after 2D transformation

y ₁ : y direction pixel position value in the bright field image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: distance moved in the x-axis direction

ty: distance moved in the y-axis direction

A method for controlling an endoscopic device including an endoscopic microscope according to the present invention includes an endoscopic microscope including the steps of obtaining a bright-field image through a first light source, obtaining a confocal image through a second light source, bringing a confocal window into contact with a sample to obtain the confocal image, calculating a relative position of the confocal image, generating a composite image by synthesizing the bright-field image and the confocal image, and outputting the composite image.

In a control method for an endoscopic device including an endoscopic microscope according to the present invention, the step of bringing a confocal window into contact with a sample to obtain the confocal image may include a step of bringing the confocal window into contact with the sample to obtain an image and thereafter moving it back into its original position.

A method for controlling an endoscopic device including an endoscopic microscope according to the present invention additionally includes a step of calculating coordinates at which the confocal image is acquired from coordinates of the bright-field image, and at this time, in the step of outputting the composite image, the coordinates of the calculated confocal image may be used to adjust transparency of the confocal image on top of the bright-field image so as to output the confocal image by overlapping it with the bright-field image.

A method for controlling an endoscopic device including an endoscopic microscope according to the present invention additionally includes a step of calculating a distance movement in the x-axis, y-axis, and z-axis from an initial position at a time point when the confocal image is acquired, and at this time, in the step of outputting the synthesized image, the calculated value of the distance movement can be periodically updated to a changed bright field image to generate a synthesized image.

In a control method of an endoscopic device including an endoscope according to the present invention, in the step of outputting the composite image, if the position of the first time point at which the confocal image was acquired cannot be found in the bright-field image of the second time point thereafter, the confocal image may be configured to separately output a position deviation from the bright-field image, remove the overlapping confocal image, and output only the bright-field image.

In a control method of an endoscopic device including an endoscopic microscope according to the present invention, the confocal window can be configured to move so as to protrude forward within a fixed angle and contact a tissue.

In a control method of an endoscopic device including an endoscopic microscope according to the present invention, the transparency of the confocal image can be configured to be adjusted to a state where the tissue structure and location can be determined through the bright field image.

In a control method of an endoscopic device including an endoscopic microscope according to the present invention, in order to calculate the distance movement in the x-axis, y-axis, and z-axis from the initial position at the time point when the confocal image is acquired, a bright-field image containing tissue structure information is acquired together at the moment when the confocal image is taken and stored as a reference image, and thereafter, a new bright-field image is acquired again at a moved position, and feature points of the reference image and the image acquired at the moved position are extracted to calculate the movement in the x and y directions and the distance movement in the z-axis.

In a control method of an endoscopic device including an endoscopic microscope according to the present invention, only a bright field image may be output when the bright field image stored as the reference image does not pass a reference score, and the synthetic image generation unit may be configured to generate a synthetic image only when the bright field image stored as the reference image passes a reference score.

In addition, another control method of an endoscopic device including an endoscopic microscope according to the present invention may include a step of obtaining a bright field image through a first light source, a step of obtaining a laser light source-based microscope image through a second light source, a step of contacting a microscope window with a sample to obtain the microscope image, a step of calculating a relative position of the microscope image, a step of generating a composite image by synthesizing the bright field image and the microscope image, and a step of outputting the composite image.

In addition, another control method of an endoscopic device including an endoscopic microscope according to the present invention, the step of generating a composite image by synthesizing the bright-field image and the confocal image may be configured such that, in generating a composite image of the confocal image and the bright-field image, a weight (α) is reflected in the pixel values of the confocal image so that the structure of the tissue can be confirmed in the bright-field image even after synthesizing the two images (i.e., the confocal image and the bright-field image), and (1-α) is reflected in the pixel values of the bright-field image.

In addition, another control method of an endoscopic device including an endoscopic microscope according to the present invention, the step of generating a composite image by synthesizing the bright-field image and the microscope image may be configured such that, in generating a composite image of the microscope image and the bright-field image, a weight (α) is reflected to the pixel values of the microscope image based on the laser light source so that the structure of the tissue can be confirmed in the bright-field image even after synthesizing the two images (i.e., the microscope image and the bright-field image), and (1-α) is reflected to the pixel values of the bright-field image.

In addition, another control method of an endoscopic device including an endoscopic microscope according to the present invention may be configured such that the step of calculating the coordinates at which the confocal image is acquired from the coordinates of the bright field image is performed by the following formula.

formula

Here,

x ₀ : x-direction pixel position value in the confocal image coordinate system before 2D transformation

y ₀ : y-direction pixel position value in the confocal image coordinate system before 2D transformation

x ₁ : x-direction pixel position value in the bright-field image coordinate system after 2D transformation

y ₁ : y direction pixel position value in the bright field image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: distance moved in the x-axis direction

ty: distance moved in the y-axis direction

In addition, another control method of an endoscopic device including an endoscopic microscope according to the present invention may be configured to further include a step of applying a fluorescent dye to the sample through a nozzle of the endoscopic microscope to obtain the fluorescence image, wherein the confocal image is a fluorescence image obtained by collecting fluorescence emitted from the sample.

According to the improved image acquisition system of a surgical endoscopic microscope and the control method thereof according to the present invention, the surgeon can confirm a high-resolution laser light source-based microscopic image or confocal image through the endoscope as a brightfield image during surgery for surgical treatment of a cancer patient, thereby enabling detection of cancer at the cellular level that is difficult to confirm with a brightfield image.

In addition, laser light source-based microscopic images or confocal fluorescence images can be viewed in real time on a single screen, like brightfield images, allowing surgeons to cut tissue or mark tissue to remember its location, enabling accurate removal of cancerous tissue.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a drawing explaining the principle of a confocal microscope.
FIG. 2a is a drawing showing an exemplary embodiment of the physical shape of a head for an endoscopic microscope according to the present invention.
Figure 2b is a conceptual diagram explaining the operation method of the confocal window of the head for an endoscopic microscope according to the present invention.
FIG. 3 is a schematic diagram showing components of an exemplary embodiment of an endoscopic microscope image acquisition system including an endoscopic microscope head according to the present invention.
FIG. 4 is a schematic diagram showing detailed components of a synthetic image generation unit of an exemplary embodiment of an endoscopic microscope image acquisition system according to the present invention.
Figure 5 is a flow chart of an exemplary embodiment of a control method for an endoscopic microscope image acquisition system according to the present invention.
Figure 6 is a schematic diagram showing the configuration of an endoscope system according to the present invention.
Figure 7 is a flow chart of another embodiment of a control method for an endoscopic microscope image acquisition system according to the present invention.

The advantages and features of the present invention, and the method for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are provided only to make the disclosure of the present invention complete and to fully inform a person skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

Corresponding components are given similar reference numerals throughout this specification.

In this specification, "and/or" includes each and every combination of one or more of the items mentioned. In this specification, the singular also includes the plural unless specifically stated otherwise. Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in the sense that they are commonly understood by those of ordinary skill in the art. In addition, terms defined in the dictionary shall not be ideally or excessively interpreted unless explicitly specifically defined.

In order to achieve the purpose of the invention as described above, a lens capable of obtaining a bright-field image and a confocal lens capable of confirming the structural information of cells can be configured together in an endoscopic microscope head. Through this, two lenses can be configured together to obtain two images at the same time. However, since real-time images can be obtained only in a very small field of view of the mm level through the confocal lens, even if a high-resolution image is obtained with the confocal lens, there is a problem that it is difficult for a surgeon to respond, such as resecting tissue or marking tissue to remember the location. One of the main purposes of the present invention is to improve this, which will be described in detail below with reference to the drawings.

In this specification, for convenience's sake and to avoid redundant explanation, the description is based on a confocal image using a fluorescent material. However, all descriptions of the present invention are applicable to a laser light source-based microscope image that is not a confocal method, except for descriptions of a nozzle (160) that may be included only when a fluorescent material is used, a step (S310) of staining cells of a living tissue with a fluorescent dye, etc. Meanwhile, a laser light source-based microscope image should be understood as a concept that naturally includes a confocal image (also a laser method).

Referring to FIGS. 2 and 3, an endoscopic microscope head (100) according to the present invention includes a first light source (110, 120) capable of implementing a bright field image, a second light source (132) capable of implementing a confocal image, and a confocal objective lens (131) that contacts a sample to obtain the confocal image.

In addition, the endoscopic microscope image acquisition system (1000) of the present invention includes a head (100) as described above, a calculation unit (140) that calculates the relative position of the confocal image, a synthetic image generation unit (150) that generates a synthetic image by synthesizing the bright field image and the confocal image, and an output unit (180) that outputs the synthetic image.

The two light sources (110, 120) of the head (100) for the endoscopic microscope according to the present invention may each be configured using one LED, and the second light source (132) capable of implementing a confocal image may be configured to irradiate a laser having a wavelength range of 400 to 800 nm.

In the present invention, it is preferable that the first light source (110, 120) capable of obtaining a bright-field image is composed of a plurality of first light sources, for example, two (reference numerals 110, 120) as in FIG. 2a. This is because the first light source (110, 120) is composed of an LED, and thus, unlike the second light source (132) capable of implementing a confocal image composed of a laser light source, it is necessary to prevent the occurrence of a shadow. However, the first light source capable of obtaining a bright-field image of the present invention is not necessarily limited to a plurality of or two light sources, and may be composed of a single first light source.

The above first light source (110, 120) is generally understood as a concept including a separate optical lens (or light source lens) in the present technical field, and therefore the first light source (110, 120) should be understood as the optical lens itself that irradiates light from the light source or a configuration including the optical lens.

Meanwhile, the second light source (132) should also be understood as a concept including a separate optical lens in the present technical field, just like the first light source (110, 120), and additionally, in the present specification, in the second light source (132) capable of implementing a confocal image, the objective lens that contacts the sample, that is, the confocal objective lens (131), is separately indicated as the reference numeral 131 in FIGS. 2A and 2B to distinguish it from the optical lens that irradiates the light of the light source, and for convenience, the configuration including the two is expressed as a confocal window (130). However, in general, in the present technical field, the terms confocal window (130), confocal objective lens (131), and second light source (132) of the confocal window are used interchangeably, and thus, in the present specification, these terms are used interchangeably as necessary.

In addition, although not shown in FIG. 3, the head (100) for an endoscope microscope may be configured to include a nozzle (160) for applying a dyeing substance to obtain a fluorescent signal using the fluorescent dyeing substance, as in FIG. 2a.

Strictly speaking, the above nozzle (160) is a component distinct from the above endoscope head (100), and can be described as forming an endoscope device together with a cable (300) that detachably connects the above endoscope head (100). The term 'detachably connected' includes 'optically connected'.

In addition, the first light source (110, 120) is optically connected to a light source lens positioned at the rear of the endoscope microscope head (100). The light source lens allows light from the light source (110, 120) to pass through and illuminate a sample.

In addition, the cable (300) and the endoscopic microscope image acquisition system (1000) included in the present specification are referred to as an endoscopic device, and the physical hardware including a display (200) capable of displaying a composite image output by the output unit (180) is referred to as an endoscopic system in the present specification (see FIG. 6). As described above, the endoscopic microscope head (100) should be understood as an element of the endoscopic microscope image acquisition system (1000).

The above fluorescence signal is intended to improve the accuracy of the confocal image, and in the present invention, it is not necessary to necessarily add a fluorescence signal to obtain a confocal image, that is, it is not necessary to necessarily use a fluorescent dye, and it is possible to obtain a confocal image without optionally adding a fluorescence signal, and therefore, in this case, it is also possible to not employ a nozzle (160) for applying a dye to obtain a fluorescence signal. In this respect, FIG. 3, which is a schematic diagram showing components of an exemplary embodiment of an internal system of an endoscopic microscope image acquisition system (1000) according to the present invention showing common basic elements, does not illustrate such a nozzle (160).

In addition, the endoscope microscope head (100) of the present invention includes an objective lens (170) that primarily forms a bright field image of a specimen formed by light irradiated through a light source lens by the light source (110, 120).

As with the above nozzle (160), the objective lens (170) is a pure hardware or physical component of the endoscopic microscope head (100) and is not separately illustrated in FIG. 3, which describes the system control components of the endoscopic microscope image acquisition system (1000). Conversely, the calculation unit (140), the synthetic image generation unit (150), and the output unit (180) are pure system control components and are not separately illustrated in FIG. 2. Therefore, understanding the hardware and software configuration of the endoscopic microscope image acquisition system (1000) should comprehensively consider FIGS. 2 and 3.

In addition, the device may be configured to additionally include a confocal objective lens guide (133) for contacting a confocal optical lens to obtain a high-resolution confocal image of a fluorescence signal while simultaneously obtaining a bright-field image, and a driving system unit (132) for changing the position of the lens.

The above driving system unit (132) is configured to be integrated with the second light source and is given the same drawing symbol, but depending on the system configuration, the second light source and the driving system unit may be configured as separate units.

Each component of the endoscopic microscope image acquisition system (1000) according to the present invention may include the function and role of one component among the other components, but conversely, the function and role of one component may be configured to be performed by two or more components.

A method for controlling an image acquisition system of a surgical endoscopic microscope in the present invention is described with reference to FIGS. 5 and 7, etc.

A method for controlling an image acquisition system of an endoscopic microscope according to the present invention or a method for controlling an endoscopic device including an endoscopic microscope (see FIG. 5) includes a step of acquiring a bright-field image through a first light source (S210), a step of acquiring a confocal image through a second light source (S220), a step of bringing a confocal window (strictly speaking, a confocal objective lens) into contact with a sample to acquire the confocal image (S230), a step of calculating a relative position of the confocal image (S240), a step of generating a composite image by synthesizing the bright-field image and the confocal image (S250), and a step of outputting the composite image (S260).

A method for controlling an image acquisition system of an endoscope microscope according to the present invention of FIG. 7 or a method for controlling an endoscopic device including an endoscope microscope is a partial abbreviation of the method of FIG. 5 and a partial abbreviation thereof and includes detailed or separate control conditions. The method of FIG. 7 additionally includes a step (S310) of staining cells of a living tissue with a fluorescent dye. This can be understood as a step of preliminary work for such image acquisition rather than a method for controlling an image acquisition system of an endoscope microscope or a method for controlling an endoscopic device including an endoscope microscope.

In addition, the confocal image acquisition/image processing (S320) corresponds to the step of acquiring a confocal image through the second light source in Fig. 5 (S220) and the step of bringing the confocal window into contact with the sample to acquire the confocal image (S230). In addition, the brightfield image acquisition step (S340) may correspond to the step of acquiring a brightfield image through the first light source in Fig. 5 (S210).

What is important is that in the embodiment of Fig. 7, depending on whether the reference score is passed in matching the reference bright field image and the feature point (S350), the output unit (180) may output only the bright field image (S370), or the synthetic image generation unit (150) may generate a synthetic image of the bright field image and the confocal image (S360), and then the output unit (180) may output the synthetic image only (S380).

The purpose of using the criterion score as above to determine whether to pass or fail is to limit cases where the feature matching algorithm extracts features from two images that are not identical and matches values that are as similar as possible, which reduces accuracy (precision).

Meanwhile, the bright field image is output on the monitor in real time so that the surgeon can view it. In order to obtain the tissue composition information dyed with fluorescent dye in the field of view of the bright field image, a dye is applied to the tissue from a fluorescent dye nozzle. The confocal image is a detailed image of the body tissue, and to obtain the image, the confocal window (130) moves to protrude forward within a fixed angle and contacts the tissue, as shown in Fig. 2b.

In the present invention, the reason why the confocal window (130) moves so as to protrude forward within a fixed angle and contacts the tissue is because a high-resolution image with a high numerical aperture (NA) can be obtained only when contact is made, and the reason why it moves at a fixed angle is to transform the coordinate system when combining a bright-field image and a confocal image. That is, if the angle is arbitrarily changed, the coordinate system cannot be predicted. At this time, the fixed angle means that the confocal objective lens (131) moves left and right in the up-and-down direction, that is, the direction looking at FIG. 2b, and cannot move up and down in the left and right direction, that is, the direction looking at FIG. 2b. The upper drawing in FIG. 2b shows the appearance when the confocal objective lens (131) protrudes, that is, approaches closely to the sample, and the lower drawing shows the appearance when the confocal objective lens (131) recedes, that is, moves away from the sample.

The above confocal objective lens (131) is an objective lens for obtaining microscopic (cellular) images of tissues, and is distinguished from an objective lens (170) for obtaining bright-field images.

Meanwhile, when in contact with body tissue, the confocal image finishes capturing within 1 second at one position (a certain field of view), and at the same time, the confocal window (130) moves to the rear, which is the initial position, as shown in the lower drawing of FIG. 2B. The coordinates at which the confocal image is acquired are calculated from the coordinates of the bright-field image, and the transparency of the confocal image is adjusted and superimposed on the bright-field image to show it. To this end, as illustrated in FIG. 4, a synthetic image control unit (1510) may be included in the synthetic image generation unit (150) of the endoscopic microscope head (100).

The transparency of the above confocal image can be configured to adjust the transparency of the confocal image to a state where the tissue structure and location can be determined through the bright field image. The confocal image can be configured as an image that expresses a probability map in the form of a heat map through image processing and deep learning operations rather than the original image. In other words, the probability map expressing the probability of cancer tissue at each location is expressed in the form of a heat map.

As the endoscope head (100) is shaken and changes position left and right and front and back by the surgical surgeon, a change occurs in the position at the time the confocal image was taken. At this time, the synthetic image update unit (1520) of the synthetic image generation unit (150) calculates the movement in the x and y directions and the distance movement in the z axis from the initial position at the time the confocal image was taken, and periodically updates the changed bright-field image to show an overlapping image. Additionally, the acquired confocal image can be configured to be shown overlapping with the bright-field image in the same way. In addition, if the position at the time when the initial confocal image was acquired cannot be found in the current bright-field image, a position deviation can be output to the monitor, the overlapping confocal image can be removed, and only the bright-field image can be output to the monitor.

Meanwhile, the above-mentioned overlapping confocal image means that at some point, a confocal image and a brightfield image were synthesized and displayed, and then, when the confocal image was moved to a position where it cannot be found on the screen, it means a confocal image synthesized with the brightfield image on the synthesized image.

Although this specification has been described based on the synthesis of a “confocal image” into a bright-field image, this is only an exemplary description and should be understood to comprehensively include any laser light source-based microscopic image as long as it can obtain an image of the specimen.

Meanwhile, when calculating the movement in the x, y directions and the distance movement in the z-axis from the initial position at the moment the confocal image was taken, the method for calculating the distance movement is to acquire a bright-field image containing the tissue structure information at the moment the confocal image was taken as a reference image. Acquire a bright-field image again at the moved position, extract feature points of the reference image and the image acquired at the most recently moved position, and calculate the movement in the x, y directions and the distance movement in the z-axis using a feature point matching algorithm.

After the above calculation, let's look at the specific cycle and updating method when periodically updating the confocal image to the brightfield image. When the slowest frame speed of the camera used for the brightfield image is set as 10 frames/s, the frame speed of the brightfield image actually shown to the user is lowered to 5 frames/s, and the update is made to 5 frames/s or more considering the calculation speed of the feature matching algorithm through the image. The update method is to replace the image synthesized on the screen with the most recently synthesized image and display it.

In the present invention, the reason for slowing down the frame rate (10 → 5 frames/s) and updating as described above is that the camera's video speed needs to be 10 frames/s to be a speed that is not awkward for humans to perceive, and the reason why the speed corresponding to half of that was set as the standard for the update speed is that if the video processing time is added to the video acquisition time, it cannot but be slower than 10 frames/s.

In other words, if the speed at which images are acquired from the camera is 0.1 seconds per frame (10 frames/s), and it takes about 0.1 seconds more to acquire the image and perform image processing (coordinate conversion, composition), the actual time it takes to update on the display screen will be 0.1 seconds + 0.1 seconds = 0.2 seconds (5 frames/s), which is slower.

Since it is based on the slowest speed of the camera, it was calculated as 10 frames/s above, but of course it is possible to acquire more than that. For example, it can be used at 20 frames/s. However, in the case of 20 frames/s, if it takes 0.05 seconds to acquire the image and 0.1 seconds to process the image, there will be a problem that the previously acquired images will pile up due to time delay.

Meanwhile, the camera is intended to obtain a bright field image and may be included as a separate component from the objective lens (170). The image obtained through the objective lens (170) may be converted into a digital signal through the camera and analyzed and post-processed in a device such as a PC including a calculation unit (140) or a generation unit (150).

In addition, the purpose of using the above-described feature point matching algorithm is to extract locations that are image-specific in order to find the same tissue structure in each image from two bright field images obtained at different locations at different times, thereby allowing one to know how much the x, y, and z-axis directions have changed from the bright field image obtained at the previous time to the next bright field image. The above-described feature point matching algorithm can generally be configured as an algorithm that extracts and matches feature points through the slope of the brightness change of the image.

The present invention is characterized by employing an algorithm for updating an image while compensating for a coordinate system in real time. The algorithm of the present invention is an algorithm for updating an image while compensating for a coordinate system in real time by transforming a coordinate system through a coordinate system transformation formula described below, synthesizing an image through a synthesis formula for synthesizing an image, and displaying it on a screen so that a user can view it.

In the present invention, in order to synthesize two images aligned in a coordinate system, two image values are added in pixel units, and when adding, a weight (α value) can be considered and calculated as in Equation 1 below.

Equation 1: Calculation formula for the composite result pixel value g(x, y)

Here,

x, y: location values of each pixel in the image

g(x,y): value of each pixel in the synthesized image

f _o (x,y): value of each pixel in the bright field image

f ₁ (x,y): value of each pixel in the confocal image

α: weight

The α value is selected so that the structure of the tissue can be confirmed in the brightfield image even after synthesizing the two images, and so that a probability heat map of the presence or absence of cancer tissue can be expressed. Using the α value, the ratio of the values of the brightfield image and the confocal image can be adjusted, and the pixel values of the two images can be adjusted, and the two adjusted values can be combined to create an image. A conceptual example of this is diagrammed as follows.

[Figure 1: Concept of synthesizing bright-field and confocal images according to the calculation formula for the pixel value g(x, y) of the synthesizing result]

In addition, in the present invention, when calculating the coordinates at which a confocal image is acquired in the bright-field image coordinates, when the scaling value between the bright-field image and the confocal image (i.e., the magnification (enlargement or reduction) value of the bright-field image and the confocal image), the rotation angle, and the movement distance (x, y) value are known, a confocal image in the bright-field image coordinate system can be obtained by calculating a 2D transformation for each pixel of the confocal image through Equation 2 below.

Equation 2:

Here,

x ₀ : x-direction pixel position value in the confocal image coordinate system before 2D transformation

y ₀ : y-direction pixel position value in the confocal image coordinate system before 2D transformation

x ₁ : x-direction pixel position value in the bright-field image coordinate system after 2D transformation

y ₁ : y direction pixel position value in the bright field image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: distance moved in the x-axis direction

ty: distance moved in the y-axis direction

The above rotation angle refers to the rotation angle of the confocal image with respect to the bright field image, and the rotation angle can be generated by moving the objective lens (170) along the confocal objective lens guide (133) using the confocal window (130).

With respect to the above equation 2, more specifically, in order to transform the coordinate value position of the confocal image into the position corresponding to the bright-field image coordinate system, the coordinate values x ₀ , y ₀ are substituted into each variable described above to calculate the coordinate values x ₁ , y ₁ , and then the pixel value of the confocal image corresponding to the coordinates x ₀ , y ₀ is substituted into the position x ₁ , y ₁ to perform the coordinate system transformation of the image.

Meanwhile, staining, fluorescence imaging, and measurement methods using fluorescent materials, which are widely used in biological research, are core technologies for examining lesions in biological tissues, and are useful for diagnostic testing by causing only specific areas of interest among biological organs to express fluorescence.

Currently, only ICG and fluorescein are used as human target fluorescent dyes for blood vessel staining, and there are attempts to utilize fluorescent wavelengths using biological tissue staining materials such as antibiotics that have secured stability in the body. However, the above technology cannot be used to determine the cause of disease in cells in the human body, etc., because the excited light exhibits fluorescent characteristics using the ultraviolet range.

Cancer ranks first in mortality worldwide, and the standard treatments for cancer are known to be surgery, chemotherapy, and radiotherapy. Among them, surgical treatment greatly contributes to increasing the survival rate of cancer patients, and is reported to be the most preferred technique in the medical field. The method of removing cancer lesions during medical treatment requires histological analysis in a pathology laboratory, even though skilled techniques are required. At this time, staining methods used include hematocin-eosin staining methods that allow visual observation, but methods using fluorescence detection methods are still difficult to apply clinically. This is because it is difficult to apply fluorescence staining and luminescent reagents to lesion search, and it is difficult to standardize diagnostic methods. Therefore, the researchers developed a technology to quickly determine the presence or absence of cancer tissue in biopsy tissue using single-photon and multi-photon to improve this problem.

Typically, in the endoscopic microscope according to the present invention, the waiting time from when the dye is applied to the tissue until an image is taken varies depending on the human tissue, but is between 1 and 5 minutes.

The technology of the endoscopic microscope of the present invention can also be applied to a hand-held type surgical microscope or a hand-held surgical probe.

In addition, it is reasonable to understand that surgical microscope technology for tissue resection, etc., rather than simply an endoscope for examination purposes, is included in the category of the endoscopic microscope of the present invention because it ultimately requires penetration into the tissue of the human body.

In addition, according to the present invention, there is a mode in which the bright field image and the confocal image are always synthesized and displayed as a single image by dividing the display mode, and a mode in which the display screen is divided into two and the bright field image and the confocal image are displayed respectively, and the mode can be configured to be switched in the middle according to the user's preference.

In addition, unlike the conventional technology in which a brightfield image and a fluorescence image are obtained as two images on the same coordinate system through one optical lens with different light sources and filters to obtain a composite image, the present invention can synthesize two images obtained from two different optical lenses.

In the present invention, since the dark-field image is a high-resolution image and the area that can be viewed at one time is a very small area of less than 2 mm, it is difficult to determine the current location in the entire area by looking only at the dark-field image, making it difficult to resect tissue using a surgical tool. Therefore, it is possible to resect tissue while looking at the entire area with a bright-field image, so that surgical convenience and accuracy can be greatly improved.

Finally, the operation process of the endoscope microscope of the present invention will be explained. When the endoscope microscope is turned on, white light is irradiated from the first light source (110, 120), which is an LED light source, and the structural information of the tissue acquired through the objective lens (170), which is an optical lens, is acquired as image information at 10 frames/s or more through the camera. While confirming this image information, the position of the endoscope microscope is adjusted around the cancer tissue, and when a trigger device connected to the nozzle (160) is triggered around the cancer tissue, a small amount of fluorescent dye or tissue dye is applied to the tissue through the nozzle (160). After the (fluorescent) dye is absorbed into the tissue (1 to 2 minutes), the confocal objective lens (131) is protruded through the control unit so that it can come into contact with the tissue to obtain a confocal image. When it is confirmed through a bright-field image that the confocal objective lens (131) is in contact with the tissue, the acquisition of the confocal image is started through the control unit. By changing the position of the endoscopic microscope and repeating the method described above on the suspicious area, the presence or absence of cancerous tissue is determined.

The term 'endoscopic microscope' used in the present invention is also used as an endoscopic microscope, a microendoscopic microscope, a microscope endoscope, etc., and can be generally understood as an endoscope with a microscope attached to it that can observe or photograph the appearance of a tissue being explored by the endoscope in an enlarged manner.

Although the embodiments of the present invention have been described in detail with reference to the attached drawings, it will be understood by those skilled in the art that the present invention may be implemented in other specific forms without changing the technical idea or essential features thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

In addition, the attached drawings are not drawn to scale, but are partially enlarged and reduced to illustrate the technical idea of the present invention.

It is also obvious that embodiments that selectively combine components and steps of the system, method, or device of the present invention described above also fall within the scope of the present invention.

100: Endoscope head
110, 120: First light source
130: Confocal window/microscope window
131: Confocal objective/microscope objective
132: Second light source
133: Confocal Objective Guide/ Microscope Objective Guide
140: Calculator
150: Synthetic image generation unit
160: Nozzle
170: Objective lens
180: Output section
1000: Endoscopic Microscope Image Acquisition System

Claims (14)

Step of obtaining a bright field image through the first light source;
Step of obtaining a microscope image through a second light source;
A step of bringing the microscope window into contact with the sample to obtain the above microscope image,
A step of calculating the relative position of the above microscope images,
A step of generating a composite image by synthesizing the above bright field image and the above microscope image, and
A step of outputting the above composite image is included,
A method for controlling an endoscopic device including an endoscopic microscope, characterized in that the step of generating a composite image by synthesizing the above-described bright-field image and the above-described microscope image is configured to reflect a weight (α) to the pixel values of the above-described microscope image so that the structure of the tissue can still be confirmed in the above-described bright-field image even after synthesizing the two images (i.e., the above-described microscope image and the above-described bright-field image), and to reflect (1-α) to the pixel values of the above-described bright-field image. In the first paragraph,
The step of bringing the microscope window into contact with the sample to obtain the above microscope image is:
A control method for an endoscopic device including an endoscopic microscope, characterized in that it includes a step of the microscope window contacting the sample to acquire an image and then retracting into position. In the first paragraph,
Additionally comprising a step of calculating the coordinates at which the above microscope image is acquired from the coordinates of the bright field image,
At this time, in the step of outputting the composite image, the control method of an endoscopic device including an endoscopic microscope is characterized in that the coordinates of the calculated microscope image are used to adjust the transparency of the microscope image on the bright field image and the microscope image is output by overlapping it on the bright field image. In the first paragraph,
It additionally includes a step of calculating the distance movement along the x-axis, y-axis, and z-axis from the initial position at the time when the above microscope image was acquired,
At this time, in the step of outputting the synthesized image, a control method for an endoscopic device including an endoscopic microscope is characterized in that the calculated value of the distance movement is periodically updated to the changed bright field image to generate a synthesized image. In the first paragraph,
A control method for an endoscopic device including an endoscopic microscope, characterized in that in the step of outputting the above-mentioned composite image, if the location of the first time point at which the above-mentioned microscope image was acquired cannot be found in the bright-field image of the second time point thereafter, the microscope image is separately output as being out of position from the bright-field image, the overlapping microscope images are removed, and only the bright-field image is output. In the first paragraph,
A control method for an endoscopic device including an endoscopic microscope, wherein the microscope window is configured to contact a tissue while moving so as to protrude forward within a fixed angle. In the third paragraph,
A control method for an endoscopic device including an endoscopic microscope, characterized in that the transparency of the above-mentioned microscopic image is adjusted to a state where the tissue structure and location can be judged through the above-mentioned bright-field image. In paragraph 4,
A control method for an endoscopic device including an endoscopic microscope, characterized in that the method comprises: acquiring a bright field image containing information on the tissue structure at the moment when the microscope image was taken in order to calculate the distance movement in the x-axis, y-axis, and z-axis from the initial position at the time when the microscope image was acquired, storing the image as a reference image, and then acquiring a new bright field image again at the moved position, extracting feature points of the reference image and the image acquired at the moved position to calculate the movement in the x-axis, y-axis, and the distance movement in the z-axis. In Article 8,
If the bright field image saved as the above reference image does not pass the reference score, only the bright field image is configured to be output.
A control method for an endoscopic device including an endoscopic microscope, characterized in that the synthetic image generating unit is configured to generate a synthetic image only when the bright field image saved as the above-mentioned reference image passes the reference score. Step of obtaining a bright field image through the first light source;
A step of obtaining a laser light source-based microscope image through a second light source;
A step of bringing the microscope window into contact with the sample to obtain the above microscope image,
A step of calculating the relative position of the above microscope images,
A step of generating a composite image by synthesizing the above bright field image and the above microscope image, and
A step of outputting the above composite image is included,
A method for controlling an endoscopic device including an endoscopic microscope, wherein the step of generating a composite image by synthesizing the above-described bright-field image and the above-described microscope image is configured to reflect a weight (α) to the pixel values of the laser light source-based microscope image so that the structure of the tissue can still be confirmed in the bright-field image even after synthesizing the two images (i.e., the above-described microscope image and the above-described bright-field image), and to reflect (1-α) to the pixel values of the above-described bright-field image. delete delete Step of obtaining a bright field image through the first light source;
Step of obtaining a microscope image through a second light source;
A step of bringing the microscope window into contact with the sample to obtain the above microscope image,
A step of calculating the relative position of the above microscope images,
A step of generating a composite image by synthesizing the above bright field image and the above microscope image, and
comprising a step of outputting the above composite image,
Additionally comprising a step of calculating the coordinates at which the above microscope image is acquired from the coordinates of the bright field image,
At this time, in the step of outputting the composite image, the coordinates of the calculated microscope image are used to adjust the transparency of the microscope image on top of the bright field image, and the microscope image is output by overlapping it on the bright field image.
A control method for an endoscopic device including an endoscopic microscope, characterized in that the step of calculating the coordinates of the acquired microscope image from the coordinates of the bright field image is configured to be performed by the following formula.
formula

Here,
x ₀ : x-direction pixel position value in the microscope image coordinate system before 2D transformation
y ₀ : y-direction pixel position value in the microscope image coordinate system before 2D transformation
x ₁ : x-direction pixel position value in the bright-field image coordinate system after 2D transformation
y ₁ : y direction pixel position value in the bright field image coordinate system after 2d transformation
Sx: scaling value in the x-axis direction
Sy: scaling value in the y-axis direction
θ: rotation angle
tx: distance moved in the x-axis direction
ty: distance moved in the y-axis direction In the first paragraph,
The above microscope image is a fluorescence image obtained by collecting the fluorescence emitted by the sample.
A control method for an endoscopic device including an endoscopic microscope, characterized in that it further includes a step of applying a fluorescent dye to the sample through a nozzle of the endoscopic microscope to obtain the fluorescence image.