KR20230125676A - Enhanced Imaging System for an Operating Endoscopic Microscope - Google Patents

Abstract

The endoscopic microscope image acquisition system according to the present invention implements a microscope image based on a first light source capable of implementing a brightfield image capable of viewing a tissue or surgical site as a whole and a laser light source capable of viewing details of the tissue or surgical site. a head for an endoscope microscope including a second light source capable of detecting and a microscope window contacting a sample to acquire the microscope image; a calculation unit that calculates a relative position of the microscope image; a synthesized image generating unit generating a synthesized image by synthesizing the brightfield image and the microscope image; and an output unit outputting the synthesized image.

Description

Image acquisition system of an improved surgical endoscopic microscope {Enhanced Imaging System for an Operating Endoscopic Microscope}

The present invention relates to an image acquisition system of an improved surgical endoscopic microscope and a control method thereof.

In addition, the present invention allows the surgeon to easily distinguish between the surgical site (cancer tissue) and other sites (normal tissue) and more intuitively and clearly grasp the location of the device or its movement path during the movement of the surgical endoscopic microscope, Head hardware for endoscopic microscope and its control process technology that can simultaneously show brightfield images and laser light source-based microscope images (or confocal fluorescence images) with excellent image quality (improvement of transparency or image resolution when switching screens, etc.) will be.

Cancer patients are treated with chemotherapy, radiation, and surgical procedures. Among them, surgical treatment completely removes cancerous tissue while the operating surgeon checks it through a surgical microscope or an endoscope. Recently, an endoscope microscope is often used to identify and remove cancerous tissue inside the human body through minimal resection to improve the patient's surgical prognosis. In order to facilitate the distinction between normal tissue and cancer tissue in an endoscopic microscope image, a fluorescence substance that reacts to cancer is injected, and a brightfield image and a fluorescence image are acquired using a lens with two different filters, and a composite image is obtained by synchronizing them. A technique to show is being studied. However, due to the limited magnification of the lens of an endoscopic microscope, it is difficult to identify fine-sized cancer tissue and it is difficult to distinguish the boundary between normal tissue and cancer tissue.

The advantage of light microscopy compared to electron microscopy is that changes in microorganisms or tissues can be taken in a living state. In the case of an electron microscope, it has the advantage of being able to observe at high magnification and high resolution, but in order to observe a sample, the sample must go through a pretreatment process such as metal coating and must be measured in a vacuum state, so the sample can be damaged after observation. The confocal laser scanning microscope has a resolution about 1.4 times better than conventional optical microscopes, and has the advantage of being able to photograph microorganisms or tissues in a living state.

In particular, confocal microscopy in the field of life science has a more important meaning. Since confocal microscopy is capable of depth-direction measurement, it is possible to measure the inside of a cell non-invasively. In other words, it is an important experimental equipment to reveal changes or lesions in the early stages by measuring structural changes in cells or tissues. Recently, in the field of life science, research on dynamic phenomena occurring in cells or subcellular units is the main focus. Since these biological dynamics occur at a very high speed, the demand for equipment capable of measuring them is rapidly increasing.

The principle of the confocal microscope can be understood by looking at the principle diagram shown in FIG. First, a beam from a light source is reflected by a beam splitter, and the beam is directed toward an objective lens. The light beam passing through the objective lens is focused on the focal plane of the sample, and the light beam focused on the focal plane is reflected, transmitted, or refracted according to the characteristics of the sample. Due to this optical phenomenon, light rays come out at locations other than the focal plane. The rays reflected on the sample pass through the objective lens, pass through the beam splitter, and go in the direction of the detector. At this time, only the rays reflected from the focal plane by the spatial filter (pinhole) present on the focal point and the conjugate point reach the detector, and the rays reflected from other places pass through the spatial filter (pinhole). ) does not affect the final image because it is blocked and not detected. In this way, it is called confocal because only the information of the conjugate point of the visual phenomenon corresponding to the position of the pinhole is obtained.

The biggest advantage of such a confocal microscope is that it removes light information from a non-focal plane and can acquire three-dimensional information, but on the other hand, it does not provide sufficient resolution to observe subcellular unit structures.

This is a problem that needs to be improved in microscopes or endoscopes using microscopic images other than confocal fluorescence imaging.

Republic of Korea Patent Registration No. 10-2203843

The present invention has been made to solve the above-mentioned problems of the prior art, and one object of the present invention is a laser light source-based microscope image or confocal image (microscopic image of a surgical site) and surgical range in cancer tissue resection surgery. It is intended to provide an endoscopic microscope and its control technology that can increase the precision and efficiency of surgery by synthesizing wide-view images (bright-field images) including .

In addition, it is intended to provide an endoscopic microscope capable of more clearly distinguishing the boundary between normal tissue and cancer tissue and a control technology thereof.

The endoscopic microscope image acquisition system according to the present invention implements a microscope image based on a first light source capable of implementing a brightfield image capable of viewing a tissue or surgical site as a whole and a laser light source capable of viewing details of the tissue or surgical site. a head for an endoscope microscope including a second light source capable of detecting and a microscope window contacting a sample to acquire the microscope image; a calculation unit that calculates a relative position of the microscope image; a synthesized image generating unit generating a synthesized image by synthesizing the brightfield image and the microscope image; and an output unit outputting the synthesized image.

In the endoscopic microscope image acquisition system according to the present invention, the microscope window may be configured to come into contact with the sample to obtain an image and then retract to the original position.

In the endoscopic microscope image acquisition system according to the present invention, the synthesized image generation unit calculates the coordinates obtained from the microscope image from the brightfield image coordinates and adjusts the transparency of the microscope image on the brightfield image to obtain the microscope image. and a synthesized image control unit overlapping the brightfield image and outputting the image through the output unit.

In the endoscopic microscope image acquisition system according to the present invention, the synthesized image generation unit calculates distance movement in the x-axis, y-axis, and z-axis from the initial position at the time when the microscope image is acquired, and periodically updates the changed brightfield image. It may include a synthesized image updating unit that generates a synthesized image.

In the endoscopic microscope image acquisition system according to the present invention, when the position of the first viewpoint from which the microscope image was acquired cannot be found in the brightfield image of the second viewpoint thereafter, the microscope image is out of position from the brightfield image. It may be configured to output separately, remove superimposed microscope images, and output only brightfield images.

The endoscope device of the present invention includes the above-described endoscopic microscope image acquisition system and a cable detachably connected to the endoscopic microscope head.

The endoscope system of the present invention includes the endoscope device as described above, and a display electrically connected to the endoscope device through the cable and capable of displaying a synthesized image output by the output unit.

In the endoscopic microscope image acquisition system according to the present invention, the microscope window may be configured to contact the tissue while moving so as to project forward at a fixed angle.

In the endoscopic microscope image acquisition system according to the present invention, the transparency of the microscope image can be adjusted to a state in which the tissue structure and position can be determined through the brightfield image.

In the endoscopic microscope image acquisition system according to the present invention, in order to calculate the x-axis, y-axis, and z-axis distance movement from the initial position at the time the microscope image was acquired, the tissue structure information of the tissue at the moment the microscope image was taken is The included brightfield image is acquired together and stored as a reference image, and then a new brightfield image is acquired again from the moved position, and the feature points of the reference image and the image acquired from the moved position are extracted to move in the x and y directions It may be configured to calculate the distance movement in the z-axis and the z-axis.

In the endoscopic microscope image acquisition system according to the present invention, only the brightfield image is output when the brightfield image stored as the reference image does not pass the reference score, and the brightfield image stored as the reference image passes the reference score. In some cases, the synthesized image generating unit may be configured to generate a synthesized image.

Another endoscopic microscope image acquisition system of the present invention includes a first light source capable of realizing a bright field image capable of viewing a tissue or surgical site as a whole, and a first light source capable of implementing a confocal image capable of viewing details of the tissue or surgical site. an endoscope microscope head including two light sources and a confocal window contacting a sample to obtain the confocal image; a calculation unit for calculating a relative position of the confocal image; a synthesized image generating unit generating a synthesized image by synthesizing the brightfield image and the confocal image; and an output unit outputting the synthesized image.

Another endoscopic device of the present invention includes another endoscopic microscope image acquisition system as described above and a cable detachably connected to the endoscopic microscope head.

Another endoscope device of the present invention may further include a nozzle for applying a fluorescent dye to the sample to obtain the fluorescent image.

In the above endoscopic microscope image acquisition systems of the present invention, the first light source may be composed of a plurality of LEDs to prevent shadow generation.

In the endoscopic microscope image acquisition system of the present invention, in generating a synthesized image of the laser light source-based microscope image and the brightfield image, the synthesized image generator generates two images (ie, the laser light source-based microscope image and the A weight (α) is reflected in the pixel value of the microscope image based on the laser light source so that the structure of the tissue can be confirmed in the brightfield image even after synthesizing the brightfield image), and the pixel value of the brightfield image is (1 -α) can be configured to reflect.

In another endoscopic microscope image acquisition system of the present invention, in generating a synthesized image of the confocal image and the brightfield image, the synthesized image generator generates two images (ie, the confocal image and the brightfield image) configured to reflect a weight (α) to the pixel value of the confocal image and to reflect (1-α) to the pixel value of the brightfield image so that the tissue structure can be confirmed in the brightfield image even after synthesizing It can be.

In the endoscopic microscope image acquisition system of the present invention, calculating the coordinates where the microscope image is acquired from the coordinates of the brightfield image may be configured to be performed by the following equation.

formula

here,

x ₀ : x-direction pixel position value in the microscope image coordinate system before 2d transformation

y ₀ : y-direction pixel position value in the microscope image coordinate system before 2d transformation

x ₁ : x-direction pixel position value in the brightfield image coordinate system after 2d transformation

y ₁ : y-direction pixel position value in the brightfield image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: movement distance in the x-axis direction

ty: movement distance in the y-axis direction

A method for controlling an endoscope device including an endoscope microscope according to the present invention includes obtaining a brightfield image through a first light source, acquiring a confocal image through a second light source, and obtaining the confocal image through a second light source. Contacting the confocal window to the sample, calculating the relative position of the confocal image, generating a synthesized image by synthesizing the brightfield image and the confocal image, and outputting the synthesized image. Including an endoscope microscope that includes.

In the control method of an endoscope device including an endoscope microscope according to the present invention, the step of contacting the confocal window to the sample to acquire the confocal image includes: contacting the confocal window to the sample to obtain an image, and then It may include retracting into position.

The control method of an endoscope device including an endoscope microscope according to the present invention further includes calculating coordinates obtained from the confocal image from the brightfield image coordinates, and at this time, in the step of outputting the synthesized image, , The confocal image may be configured to superimpose the confocal image on the brightfield image by adjusting the transparency of the confocal image on the brightfield image using the calculated coordinates of the confocal image.

The method for controlling an endoscope device including an endoscope microscope according to the present invention further includes calculating distance movement in the x-axis, y-axis, and z-axis from an initial position at the time when the confocal image is acquired, and at this time, In the step of outputting the synthesized image, a synthesized image may be generated by periodically updating the calculated value of the distance movement with the changed brightfield image.

In the method for controlling an endoscope device including an endoscope microscope according to the present invention, in the step of outputting the synthesized image, the position of the first viewpoint from which the confocal image was acquired is found in the brightfield image of the second viewpoint thereafter. If it is not possible, the confocal image may be configured to separately output the position deviation from the brightfield image, remove the overlapping confocal image, and output only the brightfield image.

In the method for controlling an endoscope device including an endoscope microscope according to the present invention, the confocal window may be configured to contact a tissue while moving so as to protrude forward at a fixed angle.

In the method for controlling an endoscope device including an endoscopic microscope according to the present invention, the transparency of the confocal image may be adjusted to a state in which the tissue structure and position can be determined through the brightfield image.

In the method for controlling an endoscope device including an endoscope microscope according to the present invention, in order to calculate distance movement in the x-axis, y-axis, and z-axis from the initial position at the time the confocal image was acquired, the moment the confocal image was taken In addition, a brightfield image containing tissue structure information is acquired and stored as a reference image, and then a new brightfield image is acquired from the moved position to extract feature points of the reference image and the image acquired from the moved position. It may be configured to calculate the movement in the x, y directions and the distance movement in the z axis.

In the control method of an endoscope device including an endoscope microscope according to the present invention, only the brightfield image is output when the brightfield image stored as the reference image does not pass the criterion score, and the brightfield image stored as the reference image The synthesized image generating unit may be configured to generate a synthesized image only when the reference score is passed.

In addition, another control method of an endoscope device including an endoscope microscope according to the present invention includes obtaining a brightfield image through a first light source, acquiring a laser light source-based microscope image through a second light source, and the microscope Contacting a microscope window to a sample to obtain an image, calculating a relative position of the microscope image, generating a synthesized image by synthesizing the brightfield image and the microscope image, and outputting the synthesized image steps may be included.

In addition, in another control method of an endoscope device including an endoscope microscope according to the present invention, the step of generating a synthesized image by synthesizing the brightfield image and the confocal image is a synthesized image of the confocal image and the brightfield image. In generating , a weight (α) is applied to a pixel value of the confocal image so that the tissue structure can be confirmed in the brightfield image even after synthesizing two images (ie, the confocal image and the brightfield image). may be reflected, and (1-α) may be reflected in the pixel value of the brightfield image.

In addition, in another control method of an endoscope device including an endoscope microscope according to the present invention, the step of generating a synthesized image by synthesizing the brightfield image and the microscope image generates a synthesized image of the microscope image and the brightfield image In doing so, even after synthesizing two images (ie, the microscope image and the brightfield image), a weight (α) is applied to the pixel value of the laser light source-based microscope image so that the structure of the tissue can be confirmed in the brightfield image. may be reflected, and (1-α) may be reflected in the pixel value of the brightfield image.

In addition, in another method of controlling an endoscope device including an endoscope microscope according to the present invention, the step of calculating the coordinates obtained from the confocal image from the coordinates of the brightfield image may be performed by the following equation.

formula

here,

x ₀ : x-direction pixel position value in the confocal image coordinate system before 2d transformation

y ₀ : y-direction pixel position value in the confocal image coordinate system before 2d transformation

x ₁ : x-direction pixel position value in the brightfield image coordinate system after 2d transformation

y ₁ : y-direction pixel position value in the brightfield image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: movement distance in the x-axis direction

ty: movement distance in the y-axis direction

In addition, in another control method of an endoscope device including an endoscope microscope according to the present invention, the confocal image is a fluorescence image obtained by collecting fluorescence emitted from a sample, and a nozzle of the endoscope microscope is used to obtain the fluorescence image. It may be configured to further include a step of applying a fluorescent dye material to the sample through.

According to the image acquisition system of an improved surgical endoscopic microscope and its control method according to the present invention, a high-resolution laser light source-based microscopic image or confocal image is obtained by an operating surgeon through an endoscope during surgery for surgical treatment of a cancer patient. Since it can be confirmed like a bright field image, it is possible to detect cancer at the cell level, which is difficult to confirm with a bright field image.

In addition, it is possible to check laser light source-based microscope images or confocal fluorescence images in real time on a single screen like brightfield images, enabling surgeons to perform tissue resection or marking to remember the location of the tissue. Cancer tissue can be removed.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram explaining the principle of a confocal microscope.
Figure 2a is a diagram showing an exemplary embodiment of the physical shape of the head for an endoscopy microscope according to the present invention.
Figure 2b is a conceptual diagram explaining the operation method of the confocal window of the endoscope microscope head according to the present invention.
3 is a schematic diagram showing components of an exemplary embodiment of an endoscopic microscope image acquisition system including an endoscopic microscope head according to the present invention;
4 is a schematic diagram showing detailed components of a synthetic image generating unit of an exemplary embodiment of an endoscopic image acquisition system according to the present invention;
5 is a flowchart of an exemplary embodiment of a control method of an endoscopic image acquisition system according to the present invention.
6 is a schematic diagram showing the configuration of an endoscope system according to the present invention.
7 is a flowchart of another embodiment of a control method of an endoscopic image acquisition system according to the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in a variety of different forms, only these embodiments make the disclosure of the present invention complete, and fully inform those skilled in the art of the scope of the invention. It is provided to note that the invention is only defined by the scope of the claims.

Throughout this specification, reference numerals of similar forms are assigned to corresponding components.

In this specification, “and/or” includes each and every combination of one or more of the recited items. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with meanings commonly understood by those skilled in the art. In addition, the terms defined in the dictionary are not interpreted ideally or excessively unless explicitly specifically defined.

In order to achieve the object of the invention as described above, a lens capable of obtaining a bright field image and a confocal lens capable of confirming structural information of cells may be configured together in the head for an endoscopic microscope. Through this, two lenses can be configured together and two images can be obtained at the same time. However, since images can be acquired in real time only with a very small field of view at the mm level through confocal lenses, even if high-resolution images are obtained with confocal lenses, the location cannot be identified in the brightfield image, so the surgeon There is a difficult problem to cope with such as resection or marking to remember the location of the tissue, and one of the main objects of the present invention is to improve this, which will be described in detail with reference to the drawings below.

In this specification, description is made based on confocal imaging using a fluorescent material in order to avoid redundant description for convenience, but all descriptions of the present invention include the nozzle 160, which can be accompanied only when a fluorescent material is used, and cells of living tissue. Except for descriptions such as the step of dyeing the fluorescent dye in S310, it is also applied to all microscopic images based on a non-confocal laser light source. On the other hand, a microscope image based on a laser light source should be understood as a concept including, of course, a confocal image (likewise a laser method).

Referring to FIGS. 2 and 3 , the endoscopic microscope head 100 according to the present invention includes first light sources 110 and 120 capable of implementing brightfield images and second light sources 132 capable of implementing confocal images. ), and a confocal objective lens 131 contacting the sample to acquire the confocal image.

In addition, the endoscopic microscope image acquisition system 1000 of the present invention combines the head 100, the calculation unit 140 for calculating the relative position of the confocal image, and the brightfield image and the confocal image to obtain It includes a synthesized image generating unit 150 that generates a synthesized image and an output unit 180 that outputs the synthesized image.

The two light sources 110 and 120 of the endoscopic microscope head 100 according to the present invention may be configured using one LED each, and the second light source 132 capable of realizing a confocal image is 400- It may be configured to irradiate a laser in the 800nm wavelength region.

In the present invention, it is preferable that the number of first light sources 110 and 120 capable of obtaining a bright field image be composed of a plurality of, for example, two (reference numerals 110 and 120) as shown in FIG. 2A. Since the first light sources 110 and 120 are composed of LEDs, unlike the second light source 132 composed of a laser light source and capable of realizing confocal images, it is necessary to prevent shadows from occurring. However, the first light source capable of obtaining a bright field image according to the present invention is not necessarily limited to a plurality of or two light sources, and may be composed of a single first light source.

The first light sources 110 and 120 are generally understood as a concept including a separate optical lens (or light source lens) in the art, and therefore, the first light sources 110 and 120 emit light from a light source. It should be understood as an optical lens itself or a configuration including it.

Meanwhile, the second light source 132, like the first light sources 110 and 120, should be understood as a concept including a separate optical lens in the art, and additionally in the present specification, a confocal image capable of implementing In the second light source 132, the objective lens in contact with the sample, that is, the confocal objective lens 131, distinguished from the optical lens for irradiating the light of the light source, is separately indicated by reference numeral 131 in FIGS. 2A and 2B, and for convenience, this A configuration comprising both is denoted as a confocal window 130 . However, in general, in this technical field, the confocal window 130, the confocal objective lens 131, and the second light source 132 of the confocal window are commonly used as interchangeable terms, and these terms are also used herein. They were used in combination as needed.

In addition, although not shown in FIG. 3 , the head 100 for an endoscope may include a nozzle 160 for applying a dye material to obtain a fluorescence signal using a fluorescent dye material, as shown in FIG. 2A.

Strictly speaking, the nozzle 160 is a component distinct from the endoscopic microscope head 100 and constitutes an endoscope device together with a cable 300 detachably connecting the endoscopic microscope head 100. may be The term 'detachably connected' includes 'optical connection'.

In addition, light source lenses are disposed behind the endoscopic microscope head 100 and are optically connected to the first light sources 110 and 120 . The light source lens allows the light from the light sources 110 and 120 to pass through and illuminate the sample.

In addition, the cable 300 and the endoscopic microscope image acquisition system 1000 are referred to as an endoscope device in this specification, and the display 200 capable of displaying the synthesized image output by the output unit 180 The included physical hardware is referred to herein as an endoscope system (see FIG. 6). As described above, the endoscopic microscope head 100 should be understood as one element of the endoscopic microscope image acquisition system 1000 .

The fluorescent signal is to improve the accuracy of confocal imaging, and in the present invention, it is not necessary to apply a fluorescent signal to obtain a confocal image, that is, a fluorescent dye is not necessarily used, and optionally It can be configured to obtain a confocal image without adding a fluorescence signal, and therefore, in this case, it is also possible not to employ the nozzle 160 for applying a dye material to obtain a fluorescence signal. In this respect, such a nozzle 160 is not shown in FIG. 3, which is a schematic diagram showing the components of an exemplary embodiment of the internal system of the endoscopic microscope image acquisition system 1000 according to the present invention showing the common basic elements.

In addition, the endoscope microscope head 100 of the present invention includes an objective lens 170 that primarily forms a bright field image of a specimen formed by light irradiated through a light source lens by the light sources 110 and 120 do.

Like the nozzle 160, the objective lens 170 is a purely hardware or physical component of the endoscopic microscope head 100, separately from FIG. Not shown, on the contrary, the calculation unit 140, the synthesized image generation unit 150, and the output unit 180 are pure system control components and are not separately shown in FIG. The understanding of the hardware and software configurations of FIGS. 2 and 3 should be comprehensively considered.

In addition, a confocal objective lens guide 133 for contacting the confocal optical lens and a driving system unit 132 for changing the lens position are additionally provided to obtain a brightfield image and obtain a fluorescence signal as a high-resolution confocal image at the same time. can be configured to include

The driving system unit 132 is a component that can be integrated with the second light source and has the same reference numerals, but depending on the system configuration, it is possible to use the second light source and the driving system unit as separate components.

Each component of the endoscopic microscope image acquisition system 1000 according to the present invention may mutually include the function and role of another component, but on the contrary, the function and role of any one component It may be configured so that the role is performed by two or more plural components.

A method of controlling the image acquisition system of the surgical endoscopic microscope according to the present invention will be described with reference to FIGS. 5 and 7 .

A method for controlling an image acquisition system of an endoscopic microscope or an endoscopic device including an endoscopic microscope according to the present invention (see FIG. 5) includes the steps of acquiring a brightfield image through a first light source (S210); Obtaining a confocal image through two light sources (S220), contacting a confocal window (strictly, a confocal objective lens) to a sample to obtain the confocal image (S230), and The method includes calculating a position (S240), synthesizing the lightfield image and the confocal image to generate a synthesized image (S250), and outputting the synthesized image (S260).

The method of controlling the image acquisition system of the endoscopic microscope according to the present invention of FIG. 7 or the method of controlling the endoscopic apparatus including the endoscopic microscope partially expresses the method of FIG. 5 by abbreviation and some includes subdivision or separate control conditions As shown, the method of FIG. 7 further includes a step (S310) of staining the cells of the biological tissue with a fluorescent dye, which is a method for controlling an image acquisition system of an endoscope microscope or an endoscope device including an endoscope microscope It can be understood as a preliminary step for acquiring such an image, rather than a method for controlling it.

In addition, the confocal image acquisition/image processing (S320) includes obtaining a confocal image through the second light source in FIG. 5 (S220), and contacting the confocal window to the sample to obtain the confocal image (S220). It corresponds to S230). In addition, the step of obtaining a brightfield image ( S340 ) may correspond to the step of acquiring a brightfield image through the first light source ( S210 ) in FIG. 5 .

Importantly, in the embodiment of FIG. 7 , the output unit 180 may output only the brightfield image (S370), depending on whether or not the reference score passes in matching the reference brightfield image and feature points (S350), and the synthesized image generator 150 After generating a synthesized image of the brightfield image and the confocal image (S360), the output unit 180 may output the synthesized image as a ratio (S380).

The purpose of determining whether to pass using the criterion score as above is that the feature point matching algorithm extracts features even if the features are not the same in the two images and matches the values that are as similar as possible. This is to limit these cases.

On the other hand, the bright field image is output from the monitor in real time so that the operating surgeon can view it. A dye is applied to the tissue from a fluorescent dye nozzle to obtain fluorescently stained tissue composition information within the brightfield image field of view. A confocal image is a detailed image of a body tissue, and since the image must be obtained by contacting the body tissue for this purpose, the confocal window 130 contacts the tissue while moving so as to protrude forward at a fixed angle, as shown in FIG. 2B.

In the present invention, the reason why the confocal window 130 contacts the tissue while moving so as to protrude forward at a fixed angle is that high-resolution images can be obtained with a high numerical aperture (NA) only when the contact is made, and at a fixed angle. The reason for the movement is to transform the coordinate system when combining brightfield and confocal images. That is, if the angle is arbitrarily changed, the coordinate system is unpredictable. At this time, the fixed angle means that the confocal objective lens 131 can move in the vertical direction, that is, in the left and right direction in the direction of looking at FIG. 2B, and in the left and right direction, that is, in the direction of looking at FIG. means there is no In FIG. 2B, the upper drawing shows the confocal objective lens 131 protruding, that is, when approaching close to the sample, and the lower drawing shows the confocal objective lens 131 retracting, that is, moving away from the sample. Show what you look like when you lose.

The confocal objective lens 131 is an objective lens for obtaining a microscopic (cell) image of a tissue, and is distinguished from the objective lens 170 for obtaining a brightfield image.

On the other hand, when the body tissue is contacted, the confocal image is taken within 1 second at one position (a certain field of view), and at the same time, the confocal window 130 moves to the initial position, the back side, as shown in the lower drawing in FIG. 2B. . The coordinates obtained from the confocal image are calculated from the coordinates of the brightfield image, and the transparency of the confocal image is adjusted and displayed over the brightfield image. For this purpose, as shown in FIG. It may be included in the synthesized image generator 150 of (100).

Transparency of the confocal image may be configured to adjust the transparency of the confocal image to a state in which the tissue structure and location can be determined through the brightfield image. The confocal image may be composed of an image expressing a probability map in the form of a heat map through image processing and deep learning operation, rather than an original image. That is, a probability map expressing a probability of being a cancer tissue at each location is expressed in a heat map form.

As the head 100 for an endoscopic microscope is shaken by the operating surgeon and the position is changed from side to side and back and forth, a change occurs in the position at the moment when the confocal image was taken. At this time, the synthesized image update unit 1520 of the synthesized image generator 150 calculates movement in the x and y directions and distance in the z axis from the initial position at the moment when the confocal image was taken, and periodically updates the changed brightfield image So you can show overlapping images. An additionally obtained confocal image may be configured to overlap with a brightfield image in the same manner. In addition, if the position at the time of acquiring the initial confocal image cannot be found in the current brightfield image, it may be configured to output the out of position to the monitor, remove the overlapping confocal image, and output only the brightfield image to the monitor. there is.

On the other hand, the meaning of the overlapped confocal image is that the confocal image and the brightfield image were synthesized and displayed at a certain point in time, and then, when the confocal image is moved to a location that cannot be found on the screen, the light on the synthesized image is displayed. It means a confocal image synthesized with a visual field image.

In the present specification, the description has been made based on the synthesis of "confocal image" with bright field image, but this is only an exemplary description, and if the image of the specimen can be obtained, all laser light source-based microscope images are comprehensively included. should be understood as

On the other hand, when calculating the movement in the x and y directions and the distance movement in the z-axis from the initial position at the moment the confocal image was taken, the distance movement calculation method is based on the bright field containing the tissue structure information of the tissue at the moment the confocal image was taken. The image is also acquired and held as a reference image. A bright field image is acquired again from the moved position, feature points of the reference image and the image obtained from the recently moved position are extracted, and movement in the x, y directions and distance in the z axis are calculated using a feature point matching algorithm.

After the above calculation, when periodically updating the confocal image to the brightfield image, looking at the specific period and updating method, the slowest frame rate of the camera used for the brightfield image is 10 frames/s. When set as a criterion, the frame rate of the brightfield image shown to the actual user is lowered to 5 frame/s and updated to 5 frame/s or more considering the feature point matching algorithm calculation speed through the image. In the update method, the synthesized image on the screen is replaced with the most recently synthesized image and displayed.

In the present invention, the reason for updating by slowing the frame rate (10 → 5 frames/s) as above is that the video rate of the camera should be 10 frame/s, which is not awkward for humans to perceive, and is half of that. The reason why is set as the criterion of the speed of updating is that the time to acquire the image and the time to process the image are inevitably slower than 10 frames/s.

In other words, if the speed at which an image is obtained from a camera is 0.1 second per frame (10 frames/s), and it takes approximately 0.1 second additional time to obtain an image and process (coordinate conversion, synthesis), it is actually displayed on the display screen. Until updating, it slows down to 0.1 sec + 0.1 sec = 0.2 sec (5 frames/s).

Since the camera's slowest speed was taken as a standard, 10 frame/s was calculated above, but it is of course possible to obtain more than that. For example, 20 frame/s can also be used. However, in the case of 20 frame/s, if it takes 0.05 seconds to acquire an image and 0.1 second to process an image, there is a problem that the previously acquired images are gradually delayed and piled up.

Meanwhile, the camera is for obtaining a brightfield image and may be included as a separate component from the objective lens 170 . An image obtained through the objective lens 170 can be converted into a digital signal through the camera, and then analyzed and post-processed in a device such as a PC including a calculator 140 or a generator 150.

In addition, the purpose of using the feature point matching algorithm is to extract an image-specific location in order to find the structure of the same tissue in each image in two brightfield images obtained at different points in time and at different locations. This is to know how much the direction of the x, y, and z axes has changed from the brightfield image obtained at the previous point in time to the next brightfield image through the above-mentioned feature point matching algorithm. It can be composed of an algorithm that extracts and matches.

In the present invention, one feature is that an algorithm for updating an image while compensating for a coordinate system in real time is employed. It is an algorithm that updates the image while compensating for the real-time coordinate system by synthesizing the image and displaying it on the screen for the user to see.

In the present invention, when two image values are added and added in units of pixels in order to synthesize two images aligned in coordinate systems, a weight value (α value) may be considered and calculated as shown in Equation 1 below.

Formula 1: Composite result pixel value g(x, y) formula

here,

x, y: position value of each pixel in the image

g(x,y): value of each pixel of the synthesized image

f _o (x,y): the value of each pixel in the brightfield image

f ₁ (x,y): the value of each pixel in the confocal image

α: weight

The value of α is selected so that the structure of the tissue can be confirmed in the brightfield image even after synthesizing the two images, and the probability heat map representation of the presence or absence of cancer tissue is possible. An image can be created by adjusting the ratio of the values of the brightfield image and the confocal image using the value of α, adjusting the pixel values of the two images, and adding the adjusted two values, and a conceptual example thereof is shown below. .

[Figure 1: Synthesis concept of brightfield image and confocal image according to the pixel value g(x, y) calculation formula]

In addition, in the present invention, when the coordinates obtained from the confocal image are calculated from the brightfield image coordinates, the scaling value between the confocal image and the brightfield image (ie, magnification (enlargement or reduction) of the brightfield image and the confocal image) value), rotation angle, and movement distance (x, y) values are known, 2d transformation is calculated for each pixel of the confocal image through Equation 2 below to obtain a confocal image in the brightfield image coordinate system. It can be.

Equation 2:

here,

x ₀ : x-direction pixel position value in the confocal image coordinate system before 2d transformation

y ₀ : y-direction pixel position value in the confocal image coordinate system before 2d transformation

x ₁ : x-direction pixel position value in the brightfield image coordinate system after 2d transformation

y ₁ : y-direction pixel position value in the brightfield image coordinate system after 2d transformation

Sx: scaling value in the x-axis direction

Sy: scaling value in the y-axis direction

θ: rotation angle

tx: movement distance in the x-axis direction

ty: movement distance in the y-axis direction

The rotation angle means the rotation angle of the confocal image with respect to the brightfield image. The rotation angle may occur as the objective lens 170 moves along the confocal objective lens guide 133 using the confocal window 130. there is.

In relation to Equation 2, more specifically, in order to convert the position of the coordinate values of the confocal image to the position corresponding to the coordinate system of the brightfield image, the coordinate values x ₀ and y ₀ are substituted for each of the variables described above to obtain the coordinate value x ₁ , y ₁ are calculated, and the coordinate system conversion of the image is performed by substituting the pixel values of the confocal image corresponding to the x ₀ , y ₀ coordinates to the x ₁ , y ₁ positions.

On the other hand, dyeing, fluorescence imaging, and measurement methods using fluorescent materials, which are widely used in biological research, are key techniques for examining lesions in living tissues, and are useful for diagnostic tests by expressing fluorescence only in specific areas of interest among biological organs. is being used

Currently, only ICG and fluorescein are used for blood vessel staining as fluorescent dyes for the human body, and there are attempts to utilize fluorescence wavelengths using biological tissue dyes such as antibiotics that have secured stability in the body. However, the above technology is not used to determine the causative bacteria of a disease in cells or the like in the human body because excitation light exhibits fluorescence characteristics using an ultraviolet region.

Cancer ranks first in mortality worldwide, and standard cancer treatments are known as surgery, chemotherapy, and radiation therapy. Double surgical therapy is reported as the most preferred technique in the medical community because it greatly contributes to increasing the survival rate of cancer patients. Histological analysis in a pathology diagnosis room is essential for the method of resecting cancer lesions during medical practice, despite skilled techniques. At this time, the staining method used includes a hematocin-eosin staining method that can be observed with the naked eye, but a method using a fluorescence detection method is still difficult to apply clinically. This is because it is difficult to apply fluorescent staining and luminescent reagents to lesion detection, and it is difficult to standardize diagnostic methods. Therefore, the present researchers improved this problem by developing a technology to quickly determine the presence or absence of cancer tissue in biopsy tissue using single-photon and multi-photon beams.

In general, in the endoscopic microscope according to the present invention, the waiting time after the staining material is applied to the tissue and then taking the image varies depending on the human tissue, but is between 1 minute and 5 minutes.

The technology of the endoscopic microscope of the present invention can also be applied to a handheld type surgical microscope or a handheld type surgical probe.

In addition, it is reasonable to understand that the surgical microscope technology for tissue resection, not the endoscope for simple examination, is included in the scope of the endoscopic microscope of the present invention because it is necessary to eventually penetrate into the human body tissue.

In addition, according to the present invention, there is a mode in which the brightfield image and the confocal image are displayed as one image by dividing the display mode and always synthesized, and a mode in which the brightfield image and the confocal image are displayed by dividing the display screen into two, respectively. Depending on the user's preference, it can be configured to be usable while changing the mode in the middle.

In addition, the present invention is different from obtaining a composite image by obtaining two images on the same coordinate system while changing the light source and filter through one optical lens for the brightfield image and the fluorescence image in the prior art. Dog images can be synthesized.

In the present invention, the dark field image is a high-resolution image, and the area that can be viewed at one time is very small, within 2 mm, so it is difficult to determine the position currently being viewed in the entire area by viewing only the dark field image. It is difficult, so it can be configured to perform tissue resection while viewing the entire area as a bright field image, so the convenience and accuracy of surgery can be greatly improved.

Finally, the operating process of the endoscopic microscope of the present invention will be described. When the power of the endoscopic microscope is turned on, white light is irradiated from the first light sources 110 and 120, which are LED light sources, and the structure of tissue obtained through the objective lens 170, which is an optical lens Information is obtained as image information through a camera at 10 frames/s or more. While checking this image information, adjust the position of the endoscopic microscope around the cancer tissue and trigger a trigger through the trigger device connected to the nozzle 160 around the cancer tissue. applied to this tissue. After the (fluorescent) staining material is absorbed into the tissue (1 to 2 minutes), the confocal objective lens 131 protrudes through the control unit so that the confocal objective lens 131 can contact the tissue to obtain a confocal image. . When the contact of the confocal objective lens 131 with the tissue is confirmed by the brightfield image, the confocal image acquisition is started through the control unit. While changing the location of the endoscopic microscope, repeat the method described above on the suspected area to determine whether or not it is a cancerous tissue.

The term 'endoscopic microscope' used in the present invention is also used as an endoscopic microscope, a microscopic endoscope, a microscopic endoscope, etc., and an endoscopic microscope is usually combined with an endoscope to enlarge and observe or photograph the tissue explored by the endoscope. can be understood as possible.

Although the embodiments of the present invention have been described in detail with reference to the accompanying drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical spirit or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting.

In addition, the accompanying drawings are not drawn to scale, but partially enlarged and reduced, in order to explain the technical idea of the present invention.

In addition, it is natural that embodiments in which the components and steps of the system, method, or device of the present invention described above are selectively combined also fall within the scope of the present invention.

100: endoscope head
110, 120: first light source
130: confocal window/microscope window
131: confocal objective / microscope objective
132: second light source
133: confocal objective guide/microscope objective guide
140: calculation unit
150: synthetic image generator
160: nozzle
170: objective lens
180: output unit
1000: endoscopic microscope image acquisition system

Claims (18)

A first light source capable of realizing a brightfield image that can see the tissue or surgical site as a whole, a second light source that can implement a microscope image based on a laser light source that can see details of the tissue or surgical site, and the microscope image an endoscope microscope head including a microscope window in contact with a sample for acquisition;
a calculation unit for calculating the relative position of the microscope image;
a synthesized image generating unit generating a synthesized image by synthesizing the brightfield image and the microscope image;
Endoscopic microscope image acquisition system comprising an output unit for outputting the synthesized image. According to claim 1,
The endoscope microscope image acquisition system, characterized in that the microscope window is configured to come into contact with the sample to acquire an image and then retract to the original position. According to claim 1,
The synthesized image generation unit calculates coordinates from which the microscope image was obtained from the brightfield image coordinates, adjusts the transparency of the microscope image on the brightfield image, superimposes the microscope image on the brightfield image, and outputs the result through the output unit. Endoscopic microscope image acquisition system, characterized in that it comprises a synthetic image control unit to. According to claim 1,
The synthesized image generating unit includes a synthesized image updating unit that generates a synthesized image by calculating distance movement in the x-, y-, and z-axes from an initial position at the time when the microscope image is acquired and periodically updating the changed brightfield image. Endoscopic microscope image acquisition system, characterized in that to do. According to claim 1,
If the position of the first viewpoint at which the microscope image was acquired cannot be found in the brightfield image of the second viewpoint thereafter, the position deviation of the microscope image from the brightfield image is separately output, and the overlapping microscope images are removed. Endoscopic microscope image acquisition system, characterized in that configured to output only the visual field image. An endoscope device comprising the endoscopic microscope image acquisition system according to claim 1 and a cable detachably connected to the endoscopic microscope head. An endoscope system comprising the endoscope device according to claim 6 and a display electrically connected to the endoscope device through the cable and displaying a synthesized image output by the output unit. According to claim 1,
The endoscope microscope image acquisition system, characterized in that the microscope window is configured to contact the tissue while moving so as to protrude forward at a fixed angle. According to claim 3,
The endoscopy image acquisition system, characterized in that the transparency of the microscope image is adjusted to a state in which the tissue structure and location can be determined through the bright field image. According to claim 4,
In order to calculate the distance movement of the x-, y-, and z-axes from the initial position at the time the microscope image was acquired, a brightfield image containing tissue structure information of the tissue was acquired as a reference image at the moment the microscope image was taken. store, and then acquire a new brightfield image again from the moved position, extract feature points of the reference image and the image acquired from the moved position, and calculate movement in the x, y directions and distance in the z axis. Endoscopic microscope image acquisition system characterized by. According to claim 10,
When the brightfield image stored as the reference image does not pass the reference score, only the brightfield image is output;
The endoscopy image acquisition system according to claim 1 , wherein the synthesized image generator is configured to generate a synthesized image only when the brightfield image stored as the reference image passes a reference score. Obtaining a first light source capable of realizing a brightfield image that can see the tissue or surgical site as a whole, a second light source that can implement a confocal image that can see details of the tissue or surgical site, and the confocal image An endoscopic microscope head including a confocal window contacting a sample to be in contact with the sample;
a calculator for calculating a relative position of the confocal image;
a synthesized image generating unit generating a synthesized image by synthesizing the brightfield image and the confocal image;
Endoscopic microscope image acquisition system comprising an output unit for outputting the synthesized image. An endoscope device comprising the endoscopic microscope image acquisition system according to claim 12 and a cable detachably connected to the endoscopic microscope head. According to claim 13,
Endoscope apparatus characterized in that it further comprises a nozzle for applying a fluorescent dye material to the sample to obtain the fluorescence image. The method of claim 1 or 14,
The endoscope microscope image acquisition system, characterized in that the first light source is composed of a plurality of LEDs to prevent shadows. According to claim 1,
In generating a synthesized image of the laser light source-based microscope image and the brightfield image, the synthesized image generator generates the composite image even after synthesizing two images (ie, the laser light source-based microscope image and the brightfield image). It is characterized in that a weight (α) is reflected in the pixel value of the laser light source-based microscope image and (1-α) is reflected in the pixel value of the bright field image so that the structure of the tissue can be confirmed in the visual field image. Endoscopic microscope image acquisition system to be. According to claim 12,
The synthesized image generating unit generates a synthesized image of the confocal image and the brightfield image, even after synthesizing the two images (ie, the confocal image and the brightfield image), the tissue structure in the brightfield image. Endoscopic microscope image acquisition system, characterized in that configured to reflect the weight (α) to the pixel value of the confocal image and to reflect (1-α) to the pixel value of the brightfield image so that can be confirmed. According to claim 3,
The endoscopic microscope image acquisition system, characterized in that the calculation of the coordinates from which the microscope image was acquired from the coordinates of the brightfield image is performed by the following formula.
formula

here,
x ₀ : x-direction pixel position value in the microscope image coordinate system before 2d transformation
y ₀ : y-direction pixel position value in the microscope image coordinate system before 2d transformation
x ₁ : x-direction pixel position value in the brightfield image coordinate system after 2d transformation
y ₁ : y-direction pixel position value in the brightfield image coordinate system after 2d transformation
Sx: scaling value in the x-axis direction
Sy: scaling value in the y-axis direction
θ: rotation angle
tx: movement distance in the x-axis direction
ty: movement distance in the y-axis direction