KR102561360B1 - Method for postprocessing fiberscope image processing not using calibration and fiberscope system performing the same - Google Patents

Abstract

Embodiments of the present application acquire at least one raw image of a sample from a camera device, binarize the raw image to separate a core and a cladding in the raw image, detect a peak point in the binarized image, and A fiberscope image processing method for interpolating a raw image based on a detected peak point in order to reconstruct a fiberscope image of , and a fiberscope system performing the same.

Description

METHOD FOR POSTPROCESSING FIBERSCOPE IMAGE PROCESSING NOT USING CALIBRATION AND FIBERSCOPE SYSTEM PERFORMING THE SAME

Embodiments of the present application relate to a technology for post-processing a fiberscope image, and more particularly, a fiberscope image processing method in which a calibration process is omitted during post-processing of a fiberscope image. and a fiberscope system that performs the same.

Fiber bundle endoscopes, also known as fiberscopes, are used in a variety of fields, including medical diagnosis and industrial inspection, as they are suitable for imaging internal organs of the human body or inside hard-to-reach industrial engines and pipes thanks to their compact probe and flexibility. .

Coherent fiber bundles used in fiber scopes consist of tens of thousands of core assemblies surrounded by cladding. This fiber bundle structure creates an artificial pattern of honeycomb structure in the fiberscope's raw image, limiting image resolution and visualization. These optical properties are eventually treated as obstacles such as noise in image processing.

High-resolution fiberscopes are also not free from the above problems. A high-resolution fiberscope consists of a bundle of high-density coherent fibers with a narrow distance between cores. These high-density fiber bundles consist of a non-uniform core cladding arrangement with non-circular cores of various sizes. Such high-density optical fiber bundles have higher crosstalk and auto-fluorescence characteristics than fiber bundles with lower density and longer core-to-core distances. Under such a high-resolution structure, crosstalk and auto-fluorescence of high-density fiber bundles occur very locally and act as highly variable noise, and consequently, more complex image processing tasks are required to deal with partial and dynamically changing noise in high-density fiber bundle images. .

In order to process partially and dynamically changing noise in a fiberscope image obtained under such a structural problem, it is common to remove a pattern of a honeycomb structure using a Gaussian smoothing filter and a spectral filter.

The Gaussian smoothing filter can easily and quickly remove the noise of the honeycomb pattern, but as the image processing process increases, the image downsampling phenomenon due to image blur becomes more severe.

The spectral filter technique can remove only the high-frequency region corresponding to the honeycomb pattern of the fiber bundle, but there is a possibility of losing important optical information of the non-honeycomb pattern contained in the high-frequency region, and the honeycomb under the condition that it is exposed to specific frequency noise. There is a problem in accurately removing the pattern of the structure.

Recently, as an attempt to solve the problems of the Gaussian smoothing filter and the spectral filter, an attempt is made to utilize an interpolation method in which intensity information of an adjacent core is filled in a position of a peripheral cladding.

Interpolation of the fiberscope image is performed using peak point information for cores in the fiber, which is obtained through calibration. Therefore, Non-Patent Document 1 (Wang P, Turcatel G, Arnesano C, Warburton D, Fraser SE, Cutrale F. Fiber pattern removal and image reconstruction method for snapshot mosaic hyperspectral endoscopic images. Biomed Opt Express. 2018 Jan 25;9(2 ):780-790.doi: 10.1364/BOE.9.000780. PMID: 29552412; PMCID: PMC5854078) in conventional embodiments for fiberscope image processing to remove the honeycomb pattern structure (Image Reconstruction). image correction techniques such as reconstruction) are essential.

Calibration is an image pre-processing method that performs core cladding separation and peak detection using a high-contrast image, such as a white reference image, where the white reference image has core pixels of maximum intensity (e.g., white). , and the cladding pixel is expressed as the minimum intensity (eg, black). Ideally, calibration only needs to be done once before imaging. However, most fiberscopes place a bundle of fibers in front of the focal plane of the objective lens, so they are very sensitive to motion during imaging, and there is always a risk of shifting the image coordinates during imaging. Also, for high-density fiber bundles, a new calibration is required whenever the experimental lighting conditions and experimental environment are changed.

The need for such repetitive calibration can burden the overall image processing and makes the development of high-resolution fiberscopes difficult. In particular, in vivo images are measured under conditions completely different from calibrated images, and different auto-fluorescence and modalities occur depending on the light source used, such as laser light or a broadband light source. Moreover, in vivo imaging situations cannot be predicted in advance and have limitations that cannot be reflected in calibration.

Wang P et al., "Fiber pattern removal and image reconstruction method for snapshot mosaic hyperspectral endoscopic images", Biomed Opt Express. 2018 Jan 25; 9(2):780-790 K. Minkyung et al., "Development of fiber-based all-optical system for neurovascular coupling mechanism study using optogenetics," in Proc.SPIE, 2020. M. Elter et al., "Physically Motivated Reconstruction of Fiberscopic Images," in 18th International Conference on Pattern Recognition (ICPR'06), 2006), 599-602. N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Transactions on Systems, Man, and Cybernetics 9, 62-66 (1979).

According to the embodiments of the present application, it is intended to provide a fiberscope image processing method that successfully interpolates fiberscope images obtained from various imaging environments including in-vivo imaging without correction, and a fiberscope image that performs the same .

A fiber scope image processing method according to an aspect of the present application is performed by a fiber scope system including a fiber bundle composed of a plurality of cores surrounded by cladding. The fiberscope image processing method includes: acquiring at least one raw image of a sample from a camera device, wherein the raw image shows a honeycomb pattern due to a structure of a fiber bundle; binarizing the raw image to separate a core and a cladding in the raw image; detecting an intensity peak point in the binarized image; and interpolating the original image based on the intensity of the detected peak point to reconstruct the fiberscope image of the sample.

In one embodiment, the step of binarizing the raw image may include selecting a candidate core pixel as a candidate core pixel and converting the intensity of the candidate core pixel into a specific intensity when a pixel on the raw image has an intensity equal to or greater than a threshold value. ; and determining non-selected pixels in the original image as cladding pixels and converting the intensity of the cladding pixel into another specific intensity.

In an embodiment, the threshold for binarization is a local threshold and may be calculated based on intensities of pixels within a local window including a peak point detected in the raw image.

In one embodiment, the threshold for the binarization is a maximum gray level value and a minimum gray level value in a local window of size w × w, formed around the position of the candidate core pixel, and a contrast value at the position of the candidate core pixel. It may be calculated based on a contrast value and a contrast threshold value preset by a user.

In an embodiment, the threshold for binarization may be calculated based on a local variance in a local window of size w×w, centered around the position of the candidate core pixel.

In an embodiment, the threshold for binarization may be calculated based on a local average in a local window of size w×w, centered around the position of the candidate core pixel.

In an embodiment, the peak point is detected in the candidate core pixel, and the core pixel in the fiberscope image may be a pixel in which a peak point is detected.

In an embodiment, the interpolating may include allocating an intensity of a corresponding core adjacent to a cladding pixel surrounding the core pixel in which the peak point is detected.

In an embodiment, the fiberscope image processing method may further include, before binarization, removing camera noise from the raw image.

In one embodiment, the fiberscope image processing method may further include, when a plurality of raw images are acquired, selecting at least one raw image from among the plurality of raw images as a reference image according to a predetermined criterion. . Only the reference image is binarized and a peak point is detected in the binarized reference image.

In one embodiment, the interpolating step may include allocating intensities of adjacent core pixels in the reference image to cladding pixels in the reference image distributed around core pixels in which peak points are detected, and as the reference image. Intensities of adjacent core pixels in the reference image may be assigned to cladding pixels in each raw image distributed around a position corresponding to a core pixel of the reference image in each of the raw images that have not been sorted.

In one embodiment, when the reference images are two or more raw images, the allocated core pixel intensity may be a representative value of core pixel intensity used for interpolation of each reference image.

A computer readable recording medium according to another aspect of the present application records a program for performing the fiberscope image processing method according to the above-described embodiments.

A fiber scope system according to another aspect of the present application includes: a probe including a fiber bundle composed of a plurality of cores surrounded by a cladding; an objective lens configured to simultaneously receive light from fibers included in the fiber bundle; a dual color fluorescence imaging system for acquiring fluorescence images of different color fields; a laser light stimulation system sharing a fiber bundle with the dual color fluorescence system; and a camera device generating raw image data of the sample, wherein the raw image shows a honeycomb pattern due to the structure of the fiber bundle; and acquiring at least one raw image of the sample from a camera device, binarizing the raw image to separate a core and a cladding in the raw image, detecting a peak point in the binarized image, and performing a fiberscope image of the sample. It may include a computing device configured to interpolate the raw image based on the detected peak point to reconstruct .

The fiber scope system according to one aspect of the present application can restore a sample image in which the core and cladding are accurately separated using only the raw image of the actually obtained sample without correction.

In addition, the fiber scope system can accurately separate the core and cladding from most of the raw images in the set, such as a plurality of image frames constituting a moving image, without correction.

As a result, the fiberscope system can accurately and quickly process raw images.

BRIEF DESCRIPTION OF THE DRAWINGS To describe the technical solutions of the embodiments of the present invention or the prior art more clearly, drawings required in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for the purpose of explaining the embodiments of the present specification and not for limiting purposes. In addition, for clarity of explanation, some elements applied with various modifications, such as exaggeration and omission, may be shown in the drawings below.
1 is a schematic diagram of a fiberscope system, according to one embodiment of the present application.
2A is an imaging view of a probe including a fiber bundle, according to one embodiment.
FIG. 2B is an enlarged view of the fiber bundle portion of FIG. 2A.
3 is a schematic diagram of a raw image obtained through a fiber bundle, according to one embodiment.
4A-4C are schematic diagrams of fiberscope image processing operations performed by computing device 50, according to one embodiment.
5 is a flowchart of a fiberscope image processing method according to another aspect of the present application.
6 shows a raw image acquired by a dual color fluorescence system, according to one embodiment.
7 is a flowchart of a binarization process using a threshold value calculated through a local gray range, according to an embodiment.
8 is a flowchart of a binarization process using a threshold calculated through local variance, according to an embodiment.
9 is a flow diagram of a binarization process using a threshold calculated through a local average, according to one embodiment.
10 is a flowchart of a fiberscope image processing method according to another aspect of the present application.
11 shows simulated images for each case according to the first experimental example.
12A and 12B show threshold evaluation for binarization using case-by-case simulated images of FIG. 11 .
13 is a raw image of a sample according to the second experimental example.
14A is a result of post-image processing through correction of the raw image of FIG. 13 .
14B is a result of post-image processing of the original image of FIG. 13 without correction.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the technical field disclosed in this specification, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when technical terms used in this specification are erroneous technical terms that do not accurately express the spirit of the technology disclosed in this specification, they should be replaced with technical terms that those skilled in the art can correctly understand.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. As used herein, “consists of.” or "includes." Such terms should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps may not be included, or additional components or steps may be included. It should be interpreted as being more inclusive.

In addition, the suffix “part” for components used in this specification is given or used interchangeably in consideration of only the ease of writing the specification, and does not itself have a meaning or role distinct from each other.

In addition, in describing the technology disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the technology disclosed in this specification, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only intended to facilitate understanding of the spirit of the technology disclosed in this specification, and should not be construed as limiting the spirit of the technology by the accompanying drawings.

A system according to embodiments may have aspects that are entirely hardware, entirely software, or partly hardware and partly software. For example, a device or system may collectively refer to hardware equipped with data processing capability and operating software for driving the hardware. In this specification, terms such as “unit”, “module”, “apparatus”, or “system” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device including a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or another processor. Also, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a fiberscope system, according to one embodiment of the present application.

Referring to FIG. 1, the fiber scope system 1 includes a) a probe; b) a dual-color fluorescence imaging system; and c) a laser light stimulation system sharing the fiber bundle 10 with the dual color fluorescence imaging system.

In certain embodiments, the fiberscope system 1 includes a probe comprising a fiber bundle 10; objective lens 21; a dichroic mirror 22, a polarizing beam splitter (PBS, 23); lens 25; a dual-color separator (27); camera device 29; and/or computing device 50 .

In addition, the fiber scope system 1 includes a first light source 31; mirror 32; and/or lenses 33, 35.

In addition, the fiberscope system 1 includes a pair of second light sources (2-1 light source 41 and 2-2 light source 42), a mirror 44, a color sorting mirror 46; lenses 47 and 49; and/or a projection device 48 .

The mirrors 32, 44, and 46 and the lenses 33, 35, 47, and 49 adjust the magnification and focus of the fiberscope system 1.

The fiber scope system 1 irradiates light to a sample through the fiber bundle 10 or acquires reflected light of the sample. The objective lens 21 has a magnification at which the light of the fibers included in the fiber bundle 10 is simultaneously incident. The magnification of the objective lens 21 may be 20 times, but is not limited thereto.

In the fiberscope system 1, the laser light stimulation system emits stimulation light from the first light source 31 to the sample. The first light source 31 may be implemented with a laser, fluorescent lamp, LD, LED, etc. having an output of sufficient intensity to stimulate the optical channel of the sample.

In the fiberscope system 1, the dual color fluorescence image system is: a path along which observation light for fluorescence image acquisition proceeds.

The pair of second light sources are configured to emit wavelength bands of different colors. In one embodiment, the 2-1st light source 41 may emit light in a green wavelength band and the 2-2nd light source 42 may emit light in a red wavelength band. For example, the 2-1 light source 41 may emit a laser having a wavelength of about 470 nm and the 2-2 light source 42 may emit a laser having a wavelength of about 530 nm.

The projection device 48 may be a spatial light modulator (SLM) capable of transmitting a pattern image in the form of a hologram by selectively reflecting or transmitting light according to an arrangement position of crystals. However, it is not limited thereto, and the projection device 48 may be implemented as an electronic device capable of forming patterned light, such as a digital micromirror device (DMD).

2A is a photographic view of a probe including a fiber bundle according to an embodiment, and FIG. 2B is an enlarged view of a portion of the fiber bundle of FIG. 2A.

As shown in FIG. 2B, the probe according to this embodiment includes a cylindrical body 11 and a fiber bundle 10 made up of a plurality of fibers 13 inside the body 11. A fiber bundle 10 is included at the front end of the probe.

The fiber bundle 10 may include a plurality of fibers 13 (eg, tens of thousands of fibers 13), and each fiber 13 has a diameter of several micrometers, which is larger than that of a single nerve cell. small.

The body 11 of the probe increases in diameter toward the rear end. As shown in FIG. 2A, the probe has a needle shape that becomes sharper towards the front end. Therefore, when inserted into the sample, damage to the sample can be minimized.

In addition, the body 11 of the probe may be made of a material that can be flexibly bent. Then, it can be appropriately used as endoscopic equipment or the like.

The objective lens 21 is coupled to the rear end of the fiber bundle 10, and the first light source 31 of the laser light stimulation system; Alternatively, a light path aligned with the rear end of the fiber bundle 10 is formed from the pair of second light sources 41 and 42 of the dual color fluorescence image system through the objective lens 21 .

The principle of the fiber scope system (1) is described in Non-Patent Document 2 (K. Minkyung and S. Hyun-joon, "Development of fiber-based all-optical system for neurovascular coupling mechanism study using optogenetics," in Proc.SPIE, 2020 ), the detailed description is omitted.

The camera device 29 collects the reflected light of the sample in the fiber scope system 1 . The camera device 29 generates raw image data of the sample based on the collected reflected light of the sample. The camera device 29 may include an image sensor such as a Charge Coupled Device (CCD) or Complementary Metal Oxide Semiconductor (CMOS).

The camera device 29 supplies raw image data of the sample to the computing device 50 . Raw images of the sample include dual fluorescence images. The camera device 29 may generate a fluorescence image when the reflected light of the collected sample is a fluorescence signal. When the pair of second light sources 41 and 42 emit green/red light, the raw image of the sample may be a green/red field image.

3 is a schematic diagram of a raw image obtained through a fiber bundle, according to one embodiment.

Referring to FIG. 3, when the front end of the fiber bundle 10 composed of a plurality of fibers 13 is brought into contact with an object, light emitted from the object is output through the rear end (when the object emits light), The light is collected by a camera device to acquire an image of the surface of the object. Due to the structural characteristics of the fiber bundle, a raw image showing a pattern of a honeycomb structure is obtained. The camera device 29 supplies the raw image in which the pattern of the honeycomb structure appears to the computing device 50 .

Computing device 50 is a device that includes one or more processors and is configured to perform image processing operations.

When the fiberscope image of the sample is acquired, the computing device 50 may process the fiberscope image.

4A-4C are schematic diagrams of fiberscope image processing operations performed by computing device 50, according to one embodiment.

Referring to FIG. 4A , the computing device 50 may binarize an image. Binarization is an image processing process that separates the pixel values of a digital image into two groups of different colors (eg, white and black). Computing device 50 may binarize the fiberscope image to separate the core and cladding in the fiberscope image. Binarization will be described in more detail with reference to FIGS. 7 to 9 below.

Referring to FIG. 4B , the computing device 50 may detect a peak point of an area corresponding to a specific object part in an image. In certain embodiments, the computing device 50 may detect peak points for the cores of the fiber bundle 10 .

Also, referring to FIG. 4C , the computing device 50 may interpolate an image. The computing device 50 may reconstruct a fiberscope image by interpolating a cladding area around a core pixel. By interpolation, the strength of the core adjacent to the position of the surrounding cladding is assigned instead of the actual cladding strength.

Also, the computing device 50 may be further configured to remove camera noise of the camera device 29 . The camera noise removal process corresponds to a pre-processing process for the image processing process with reference to FIG. 4 described above. That is, the above-described binarization and the like correspond to image post-processing.

An image preprocessing/postprocessing process of the computing device 50 will be described in more detail with reference to FIG. 5 below.

In addition, the computing device 50 may acquire a plurality of raw images for implementing a sequence from the camera device 29 and process each of the plurality of raw images without correction. When the camera device 29 generates video data for the sample and supplies it to the computing device 50, the computing device 50 processes the image of the video frame to provide a clear video with the honeycomb pattern removed. . This will be described in detail with reference to FIG. 10 below.

It will be apparent to those skilled in the art that the computing device 50 may include other components not described herein. For example, the computing device 50 may include other hardware components required for the operations described herein, including a network interface, input devices for data entry, and output devices for display, printing, or other presentation of data. You may.

The fiber scope image processing method according to another aspect of the present application may be performed by a computing device including a processor. In certain embodiments, the fiberscope image processing method may be performed by at least some components (eg, the computing device 50) of the fiberscope system of FIG. 1 .

5 is a flowchart of a fiberscope image processing method according to another aspect of the present application.

Referring to FIG. 5 , the fiberscope image processing method (hereinafter referred to as “image processing method”) includes obtaining at least one raw image data of a sample (S100). The raw image data of the sample is raw image data received from the camera device 29 . The image data includes raw image data acquired by the dual color fluorescence system in the fiber scope system of FIG. 1 or raw image data acquired by the laser light stimulation system.

In step S100, a raw image including a pattern of a honeycomb structure is acquired.

6 shows a raw image acquired by a dual color fluorescence system, according to one embodiment.

Referring to FIG. 6 , raw image data consisting of green light/red light is acquired by the dual color fluorescence system in the fiberscope system of FIG. 1 . When the green light region of FIG. 6 is enlarged, the pattern of the honeycomb structure in the original image is expressed. The honeycomb pattern structure acts as noise in analyzing the sample image.

Further, the image processing method includes: binarizing the raw image including the honeycomb pattern obtained in step S100 (S400). In step S400, binarization is performed to separate the core and the cladding from the fiberscope image.

In one embodiment, the raw image may be binarized based on the pixel intensities of the raw image (S400). The step (S400) may include: selecting a pixel on the raw image as a candidate core pixel and converting the intensity of the candidate core pixel into a specific intensity when the pixel on the raw image has an intensity equal to or greater than a threshold value; and determining non-selected pixels in the raw image as cladding pixels and converting the intensity of the cladding pixel into another specific intensity.

As shown in the partially enlarged view of FIG. 6 , the core portion in the fiberscope image tends to be brighter than the surrounding cladding because it is a path that transmits intensity information of the region of interest. Thus, core pixels have higher intensities and cladding pixels have lower intensities.

Based on this trend, a pixel having an intensity higher than the threshold may be a candidate pixel with a high probability of being a core pixel. In particular, if a threshold value is appropriately set, only core pixels may be selected. Pixels that are not sorted may be treated as cladding pixels and filtered out.

Binarization using the threshold value T(x,y) proceeds through the following equation.

where B(x, y) is a binary image and I(x, y) ∈ [0,1] is the intensity of the pixel at position (x, y) in image I. In the raw image, pixels with intensity values above the threshold and pixels with intensity values below the threshold are classified.

Then, the intensity of the selected candidate core pixel is converted to a specific value (eg, intensity 1), and the intensity of the filtered cladding pixel is converted to another specific value (eg, intensity 0).

In one embodiment, the computing device 50 binarizes the raw image with a preset local threshold to separate the core portion and the cladding portion from the raw image. A local threshold is calculated based on the intensities of pixels in the local window including the peak point detected in the raw image. When multiple peak points are detected, different threshold values may be used for each local window area including each peak point.

A process of calculating the local threshold will be described in detail with reference to FIGS. 7 to 9 below.

The image processing method includes: detecting a peak point in the binarized image of step S400 (S500). The peak point is a point having a peak intensity in the binarized image area.

When the binarization operation is performed through Equation 1 using the threshold value T(x,y) for all fiberscope image pixels (S400), the pixel of intensity 1 in the fiberscope image is regarded as a candidate core pixel and the pixel of intensity 0 Each pixel is regarded as a candidate cladding pixel (S400). Then, as a result of binarization, peak points for finding the center of the fiber core are detected from the candidate core pixel having an intensity of 1 (S500). A candidate core pixel in which the peak point is detected is used as a core pixel in the reconstructed image.

In an embodiment, the step of detecting a peak in the binarized image (S500) includes: generating a binary mask composed of candidate core pixels having an intensity of 1 before detecting the peak; and detecting peak points in pixels selected with a binary mask.

The peak point is Non-Patent Document 3 (M. Elter, S. Rupp, and C. Winter, "Physically Motivated Reconstruction of Fiberscopic Images," in 18th International Conference on Pattern Recognition (ICPR'06), 2006), 599-602. ) may be obtained using a peak detection algorithm based on

The core pixel follows a Gaussian distribution according to the beam profile of the optical fiber according to the distance r from the center, and the cladding pixel expresses the calculated light spread in the core pixel by adjusting the intensity level through the peak intensity.

In one embodiment, the detecting of the peak point in the pixels selected by the binary mask may include: selecting a pixel following a 2-dimensional Gaussian distribution from among the pixels selected by the binary mask; Calculating a score for each pixel following a selected, 2-dimensional Gaussian distribution; and detecting a peak point corresponding to the center of the core through the pixel-by-pixel score.

A Gaussian distribution value is calculated only for pixels selected with a binary mask, and a peak point corresponding to the center of the core is detected through a score for each pixel following a 2D Gaussian distribution. The pixel-by-pixel score is a score indicating how well the symmetric 2D Gaussian fits the core location, and is determined for each candidate core point. In calculating the score per pixel, the size of the core of the fiber bundle used in the fiber scope and the distance between adjacent cores (core-to-core distance) are considered.

When the coordinates of the candidate core center point are (x _c , y _c ), the candidate core center point S _c may be calculated through Equation 2 below.

Here, G(x,y) can be expressed by Equation 3 below, and N can be expressed by Equation 4 below.

where r is the distance from the candidate core center point (x _c , y _c ) to the candidate core pixel (x, y) and d represents the core radius. N is the set of pixels around the candidate core pixel. G (x, y) is a quasi-Gaussian surface of the candidate core pixel surface, and is a surface of size N calculated based on values for each candidate core pixel in pixel set N. a is an optimized value for each image, and the value for a is calculated through Equations 5 to 7 below.

where mid is the midpoint of the y-axis of the image, j is the number of peaks measured on the midline, and x _j is the x-coordinate of the measured peak.

In step S500, only peaks having a specific intensity value or higher are extracted. Subsequently, a Gaussian distribution according to each extracted peak I(x _j , mid) is calculated through Equation 5 within the range of a _j , and the smallest a _j is selected through Equation 6. Subsequently, the average value of a _j calculated for each peak by Equation 7 is applied to the entire image. Then, B(xy,) is placed at the peak point C(x, y) in the order of the highest candidate core centroid score S _c among all candidate core pixels having a value of 1. If the coordinates of the previously placed candidate core pixels are (x _m , y _m ), candidate core pixels satisfying Equation 8 are arranged in order.

where D is the core-to-core distance adjacent to each other.

Starting from the highest ranked candidate core centroid, each candidate core centroid (x _c ,y _c ) is evaluated only if the candidate core centroid is farther than the minimum distance of a single core diameter from all already placed core centroids (x _m ,y _m ). Place it on the center plot. That is, a candidate core center that satisfies Equation 8 is selected as the core center, and a peak point corresponding to the core center is detected. Then, the peak image shown in FIG. 4B is obtained.

Also, the image processing method includes interpolating a raw image based on the detected peak point (S600). In step S500, the intensity information of the cladding pixels is filled by allocating the intensity of the corresponding adjacent core pixel to the cladding pixels distributed around the core pixel whose peak point is detected. Then, intensity information of the cladding region not selected as a candidate core pixel in the previous steps (S400 and S500) is interpolated.

Specifically, a fiberscope image is reconstructed using a linear interpolation technique based on the peak point map C(x, y) obtained in the peak detection step. For example, the scatteredInterpolant function of the MATLAB program installed in the computing device 50 may be used as a linear interpolation technique.

When this interpolation is completed, a fiberscope image obtained by reconstructing the raw image of the sample is obtained. In the fiberscope image of the reconstructed sample, the honeycomb pattern of the fiber bundle is removed through interpolation, and a fiberscope image with improved visibility is obtained as shown in FIG. 4c.

In certain embodiments, the image processing method may further include removing camera noise (S200). Camera noise is a preprocessing operation performed before the binarization step (S400).

Upon receiving the raw image data, the computing device 50 selectively removes, among the raw image pixels, pixels having characteristics to be treated as noise of the camera device 29 in advance. The characteristics treated as the noise may include intensity characteristics. Then, the computing device 50 may remove pixels in the raw image having preset intensity characteristics as being regarded as noise of the camera device 29 (S200).

In one embodiment, the preset intensity characteristics may include whether the camera device 29 has an intensity four times higher than the average intensity of all target images when the camera device 29 is a CCD camera. If a specific pixel in all frames of the raw image has an intensity four times higher than the average intensity of all pixels in the raw image, that pixel may be removed. Otherwise, the corresponding pixel is not removed.

In one embodiment, the computing device 50 may remove the corresponding pixel using a 2D filter when the pixel has the preset intensity characteristic. In some embodiments, the computing device 50 may selectively remove pixels having an intensity at least 4 times the average intensity of the entire image by using a 3x3 2D median filter.

In certain embodiments, the image processing method may further include: determining an appropriate threshold value T(x,y) for binarization (S400).

The threshold for binarization is classified into a global threshold based on all pixels constituting the raw image and a local threshold based on pixels in a desired region.

The raw image of the sample obtained in step S100 is not an image having a uniform contrast distribution of background and foreground like a document image. The raw image of the sample contains many pixels that cannot be easily classified as foreground or background, either because the background noise is high or because it has variable contrast and illumination. Considering the characteristics of these raw images, a local threshold is appropriate for the threshold for binarization.

In certain embodiments, the threshold value T(x, y) as the local threshold value is a parameter such as the range, variance or surface-fitting parameters of adjacent pixels within a local window of size w × w. It is computed for each pixel located in the local window based on local statistical elements.

7 is a flowchart of a binarization process using a threshold value calculated through a local gray range, according to an embodiment.

Referring to FIG. 7 , the threshold for binarization may be calculated through a local gray range. The threshold for the binarization is the maximum gray level value and the minimum gray level value in a local window of size w × w, formed around the position (x, y) of the candidate core pixel, and the position (x, y) of the candidate core pixel. ), and a contrast threshold value preset by the user.

For example, the local threshold T(x,y) at location (x,y) is the size formed around the location (x,y) of the candidate core pixel on the raw image. It is calculated by the following equation within a local window of w × w.

Here, max _w and min _x denote the maximum gray level value and the minimum gray level value in a local window of size w × w formed around the position (x, y), respectively. The subscript w represents the window index.

C _w is a contrast value at the position (x, y) of the candidate core pixel, and T _c is a contrast threshold value. T _c is a hyperparameter preset by the user.

A local window is formed for the position (x,y) of the candidate core pixel of the raw image of the sample, and the calculated _Cw is compared with a preset Tc to calculate a local threshold value T(x,y).

8 is a flowchart of a binarization process using a threshold calculated through local variance, according to an embodiment.

Referring to FIG. 8 , the threshold for binarization may be calculated using local variance. The local threshold T(x, y) is calculated based on the local mean m(x, y) and the local standard deviation δ(x, y) within the window size w × w.

For example, the local threshold value T(x, y) may be calculated by the following equation within a window area of size w × w formed around the position (x, y) of the candidate core pixel on the raw image. .

Here, k is the bias value of system 1 and w is the window index.

9 is a flow diagram of a binarization process using a threshold calculated through a local average, according to one embodiment.

Referring to FIG. 9 , the threshold for binarization may be calculated using a local average. The local threshold value T(x, y) is calculated based on the local mean m(x, y) within the local window area of size w × w formed around the position (x,y) in the raw image.

For example, the local threshold value T(x, y) at location (x, y) may be calculated by the following equation within a window of size w × w.

Here, k is the bias value of system 1.

The size of the window may be determined by statistical results. For example, a window with a pixel size of w=7 may be used.

This image processing method and the fiberscope system 1 for performing it effectively reduce the overall processing time while greatly reducing the complexity of core peak detection implementation compared to conventional embodiments utilizing pre-correction.

Specifically, in order to remove the noise of the honeycomb pattern, conventional image processing for correcting the sample image preprocesses the sample image using a white reference image, which is a high-contrast image taken in advance before the experiment, Experimental images (ie, target images) were interpolated using the fiber core coordinates obtained in this process.

On the other hand, according to the embodiments of the present application, it is possible to reconstruct a sample image in which the core and cladding are accurately separated using only the raw image of the actually obtained sample without correction.

According to another aspect of the present application, a fiberscope image processing method performed by a computing device including a processor may process images of a plurality of frames to implement a sequence.

10 is a flowchart of a fiberscope image processing method according to another aspect of the present application.

Since the steps of the fiberscope image processing method of FIG. 10 are similar to those of the fiberscope image processing method of FIG. 5, differences will be mainly described.

The fiberscope image processing method includes: acquiring a plurality of raw images of a sample (S1100). Here, the plurality of raw images are sequence images, and may be frames of a video, for example. The plurality of raw images are images obtained in an environment where the view from the fiberscope system 1 (eg, the objective lens 21) to the sample is the same.

The image processing method includes: selecting at least one raw image as a reference image among a plurality of raw images according to a predetermined criterion (S1300). The selected reference image is an image used for interpolation of the remaining raw images that have not been selected, and is subject to binarization.

In one embodiment, the preset criterion may be a frame number. A frame that meets the criterion of the frame number may be a frame having a specific single number (eg, the first frame) or frames having numbers within a specific range (eg, the first to nth frames).

The image processing method may include: binarizing each selected raw image (S1400); Detecting a peak point (S1500); and interpolating (S1600). Since these steps (S1400, S1500, and S1600) are similar to the operations of the steps (S400, S500, and S600) of FIG. 5, detailed descriptions are omitted.

In the image processing method of FIG. 10 , the remaining raw images that are not selected are interpolated using the intensities of core pixels allocated for interpolation of the reference image.

Referring to the above step (S600), a peak point is detected in the selected reference image (S1500), and for cladding pixels in the reference image distributed around the core pixel in which the peak point is detected, adjacent in the reference image The intensity of the core pixel is assigned.

Since the view from the fiberscope system 1 to the sample is the same for all of the multiple raw images, the relative positions of the individual honeycombs, i.e., the core/cladding, of the honeycomb pattern structure shown in each raw image are the same. Therefore, the position of each of the remaining raw images corresponding to the position of the peak point detected in the reference image can be regarded as a peak point without separate peak detection.

Intensities of adjacent core pixels in the reference image are assigned to cladding pixels in each raw image distributed around a position corresponding to a core pixel of the reference image in each of the remaining raw images that have not been sorted. That is, the cladding area of the remaining raw image is interpolated with the intensity of the core pixel of the reference image (S1600).

In one example, when the reference image is a single raw image, cladding pixels of the remaining raw images distributed around positions corresponding to core pixels on the single reference image that have already been interpolated are assigned the intensities of the core pixels of the single reference image. may be

In another example, when some raw images (eg, 10 images) are selected, binarized, and interpolated in step S13000, the remaining raw images distributed around positions corresponding to core pixels on a plurality of reference images that have already been interpolated Representative values of intensities of core pixels of the plurality of interpolated reference images may be assigned to the cladding pixels. Here, the assigned representative value may be an average value of intensities of core pixels of a plurality of reference images, but is not limited thereto and may be other representative values such as minimum, maximum, and mode values.

In this way, by interpolating the remaining raw images using the result of interpolating only the reference image (eg, the first raw image), multiple fiberscope images can be quickly restored from multiple raw images.

In alternative embodiments, some of the plurality of raw images obtained in step S100 are obtained in a first view into the sample from fiberscope system 1 (eg, objective lens 21 ) and , other parts are obtained in the second view from the fiberscope system 1 (eg, the objective lens 21) to the sample, the reference images may be selected and used for subgroup interpolation for each view. .

At least one first reference image may be selected and used for the original image of the first view. The first reference image is selected from raw images of the first view.

At least one second reference image may be selected and used for the original image of the second view. The second reference image is selected from raw images of the second view.

In alternative embodiments, the fiberscope image processing method may further include removing camera noise (S1200). Since step S1200 is similar to step S200, a detailed description thereof will be omitted.

Experimental Example 1

In order to clarify the effect of the fiber scope system 1 of FIG. 1 and the method of FIG. 6, image processing results using the global threshold value and image processing using the local threshold value according to Equations 9 to 11 in the above experimental example Results were compared.

The global threshold was calculated according to Non-Patent Document 4 (N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Transactions on Systems, Man, and Cybernetics 9, 62-66 (1979)). Based on Non-Patent Document 4, the threshold T (x, y) was automatically selected by minimizing the variance within the class of white and black pixels. A global threshold is specified as a single threshold for the entire image. Each pixel is compared against a specified global threshold at conversion time.

In the above experimental example, simulated images similar to raw images of real fiber scopes were generated to evaluate whether each threshold value could accurately select only core pixels. The core pixel follows a Gaussian distribution according to the beam profile of the fiber according to the distance r from the center, and the cladding pixel expresses the light spread calculated in the core pixel by adjusting the intensity level through the peak intensity. That is, in the experimental example, the simulated image controls not only the intensity level of the core but also the intensity level of the cladding. The intensity level of a core pixel C(x, y) having a core diameter following a Gaussian distribution for a peak point Peak (x, y) may be expressed by the equation below.

n is the square matrix size of the peak intensity, the variable times is a multiple of the total core intensity, r is the distance between the peaks and the value is 0 < r < core diameter. a may be an empirical value for each image calculated by Equations 5 to 7. The cladding value is specified through interpolation of the point Peak (x, y) Х cladding parameter.

FIG. 11 shows simulated images for each case according to the first experimental example, and FIGS. 12A and 12B show threshold value evaluation for binarization using the simulated images for each case in FIG. 11 .

In the case-by-case image of FIG. 11 , simulated core pixels are displayed in white, and simulated cladding pixels are displayed in black. Also, among the pixels selected as candidate core pixels by each threshold, simulated core pixels are displayed in blue and simulated cladding pixels are displayed in red.

Referring to FIG. 12A , in the case of using the global threshold, the largest number of cladding pixels selected for binarization (ie, candidate core pixels) is found compared to other local threshold values. On the other hand, cases using the local threshold have a relatively small number of cladding pixels incorrectly selected as candidate core pixels, and in particular, the cases using the local threshold according to Equations 10 and 11 almost match the number of simulated core pixels. A candidate core pixel is selected.

12a confirms that binarization using a global threshold results in storage results with high distortion and loss for a simulated test image in which the intensity distribution is not uniform and varies over the entire region of the raw image including the cladding.

12B shows the result of measuring and comparing average runtimes of post-image processing tasks for each threshold.

As shown in Fig. 12b, the processing time of the peak detection step is reduced in cases using the local threshold. Since most of the entire time of the image processing process is used for peak detection, it is confirmed that the embodiments of the present application using a local threshold value save time and resources compared to conventional embodiments using a global threshold value. .

Second Experimental Example

In the second experimental example, a post-image processing result through correction and a post-image processing result requiring no correction according to the embodiments of the present application were compared.

13 is a raw image of a sample according to the second experimental example.

In the second experimental example, post-image processing through correction was performed by applying interpolation to core peak points already obtained through correction using a reference image obtained by cropping the green light portion in the raw image of FIG. 6 .

In post-image processing according to embodiments of the present application, binarization (S400), peak detection (S500), and interpolation (S600) are sequentially applied using only the raw image of FIG. 13 . Here, the value calculated by Equation 10 is used as an example of the local threshold.

FIG. 14A is a result of post-image processing of the raw image of FIG. 13 through correction, and FIG. 14B is a result of post-image processing of the raw image of FIG. 13 without correction.

It can be visually confirmed that the interpolation result using the binarization based on the local threshold of FIG. 14b produced a clearer reconstructed image than the interpolation result using the correction of FIG. 14a.

The fiberscope image processing method according to the embodiments described above and the operation by the fiberscope system 1 for performing the method may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented together with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be any device that may be integrated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device that has one or more alternative and special purpose processors, memory, storage, and networking components (whether wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, Apple's OS X or iOS, a Linux distribution, or an operating system such as Google's Android OS.

The computer-readable recording medium includes all types of recording and identification devices in which data readable by a computer is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage and identification devices, and the like. In addition, computer-readable recording media may be distributed in computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

Those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: fiber bundle
11: body
13: fiber
21: objective lens
22: color sorting mirror
23: polarization beam splitter
25: lens
27: dual color separator
29: camera device
31: first light source
41, 42: second light source
50: computing device

Claims (14)

In the fiber scope image processing method performed by a fiber scope system including a fiber bundle composed of a plurality of cores surrounded by cladding,
acquiring at least one raw image of the sample from a camera device, wherein the raw image shows a honeycomb pattern due to the structure of the fiber bundle;
In order to separate the core and the cladding in the raw image, the raw image is binarized based on the intensity of pixels of the raw image and a threshold value, and when a pixel on the raw image has an intensity value greater than or equal to the threshold value, Candidate core pixels are selected, the intensity of the selected candidate core pixels is converted into a specific value, and when a pixel on the raw image has an intensity value less than a threshold value, a cladding pixel is selected, and the intensity of the selected cladding pixel is determined. Binarizing the raw image by converting it into another specific value;
detecting a peak point in the binarized image, generating a binary mask composed of the candidate core pixels, and calculating a Gaussian distribution value only for pixels selected with the binary mask to detect the peak point; and
In order to reconstruct the fiberscope image of the sample, interpolating a raw image based on the detected peak point, and allocating an intensity of a core pixel adjacent to a cladding pixel around the core pixel at which the peak point is detected Scope image processing method.
delete According to claim 1,
The threshold for binarization is a local threshold, and is calculated based on intensities of pixels in a local window including a peak point detected in the raw image.
According to claim 3,
The threshold for the binarization is a maximum gray level value and a minimum gray level value in a local window of size w × w, formed around the position of the candidate core pixel, a contrast value at the position of the candidate core pixel, and a fiberscope image processing method characterized in that the calculation is based on a contrast threshold value preset by a user.
According to claim 3,
The threshold for the binarization is calculated based on the local variance in a local window of size w × w formed around the position of the candidate core pixel, and calculated based on the local average and local standard deviation within the local window Fiberscope image processing method, characterized in that.
According to claim 3,
The threshold value for binarization is calculated based on a local average in a local window of size w × w formed around the position of the candidate core pixel.
delete delete According to claim 1,
Prior to binarization, the fiberscope image processing method further comprising the step of removing camera noise from the raw image.
According to claim 1,
When a plurality of raw images are acquired, further comprising selecting at least one raw image as a reference image from among the plurality of raw images according to a preset criterion;
A fiberscope image processing method, characterized in that for binarizing only the reference image and detecting a peak point in the binarized reference image.
11. The method of claim 10, wherein the interpolating step,
For cladding pixels in the reference image distributed around core pixels in which peak points are detected, intensities of adjacent core pixels in the reference image are allocated, and in each of the remaining raw images not selected as the reference image, the reference image A fiberscope image processing method, characterized in that allocating intensities of adjacent core pixels in the reference image to cladding pixels in each raw image distributed around a position corresponding to a core pixel in the reference image.
According to claim 11,
When the reference images are two or more raw images, the allocated core pixel intensity is a representative value of core pixel intensity used for interpolation of each reference image.
A computer-readable recording medium recording a program for performing the fiberscope image processing method according to any one of claims 1, 3 to 6, and 9 to 12.
In the fiberscope system,
a probe including a fiber bundle composed of a plurality of cores surrounded by a cladding;
an objective lens configured to simultaneously receive light from fibers included in the fiber bundle;
a dual color fluorescence imaging system for acquiring fluorescence images of different color fields;
a laser light stimulation system sharing a fiber bundle with the dual color fluorescence system;
A camera device generating raw image data of the sample, wherein the raw image exhibits a honeycomb pattern due to the structure of the fiber bundle; and
Acquiring at least one raw image of the sample from a camera device, and binarizing the raw image based on pixel intensity and a threshold value of the raw image to separate a core and a cladding in the raw image, If a pixel on the raw image has an intensity value greater than or equal to a threshold value, it is selected as a candidate core pixel, the intensity of the selected candidate core pixel is converted to a specific value, and the pixel on the raw image has an intensity value less than the threshold value. , select cladding pixels, convert intensities of the selected cladding pixels to other specific values to binarize the raw image, detect peak points in the binarized image, and create a binary mask composed of the candidate core pixels; The peak point is detected by calculating the value of the Gaussian distribution only for the pixels selected with the binary mask, and the raw image is interpolated based on the detected peak point to reconstruct the fiberscope image of the sample, but the peak point is detected A fiberscope system comprising a computing device configured to interpolate a raw image by assigning intensities of adjacent core pixels to surrounding cladding pixels of the adjacent core pixels.

Images