JP7189173B2 - IMAGE PROCESSING METHOD, IMAGING DEVICE AND IMAGE PROCESSING DEVICE - Google Patents

Description

The present invention relates to an image processing method, an imaging device, and an image processing device.

Endoscopes with image guide fibers that include multiple cores are known. Each of the plurality of cores transmits incident light after being reflected by the surface being observed. The gaps between the multiple cores are filled with clad. Incident light is transmitted while repeating total reflection at the interface between the core and the clad, and is confined within the core. On the other hand, since the clad hardly transmits light, the luminance of the region corresponding to the clad is low in the region of the image of the observation target displayed on the display device connected to the endoscope. The observer thus sees a so-called mesh pattern.

Patent Literature 1 discloses a technique for removing such a mesh pattern. Specifically, in the displayed image of the observation target, the colors of the region corresponding to the core and the region corresponding to the outer cladding are replaced with the color of one point in the region corresponding to the core. According to this document, the color of the region corresponding to the cladding is interpolated based on the color of the region corresponding to the core, and the mesh pattern can be removed or reduced.

JP-A-8-191439

However, in the method disclosed in Patent Document 1, by replacing the color of the region corresponding to the core and the region corresponding to the outer cladding with the color of one point in the region corresponding to the core, Since the different luminance and chromaticity information for each point in the region is replaced with a single luminance and chromaticity, the resolution of the displayed image is degraded.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing method capable of substantially removing a mesh pattern while suppressing degradation in resolution of an image displayed by an endoscope system.

The main invention for achieving the above object is an endoscope having an image guide fiber including a plurality of cores that transmit incident light after being reflected on the surface of an observation target, and the endoscope, and connectable to the endoscope, an imaging device having an image sensor that detects the light and converts it into an electrical signal; An image processing method in an endoscope system comprising: a step of photographing a reference plane to generate reference image data; deriving a luminance unevenness correction operation for uniforming luminance components; photographing an observation target to generate observed image data; and performing an operation. Other features of the present invention will be clarified by the description of the specification and drawings described later.

According to the present invention, it is possible to provide an image processing method capable of substantially removing a mesh pattern while suppressing degradation in resolution of an image displayed by an endoscope system.

BRIEF DESCRIPTION OF THE DRAWINGS It is the schematic explaining the whole structure of the endoscope system of embodiment. It is a figure which shows the structure of the endoscope of embodiment. It is a figure which shows the cross section of the insertion part in the endoscope of embodiment. It is a figure which shows the cross section of the insertion part in the endoscope of embodiment. 1 is a functional block diagram of an imaging device according to an embodiment; FIG. It is a figure explaining the path|route of the light which reaches|attains the image sensor of embodiment. 4 is a flowchart of an image processing method according to an embodiment; 4 is a table showing image processing conditions and evaluation results of Examples. 10 is a display image when image processing is performed on the reference image under the conditions of Example 1. FIG. 4 is a display image obtained by subjecting an observed image to image processing under the conditions of Example 1. FIG. It is a display image when image processing is performed on the reference image under the conditions of Example 2. FIG. It is a display image when image processing is performed on the observed image under the conditions of Example 2. FIG. It is a display image when image processing is performed on the reference image under the conditions of Comparative Example 1. FIG. It is a display image when image processing is performed on the observed image under the conditions of Comparative Example 1. FIG. It is a display image when image processing is performed on the reference image under the conditions of Comparative Example 2. FIG. It is a display image when image processing is performed on the observed image under the conditions of Comparative Example 2. FIG. It is a display image when image processing is performed on the reference image under the conditions of Example 3. FIG. It is a display image when image processing is performed on the observed image under the conditions of Example 3. FIG. It is a display image when image processing is performed on the reference image under the conditions of Comparative Example 3. FIG. It is a display image when image processing is performed on the observed image under the conditions of Comparative Example 3. FIG. It is a display image when image processing is performed on the reference image under the conditions of Comparative Example 4. FIG. It is a display image when image processing is performed on the observation image under the conditions of Comparative Example 4. FIG. 10 is a display image when image processing is performed on the reference image under the conditions of Comparative Example 5. FIG. It is a display image when image processing is performed on the observed image under the conditions of Comparative Example 5. FIG.

<Embodiment>
== Endoscope System ==
FIG. 1 is a diagram illustrating the configuration of an endoscope system 1 according to this embodiment. The endoscope system 1 includes an endoscope 2 , an imaging device 3 , a light source device 4 and a display device 5 .

[Endoscope]
FIG. 2 is a diagram illustrating the configuration of the endoscope 2 of this embodiment. The endoscope 2 is, for example, a medical device intended to observe the inside of a living body (that is, an observation target) such as the lacrimal duct. The endoscope 2 of this embodiment has an insertion section 21 , a grip section 22 and a lead section 23 .

The insertion section 21 is an elongated member that is inserted into an observation target. As shown in FIGS. 1 and 2, the insertion portion 21 is connected to the grip portion 22 at its proximal end.

FIG. 3 is a cross-sectional view of a plane perpendicular to the longitudinal direction of the insertion portion 21 and through which the image guide fiber 212 passes. FIG. 4 is a cross-sectional view of a plane passing through the axis of the insertion portion 21. As shown in FIG. The insertion section 21 has an objective lens 211 , an image guide fiber 212 , a light guide fiber 213 and an exterior pipe 214 .

The objective lens 211 is a lens for condensing the light reflected by the surface of the observation target. The objective lens 211 is provided at the distal end of the insertion section 21 .

The image guide fiber 212 transmits the light incident through the objective lens 211 to the imaging device 3 . Image guide fiber 212 includes a plurality of cores 212a, claddings 212b and jackets 212c. Each of the plurality of cores 212 a transmits light that is reflected by the surface of the observation target and has entered through the objective lens 211 . The material of each of the plurality of cores 212a is, for example, quartz, glass, plastic, or the like.

The cladding 212b fills the gaps outside the plurality of cores 212a. The material of the clad 212b is, for example, quartz, glass, plastic, or the like, and has a lower refractive index than the core 212a. A jacket 212c covers the plurality of cores 212a and claddings 212b.

A light guide fiber 213 illuminates the observation target. In this embodiment, the insertion section 21 has five light guide fibers 213 . The light guide fiber 213 of this embodiment includes a core 213a and a cladding 213b. The core 213 a transmits light from the light source device 4 . Cladding 213b fills the outside of core 213a. Materials for the core 213a and the clad 213b are not particularly limited. For example, it is made of the same material as the core 212a and the clad 212b.

The exterior pipe 214 is a tubular member in which the image guide fibers 212 and the light guide fibers 213 are arranged. The exterior pipe 214 is made of metal such as stainless steel. Inside the exterior pipe 214, the outsides of the objective lens 211, the image guide fiber 212 and the light guide fiber 213 are filled with a filler 214a.

The grip portion 22 is a portion that is gripped by the observer when using the endoscope 2 . A base end portion of the insertion portion 21 is connected to one end of the grip portion 22 . A lead portion 23 is connected to the other end of the grip portion 22 .

The lead portion 23 is connected to the grip portion 22 . The lead portion 23 is branched into two via a branch portion 23a. An image guide plug 23b is provided at one branched end, and a light guide plug 23c is provided at the other end.

An end portion of an image guide fiber 212 extending from the insertion portion 21 through the grip portion 22 and the branch portion 23a is connected to the image guide plug 23b. An end portion of a light guide fiber 213 extending from the insertion portion 21 through the grip portion 22 and the branch portion 23a is connected to the light guide plug 23c.

[Imaging device]
FIG. 5 is a functional block diagram of the imaging device 3 of this embodiment. FIG. 6 is a diagram illustrating the path of light reaching the image sensor 31 of this embodiment. The imaging device 3 detects light from the endoscope 2, generates image data of an observation target, and applies image processing to the image data. Then, an output signal corresponding to the image data subjected to image processing is output to the display device 5 . The imaging device 3 is provided with an endoscope terminal 3a (see FIG. 1). The endoscope 2 is connected to the imaging device 3 by connecting the image guide plug 23b of the endoscope 2 to the endoscope terminal 3a. The imaging device 3 has an eyepiece lens 30 , an image sensor 31 , an image generator 32 , an image processor 33 and an output signal generator 34 .

(eyepiece)
The eyepiece 30 collects the light transmitted to the image guide fiber 212 and forms an enlarged image of the observation target on the image sensor 31, as shown in FIG.

(image sensor)
The image sensor 31 detects light through the eyepiece lens 30 and converts it into an electrical signal corresponding to the intensity of the light. As the image sensor 31, for example, CMOS, CCD, or the like can be used.

The image sensor 31 is provided with a plurality of pixels. A plurality of pixels are arranged in a matrix. In this embodiment, each of a plurality of pixels is provided with one light receiving element for detecting red, green, or blue light. For example, a photodiode or the like can be used as the light receiving element.

(Image generator)
The image generator 32 generates image data based on the electrical signal converted by the image sensor 31 . Image data is data for representing an image. Image data is composed of a plurality of pixel data. Each of the plurality of pixel data has pixel information and color information.

The pixel information is information specifying a pixel corresponding to each of the plurality of pixel data. Pixel information may be represented by, for example, one integer i (i=1 to Np). Np is a value corresponding to the number of pixels provided in the image sensor 31 . Alternatively, the pixel information may be represented by two integers m and n (m=1 to Nr, n=1 to Nc) indicating the rows and columns in which the pixels are arranged. Nr and Nc are values corresponding to the numbers of rows and columns of a plurality of pixels arranged in a matrix on the image sensor 31, respectively.

The color information is information expressing the color of light detected by the light-receiving element arranged in the pixel specified by the pixel information. For color information, representation using the RGB space (hereinafter referred to as RGB representation) is used. Specifically, color information represented by RGB is represented by luminance values of R, G, and B, respectively. The luminance values of R, G, and B are each represented by 256 gradations within a range of 0 to 1, for example.

In this embodiment, each of the plurality of pixel data has color information (R(i), G(i), B(i)) expressed in RGB. i is the corresponding pixel information. R(i), G(i), and B(i) are based on electrical signals output from light receiving elements that detect red, green, and blue light, respectively, arranged in pixels specified by pixel information i. Determined.

In the image processing method of this embodiment, which will be described later, the image data generated by the image generator 32 are reference image data and observation image data. The reference image data is image data for calibrating the endoscope system 1 . The reference image data is image data generated by photographing a reference plane prepared in advance. The reference surface is the surface of a flat plate or the like which is colored in a monochromatic and achromatic color.

Observation image data is image data generated by photographing an observation target. A plurality of observation image data are generated at predetermined time intervals. Each of the plurality of observed image data is image data that forms a frame for displaying a moving image of an observation target on the display device 5 .

Hereinafter, when simply referred to as "image data", both reference image data and observation image data are applicable.

(Image processing unit)
The image processing unit 33 performs luminance unevenness correction, color unevenness correction, and smoothing on the image data generated by the image generation unit 32 .

Brightness unevenness correction is processing performed on image data in order to remove or reduce brightness unevenness. The luminance unevenness refers to variations in brightness at various locations in an image displayed on the display device 5 (hereinafter referred to as a display image). The cause of uneven brightness will be described below.

The light incident on the core 212a propagates while repeating total reflection at the interface between the core 212a and the clad 212b. However, due to variations in the manufacturing process of the image guide fiber 212 and deformation and deterioration due to long-term use, the light propagating through the core 212a is not totally reflected at the interface between the core 212a and the clad 212b, and some part of the light may enter the clad 212b side. In such a case, even if the reference surface or the observation object is monochromatic, the amount of light that is reflected on the surface, enters the core 212a, propagates, and reaches the image sensor 31 varies from core 212a to core 212a. .

Therefore, if the image data is not subjected to luminance unevenness correction, luminance unevenness may occur in the displayed image. A specific processing procedure for luminance unevenness correction will be described later.

It should be noted that the so-called mesh pattern can be substantially removed by the luminance unevenness correction of this embodiment. A mesh pattern refers to a mesh-like pattern that occurs in a display image due to the cladding 212b. The cause of the generation of the mesh pattern will be described below.

Part of the light incident on the clad 212 b side propagates through the clad 212 b and reaches the image sensor 31 . The amount of light that propagates through the cladding 212b and reaches the image sensor 31 is smaller than the amount of light that propagates through the plurality of cores 212a and reaches the image sensor 31 . Therefore, in the image of the observation target displayed on the display device 5, the brightness of the region corresponding to the clad 212b is lower than the brightness of other regions. As a result, the region corresponding to the clad 212b is visually recognized as a mesh pattern.

Further, color unevenness correction is processing performed on image data in order to remove or reduce color unevenness. Color unevenness refers to variations in color at various locations in a displayed image. The cause of color unevenness will be described below.

As described above, the light propagating through the core 212a may not be totally reflected at the interface between the core 212a and the clad 212b, and part of it may enter the clad 212b side. At this time, only light in a partial wavelength range may enter the clad 212b side. In such a case, even if the surface of the observation target is achromatic, the light that is reflected by the surface of the observation target, enters the core 212a, propagates, and reaches the image sensor 31 may become chromatic. be.

Therefore, if the image data is not subjected to color unevenness correction, color unevenness may occur in the image of the observation target displayed on the display device 5 . A specific processing procedure for color unevenness correction will be described later.

Smoothing is a process for removing noise from the image of the observation target displayed on the display device 5 or blurring the image. For example, smoothing is performed by converting the color information of pixel data corresponding to each of a plurality of pixels arranged in a matrix to the color of pixel data of a plurality of pixels (for example, 9 or 25 pixels) centering on that pixel. It can be realized by replacing with an average value of information. Smoothing is performed using a predetermined smoothing filter. A predetermined smoothing filter is determined based on the diameter of the core 212 a , the density of the core 212 a , the image magnification of the eyepiece 30 and the pixel density of the image sensor 31 .

The density of the cores 212a is the number density of the cores 212a inside the jacket 212c. Specifically, it is the number obtained by dividing the number of cores 212a by the cross-sectional area inside the jacket 212c. The pixel density is, for example, a density defined by the number of pixels arranged per inch (ppi).

In smoothing, the greater the number of pixels for taking the average value, the greater the reduction of the mesh pattern, but the greater the deterioration of the resolution of the displayed image.

In particular, in smoothing, by increasing the number of pixels to be averaged, the cross-hatch pattern can be substantially eliminated, but the resolution of the displayed image is significantly degraded.

On the other hand, when the smoothing (weak smoothing) is performed under the condition that the degradation of the resolution of the display image cannot be visually recognized with the naked eye, a mesh pattern that is easily visible with the naked eye remains. However, when luminance non-uniformity correction is performed along with weak smoothing, the mesh pattern can be removed while suppressing deterioration of the resolution of the displayed image. A specific processing procedure for smoothing will be described later.

In this specification, the image data immediately after being generated by the image generation unit 32 is referred to as "image data subjected to image processing" or the like. The “image processing” here includes any one or more of image processing of brightness unevenness correction, color unevenness correction, and smoothing.

Before describing the procedure for the image processing unit 33 to perform luminance unevenness correction and color unevenness correction, YUV representation will be described. In the following description, the procedure for performing luminance unevenness correction and color unevenness correction will be described using a method using the YUV color space, but the color space used is not limited to this. Any color space may be used as long as it can numerically represent the index of luminance and the index of saturation. For example, RGB space, CIE-XYZ color system, CIE-L*a*b Color space, HSI space hexagonal pyramid model (HSV), HSI space bihexagonal pyramid model (HSL), etc. may be used. Also, a color space obtained by performing linear or nonlinear conversion of color information in each color space may be used. The image processing unit 33 performs processing using YUV representation when performing luminance unevenness correction and color unevenness correction. Specifically, the image processing unit 33 first converts color information of image data represented by RGB into color information represented by YUV.

Color information in YUV representation is indicated by Y, U and V. Y, U, and V are obtained by applying a predetermined linear transformation to color information represented by RGB. Several definitions of the given linear transformation have been proposed. In the following description, the definition of ITU-R BT.709 will be used, but the definition is not limited to ITU-R BT.709. 601, PAL standard, or the like may be used. Color information in YUV representation defined by ITU-R BT.709 is obtained by performing the following linear transformation on color information in RGB representation.

Y=0.2126R+0.7152G+0.0722B
U=0.5389(BY)=-0.1146R-0.3854G+0.5000B
V=0.6350(RY)=0.5000R-0.4542G-0.0458B

According to the definition of ITU-R BT.709, color information represented by RGB is obtained by performing the following linear transformation on color information represented by YUV.
R=Y+1.575V
G=Y-0.187U-0.468V
B = Y + 1.856U

Y is a value indicating the brightness of the color represented by the color information. Color information (Y, U, V)=(Y, 0, 0) in YUV representation when U and V are 0 corresponds to an achromatic color whose luminance is Y. Also, color information (Y, U, V)=(0, U, V) in YUV representation when Y is 0 is (R, G, B)=(0, 0, 0) in RGB representation. Therefore, it corresponds to black.

U is a value that indicates the relative B luminance when compared to the color information (Y, 0, V) when it is 0; V is a value that indicates the relative R luminance when compared to the color information (Y, U, 0) when it is 0;

In this specification, Y of color information (Y, U, V) in YUV representation may be referred to as "luminance component", and U and V may be referred to as "chrominance components".

Color information (Yc(i), Uc(i), Vc(i)) of each of the plurality of pixel data constituting the reference image data is used in the process of luminance unevenness correction and color unevenness correction. Here, i is a number for specifying a pixel, and as described above, i is an integer from 1 to Np (the same applies hereinafter).

The color information of light reflected from a monochromatic reference surface is uniform across the reference surface. However, the brightness of the light reflected by the reference surface, propagated through the plurality of cores 212a, and reached the image sensor 31 is not necessarily uniform for the reason described above.

Therefore, even if the reference plane is monochromatic, the luminance component Yc(i) of the light reaching the image sensor 31 is not always constant for all i (i=1 to Np).

Further, the color difference component of the color information of the light reflected by the achromatic reference plane is zero. However, as described above, the light that is reflected by the reference surface, propagates through the plurality of cores 212a, and reaches the image sensor 31 may become chromatic.

Therefore, even if the reference surface is achromatic, the color difference components Uc(i) and Vc(i) of the light reaching the image sensor 31 are not always zero.

A specific process procedure for luminance unevenness correction performed by the image processing unit 33 will be described below. The uneven brightness correction is realized by the image processing unit 33 deriving a brightness unevenness correction calculation and applying the brightness unevenness correction calculation to the image data.

The luminance unevenness correction calculation is a calculation that, when applied to the reference image data, uniforms the luminance components of the color information of each of the plurality of pixel data constituting the reference image data.

Hereinafter, the process of deriving the uneven brightness correction calculation by the image processing unit 33 will be described, and then the process of applying the uneven brightness correction calculation to the image data will be described.

The brightness non-uniformity correction calculation is a calculation using a brightness non-uniformity correction value. In order to derive the luminance non-uniformity correction calculation, first, the luminance non-uniformity correction value is obtained.

The brightness unevenness correction value is the brightness of Y0, which is a brightness value set in advance, and the color information (Yc(i), Uc(i), Vc(i)) of each of the plurality of pixel data constituting the reference image data. component Yc(i).

Specifically, the luminance unevenness correction calculation is performed by adding Y0/ By multiplying Yc(i), it is an operation for converting into color information (Y(i)Y0/Yc(i), U(i), V(i)).

When luminance unevenness correction calculation is applied to each of a plurality of pixel data constituting the reference image data, the color information (Yc(i), Uc(i), Vc(i)) becomes (Y0, Uc(i) , Vc(i)). In other words, as described above, the luminance unevenness correction calculation makes the luminance components of all the color information of the plurality of pixel data forming the reference image data uniform.

Next, a specific processing procedure of color unevenness correction executed by the image processing unit 33 will be described. The color unevenness correction is realized by the image processing unit 33 deriving the color unevenness correction calculation and applying the color unevenness correction calculation to the image data.

The color unevenness correction operation is an operation for removing the color difference component of the color information of each of the plurality of pixel data forming the reference image data when it is applied to the reference image data.

Hereinafter, the process of deriving the color unevenness correction value calculation by the image processing unit 33 will be described, and then the process of applying the color unevenness correction calculation to the image data will be described.

The color nonuniformity correction calculation is a calculation using color nonuniformity correction values. In order to derive the color non-uniformity correction calculation, first, the color non-uniformity correction value is obtained.

The color unevenness correction value is the color difference components Uc(i) and Vc(i) of the color information (Yc(i), Uc(i), Vc(i)) of each of the plurality of pixel data constituting the reference image data. be.

Specifically, the color unevenness correction calculation is performed by correcting U(i) of the color information (Y(i), U(i), V(i)) of each of the plurality of pixel data constituting the image data. This is an operation for subtracting the value Uc(i) and subtracting V(i) by the color difference correction value Vc(i).

When the color unevenness correction calculation is applied to each of the plurality of pixel data constituting the reference image data, the color information (Yc(i), Uc(i), Vc(i)) is converted to (Yc(i), 0 , 0). That is, as described above, the color difference components of all the color information of the plurality of pixel data forming the reference image data are removed by the color unevenness correction calculation.

Although brightness unevenness correction, color unevenness correction, and smoothing have been described above, the image processing unit 33 can also perform image processing such as brightness correction, contrast correction, white balance correction, and gamma correction. A known technique can be used for these processes.

(output signal generator)
The output signal generator 34 generates an output signal based on the image data processed by the image processor 33 and outputs the output signal to the display device 5 .

[Light source device]
The light source device 4 supplies the light guide fiber 213 with light for illuminating the observation target. The light source device 4 is provided with a socket 4a (see FIG. 1). The endoscope 2 is connected to the light source device 4 by connecting the light guide plug 23c of the endoscope 2 to the socket 4a. This allows the light source device 4 to supply the light guide fiber 213 with light for illuminating the observation target. The light source device 4 has a light source such as an LED or a halogen lamp that generates light.

[Display device]
The display device 5 displays the image of the observation target based on the output signal output by the output signal generator 34 .

<Image processing method>
The image processing method executed by the imaging device 3 of this embodiment will be described in detail. FIG. 7 is a flowchart for explaining the image processing method of this embodiment.

The image processing method of the present embodiment comprises step S1 of generating reference image data, step S2 of deriving brightness non-uniformity correction calculation, step S3 of deriving color non-uniformity correction calculation, and step S4 of generating observed image data. , a step S5 of performing brightness unevenness correction calculation, a step S6 of performing color unevenness correction calculation, a smoothing step S7, and a step S8 of displaying on the display screen. Each will be described in detail below.

1. Step S1 of generating reference image data
In step S1, the image generator 32 generates reference image data.

Specifically, when the observer switches on the light source device 4 while the endoscope 2 is connected, the light source device 4 generates light and supplies it to the light guide fiber 213 . The light guide fiber 213 propagates the supplied light and emits it from the distal end of the endoscope 2, so the observer directs the endoscope 2 toward the reference plane. Thereby, the light source device 4 illuminates the reference plane through the light guide fiber 213 . In this state, the observer presses a calibration button (not shown) of the imaging device 3 . In response to this, the image generator 32 generates reference image data. In other words, the image generator 32 generates pixel information and color information (Rc(i), Gc(i), Bc(i)) of each of the plurality of pixel data forming the reference image data.

2. Step S2 of deriving luminance unevenness correction calculation
In step S2, the image processing unit 33 derives luminance unevenness correction calculation.

Specifically, first, the color information (Rc(i), Gc(i), Bc(i)) of each of the plurality of pixel data constituting the reference image data generated by the image generation unit 32 in step S1 is Convert to color information (Yc(i), Uc(i), Vc(i)) expressed in YUV space.

Then, the image processing unit 33 acquires the preset luminance value Y0 and the luminance component Yc(i) of the color information as the luminance unevenness correction value.

3. Step S3 for Deriving Color Unevenness Correction Calculation
In step S3, the image processing unit 33 derives color unevenness correction calculation.

First, the image processing unit 33 calculates the color difference component Uc( i) and Vc(i).

4. Step S4 of generating observation image data
In step S4, the image generator 32 generates observation image data.

Specifically, in step S4, the observer keeps the switch of the light source device 4 on. Then, the light from the light source device 4 propagates through the light guide fiber 213 of the endoscope 2 and is maintained in a state of being emitted from the distal end portion of the endoscope 2, so that the observer observes the endoscope 2. Aim at the target. Thereby, the light source device 4 illuminates the observation target through the light guide fiber 213 . In this state, when the observer presses a shooting button (not shown) of the imaging device 3, the imaging device 3 starts observation. In response to this, the image generator 32 generates image data of the observation target. That is, the pixel information and color information (Y'(i), U'(i), V'(i)) of each of the plurality of pixel data constituting the observed image data are obtained.

5. Step S5 of performing luminance unevenness correction calculation
In step S5, the image processing unit 33 performs luminance unevenness correction calculation on each of the plurality of pixel data constituting the image data of the observation target.

Specifically, the image processing unit 33 performs the uneven brightness correction calculation derived in step S2 on each color information (Y'(i), U'(i), V'(i)) of the pixel data. , (Y'(i)Y0/Yc(i), U'(i), V'(i)).

6. Step S6 for color unevenness correction calculation
In step S6, the image processing unit 33 performs color unevenness correction calculations on each of the plurality of pixel data constituting the image data of the observation target.

Specifically, the color unevenness correction derived in step S3 is applied to each of the color information (Y'(i), U'(i), V'(i)) of the plurality of pixel data constituting the image data to be observed. Perform calculations and convert as follows.
(Y'(i), U'(i), V'(i)) → (Y'(i), U'(i)-Uc(i), V'(i)-Vc(i))

7. Smoothing step S7
In step S7, the image processing unit 33 smoothes each of the plurality of pixel data forming the image data of the observation target.

8. Step S8 of displaying on the display device
In step S8, the output signal generator 34 generates an output signal based on the observed image data subjected to image processing in steps S5 to S7, and outputs the output signal to the display device 5. FIG. The observed image data that has been subjected to image processing in steps S5 to S7 is assumed to be image data of one frame of the observation target. The display device 5 displays a one-frame image of the observation target based on the output signal.

Through the above steps, the image processing method of this embodiment is executed for one frame of image data. If the observation of the observation target is to be continued without ending (step S9: No), the process returns to step S4, and similar processing is performed on the image data of subsequent frames. That is, the image data of the succeeding frame is also subjected to image processing using the color unevenness correction calculation derived in step S2 and the color unevenness correction calculation derived in step S3.

Note that each step of the image processing method by the endoscope system 1 of the present embodiment is not limited to the order described above. For example, the order of performing the step S5 of performing luminance unevenness correction calculation, the step S6 of performing color unevenness correction calculation, and the smoothing step S7 is arbitrary.

Further, the step S6 of performing color unevenness correction calculation and the step S7 of smoothing are optional steps.

As is clear from the above, the image processing method of the present embodiment includes an endoscope 2 having an image guide fiber 212 including a plurality of cores 212a that transmit incident light after being reflected by the surface of an observation object, and an endoscope. An image composed of a plurality of pixel data based on an image sensor 31 to which a mirror 2 can be connected and which detects light through an eyepiece lens 30 and converts it into an electrical signal and the electrical signal converted by the image sensor 31. An image processing method in an endoscope system 1 comprising an imaging device 3 having an image generation unit 32 for generating data, comprising: a step of capturing a reference plane to generate reference image data; and a step of forming the reference image data. a step of deriving a luminance non-uniformity correction operation that uniforms luminance components of each of a plurality of pixel data, a step of photographing an observation target to generate observation image data, and a step of generating observation image data; each of which includes a step of performing a brightness unevenness correction operation.

According to such an image processing method, among the plurality of pixel data, each of the plurality of pixel data in the region corresponding to the clad 212b is also subjected to luminance unevenness correction. The pre-correction brightness of the plurality of pixel data in the region corresponding to the cladding 212b is lower than the pre-correction brightness of the plurality of pixel data in the region corresponding to the core 212a. However, due to the luminance unevenness correction, the corrected luminance of the plurality of pixel data in the region corresponding to the cladding 212b is approximately the same as the corrected luminance of the plurality of pixel data in the region corresponding to the core 212a. As a result, the mesh pattern of the display image displayed on the display device 5 is substantially eliminated. In addition, since luminance non-uniformity correction calculation is applied to each of the plurality of pixel data, it is possible to suppress degradation of the resolution of the display image.

That is, according to such an image processing method, it is possible to substantially remove the mesh pattern while suppressing degradation in resolution of the image displayed by the endoscope system 1 . In addition, substantially removing the mesh pattern refers to reducing the mesh pattern to such an extent that it is difficult to visually recognize the mesh pattern with the naked eye.

Further, in the above image processing method, the reference surface is achromatic, and the image processing method includes the step of deriving a color unevenness correction operation for removing color difference components of each of the plurality of pixel data constituting the reference image data; and a step of performing a color unevenness correction operation on each of the plurality of pixel data constituting the observed image data.

According to such an image processing method, color unevenness in the image displayed by the endoscope system 1 can be further removed.

Further, in the above image processing method, each of the plurality of pixel data constituting the observed image data is determined based on the diameter of the core 212a, the density of the core 212a, the image magnification of the eyepiece 30, and the pixel density of the image sensor 31. smoothing using a smoothing filter.

According to such an image processing method, the mesh pattern of the image displayed by the endoscope system 1 can be further removed.

The imaging device 3 of this embodiment includes an image sensor 31 that detects light and converts it into an electric signal, and image data composed of a plurality of pixel data based on the electric signal converted by the image sensor 31. The image generation unit 32 generates an image, and a brightness unevenness correction operation is derived to uniform the brightness components of each of the plurality of pixel data constituting the reference image data obtained by capturing the reference plane, and the observation target is captured. An image processing unit 33 is provided for performing luminance unevenness correction calculation on each of a plurality of pixel data constituting the obtained image data of the observation target.

According to such an imaging device 3, deterioration of the image quality of the image displayed on the display device 5 can be suppressed, luminance unevenness can be removed, and the mesh pattern can be substantially removed.

Further, in the imaging device 3, the image processing unit 33 derives a color unevenness correction operation for removing the color difference components of each of the plurality of pixel data constituting the reference image data, Each of the pixel data is further subjected to a color unevenness correction operation.

According to such an imaging device 3 , it is possible to further eliminate color unevenness in the image displayed on the display device 5 .

Further, in the imaging device 3, the image processing section 33 smoothes each of the plurality of pixel data constituting the image data of the observation target.

According to such an imaging device 3, the mesh pattern of the image displayed on the display device 5 can be further removed.

<Modification>
Although the imaging device 3 has the image processing unit 33 in this embodiment, the present invention is not limited to this. As a modification, an image processing device connectable to the imaging device 3 is provided as a configuration separate from the imaging device 3, and the image processing device includes an image processing section having the same function as the image processing section 33 of the present embodiment. It is good also as a mode provided.

The image processing apparatus of the modified example derives a luminance non-uniformity correction calculation that uniforms the luminance components of each of the plurality of pixel data constituting the reference image data obtained by photographing the reference plane, and photographs the observation target. A luminance non-uniformity correction operation is performed on each of a plurality of pixel data constituting the obtained observed image data.

According to such an image processing apparatus, it is possible to suppress the degradation of the image quality of the image displayed on the display device 5, remove the luminance unevenness, and substantially remove the mesh pattern.

Further, in the above image processing apparatus, a color unevenness correction operation for removing color difference components of each of the plurality of pixel data constituting the reference image data is derived, and each of the plurality of pixel data constituting the image data of the observation object is colored. Non-uniformity correction calculation is further performed.

According to such an image processing apparatus, it is possible to further eliminate color unevenness in the image displayed on the display device 5 .

Further, in the image processing apparatus, each of the plurality of pixel data forming the image data of the observation target is smoothed.

According to such an image processing device, it is possible to further remove the mesh pattern from the image displayed on the display device 5 .

An observation target was photographed, and an image of the observation target displayed on the display device 5 (hereinafter referred to as a display image) was evaluated. Specifically, compared with the displayed image when no image processing is performed, the degree of reduction in luminance unevenness, the degree of color unevenness, the degree of reduction in mesh patterns, and the deterioration in resolution of the displayed image under each image processing condition. I evaluated the condition.

These evaluations were made visually by an observer. FIG. 8 is a table summarizing the image processing conditions of Examples 1 to 3 and Comparative Examples 1 to 5 and the evaluation results of the display images obtained. A detailed description will be given below.

[Common conditions]
Conditions common to Examples 1 to 3 and Comparative Examples 1 to 5 will be described. The surface of a white plate was used as a reference plane for obtaining a reference image. A human palm was used as an observation object. The palm was observed with the palm open and the insertion site of the endoscope facing the center of the palm.

In this example, a lacrimal fiberscope manufactured by Fibertech was used as an endoscope. A 3CMOS HD camera FC-304 manufactured by Fibertech Co., Ltd. was used as the imaging device, and the program for controlling the image processing unit was adjusted to execute the same control as in the above embodiment. A light source device FL-301 manufactured by Fiber Tech was used as the light source device. As the display device, MLW-2425C manufactured by Ikegami Tsushinki Co., Ltd. was used.

In an endoscope, the diameter of the image guide fiber is 0.35 mm. The image guide fiber has 10000 cores. Each core has a diameter of 3.5±0.5 μm. The core density is 100,000/mm ² .

In the imaging device, the image sensor has a plurality of pixels arranged in an area of 3 mm×5.4 mm. A total of 2,073,600 pixels are arranged in a matrix of 1,080 rows and 1,920 columns. The pixel density is 900-920 ppi. An eyepiece lens is provided between the image guide fiber and the image sensor, and the image formed by the light transmitted to the image guide fiber is enlarged and formed on the image sensor. The image magnification of the eyepiece lens is 5.5. ±1 times.

[Individual conditions]
FIG. 8 shows the presence or absence of luminance unevenness correction and color unevenness correction in each example and each comparative example. Specifically, "Yes" indicates that the process was executed, and "No" indicates that the process was not executed. As for the smoothing, "none" indicates that the process was not performed, and "strong" or "weak" indicates that it was performed. ``Weak'' and ``Strong'' are smoothing methods that replace the color information of pixel data corresponding to each of a plurality of pixels with the average value of the color information of pixel data of 25 and 49 pixels centering on that pixel, respectively. correspond to "Weak" and "strong" are referred to as "weak smoothing" and "strong smoothing" respectively.

[Evaluation method]
The degree of reduction in mesh pattern, the degree of reduction in luminance unevenness, the degree of reduction in color unevenness, and the degree of deterioration in resolution were evaluated in four stages of double circles, single circles, triangles, and crosses.

Here, the degree of reduction in mesh pattern, the degree of reduction in luminance unevenness, and the degree of reduction in color unevenness were evaluated according to the following criteria. A cross indicates a result rated as significantly visible. Triangles indicate results evaluated as reduced compared to crosses but easily visible to the naked eye. A single circle indicates a result evaluated as being substantially removed. The term "substantially removed" indicates a result evaluated as having been reduced to such an extent that it is difficult to visually recognize with the naked eye, although it is not necessarily removed. Double circles indicate results evaluated as removed. “Removed” indicates a result evaluated as being reduced to the extent that it cannot be visually recognized with the naked eye.

Further, regarding the degree of degradation of resolution, Comparative Example 1, which is one of the conditions in which no smoothing is performed and was evaluated as having good resolution, is marked with a double circle, and the evaluation was performed based on this. A double circle indicates a result evaluated as not degraded compared to Comparative Example 1. Here, "not degraded" means that degradation cannot be visually recognized with the naked eye. A single circle indicates a result evaluated as not substantially degraded compared to Comparative Example 1. Here, "substantially not degraded" means that although degraded, it is difficult to visually recognize the degradation with the naked eye. Triangles indicate results evaluated as degraded to the extent that they can be easily visually recognized with the naked eye. A cross indicates a result evaluated as significantly deteriorated.

[Evaluation results]
FIG. 8 shows the evaluation results of the degree of reduction of mesh patterns, luminance unevenness and color unevenness, and the degree of deterioration of resolution. 9A to 16B show displayed images of the reference plane and the observation object obtained under the conditions of Examples 1 to 3 and Comparative Examples 1 to 5. FIG.

(Effect of luminance unevenness correction)
By comparing the evaluation results of Example 1 and Comparative Example 1, the effect of luminance unevenness correction will be considered. FIG. 9A is a display image when image processing is performed on the reference image under the conditions of the first embodiment. 9B is a display image when image processing is performed on the observation image under the conditions of Example 1. FIG. FIG. 11A is a display image when the reference image is subjected to image processing under the conditions of Comparative Example 1, that is, when no image processing is performed. FIG. 11B is a display image obtained by subjecting the observation image to image processing under the conditions of Comparative Example 1, that is, when no image processing is performed.

Comparing the display images of Example 1 and Comparative Example 1, it can be seen that not only the luminance unevenness is removed but also the mesh pattern is substantially removed by performing luminance unevenness correction on the image data of the reference image and the observation image. I understand.

(Effect of color unevenness correction)
The evaluation results of Example 2 and Example 1 are compared to discuss the effect of color unevenness correction. FIG. 10A is a display image when image processing is performed on the reference image under the conditions of the second embodiment. 10B is a display image when image processing is performed on the observed image under the conditions of Example 2. FIG.

Comparing the display images of Example 2 and Example 1, it can be seen that color unevenness is removed by applying color unevenness correction to the image data of the reference image and the observed image. Moreover, it can be seen that the mesh pattern is not removed or reduced even if the color unevenness correction is applied to the image data of the reference image and the observation image.

(Synergistic effect of luminance unevenness correction and weak smoothing)
By comparing the evaluation results of Example 2, Example 3, Comparative Example 3, and Comparative Example 4, the synergistic effect of luminance unevenness correction and weak smoothing will be considered. FIG. 10A is a display image when image processing is performed on the reference image under the conditions of the second embodiment. 10B is a display image when image processing is performed on the observed image under the conditions of Example 2. FIG. 13A is a display image when image processing is performed on the reference image under the conditions of Example 3. FIG. 13B is a display image when image processing is performed on the observed image under the conditions of Example 3. FIG. 14A is a display image when image processing is performed on the reference image under the conditions of Comparative Example 3. FIG. 14B is a display image when image processing is performed on the observed image under the conditions of Comparative Example 3. FIG. 15A is a display image when image processing is performed on the reference image under the conditions of Comparative Example 4. FIG. 15B is a display image when image processing is performed on the observed image under the conditions of Comparative Example 4. FIG.

From Example 2 and Comparative Example 3, it can be seen that it is not possible to evaluate that the mesh pattern has been removed simply by applying either luminance unevenness correction or weak smoothing to the image data. In other words, it can be understood that it cannot be evaluated that the mesh pattern has been reduced to the extent that it cannot be visually recognized with the naked eye (that is, double circles) by only one of these image processes.

On the other hand, it can be seen from Comparative Example 4 that the mesh pattern is removed by applying strong smoothing to the image data. However, it can be seen that applying strong smoothing to the image data degrades the resolution of the image.

From Example 3, it can be seen that the mesh pattern is removed without deteriorating the resolution of the image by applying luminance unevenness correction and weak smoothing to the image data.

1: Endoscope system 2: Endoscope 21: Insertion section 211: Objective lens 212: Image guide fiber 212a: Core 212b: Clad 212c: Jacket 213: Light guide fiber 213a: Core 213b: Clad 214: Exterior pipe 214a: Filler 22: Grip 23: Lead 23a: Branch 23b: Image guide plug 23c: Light guide plug 3: Imaging device 3a: Endoscope terminal 30: Eyepiece 31: Image sensor 32: Image generator 33: Image Processing unit 34: Output signal generation unit 4: Light source device 4a: Socket 5: Display device

Claims (9)

An endoscope having an image guide fiber that includes a plurality of cores that transmit incident light after reflection off a surface to be observed;
An image sensor connectable to the endoscope that detects the light through an eyepiece and converts it into an electrical signal, and an image that generates image data composed of a plurality of pixel data based on the electrical signal. An image processing method in an endoscope system comprising an imaging device having a generation unit,
imaging a reference plane to generate reference image data;
a step of deriving a luminance non-uniformity correction calculation that uniforms luminance components of each of a plurality of pixel data constituting the reference image data;
a step of photographing an observation target to generate observation image data;
and a step of performing the luminance unevenness correction operation on each of a plurality of pixel data constituting the observed image data. The reference surface is achromatic, and the image processing method includes
deriving a color unevenness correction operation for removing color difference components of each of a plurality of pixel data constituting the reference image data;
performing the color unevenness correction operation on each of a plurality of pixel data constituting the observed image data;
The image processing method according to claim 1, comprising: Each of the plurality of pixel data constituting the observed image data is processed using a smoothing filter determined based on the diameter of the core, the density of the core, the image magnification of the eyepiece, and the pixel density of the image sensor. 3. An image processing method according to claim 1, comprising a smoothing step. An endoscope having an image guide fiber that includes a plurality of cores that transmit incident light after reflection off a surface to be observed;
an image sensor connectable to the endoscope, which detects light through an eyepiece and converts it into an electrical signal;
an image generator that generates image data composed of a plurality of pixel data based on the electric signal converted by the image sensor;
Deriving a brightness non-uniformity correction operation for uniforming brightness components of each of a plurality of pixel data constituting reference image data obtained by photographing a reference plane,
an image processing unit that performs the brightness unevenness correction operation on each of a plurality of pixel data constituting observation image data obtained by photographing an observation target;
an imaging device having
An endoscope system with a The image processing unit
deriving a color unevenness correction operation for removing color difference components of each of a plurality of pixel data constituting the reference image data;
5. The endoscope system according to claim 4, wherein said color unevenness correction operation is applied to each of a plurality of pixel data constituting said observation image data. The image processing unit
6. The endoscope system according to claim 4, wherein each of a plurality of pixel data constituting said observation image data is smoothed. An endoscope having an image guide fiber that includes a plurality of cores that transmit incident light after reflection off a surface to be observed;
An image sensor that can be connected to the endoscope and detects light through an eyepiece and converts it into an electric signal, and a plurality of pixel data based on the electric signal converted by the image sensor. an imaging device having an image generator that generates image data;
connectable to the imaging device;
Deriving a brightness non-uniformity correction operation for uniforming brightness components of each of a plurality of pixel data constituting reference image data obtained by photographing a reference plane,
an image processing device for performing the brightness unevenness correction operation on each of a plurality of pixel data constituting observation image data obtained by photographing an observation target ;
An endoscope system with a The image processing device is
deriving a color unevenness correction operation for removing color difference components of each of a plurality of pixel data constituting the reference image data;
8. The endoscope system according to claim 7, wherein said color unevenness correction operation is applied to each of a plurality of pixel data constituting said observation image data. The image processing device is
9. The endoscope system according to claim 7, wherein each of a plurality of pixel data forming said observation image data is smoothed.

Images