KR20190058936A - Method and system for retinex processing of endoscopic image, and a recording medium having computer readable program for executing the method - Google Patents

Abstract

The endoscopic image processing system comprises a light source unit, an image sensing unit, an image reconstruction unit, a retinex image processing unit, and a retinex image processing control unit, wherein the retinex image processing unit comprises an image separation unit, an illumination processing unit, and an image synthesis unit. The light source unit illuminates an object with light, the image sensing unit acquires sensor image data of the object, the image reconstruction unit reconstructs an image of the object from the acquired image, the retinex image processing unit applies retinex image processing to the output image from the image reconstruction unit, and the retinex image processing control unit calculates a control parameter for controlling the retinex image processing unit. Here, the image separation unit separates an input image, and, for each pixel of the input image, calculates a reflectance map for a reflectance component and an illumination map for an illumination component, the illumination processing unit modifies illumination brightness on the illumination map, and the image synthesis unit produces a synthesized image of the processed reflectance map and the illumination map. Thus, a clear image can be provided.

Description

TECHNICAL FIELD [0001] The present invention relates to a retinex endoscopic image processing system, a retinex endoscope image processing system, a retinex endoscope image processing system, a retinex endoscope image processing system, a retinex endoscope image processing system, a retinex endoscope image processing system,

The present invention relates to an image processing system and method, and more particularly, to a system and method for processing an endoscopic image.

The general endoscopic imaging system includes a light source unit and an image sensor unit for providing illumination for observing the inside of the organ, and an image signal processing (ISP) processor unit for transforming the image raw data into an image to be observed. Converted video format to the display and storage device.

The ISP mainly includes functions such as debayering, white balance, color correction, gamma correction, etc., which restore RGB channel images from RGB pixel arrays. 1 is a schematic block diagram of a conventional endoscopic imaging system.

On the other hand, in the endoscopic image, a sharp image having no dark portion is very important. Particularly, the importance of surgical endoscopes is further increased, because if the bleeding part or the visibility of surgical instruments is poor, it can lead to medical accidents.

However, in the conventional art as shown in FIG. 1, when the intensity of the light source is increased, a saturation phenomenon occurs in the bright region, and when the intensity of the light source is decreased, the peripheral regions over a certain distance from the light source are darkened, It has been very difficult to obtain an endoscopic image having a sharp image quality without a dark part.

US 8979741 B2

SUMMARY OF THE INVENTION The present invention has been made in order to solve the conventional problems described above, and it is an object of the present invention to provide an endoscopic image processing system and method capable of improving a difference in light quantity of illumination illuminated by each site, .

In order to achieve the above object, an endoscope image processing system according to the present invention includes a light source unit, an image sensing unit, an image reconstruction unit, a Retinex image processing unit, and a Retinex image processing control unit, An illumination processing unit, and an image synthesizing unit.

The retinex image processor reconstructs the image of the subject from the acquired sensor image data, and the retinex image processor converts the output image of the image reconstruction unit into retinex image data, And the Retinex image processing control unit calculates control parameters for controlling the Retinex image processing unit.

At this time, in order to decompose the input image signal, the image separating unit divides the reflection map of the reflection component of the subject and the illumination map of the incident illumination component into correspondence to each pixel of the input image, And the illumination processing section modifies the illumination brightness in the illumination map, and the image synthesis section calculates the composite image of the processed reflection map and the modified illumination map.

According to such a configuration, the input image is divided into the reflection component and the illumination component, and the non-uniform illumination component is corrected and synthesized, thereby improving the difference in the amount of light irradiated for each site, .

At this time, the Retinex image processing control unit can control the degree of information loss in the bright region of the input image by calculating the Gaussian kernel size of the Gaussian filter applied by the image separator for generating the illumination map.

Also, the Retinex image processing control unit can control the brightness amplification degree of the dark region of the input image by calculating the alpha value necessary for determining the gamma value for the gamma correction process applied by the illumination processing unit to perform the deformation of the illumination brightness have.

In addition, the Retinex image processing control unit can control the noise of the composite image by calculating the Epsilon value applied by the image separating unit to generate the reflection map.

Also, the Retinex image processing control unit may include a dark image processing unit for transmitting a Gaussian filter output for an image having the minimum value among red, green, and blue corresponding to each pixel of the input image to the illumination processing unit.

Also, the Retinex image processing control unit may calculate the manually selected control parameter according to the user input, and may automatically calculate the control parameter using the learned information from the plurality of input images already set.

Further, the light source section can irradiate light of a plurality of spectra, and can convert the wavelength of the light source.

At this time, the Retinex image processing unit may perform Retinex image processing for each of a plurality of spectral lights.

The apparatus may further include an image combining unit for combining the retinex image-processed images with respect to each of the plurality of spectral lights.

The apparatus may further include an image combining unit configured to synthesize output images of a plurality of image reconstructing units for each of a plurality of spectrums of light and to input the resultant image to a Retinex image processing unit.

In addition, an invention in which the system is implemented in the form of a method and a recording medium recording a computer-readable program for executing the method are disclosed.

According to the present invention, the input image is divided into the reflection component and the illumination component, and the non-uniform illumination component is corrected and synthesized, thereby improving the difference in the amount of light irradiated for each site, .

In addition, it is possible to improve the output image more effectively by controlling various parameters of the Retinex image processing.

1 is a schematic block diagram of a conventional endoscopic imaging system;
2 is a schematic block diagram of an endoscopic image processing system according to an embodiment of the present invention;
3 is a schematic block diagram of an endoscopic image processing system according to another embodiment of the present invention.
Figs. 4 to 6 show an example of the configuration of a retinex unit in image processing for a plurality of light sources. Fig.
7 is a block diagram schematically showing an example of the configuration of a Retinex image processing unit.
8 is a diagram showing an example of a method of adjusting control parameters so as to adjust the degree of information loss in a bright region.
9 is a diagram showing an example of a method of adjusting a control parameter capable of adjusting the degree of brightness amplification in a dark region.
10 is a diagram showing an example of a method of adjusting a control parameter capable of adjusting the amount of noise.
Fig. 11 shows an example of a method of adjusting a control parameter capable of adjusting the degree of elimination of a fog effect; Fig.
12 and 13 are views of a conventional endoscopic image and an endoscopic image according to the present invention, respectively.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

2 is a schematic block diagram of an endoscopic image processing system according to an embodiment of the present invention. The endoscope image processing system 100 according to the present invention includes a light source unit 110, an image sensing unit 120, an image reconstruction unit 130, a Retinex image processing unit 140, and a Retinex image processing control unit 150 And the retine image processing unit 140 further includes an image separating unit 142, an illumination processing unit 144, an image synthesizing unit 146, and a reflection processing unit 148.

At this time, all components of the endoscopic image processing system 100 can be implemented by hardware alone, but they are generally implemented together with hardware and hardware that operates on hardware.

2, the light source unit 110, the image sensing unit 120, the image reconstruction unit 130, the Retinex image processing unit 140, and the Retinex image processing control unit 150 are respectively provided with a light source unit, Processor unit, a Retinex processor unit, and a Retinex image quality control unit.

1, when the Retinex processor unit 140 is inserted between the output terminal of the ISP processor unit (video processor unit) 130 and the display device, and the Retinex image quality control unit 150 is connected again to the Retinex processor unit 140 . 2, the Retinex image quality controller 150 is separately formed from the Retinex processor unit 140 and is configured to be connected to each other. However, the Retinex image quality controller 150 may be included in the Retinex processor unit 140 so as to be integrally formed.

In FIG. 2, the interface between the ISP processor unit 130 and the Retinex processor unit 140 is configured as a standard video interface. At this time, both the ISP processor unit 130 and the Retinex processor unit 140 may be implemented in the form of GPU-based parallel processing or FPGA hardware, respectively, so that four different hardware combinations are possible.

3 is a schematic block diagram of an endoscopic image processing system according to another embodiment of the present invention. 3, the image reconstructing unit 130, the Retinex image processing unit 140, and the Retinex image processing control unit 150 respectively include an ISP processor module 130, a Retinex processor module 140, a Retinex image quality control module 150, Which is implemented as modules included in the video processing unit. At this time, the video processing unit may be implemented as one of an FPGA or a PC-based GPU.

The light source unit 110 irradiates the subject with light. The light source unit 110 can irradiate light of a plurality of spectra. At this time, the light of a plurality of spectra may be, for example, wide band visible light having a wavelength of 400 nm to 700 nm, infrared light having a wavelength of 800 nm or more, or narrow band light such as 410 nm, 415 nm and 540 nm.

The light of the plural spectra may be irradiated from each of the plurality of light sources, but may be irradiated by changing the wavelength of the light source of a single wavelength. To do this, you can also perform quantum dot light source modulation.

Figs. 4 to 6 are diagrams showing examples of configurations of a retinex unit in image processing for a plurality of light sources. Fig. In FIGS. 4 and 5, it can be seen that different retinex units 140 are connected to the plurality of light sources. In this manner, the Retinex image processing unit 140 can perform a plurality of retinex image processing corresponding to each of a plurality of spectral lights.

In this case, the plurality of images may be outputted through different display devices as shown in FIG. 4, or may be output to different areas of one display device (PNP), respectively.

In addition, a plurality of images may be synthesized and output to a single display device. To do so, the image processing system 100 synthesizes an image subjected to Retinex image processing for each of a plurality of spectral lights as shown in FIG. 5 And an image combining unit 160 may be further included.

6, the image combining unit 160 synthesizes output images of a plurality of image reconstructing units 130 with respect to a plurality of lights of a plurality of spectra, and outputs them to the Retinex image processing unit 140, As shown in FIG.

The Retinex image processing unit 140 receives the image of the subject from the image reconstructing unit 130. The image sensing unit 120 acquires the image of the subject, the image reconstructing unit 130 reconstructs the image of the object from the acquired image, The image is processed by Retinex image.

To this end, the image separating unit 142 separates the input image and calculates a reflection map for the reflection component and an illumination map for the illumination component corresponding to each pixel of the input image, The processing unit 144 modifies the illumination brightness in the illumination map, and the image synthesizing unit 146 calculates a composite image of the processed reflection map and the illumination map.

7 is a block diagram schematically showing an example of the configuration of the Retinex image processing unit. 7, the Retinex image processing system 140 includes an image separation unit 142, an illumination processing unit 144, a reflectance processing unit 148, and an image synthesis unit 146.

The image separating unit 142 decomposes the input image, calculates reflectance, which is a reflection component for each pixel, and illumination value, which is an illumination component, and stores a reflection map and an illumination map in a memory.

To this end, the illumination processing unit 142 uses a gamma correction curve to adjust the brightness of the illumination component. At this time, the gamma correction curve can be varied according to the statistical characteristics (average value of brightness, variance / standard deviation, histogram, etc.) of the image.

The Illumination Map is a Map associated with the illumination brightness, and the Illumination processing unit 144 processes the separated Illumination Map close to the Uniform Illumination Map. To do this, gamma correction is applied to the illumination map or a similar correction curve is used to vary the brightness of the illumination.

Unlike the Illumination Map, the reflectance processing unit 148 performs a process of not performing any processing on the Reflectance Map or correcting the color, and the image combining unit 146 multiplies the processed Reflectance Map and Illumination Map to calculate the synthesized image.

The Retinex image processing control unit 150 calculates a control parameter for controlling the Retinex image processing unit 140. [ At this time, the Retinax image processing control unit 150 can control the information loss information of the bright region of the input image by calculating the Gaussian kernel size of the Gaussian filter applied by the image separation unit 142 to generate the illumination map .

8 is a diagram showing an example of a method of adjusting control parameters so as to adjust the degree of information loss in a bright region. In FIG. 8, when the Gaussian kernel size is increased, the loss of the bright area is reduced. When the Gaussian kernel size is decreased, the loss of the bright area is increased.

In addition, the Retinex image processing control unit 150 can control the brightness amplification degree of the dark region of the input image by calculating the alpha value necessary for determining the gamma value applied by the illumination processing unit 144 to perform the deformation of the illumination brightness have.

9 is a diagram showing an example of a method of adjusting a control parameter capable of adjusting the degree of brightness amplification in a dark region. In FIG. 9, the gamma determination can be performed by the equation (L (x, y) + Alpha) / (1 + Alpha). When Alpha is adjusted to a small value, the degree of brightness amplification is increased. The degree of amplification is reduced.

Also, the Retinex image processing control unit 150 can control the noise of the composite image by calculating the Epsilon value applied by the image separation unit 142 to generate the reflection map.

10 is a diagram showing an example of a method of adjusting a control parameter capable of adjusting the amount of noise. In FIG. 10, the divider can be performed by the following equation, where the noise of the final Retinex output image is increased when the Epsilon is reduced, and the noise of the final Retinex output image is decreased when the Epsilon is increased.

Reflectance_R = R (x, y) / (L (x, y) + epsilon)

Reflectance_G = G (x, y) / (L (x, y) + epsilon)

Reflectance_B = B (x, y) / (L (x, y) + epsilon)

The retinex image processing control unit 150 may include a dark image processing unit 152 for transmitting the Gaussian filter output of the minimum value image among the red, green, and blue of the input image to the illumination processing unit 144. 11 is a diagram showing an example of a method of adjusting a control parameter capable of adjusting the degree of eliminating the fog effect.

At this time, the Retinax image processing control unit 150 may calculate the control parameter according to the user input, and may calculate the control parameter using the learned information from the plurality of input images already set.

That is, the control parameter determination method can be determined manually through a SW GUI or a physical button, and can be determined to be fully automatic, such as automatically determining a parameter for each frame through machine learning, And semi- automatic (semi-automatic) which can switch between manuals. At this time, the control parameter adjustment range can be implemented as a Continuous Value or a Discrete Value (Preset).

12 and 13 are images of a conventional endoscope and images of an endoscope according to the present invention, respectively. In FIG. 13, it is possible to improve the difference in the light amount of the illuminated light for each site, and to see a clear image without dark portions.

In summary, the present invention is a modular Retinex endoscopic imaging system capable of image quality control, comprising: a light source unit for illuminating one or more spectra of light; at least one image sensor unit for observing an internal cavity; One or more ISP units for reconstructing the image, and an additional Retinex image processing unit for improving the image.

In order to control the Retinex image processing unit, a control parameter capable of adjusting the degree of brightness amplification in the dark region, a control parameter capable of adjusting the degree of information loss in the bright region, a control parameter capable of adjusting the amount of noise, And a control parameter for controlling whether to apply Retinex image processing or not, and a Retinex image processing control unit capable of adjusting at least one of control parameters for controlling whether or not Retinex image processing is applied.

According to the present invention, the retinex image processing can overcome the conventional image quality problem by dividing the input image into the reflection component and the illumination component and correcting the uneven illumination component. In addition, it can be modularized into an endoscope, and real-time performance can be realized for a high-resolution image by implementing a GPU or an FPGA.

Therefore, the conventional surgical endoscope can lead to medical accidents if the amount of light emitted by each part is greatly different and the bleeding part or the visibility of the surgical instrument is poor. However, the proposed structure is combined with the endoscope In one system, this problem is solved and more safe operation is possible.

Although the present invention has been described in terms of some preferred embodiments, the scope of the present invention should not be limited thereby but should be modified and improved in accordance with the above-described embodiments.

100: Retinex Endoscopy Image Processing System
110: light source
120: image sensing unit
130: Image reconstruction unit
140: Retinex image processing unit
142:
144:
146:
148:
150: Retinex image processing control unit
152: Dark image processing unit
160:

Claims (26)

A light source unit for irradiating light to a subject;
An image sensing unit for acquiring an image of the subject;
An image reconstruction unit reconstructing an image of the object from the acquired image;
A Retinex image processing unit for processing a retinal image of an output image of the image reconstructing unit; And
A retinex image processing control unit for calculating a control parameter for controlling the retinex image processing unit,
Wherein the retinex image processor comprises:
An image separator for separating an input image and calculating a reflection map for a reflection component and an illumination map for an illumination component corresponding to each pixel of the input image;
An illumination processor for modifying illumination brightness in the illumination map; And
And an image combining unit for calculating a composite image of the processed reflection map and the illumination map.
The method according to claim 1,
Wherein the retinex image processing control unit calculates the Gaussian kernel size of the Gaussian filter applied by the image separating unit to generate the illumination map,
Wherein information loss information of an area of the input image that is brighter than a predetermined reference is controlled.
The method according to claim 1,
Wherein the retinex image processing control unit calculates an alpha value necessary for determining a gamma value applied to perform the deformation of the illumination brightness,
Wherein the control unit controls the brightness amplification degree of the input image in a region darker than a predetermined reference.
The method according to claim 1,
Wherein the retinex image processing control unit calculates an epsilon value applied by the image separating unit to generate the reflection map,
And controls the noise of the synthesized image.
The method according to claim 1,
Wherein the retinex image processing control unit includes a dark image processing unit for transmitting a Gaussian filter output of a minimum value image among red, green, and blue of the input image to the illumination processing unit.
The method according to claim 1,
Wherein the image processing control unit calculates the control parameter according to a user input.
The method according to claim 1,
Wherein the image processing control unit calculates the control parameter using information learned through an artificial neural network or the like using a plurality of input images already set.
The method according to claim 1,
Wherein the light source unit emits a plurality of spectral lights.
The method of claim 8,
Wherein the light source unit converts the wavelength of the light source.
The method of claim 8,
Wherein the retinex image processing unit performs a plurality of retinex image processing corresponding to each of the plurality of spectral lights.
The method of claim 8,
Further comprising an image combining unit configured to synthesize the retinex image processed for each of the plurality of spectral lights.
The method of claim 8,
Further comprising an image combining unit for synthesizing output images of a plurality of the image reconstructing units for each of the plurality of spectral lights and inputting the synthesized output images to the Retinex image processing unit.
The method according to claim 1,
Wherein the Retinex image processing unit is implemented as a GPU parallel processing software or a FPGA hardware circuit to realize a real-time processing module.
An image processing method performed by a Retinex endoscope image processing system,
A light irradiation step of irradiating light to a subject;
An image sensing step of acquiring an image of the subject;
An image reconstruction step of reconstructing an image of the subject from the acquired image;
A Retinex image processing step of processing the reconstructed image by Retinex image processing; And
A retinex image processing control step of calculating a control parameter for controlling the retinex image processing step,
Wherein the retinex image processing step comprises:
An image separation step of decomposing an input image and calculating a reflection map for a reflection component and an illumination map for an illumination component corresponding to each pixel of the input image;
An illumination processing step of modifying the illumination brightness in the illumination map; And
And a synthesized image of the processed reflection map and the synthesized image of the illumination map.
15. The method of claim 14,
Wherein the retinex image processing control step includes a step of calculating a Gaussian kernel size of a Gaussian filter applied by the image separation step for generating the illumination map,
Wherein information loss information of an area of the input image that is brighter than a preset reference is controlled.
15. The method of claim 14,
Wherein the retinex image processing control step comprises: calculating the alpha value necessary for determining the gamma value applied to perform the deformation of the illumination brightness,
Wherein the control unit controls the degree of brightness amplification of the input image in a region darker than a preset reference.
15. The method of claim 14,
Wherein the retinex image processing control step includes a step of calculating an epsilon value applied by the image separation step to generate the reflection map,
And controlling the noise of the synthesized image.
15. The method of claim 14,
Wherein the retinex image processing control step includes a dark image processing unit for transmitting a Gaussian filter output of a minimum value image among red, green, and blue of the input image to the illumination processing step.
15. The method of claim 14,
Wherein the image processing control step calculates the control parameter according to a user input.
15. The method of claim 14,
Wherein the image processing control step calculates the control parameter using information learned from a plurality of input images that have been already set.
15. The method of claim 14,
Wherein the light irradiation step irradiates a plurality of spectral lights.
23. The method of claim 21,
Wherein the light irradiation step converts the wavelength of the light source.
23. The method of claim 21,
Wherein the retinex image processing step performs a plurality of retinex image processing corresponding to each of the plurality of spectral lights.
23. The method of claim 21,
Further comprising an image combining step of synthesizing an image subjected to Retinex image processing for each of the plurality of spectral lights.
23. The method of claim 21,
Further comprising an image combining step of synthesizing output images of the plurality of image reconstruction steps for each of the plurality of spectral lights and inputting the resultant image to the Retinex image processing step.
A recording medium on which a computer-readable program for executing the method of any one of claims 14 to 25 is recorded.

Images