CN113139929A

CN113139929A - Gastrointestinal tract endoscope image preprocessing method comprising information screening and fusion repairing

Info

Publication number: CN113139929A
Application number: CN202110283777.3A
Authority: CN
Inventors: 李胜; 喻东; 何熊熊; 李静; 陈铭; 司鹏
Original assignee: Zhejiang University of Technology ZJUT
Current assignee: Zhejiang University of Technology ZJUT
Priority date: 2021-03-17
Filing date: 2021-03-17
Publication date: 2021-07-20

Abstract

A gastrointestinal endoscope image preprocessing method comprising information screening and fusion repairing comprises the steps of firstly, extracting effective information areas in a data set picture twice by using a key information extractor, wherein the results of the two extractions are mutually supplemented and stored as a new picture data set; brightness balance and promotion are carried out on the new image data set by using an improved EACE method, the defect of uneven brightness is improved, the overall brightness is promoted, and more information beneficial to identifying polyps is obtained; and (4) performing light spot detection and light spot area repair on the enhanced result by using a two-image matching fusion repair method. The invention reduces the interference of light spots in the endoscope image on the polyp detection result and reduces the misdiagnosis rate when a doctor or a computer detects the polyp.

Description

Gastrointestinal tract endoscope image preprocessing method comprising information screening and fusion repairing

Technical Field

The invention relates to the field of polyp detection of gastrointestinal endoscopy, in particular to a gastrointestinal endoscope image preprocessing method combining information screening and fusion restoration.

Background

The development of cancer caused by digestive diseases and their further deterioration is one of the killers of human health. According to related reports, three and four cancers with incidence and fatality rates ranked in the top five in China belong to digestive tract cancer respectively. Polyps are a neoplasm of the surface of the small intestine in the digestive tract, most of which are benign, but if their size exceeds 1cm, the likelihood of their canceration will be greatly increased. Thus, early detection and timely treatment of polyps can reduce the likelihood of a patient developing cancer.

Currently, most of polyps are detected by using a gastrointestinal endoscope detection method, and a large number of gastrointestinal tract pictures can be acquired by the endoscope in one detection, wherein the data volume of the pictures can reach thousands or even tens of thousands. It is not easy for doctors to screen such huge amount of pictures one by one, and there may be high risks of missed diagnosis and misdiagnosis. With the continuous update and the increasing maturity of Computer Aided diagnosis technology, it has become a feasible method to use a Computer Aided System (CAS) to help doctors detect polyp in endoscopic pictures.

CAS is a "good helper" to help physicians perform gastrointestinal polyp detection. On one hand, the device saves the time for reading and screening for doctors, on the other hand, the device accelerates the detection efficiency and wins precious treatment time for some critically ill patients. However, the accuracy of the CAS for identifying polyps is related to the quality of gastrointestinal tract pictures input into the system, because the endoscope is limited to a severe imaging environment when performing a photographing operation inside the gastrointestinal tract, and most of photographed pictures have problems of low brightness, blurring or spot interference and the like. These problems can affect the accuracy with which subsequent computer-aided systems identify polyps.

Disclosure of Invention

In order to solve the problem that invalid information, uneven brightness and light spot interference of an endoscope image influence the speed and precision of polyp identification by CAS, the invention provides a gastrointestinal tract endoscope image preprocessing method combining information screening and fusion restoration. In order to realize information screening, the key information extractor is designed to screen the effective information in the endoscope image twice, and screening results are complementary; secondly, in order to improve the brightness of the screened image, the invention provides an improved Adaptive Contrast Enhancement method (EACE) for balancing and improving the overall brightness of the image, and the CAS is Enhanced to detect the low-brightness area; finally, the invention designs a method for matching, fusing and repairing the two images to repair the light spot area, eliminates the influence of the light spot on the CAS recognition result and improves the accuracy of the result.

The technical scheme of the invention is as follows:

a gastrointestinal endoscope image preprocessing method including information screening and fusion repair, the method comprising the steps of:

step 1, acquiring a picture to be processed, wherein an experimental data set is a partial endoscopic picture data set provided by a cooperative doctor and comprises pictures with polyps or pictures without polyps in various types, shapes and colors;

step 2, the key information extractor screens the endoscope picture information;

the key information extractor automatically detects the effective information area of the image according to the input image information of the endoscope, calculates the information of the top point and the side length of the maximum inscribed square inside the image, and extracts all the information in the square as the information

And the key information extractor performs second extraction by utilizing a rotation conversion matrix T:

wherein θ is the rotation angle;

rotating the original image by 45 degrees by taking the central pixel point as the center of a circle, removing the invalid filling part generated by the rotation to extract

Method for extracting new effective information area

Step 3, using improved EACE method to balance the whole brightness of the picture

For pixel point X in the graph, its pixel value is f_x(i, j) calculating its local mean m_x(i, j) and local standard deviation σ_x(i, j) is calculated by the following formula to obtain f_x(I, j) enhanced pixel value I_x(i,j)：

I_x(i,j)＝m_x(i,j)+G(f_x(i,j)-m_x(i,j)) (2)

Wherein M is a global average value and alpha is an enhancement parameter; σ of high frequency region_x(i, j) is large, gain is small, and the enhancement is not too bright, and sigma of low frequency part_x(i, j) is small, the gain G is large, and the brightness value of the area is improved; performing gamma conversion on the V channel of the picture, and finally merging the channel generation results to obtain an endoscope picture with better brightness;

and 4, designing a light spot area in the two images by a matching fusion repairing method.

Further, in the step 4, the two-image matching fusion repairing method repairs the light spot area in the endoscope image, and the processing process is as follows:

4.1 detecting the position of the light spot in the positioning picture through a threshold value and correspondingly positioning the light spot on a light spot mask

Marking;

4.2 masking the light spots

Performing a closing operation to obtain an expanded mask

By using

And

obtaining an edge mask

By using

Acquiring color parameters of a normal area around the light spot, and filling the color parameters with the normal color parameters

Marked spot pixels;

4.3 filling of the color data after preliminary restoration with Normal color parameters

Carrying out Gaussian filtering to balance the repaired facula area with the whole area;

4.4 by means of masks

Extracting the repaired spot part, and the repaired spot part and

matching and fusing;

and 4.5, repairing the fusion trace at the edge, eliminating the unnatural repairing trace, and finishing the pretreatment.

The beneficial effects of the invention are as follows:

(1) compared with the traditional method for extracting the effective information of the gastrointestinal endoscope through manual setting, the invention designs the key information extractor which can be automatically processed in batch under the scene of the gastrointestinal endoscope, only extracts the effective information in the image of the endoscope twice, the contents of the two extractions are complementary, the diagnosis time consumption of a computer-aided system is reduced, and simultaneously, an original data set can be expanded, so that the computer-aided system can obtain more training samples;

(2) the improved ACE gets rid of the defect that the traditional method loses color information to achieve the purpose of enhancement. The color characteristic is an important basis of medical diagnosis, and the improved ACE method is adopted to improve the content of a low-brightness area and ensure that the color of a normal area is not influenced at the same time, so that the overall balanced effect is achieved.

(3) Compared with other spot removing methods which are easy to form mosaic phenomenon, the invention designs a two-image matching fusion repairing method to repair the spots, uses normal pixel information of the image to repair the spot region, extracts the repaired spot region to match and fuse with the original image, so that the repaired image has more natural visual effect, and greatly reduces misdiagnosis risk of identifying the spots as polyps by a computer auxiliary system.

Drawings

FIG. 1 is a general flow diagram of the present invention;

fig. 2 (a) is a test picture used in the present invention, in which a circular area contains valid information required for diagnosis, the outside of the circle is filled with a background color, and the background is invalid information. (b) The pictures are rotated by 40 degrees, and (c) and (d) are respectively extracted

And

in FIG. 3, (a) is

(b) Result light spot mask generated for test picture through light spot detection

(c) Is that

Expanded mask obtained after closing operation

(d) Is that

After the reverse color is finished with

The result of performing a bit OR operation

FIG. 4 is a drawing showing

And

the effect graph after the contrast is improved by using an automatic contrast enhancement method;

fig. 5 is the final result of the enhancement and repair of the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings.

Referring to fig. 1-5, a method for gastrointestinal endoscopic image pre-processing including information screening and fusion repair, the method comprising the steps of:

step 1, obtaining a picture to be processed: the experimental data set is a partial endoscopic picture data set from a collaborator providing, containing pictures of various types, shapes and colors, containing polyps or pictures without polyps;

and 2, designing a key information extractor to screen and extract an effective information area.

The endoscope picture is different from pictures with common formats, an effective information area in the endoscope picture is located in a circular area in the center of an image, an invalid information area filled with black pixels is arranged outside the effective information area, and information needed by diagnosis is not arranged in the invalid information area. When the computer-aided diagnosis is carried out, the detection of the invalid information area wastes the computing resources of the computer, and the running time of a computer-aided diagnosis system is prolonged; discarding invalid information regions to detect only valid regions may improve the efficiency of computer diagnostics.

The key information extractor designed by the invention can automatically extract and store the content of the maximum inscribed square area in the circular effective information area. The width, height and number of channels of the pictures collected in the first step are 534 × 534 × 3, and the pictures are stored in a bitmap (bmp) format. The coordinate of a central pixel point (267) in an original image is taken as a central point, according to mathematical knowledge, the pixel point with the coordinate positioned at (78,78) can be taken as the upper left vertex of an inscribed square, and the side length of the square is 377. The key information extractor extracts the content of this square area as

The method for extracting the inscribed maximum square in the circle inevitably loses a part of effective information, and in order to keep more effective information and reduce the risk of missed diagnosis, the effective information is extracted for the second time in the next step. The central pixel point (267) of the original picture (534 multiplied by 3) is taken as the origin, and the time hand is rotated by 45 degrees. Cutting the picture with length and width changed by rotating and filling into the size of the original picture, and cutting out a square area with side length of 377 by taking (78,78) as the upper left vertex

Before the picture is rotated, a rotation transformation matrix T needs to be determined, and the rotation transformation matrix used by the invention is as follows:

wherein θ is the rotation angle;

the picture is stored in the computer in a matrix form, and in the rotation process of the picture, the original coordinate points and the color information are mapped to a new picture through a rotation matrixAt the coordinate point (pixel point), the new coordinate point will be stored into a minimum (size) matrix which can contain all coordinates according to the transformation rule, and the pixel point which has no mapping relation with the original image pixel point in the minimum matrix will be automatically filled with black. In the application scene of gastrointestinal endoscope polyp detection, the effective information region to be extracted is a circular region in the image, and the range and the position of the effective information region are not changed too much after the effective information region is rotated. What changes is that the surrounding black background area becomes "thick" and the black background belongs to invalid information. In order to simplify the complexity of the processing operation, the format information of the original picture is used to remove the filling area generated by rotation, and a picture with the same size as the original picture and only the middle circular effective information area rotated by 45 degrees is obtained. After the effective information area rotates 45 degrees, the key information extractor extracts the information by the original parameter information

Extracted this time

Content and

in contrast to this, the first and second,

in therein will be

Has no information collected in the information acquisition system,

and

can supplement each other, reduce the loss of the effective information of the original image.

And step 3: equalizing pictures using improved EACE method

And

the brightness of (2).

Due to the limitations of endoscopic imaging conditions, some or most regions of a significant image are not strongly contrasted, and the less strongly contrasted regions of the image may hide polyps but are not easily detected by the CAS. The traditional Adaptive Contrast Enhancement (ACE) algorithm can only balance the image brightness, and in order to improve the balanced image brightness and improve the polyp detection accuracy, the invention provides an improved EACE method to balance and improve the overall brightness of an original image. The principle of the EACE method is to divide an endoscopic image into two parts: one is the low frequency part, which can be obtained by low pass filtering (smooth blurring) of the image; and the second is a high-frequency part which can be obtained by cutting off a low-frequency part from an original image. In order to enhance the high frequency parts representing the details, the high frequency parts are multiplied by a certain gain value, and an enhanced image is obtained after recombination. Suppose that the pixel value of a pixel point x in the endoscopic picture is f_x(i, j), then the local mean m of the region centered at (i, j) and having a window size of (2n +1) × (2n +1)_x(i, j) and variance

Can be expressed as:

mean value m_xCan be approximated as a background portion when f_x(k,l)-m_x(i, j) is the high frequency detail part, and the gain product is made for the high frequency, which comprises:

I_x(i,j)＝m_x(i,j)+G(i,j)[f_x(i,j)-m_x(i,j)] (6)

two schemes for gain G can be chosen, one is that G takes a constant greater than 1:

I_x(i,j)＝m_x(i,j)+C[f_x(i,j)-m_x(i,j)] (2)

scheme two, G is represented as a variation value inversely proportional to the global mean variation:

in order to enhance the low-brightness area of the image, the value taking mode of G in the second scheme is adopted, and then the pixel point f_x(i, j) the enhanced pixel values are:

in a high-frequency area of the image, the local standard deviation is large, the gain value is small at the moment, and the situation of over-brightness cannot occur as a result. But in a smooth region of the image. The local variance is small, and the gain value is large. To avoid the problem of false colors caused by enhancement in the RGB color space, we will do

And (3) in the HSV color space (the space conversion method is the same as the fourth conversion method), the brightness parameters of the whole image are balanced by using an automatic contrast enhancement method for the separated V channel, and after the V channel is set, the channel parameters are converted by using a gamma conversion method, wherein the gamma conversion formula is as follows:

and gamma is a constant which will be empirically determined

And gamma are set to values of 1.13 and 0.89, respectively. And after the conversion is finished, the channels are combined to generate a result. The EACE algorithm realizes the color matching without distortion

And

the brightness equalization and boosting are performed as a whole.

And 4, repairing the light spot area in the image by using the two-image matching fusion repairing method instead of the traditional method.

In addition to contrast problems, during imaging, the endoscope's own light source shines on the smooth gastrointestinal wall where body fluids adhere, possibly causing specular reflection, which in turn creates spots in the endoscopic image. When the CAS detects the image, the light spot region may be misjudged as a polyp part, which affects the accuracy of the computer for identifying polyps, resulting in a risk of misdiagnosis. Most of the traditional facula removing methods cause the mosaic phenomenon of facula areas and also influence the detection result of the CAS. In order to improve the undesirable repairing effect, the invention designs a two-image matching fusion repairing method to repair the light spot area in the image, and the processing flow of the two-image matching fusion repairing method is as follows:

4.1 light spot detection. The image is difficult to distinguish the facula pixel point and the normal pixel point in the original RGB color space, and the image in HSV format is easy to distinguish the facula pixel point and the normal pixel point. When the picture in the original RGB color space is converted into the HSV color space, the conversion formula of the pixel value when the HSV space is converted into the RGB space is as follows:

R′＝R/255 (11)

G′＝G/255 (12)

B′＝B/255 (13)

C max＝max(R′,G′,B′) (14)

C min＝min(R′,G′,B′) (15)

Δ＝C max-C min (16)

color tone:

saturation degree:

lightness:

V＝C max (19)

after the color space conversion is completed, it can be known through looking up related materials and experience that when the Saturation (Saturation) of a certain pixel point in the graph is lower than 55 and the intensity (Value) is higher than 176, the pixel point can be identified as a light spot with a high probability. The facula points in the graph can be positioned by pixel detection after setting the threshold value, and in order to repair, the facula mask with the same size as the original graph is generated after pixel detection

At the spot mask

And marking the pixel point in the middle normal area as 0, and marking the pixel point at the light spot as 255.

4.2 mean value repair. The invention uses the average value of the surrounding pixel points at the light spot as the reference to fill the color information at the light spot. A transition area exists between the light spot and the surrounding normal area, and parameter information of pixel points in the transition area can be influenced by mirror reflection and does not accord with normal parameters. The transition area is also included in the range of the spot area to be repaired, so that the influence of specular reflection on the picture can be reduced to the minimum. Using structural element SE of 3x3_3×3Masking the spots obtained in 4.1

Performing morphological closing operation to respectively obtain expanded masks of expanded light spots

The close operation is defined as follows:

wherein is

The operation of the dilation operation is carried out,

is the erosion operation operator.

Will be provided with

Making a reverse color change (black-white interchange), and then making a color change with

Performing AND operation to obtain edge mask

Edge mask

The pixel points marked as white are normal pixel points of the adjacent light spot area, and the edge mask can be obtained by utilizing the mean () function of OpenCv

Color mean around lower spot, control

De-filling with the obtained color mean information

The corresponding spot area.

4.3 fusion repair. Even if the color values of the continuous pixels in the flat area in the normal image are different, the color values of the continuous pixels in the flat area are different. After the initial repair of the spot position in 5.2, the color information and the integrity of the spot position are repairedThe volumes have tended to be consistent but there is still some inconsistency. The Gaussian filter has obvious denoising and smoothing effects on the flat area. In order to better integrate the spot repairing area, firstly, the primarily repaired picture is smoothed once by Gaussian filtering to obtain a primarily repaired image

The smoothed repair area is preliminarily fused with the surrounding area. Gaussian filtering also loses some of the edge information in the image. Considering that in the endoscope scene, most of the light spots are in a flat area of the image, the method utilizes

Corresponding interception

Repairing the smoothed region

The addWeighted () function using OpenCV will

And

corresponding location matching fusion, wherein the addWeighted function is to

Is set to 0.67,

the fusion weight coefficient is set to 0.33. After the fusion is completed, the corresponding is obtained

4.4 Final repair. After the steps, the repairing work is nearly completed, and finallyIn order to eliminate the trace of artificial patching fusion and make the picture more balanced and natural, an inpainting () function pair of OpenCV is used

Performing a final repair operation, wherein the repair masks of the inpainting () function are respectively

Of the output

Saved in the new dataset as the final result. In the same way, pair

The same operation can be performed.

The final effect of the present invention can be further illustrated by the following experiments.

1) The experimental conditions are as follows:

the experimental part of the invention is implemented on Visual Studio 2012 software by using vc10+ opencv2.4.13 programming. The operating system of the PC for the experiment is Windows 10 professional edition, the processor is Intel (R) core (TM) i5-4210U CPU @1.70GHz 2.39GHz, and the installation memory is 4 GB.

2) The experimental results are as follows:

the experimental result is shown in fig. 5, and compared with the test picture, the picture which is processed and repaired by the method of the invention eliminates the black filling background part; the contrast of the area with lower brightness is improved, and the originally blurred vascular tissue becomes clear to a certain extent; the obvious facula part in the original image is repaired in the result, compared with the original image, the visual effect of the result picture is better, which is beneficial for the doctor to recheck the diagnosis result of the computer auxiliary system, and has good practical engineering application value.

The matters described in this specification are merely examples of implementations of the inventive concept and are for illustration purposes only. The scope of the present invention should not be construed as being limited to the particular forms set forth in the embodiments, but is also based on the technical equivalents which may be conceived by one of ordinary skill in the art based on the concept of the present invention.

Claims

1. A gastrointestinal endoscope image preprocessing method including information screening and fusion repair, the method comprising the steps of:

step 1, acquiring a picture to be processed, wherein an experimental data set is a partial endoscope picture data set provided by a cooperative doctor and comprises pictures with polyps or pictures without polyps in various types, shapes and colors;

the key information extractor automatically detects the effective information area of the image according to the input image information of the endoscope, calculates the peak and side length information of the maximum inscribed square inside the image, and extracts all information in the square as the information

wherein θ is the rotation angle;

Method for extracting new effective information area

Step 3, using improved EACE method to balance picture

And

brightness of (d);

I_x(i,j)＝m_x(i,j)+G(f_x(i,j)-m_x(i,j)) (2)

2. The gastrointestinal endoscope image preprocessing method including information screening and fusion restoration according to claim 1, wherein in the step 4, the two-image matching fusion restoration method restores the light spot area in the endoscope image, and the processing procedure is as follows: