CN111161852B - Endoscope image processing method, electronic equipment and endoscope system - Google Patents

Abstract

The present invention relates to the field of medical devices, and in particular, to an endoscope image processing method, an electronic device, and an endoscope system. The endoscope image processing method comprises the following steps: identifying a specific region in a target image, wherein the target image is an image arbitrarily selected from a series of images taken by an object under illumination by an illumination unit; calculating the offset range which can be generated by the specific area in other images in the series of images; the series of images are aligned to the particular region within the offset range. The invention greatly reduces the calculated amount of a series of spectrum images shot by the alignment endoscope and improves the alignment effect of a specific area; after alignment, an accurate spectrum data cube or spectrum curve can be generated, or images with clear features can be generated through fusion, so that the accuracy of spectrum pathological analysis and diagnosis is improved.

Description

An endoscope image processing method, electronic equipment and endoscope system

technical field

The invention relates to the field of medical instruments, in particular to an endoscope image processing method, electronic equipment and an endoscope system.

Background technique

The endoscope system is an important medical device for the diagnosis and treatment of early diseases of the human cavity, which includes an endoscope body and a lighting unit. When in use, the soft or hard tubular endoscope body is inserted into the human body, and the camera is taken under the illumination of the lighting unit, so as to observe the tissue morphology and pathological changes of human organs with the help of images for diagnosis. Since human tissues (including diseased tissues) are sensitive to certain specific wavelengths of light but not sensitive to certain specific wavelengths of light, in order to highlight the submerged characteristic information under white light irradiation, a variety of different wavelengths of light are often used Irradiate on the object to be inspected, take a series of images, and obtain the spectral atlas or spectral data cube of human tissue (including diseased tissue). By studying the characteristics of the spectral curve and analyzing the spectral pathology, the dependence on sampling and pathological testing in diagnosis can be reduced.

At present, the way to analyze and process a series of images is to select multiple images randomly from these images for image recognition, alignment and false color assignment, and finally form a color image. However, when a series of images of the inspected object are taken continuously, human respiration, heartbeat, gastrointestinal peristalsis, etc. will cause the object to be inspected to move or deform during the interval between two illumination imaging, and the user operates the endoscope at the far end It will cause the endoscope to vibrate or rotate, resulting in a significant offset between each image in a series of images taken continuously; in addition, the endoscope has a large field of view with barrel distortion, which increases the image quality. offset between. At present, the selected images are directly aligned within the entire image range, which requires a large amount of calculation and it is difficult to ensure the alignment effect of the area of interest, which leads to the inability to obtain an accurate spectral curve at a certain position, resulting in a significant increase in the accuracy of subsequent spectral pathological analysis. lower, or even outright wrong.

Contents of the invention

In view of the above-mentioned defects in the background technology, the purpose of the present invention is to provide an endoscope image processing method, electronic equipment and an endoscope system to solve the problem of poor image processing effect in the use of the existing endoscope system, which cannot be used for follow-up research. Provide accurate data and a good image of the problem.

In order to solve the above technical problems, the present invention provides a method for processing endoscopic images, including:

Identify a specific area in the target image, wherein the target image is an image selected arbitrarily from a series of images taken by the inspected object under the illumination of the lighting unit; calculate other values of the specific area in the series of images A range of offsets that can be generated within the images; aligning the series of images according to the specific area within the offset range.

In order to solve the above-mentioned technical problems, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, The steps of the above-mentioned endoscopic image processing method are realized.

In order to solve the above technical problems, the present invention further provides an endoscope system, which includes an endoscope body, and also includes the above-mentioned electronic device, the processor communicates with the camera module in the endoscope body connect.

The technical scheme of the present invention has the following beneficial effects:

In the endoscopic image processing method of the present invention, a specific area is identified in an image randomly selected in a series of images, and the offset range that the specific area can produce in other images in the series of images is calculated. Align a series of images according to a specific area within the offset range; the endoscopic image processing method calculates the offset range of a specific area in other images in a series of images, within the offset range that can be generated Aligning the series of images according to specific areas greatly reduces the amount of calculation and improves the alignment effect of specific areas; using the aligned images, accurate spectral data cubes or spectral curves can be generated for diagnosis and research Provide accurate data; after alignment, images with rich feature information can also be fused, and the fused images have distinct features, which is conducive to rapid and accurate diagnosis.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flow chart of an embodiment of endoscopic image processing method of the present invention;

Fig. 2 is a schematic diagram of the spectrum of a combination of multiple illumination lights irradiated by the illumination unit in the embodiment shown in Fig. 1;

3 is a schematic diagram of the spectrum of another combination of multiple illumination lights irradiated by the illumination unit in the embodiment shown in FIG. 1;

Fig. 4 is a corresponding diagram of a series of images in Fig. 1 and various illumination lights shown in Fig. 2;

Fig. 5 is a schematic diagram of an embodiment of identifying a specific region in a target image;

Fig. 6 is a schematic diagram when there is only one pixel in a certain area artificially specified in the target image;

Fig. 7 is a schematic diagram of another embodiment of identifying a specific region in a target image;

Fig. 8 is a flowchart of yet another embodiment for identifying a specific region in a target image;

Fig. 9 is a schematic diagram of processing results corresponding to the process shown in Fig. 8;

Fig. 10 is a schematic diagram of identifying multiple specific regions in a target image;

Fig. 11 is a schematic diagram of an embodiment of calculating the offset range of the specific region shown in Fig. 5;

Fig. 12 is a schematic diagram of another embodiment for calculating the offset range of the specific region shown in Fig. 5;

Fig. 13 is the selected pixel points to be marked in the real-time image range;

Fig. 14 is a schematic diagram of the calibration process of the offset range circle of the pixel point to be calibrated at time Δt;

Fig. 15 is a flowchart of an embodiment of the alignment step shown in Fig. 1;

Fig. 16 is a schematic diagram of alignment according to the method shown in Fig. 15;

Figure 17 is a flowchart of another embodiment of the alignment step shown in Figure 1;

Fig. 18 is a flowchart of another embodiment of the endoscopic image processing method of the present invention;

Fig. 19 is a schematic diagram of preliminary alignment of a series of images in the manner shown in Fig. 18;

Fig. 20 is a schematic diagram of the displacement transformation included in the similarity transformation;

Fig. 21 is a schematic diagram of the rotation transformation included in the similarity transformation;

Fig. 22 is a schematic diagram of scaling transformation included in similarity transformation;

FIG. 23 is a flowchart of an embodiment of the image alignment adjustment step shown in FIG. 18;

Fig. 24 is a schematic diagram of displacement trajectories after preliminary alignment of a group of images shown in Fig. 19;

Fig. 25 is a schematic diagram of the rotation trajectory after the initial alignment of a group of images shown in Fig. 19;

Fig. 26 is a schematic diagram of the zooming trajectory after a group of images shown in Fig. 19 are initially aligned;

Fig. 27 is a schematic structural view of an embodiment of the endoscope system of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In the description of the embodiments of the present invention, it should be noted that, unless otherwise specified and limited, the terms "first" and "second" are for the purpose of clearly describing the numbering of product components and do not represent any substantial difference. The directions of "up", "down", "left" and "right" are all subject to the directions shown in the attached drawings. Those of ordinary skill in the art can understand the specific meanings of the above terms in the embodiments of the present invention according to specific situations.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediary, and it can be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

Fig. 1 is the flowchart of an embodiment of the endoscope image processing method of the present invention, and the endoscope image processing method shown in Fig. 1 comprises steps:

Step S110, identifying a specific area in the target image, wherein the target image is an image randomly selected from a series of images taken by the object under inspection under the illumination of the lighting unit.

Step S120, calculating the offset range that the specific area can produce in other images in the series of images.

Step S130, aligning a series of images according to specific regions within the offset range.

Wherein, the lighting unit may use a xenon lamp as a light source, or may use a wide-spectrum LED or a tungsten lamp or other wide-spectrum light sources as a light source. The light source passes through the optical filter group, the optical filter wheel and the optical filter group control mechanism to successively filter out various illumination lights. Specifically, narrow-band filters with different central wavelengths or broadband filters with different wavelength ranges or no filters can be selected as required to form a filter group; the filters are in a certain order, such as the central wavelength is small To large, it is inlaid on the filter wheel; the light transmittance of the filter can be different to adjust the intensity of each illumination light; the filter control mechanism controls the rotation of the filter wheel, and the filter can be switched to filter Various lighting lights required. Fig. 2 is a schematic diagram of the spectrum of a combination of various illumination lights irradiated by the illumination unit in the embodiment shown in Fig. 1, which includes 25 kinds of narrow-band light 201-225 with only one central wavelength, covering the visible light range of 400-760nm continuously in total , the filters are designed with different light transmittances, and the illumination intensity is adjusted so that the brightness of a series of images captured by the camera unit in the endoscope body is as balanced as possible.

In addition, the light source of the lighting unit can also be realized by using a variety of LEDs corresponding to the required lighting lights. In this implementation, the filter can be omitted, and each light in the lighting path can be illuminated sequentially by electronic control. LEDs, or mechanically switch each LED into the lighting path in turn. In addition, when LEDs are used as the light source, the LEDs can be installed on the top of the endoscope body, but the space at the top of the endoscope body is narrow, and the number of LEDs that can be installed is limited.

It should be noted that the various illumination lights mentioned above are not limited to the 25 kinds of narrow-band lights shown in Figure 2. In addition, each illumination light can have more than one central wavelength and bandwidth, that is, each illumination light The light can be any combination of narrowband light with one or more central wavelengths, and/or broadband light with one or more wavelength ranges, especially illumination that may be sensitive to a particular condition or may accentuate an aspect of the subject being examined Light. Each illumination light can appear multiple times at different intensities during the capture of a series of images. 3 is a schematic diagram of the spectrum of another combination of various illumination lights irradiated by the lighting unit, including a narrow-band light 201 with one central wavelength, a narrow-band light 226 with two central wavelengths, and a narrow-band light 226 with a wavelength range A broadband light 227 and a broadband light 228 having two wavelength ranges.

FIG. 4 is a corresponding diagram of a series of images in FIG. 1 and various illumination lights shown in FIG. 2 . The illuminating light 201 is guided by the endoscope mirror body and irradiates on the object to be inspected. The camera module captures a corresponding image 301 to complete one illumination imaging; the illuminating unit switches to the next illumination light 202, and the camera module captures the corresponding another image 302 to complete another lighting imaging; the lighting unit continues to switch the next lighting light 203, and the camera module captures a corresponding image 303 to complete another lighting imaging; in this way, the lighting unit switches the lighting light sequentially, and the camera module sequentially A corresponding image is captured until the lighting unit switches to the illumination light 225 , and the camera module captures the corresponding image 325 to complete the lighting imaging of a series of images, and the series of images constitute a group of images 330 . Let the time interval between two adjacent lighting imaging be Δt. It should be noted that the endoscopic image processing method provided by the present invention is also applicable to multiple images taken under single-wavelength illumination.

According to the time sequence of image capture, if the target image is the first image captured, the offset range of the specific area in the second image can be calculated first, and the second image can be compared with the target image within the calculated offset range Align to obtain the exact corresponding area of a specific area in the second image, then calculate the offset range of the third image compared to the corresponding area, and align the third image to the target image within the offset range , and so on until all images are aligned to the target image. In addition, the offset range of each image other than the target image relative to the target image can also be calculated first, and then these images are aligned to the target image within the offset range one by one. Of course, if the target image is the i-th image captured, use the method consistent with the above to calculate the offset range and align the image after the i+1th image, and use the reverse direction when calculating the offset range of the first i-1 image way of operation.

For the convenience of description, a series of images in FIG. 4 will be combined and the first image will be used as the target image for specific description later, and the situation of using other images as the target object will not be repeated.

Fig. 5 is a schematic diagram of an embodiment of identifying a specific region in a target image. Manually designate a certain area in the target image as a specific area. Specifically, a region 403 is defined in the first image 301 of the group of images 330 by means of the mouse 401 , and this region 403 is used as a specific region. Wherein, the area 403 includes a polyp 402 of interest. Images of specific regions were designated as target images for subsequent alignment. In addition, a specific region may also be defined in any other image of the group of images 330 . In addition to the mouse 401, other arbitrary input devices such as a touch panel, a keyboard, a voice input device, a camera, and a scanner can also be used to assist manual designation of a specific area; Specify a specific area by means of a set area, moving a selection box of a specific shape, etc.

Wherein, when the user specifies an area through an input device, the user may only specify a particularly small area. For example, as shown in FIG. 6 , the artificially specified area 409 in the target image has only one pixel. A pixel is very susceptible to noise interference, and subsequent alignment cannot be performed by only one pixel. In this case, the area 409 is centered and expanded to a 5x5 pixel area 410 as the final specific area. Alternatively, it can be expanded to 9x9 or 13x13 or other slightly larger areas.

Alternatively, other methods may also be used to identify a specific region in the target image. Fig. 7 is a schematic diagram of another embodiment of identifying a specific region in a target image without using an input device. Specifically, first preset a square area 404 with a side length of 100 pixels in the center of the real-time image, and use a dotted frame to prompt; The direction and position of the top, so that the square area is aligned with the area of interest, that is, the polyp 402 is aligned, and the polyp 402 is filled with the square area 404; then the image is taken, and the square area 404 in the first image 301 is used as a specific area . Wherein, the polyp 402 may not fill the square area 404 as long as it falls within the square area 404 . In addition, an area of any shape and size can be preset at any position of the real-time image, and the area of any image can also be used as a specific area.

Fig. 8 is a flow chart of another embodiment for identifying a specific region in a target image, and Fig. 9 is a schematic diagram of a processing result corresponding to the flow shown in Fig. 8 , which also does not require an input device. Step S410, using the canny operator, sobel operator, laplacian operator, roberts operator or prewitt operator to perform edge detection on the first image 301 of the captured group of images 330; step S420, based on the breakpoint The closed contour extraction method of keeping the edge direction assuming that the broken edges are connected to obtain a closed contour, and the closed contour is extracted to extract the contour of the polyp 402; step S430, and then perform a certain ratio or number of pixels on the extracted polyp contour The region 408 is obtained by the expansion of , and the region 408 is selected as a specific region.

In addition, two or more of the user input position designation means, the preset position designation means, and the automatic identification position designation means can also be combined to designate the concerned area as a specific area more quickly and accurately. In one embodiment, a plurality of regions are first selected by means of automatic position identification, and then one region is designated from the plurality of regions by means of user input position designation, and then perfected to be selected as a specific region.

Generally, the specific regions 403 , 404 , and 408 illustrated in FIG. 5 , FIG. 7 , and FIG. 9 are relatively large, and do not need to be enlarged by the method shown in FIG. 6 .

The specific area is often the area that the user pays attention to. In the embodiments shown in Fig. 5, Fig. 6, Fig. 7 and Fig. 9, only one area is designated as the specific area, and sometimes the user also needs to designate another area As a specific area, it is used as a reference area for the area of interest. For example, the user wants to obtain the spectral curve of the diseased part, and at the same time wants to compare it with the spectral curve of the normal part. worse than the region of interest). FIG. 10 is a schematic diagram of identifying multiple specific areas in the target image. The specific area 501 is an attention area, including a polyp 402; the specific area 502 is a reference area, and no lesion is found. In order to ensure the alignment effect of the attention region 501 and take into account the alignment effect of the reference region 502 , different alignment weights are applied to the attention region 501 and the reference region 502 for subsequent alignment. For example, an implementation scheme for applying weights is that the user inputs the degree of attention of each specific region according to a predetermined rule through an input device; an alignment weight of a corresponding ratio is applied according to the ratio of the degree of attention of each specific region. The specific area used as the reference area is often small, so the area of each specific area can also be calculated, and the alignment weight of the corresponding ratio can be applied according to the ratio of the area, as an implementation of automatically applying the alignment weight. When calculating the offset range, the offset range calculation is performed separately for each specific area.

In the time interval Δt between two adjacent illumination imaging, the specific area in the target image will be shifted under the influence of visceral peristalsis, endoscope mirror body shaking or rotation, etc., and it is necessary to calculate the The range of offsets that can be generated. With regard to the above-mentioned different methods for identifying a specific area, the specific area shown in FIG. 5 is used as an example for a specific introduction below. Other methods for identifying a specific area are the same and will not be described again.

FIG. 11 is a schematic diagram of an embodiment of calculating the offset range of the specific area shown in FIG. 5 . First, in the specific area 403 specified in the first image 301 of a group of images 330, determine the minimum x coordinate, the maximum x coordinate, the minimum y coordinate, and the maximum four pixel points 601 in total; read the four pixel points 601 In the offset range circle 602 at time Δt, determine the four positions 603 where the four pixel points 601 may deviate from the maximum in the x and y directions from the specific area 403; through the four positions 603, determine a side parallel to the x-axis and y-axis respectively The rectangular area 604 is the calculated offset range that the specific area 403 can generate in the second image 302 . It should be noted that the number of pixels is not less than three, and three pixels can determine a triangular area, which is used as the offset range that the specific area 403 can produce in the second image 302; when it is When there are five or six points or more, a polygonal area can be determined, and this polygonal area is used as 403 the offset range that can be generated in the second image 302 .

The offset range circle 602 of the pixel point at time Δt is obtained through calibration in advance, stored in the storage module, and read from the storage module when needed. Within the offset range 604 , the image to be aligned 302 is aligned to the target image 301 according to the specific region 403 . After the alignment, the exact corresponding area of the specific area 403 in the second image 302 can be obtained, and then the range of offset that the specific area 403 can produce in the third image 303 can be estimated according to the corresponding area; Within the range, the image to be aligned 303 is aligned to the target image 301 according to a specific area 403 . By reciprocating in this way, the images 302 - 325 can be sequentially aligned to the target image 301 according to the specific area 403 within the corresponding offset range.

FIG. 12 is a schematic diagram of another embodiment for calculating the offset range of the specific area shown in FIG. 5 . Read the offset range circle 602 of each pixel point 601 on the boundary of the specific area 403 at the time Δt from the storage module, superimpose all the offset range circles 602, and determine a new area 605 by the outer boundary of the superimposed area, which is The calculated offset range that the specific area 403 can produce in the second image 302, and an area 606 is also determined by the inner boundary of the superimposed area, which can be used as an additional restriction during alignment, that is, the pixels on the boundary of the specific area 403 Point 601 limits the inward offset. It should be noted that, a plurality of pixel points selected at intervals on the boundary of the specific area 403 can also be used to read the offset range circles 602 of these pixel points at the time Δt, so as to circumscribe the boundaries of these offset range circles 602 at the same time As the calculated offset range that the specific region 403 can produce in the second image 302, the number of pixels is not less than three, and a circumscribed circle can be uniquely determined by the three offset range circles 602 or by The outer boundary formed by the boundary of the specific area 403 is scaled up. Certainly, the plurality of pixel points may all be set on the boundary of the specific area 403 ; or part of them may be adjacent to the boundary of the specific area 403 , and the other part may be located on the boundary of the specific area 403 , which is not specifically limited in this embodiment of the present invention.

During image capture, human respiration, heartbeat, gastrointestinal peristalsis, etc. will cause the movement or deformation of the inspected object; the user operates the endoscope at the far end, although he will try to hold the endoscope body as steady as possible, but the endoscope body will There will still be a slight shake or rotation of the tip; coupled with the barrel distortion of the endoscope, the combination of the above factors leads to a significant offset between each image in a series of consecutive images, but the above offset is not infinite Instead, it has an estimable range. By estimating the offset range and aligning the images within the offset range, the calculation amount of alignment can be greatly reduced and the alignment accuracy can be improved.

FIG. 13 and FIG. 14 explain how to calibrate the offset range circle 602 of the pixel point at time Δt. As shown in FIG. 13 , 15×15 pixel points 601 to be calibrated are selected uniformly distributed within the entire image range. FIG. 14 is a schematic diagram of the calibration process of the offset range circle of the pixel points to be calibrated at time Δt. At a typical viewing distance, the first pixel to be calibrated 601 is aligned with a point-shaped site with obvious features in the object under inspection, such as a small polyp in the stomach, and two images are taken at a time interval not less than Δt. Manually find out the central position point 701 of the small polyp in the first image, find out the central position point 702 of the small polyp in the second image, determine the coordinate offset of the position point 702 relative to the position point 701, and complete the first Acquisition of the first set of calibration data of the pixels to be calibrated 601. In this way, for the first pixel point 601 to be calibrated, 100 sets of calibration data are obtained, and before each set of calibration data is obtained, the typical observation distance and the point-shaped part with obvious characteristics are randomly changed. Draw the 100 sets of calibration data together, keep the coordinate offset of all position points 702 relative to position point 701, and make all position points 701 coincide; draw a circle with 701 as the center of the circle, just include all position points 702, A circle 703 of radius r is obtained. Because there is an error when finding out the center position points 701 and 702 of the point-shaped parts with obvious characteristics, it is also necessary to multiply the radius r by a margin coefficient, such as 1.2, to obtain a circle 602 with a radius of R, which is the Mark the offset range circle 602 of the pixel point 601 at time Δt. Reciprocating in this way, each to-be-marked pixel point 601 is marked in turn. For an unmarked pixel point, according to the distance to the adjacent marked pixel point, the inverse distance weighted interpolation is performed on the offset range circle radius of the adjacent marked pixel point to obtain the offset range circle radius of the pixel point, Further, the offset range circle 602 of the pixel point is obtained.

Among them, the number of pixels to be calibrated is not limited to 15×15, and can be adjusted, increased or decreased according to the specific image resolution; the pixels to be calibrated can also be distributed in other arrangements within the entire image range. The 100 sets of calibration data do not need to be acquired at one time, but can be accumulated during clinical examination, or extracted from clinical video and image data, and the number of data sets is not limited to 100. The margin coefficient can be reasonably adjusted according to the data acquisition situation and the number of data groups. It can also be calibrated separately for different types of endoscopes and organs to improve the calibration accuracy, such as gastroscope and stomach, colonoscope and colon, etc.

It should be noted that, in the same manner as the calibration of the offset range circle within the above Δt time, the calibration of the offset range circle within the NΔt (N≥2) time can also be performed, and then the offset range circle within the NΔt time is used to The offset range of a specific area is determined for each image at intervals of N−1 images. For example, it is possible to calibrate the offset range circles in multiple times such as Δt, 2Δt, and 3Δt at the same time, and divide a series of images collected into two categories: images with relatively rich features and images with relatively fuzzy features, and then first analyze the images with relatively rich features The image is processed, and each image carries acquisition time information. According to the time interval distance between it and the target image, select the offset range circle in the corresponding time period to determine the offset range that a specific area can produce in the image, and then carry out alignment. Repeat this process until all the images with relatively rich features are aligned, and then use the same method to align the images with relatively blurred features in turn.

It should also be noted that in the above calibration, not only circles can be used to define the offset range of pixels to obtain the offset range circle, but other arbitrary shapes, such as squares, can also be used to obtain the offset range square.

In addition, for the method of identifying a specific area in the target image shown in Figure 7, since the specific area is preset, the specific area can be pre-calculated according to the offset range within the calibrated NΔt (N≥1) time The offset range in other images in the series of images. In this way, after the series of images are captured, step S120 shown in FIG. 1 can be omitted, and step S130 can be directly performed.

FIG. 15 is a flowchart of an embodiment of the alignment step shown in FIG. 1 . FIG. 16 is a schematic diagram of aligning images according to the specific area shown in FIG. 5 within the offset range shown in FIG. 11 according to the method shown in FIG. 15 . The alignment step specifically includes: step S810, within the specific region 403 in the target image 301 and the offset range 604 in the image to be aligned 302, respectively, using the Speed Up Robust Feature (SURF for short) algorithm to search for features point, the feature point 810 in Figure 16 is one of many feature points; step S820, use the Fast Library for Approximate Nearest Neighbors (FLANN for short) to match feature point pairs, and obtain many feature points such as 804-809 Right; step S830, use the Random Sample Consensus (RANSAC for short) algorithm to screen feature point pairs, for example, filter out feature point pairs 805 to 808; step S840, use the coordinates of the screened out feature point pairs, and the simultaneous equation The group solves the homography transformation matrix from the image 302 to the image 301 as the alignment transformation matrix, that is, the alignment mapping relationship is obtained, so as to align the image 302 to be aligned to the target according to the specific area 403 within the offset range 604 Image 301. After alignment, the accurate corresponding area 821 of the specific area 403 in the second image 302 can be obtained, and then the range of offset that the specific area 403 can produce in the third image 303 can be estimated according to the corresponding area 821; and then in Within the offset range in the image 303 , the image to be aligned 303 is aligned to the target image 301 according to a specific region 403 . By reciprocating in this way, the images 302 - 325 can be sequentially aligned to the target image 301 according to the specific area 403 within the corresponding offset range.

The endoscopic image processing method provided by the present invention only searches for feature points in a specific area and the corresponding offset range and performs subsequent processing. Compared with processing the entire image, the amount of calculation is greatly reduced, and the offset range The limitation of , guarantees the alignment effect in a specific area. Alternatively, the alignment transformation unit 803 can also use the coordinates of the selected feature point pairs and the simultaneous equations to solve the similarity transformation or affine transformation matrix as the alignment transformation matrix to align the image to be aligned with the target image .

In addition, remote sensing image registration based on Marr wavelet improved SIFT algorithm, or infrared and visible light image registration algorithm based on saliency and ORB can be used to align the image to be aligned to the target in a specific area within the offset range image. Among them, the remote sensing image registration based on the Marr wavelet improved SIFT algorithm is to use the Marr wavelet under the scale space theory to extract the features of the target image and the image to be aligned, and then use the Euclidean distance to compare the features of the target image and the image to be aligned Points are initially aligned, and then the results of the initial alignment are fine-aligned according to the random sampling consensus method. The infrared and visible light image registration algorithm based on saliency and ORB is to use the optimized HC-GHS saliency detection algorithm to obtain the saliency structure map of the image, and then use the ORB algorithm to detect the feature points on the saliency structure map. The feature points with strong robustness are screened out, and the group matching is performed according to the direction of the feature points, and finally the matching of the feature points is realized by using the Hamming distance. Alternatively, it is also possible to use a specific area as an alignment template, sweep within the corresponding offset range, and use mutual information as a similarity measure for alignment; or, use B-spline free deformation or demons algorithm based on grayscale The elastic alignment method is used for alignment.

Fig. 17 is a flow chart of another embodiment of the alignment step shown in Fig. 1. Compared with the embodiment shown in Fig. 15, before searching for feature points, step S800 is performed first, that is, according to the specific endoscope model, the data stored in the The distortion coefficient in the storage module is used to correct the distortion of the specific area and the corresponding offset range to eliminate the barrel distortion caused by the camera module of the endoscope. The distortion coefficient can be calibrated by using the Zhang camera through the checkerboard calibration board The method is obtained by pre-calibrating the endoscope. After the image alignment is completed, step S850 is performed, that is, the distortion coefficient is used again to convert the alignment mapping relationship to before distortion correction, so as to realize the alignment of the image to be aligned to the target image.

When there are multiple identified specific regions, different alignment weights need to be applied to the multiple specific regions. The following takes the image alignment process shown in FIG. 15 as an example to illustrate how to use the alignment weights for alignment during image alignment. First, when screening feature point pairs, it is ensured that the ratio of the number of feature point pairs screened out from each specific area is equal to the ratio of the alignment weights. For example, the alignment weights of the specific area 501 and the specific area 502 in FIG. 10 are 1 and 0.33 respectively, and 4 pairs of feature points need to be screened out, then 3 pairs are screened from the specific area 501, and only 1 pair is screened from the specific area 502. In this way, when the simultaneous equations solve the aligned homography transformation matrix, the specific area 501 plays a leading role, and the specific area 502 plays an auxiliary role, which ensures the alignment effect of the specific area 501 and takes into account the alignment effect of the specific area 502. quasi effect. In addition, any set of images stored in the memory module can also be recalled to re-designate a specific area, re-estimate the offset range, and re-align.

Fig. 18 is a flowchart of another embodiment of the endoscopic image processing method of the present invention. Compared with the endoscopic image processing method shown in Fig. 1, step S1010 is performed before alignment, that is, preliminary image alignment, Step S1020, that is, image alignment adjustment, is performed after preliminary alignment to achieve a better alignment effect. FIG. 19 is a schematic diagram of preliminary alignment of a series of images in the manner shown in FIG. 18 . Within the corresponding offset range, a group of images 1130 are preliminarily aligned according to a specific area 1131 using similar transformation, that is, images 1102 to 1125 to be aligned (only 1112, 1113, and 1114 are shown in FIG. 19 ) to The 24 similar transformation matrices of the target image 1101 preliminarily align the images 1102-1125 to be aligned to the target image 1101, and the images 1101-1125 respectively correspond to the 25 kinds of illumination lights 201-225 shown in FIG. 2 . After preliminary alignment, the corresponding regions 1132-1155 of the specific region 1101 in the images 1102-1125 to be aligned can be obtained (only 1142, 1143, 1144 are shown in FIG. 19).

Since human tissues (including diseased tissues) are not sensitive to light of certain specific wavelengths, images taken when insensitive light may be darker or have fewer features. For example, the image 1113 in FIG. 19 is dark and the feature information is not obvious. However, the information of this image is necessary when calculating the spectral curve, and the accuracy of its alignment affects the accuracy of the subsequent spectral pathological analysis. Although there is a limit to the range of offset, due to the inconspicuous feature information, large deviations may still occur during preliminary alignment, for example, the corresponding area 1143 of a specific area 1101 in the image 1113 has obvious deviations; however, during image capture, various The image offset caused by factors is continuous, and there should be no excessive mutation. Therefore, the offset trajectory of a group of images 1130 can be calculated according to the 24 similar transformation matrices initially aligned, and the alignment mapping relationship of images with obvious deviations can be adjusted according to the offset trajectory.

Specifically, the similarity transformation can be decomposed into three basic transformations: displacement transformation, rotation transformation and scaling transformation. Figure 20, Figure 21, and Figure 22 are schematic diagrams of displacement transformation, rotation transformation, and scaling transformation included in similar transformation, wherein 1201 is the image before transformation, 1202 is the image after displacement transformation, 1203 is the image after rotation transformation, and 1204 is the transformed image. Correspondingly, the offset track of a group of images 1130 may be composed of a displacement track, a rotation track and a scaling track.

FIG. 23 is a flowchart of an embodiment of the image alignment adjustment step shown in FIG. 18 . Specifically, in step S1310, first, according to the 24 similar transformation matrices from images 1102-1125 to the target image 1101, respectively calculate the offset trajectories of a group of images 1130, ie displacement trajectories, rotation trajectories and scaling trajectories. Fig. 24, Fig. 25, and Fig. 26 are the schematic diagrams of the displacement trajectory, the rotation trajectory and the zoom trajectory after preliminary alignment respectively; Corresponding to the images 1112, 1113, and 1114; the vertical coordinates are displacement, rotation, and scaling respectively, 1401 is the displacement track, 1402 is the rotation track, and 1403 is the scaling track. Step S1320, calculate the variance of each site that constitutes the offset trajectory, or calculate the gradient of two adjacent sites, and determine that the site 1410 whose variance or gradient exceeds the set threshold is an abnormal site, and the corresponding abnormal site 1410 Image 1113 is an image with obvious deviation on the corresponding offset track. Step S1330, using the displacement transformation matrix from the image 1112 and the image 1114 to the image 1101, interpolating to calculate a new displacement transformation matrix; using the rotation transformation matrix from the image 1112 and the image 1114 to the image 1101, interpolating to calculate a new rotation transformation matrix; using The scaling transformation matrix from image 1112 and image 1114 to image 1101 is interpolated to calculate a new scaling transformation matrix. Step S1340: Synthesize the new displacement transformation matrix, rotation transformation matrix and scaling transformation matrix into a new similarity transformation matrix as a new alignment transformation matrix from image 1113 to image 1101, and complete the alignment mapping of image 1113 with obvious deviation relationship adjustments. If abnormal points are detected in only one or two kinds of offset trajectories, it is only necessary to adjust the corresponding basic transformation matrix, and then synthesize a new similar transformation matrix; or only calculate one kind of offset trajectories, only one Detection and adjustment of the underlying transform dimensions.

In addition, affine transformation or homography transformation can also be used to initially align a series of images according to a specific area within the offset range. In addition to displacement, rotation, and scaling trajectories, other offset trajectories can also be calculated, such as Shear track or perspective track, detect and adjust the corresponding base transform dimension.

By means of a software program, a set of aligned images is arranged in the order of the corresponding center wavelength of the illumination light to generate a spectral data cube of a specific area. In addition, the user can also select any pixel point through the input device, or automatically select the pixel point of the gray center of gravity in a specific area, take the center wavelength as the abscissa, and the gray value as the ordinate, to generate the spectrum of the pixel point curve. Among them, the spectral curves of all pixels in a specific area can be directly generated; or the specific area and the corresponding areas in other images are respectively gray-scale averaged, and then the average spectral curve of the specific area is generated. For the case where multiple specific regions are specified as shown in FIG. 10 , spectral data cubes or spectral curves for each specific region can be generated separately.

A software program is used to fuse all or part of an aligned set of images to produce a single image. One implementation is: separately calculate the gray level variance of a specific area and the corresponding area in other images, sort all the images according to the variance from large to small, and automatically select the top 50% of the images for fusion, that is, select the gray level Images with relatively large differences are fused. Another implementation is: with the same threshold standard, re-screen feature points from specific regions and corresponding regions in other images, sort by the number of feature points contained in descending order, and automatically select the top 30% of the images For fusion, that is, to select images with rich feature information for fusion. In addition, the user can also select any number of images through the input device, and through pseudo-color assignment, they can be fused to generate a color image. Or, treat all images as one color component, perform white balance, and fuse to generate a white light image.

In addition, an embodiment of the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and operable on the processor. Wherein, the memory and the processor communicate with each other through the bus, and the processor is used to call the program instructions in the memory to perform the following method: identify a specific area in the target image, wherein the target image is irradiated from the object under inspection in the lighting unit An image randomly selected from a series of images taken below; calculate the offset range that a specific area can produce in other images in a series of images, and align a series of images according to a specific area within the offset range allow.

The electronic device provided by the embodiment of the present invention can execute the specific steps in the endoscopic image processing method, and can achieve the same technical effect, which will not be described in detail here.

In the above embodiments, the memory is a separate module, including the above-mentioned storage module, which can be implemented based on non-volatile memory, such as Electrically Erasable Programmable Read Only Memory (EEPROM for short) , flash memory (Flash memory), etc., also can be hard disk, optical disk or magnetic tape. In another embodiment, the memory is only used to store the programmed endoscopic image processing method, and the above-mentioned storage module belongs to another storage unit independent of the memory. When the endoscopic image processing method is executed, If the data needs to be retrieved, it is retrieved from the storage module.

Fig. 27 is a schematic structural view of an embodiment of the endoscope system of the present invention. The endoscope system provided in FIG. 27 includes an illumination unit 1501 , an endoscope body 1502 and electronic equipment 1506 . Wherein, the illumination unit 1501 generates various illumination lights, which are guided by the endoscope body 1502 and irradiated on the inspected object 1520 respectively. The endoscope mirror body 1502 includes a camera module 1504, which captures a series of images 1530 of the inspected object 1520 corresponding to the various illumination lights; the camera module 1504 has an optical imaging lens 1505, and the optical imaging lens 1505 is located in the endoscope The top of body 1502. The processor in the electronic device executes the endoscopic image processing method to designate a specific area in any image of the series of images 1530; estimate the displacement of the specific area in other images of the series of images 1530 range; align a series of images 1530 according to the specific area within the offset range; use the aligned series of images to generate a spectral data cube or spectral curve for the specific area; All or part of a series of images aligned as described above are fused to generate an image. A memory for storing the image, specific regions, offset ranges, spectral data, etc. The electronic equipment has its own display device. In addition, the electronic equipment can also be connected with an external display device for displaying the image, specific area and spectral data, etc. The display device can be realized by selecting Sony's medical display LMD-2451MC or other displays matching the image resolution. In order to facilitate the user's manual operation, the electronic device can also be provided with an input device, such as a touch panel, a keyboard, a voice input device, a camera, a scanner and other arbitrary input auxiliary devices. It should be noted that the inspected object 1520 in FIG. 27 is only an example, and the actual inspected object may be stomach, intestine, and any other organs or parts to which the endoscope is applicable.

Electronic devices can be implemented based on FPGA (Field Programable Gate Array, Field Programmable Logic Gate Array). In addition, it can also be implemented based on SoC (System on Chip, system on chip), or ASIC (Application Specific Integrated Circuit, application specific integrated circuit), or embedded processor, or directly use a computer, or combine one or more of the above solutions to fulfill.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (8)

1. An endoscope image processing method, characterized in that, comprising: Identify a specific area in the target image, wherein the target image is an image selected arbitrarily from a series of images taken by the inspected object under the illumination of the lighting unit; calculate other values of the specific area in the series of images A range of offsets that can be generated within an image; within the range of offsets, aligning the series of images according to the specific area; Wherein, calculating the offset range that the specific area can produce in other images in the series of images includes: calculating the range of offset of the specific area in other images based on the pixel offset range that can be generated by pre-marked pixels. The range of offsets that can be generated within the image; Before aligning the series of images according to the specific region within the offset range, it also includes calculating an offset track of the series of images, and adjusting a certain image in the series of images based on the offset track. The alignment mapping relationship of these images. 2. The endoscope image processing method according to claim 1, wherein identifying a specific region in the target image comprises: Manually designate a certain area in the target image as a specific area; or adjust the endoscope body when shooting the target image so that the target object falls into the preset area, and use the preset area as the specific area; or in the target image automatically recognizes the area where the target object is located as a specific area. 3. The endoscopic image processing method according to claim 1 or 2, characterized in that, the specific regions include a plurality of different alignment weights are applied to a plurality of specific regions, based on the alignment weights in the The series of images are aligned within the offset range. 4. The endoscopic image processing method according to claim 1, characterized in that, the calculation of the range of pixel offsets that can be produced by the specific region in other images based on the pre-marked pixel points can produce Offset ranges include: Obtain a pre-marked pixel offset range of at least three pixels; Taking the closed area formed by the intersection of the outermost tangent lines of the pixel offset range as the offset range, or taking the outer boundary of the pixel offset range as the offset range; Wherein, the at least three pixel points include three pixel points located on the boundary of the specific area. 5. The endoscopic image processing method according to claim 1, wherein aligning the series of images according to the specific region within the offset range comprises: Search for feature points and match feature point pairs within a specific area of the target image and the offset range of the image to be aligned, and calculate an alignment mapping relationship from the image to be aligned to the target image based on the feature point pairs. 6. The endoscopic image processing method according to claim 5, characterized in that, before searching for feature points, it also includes: carrying out distortion correction based on the distortion coefficient to the specific region of the target image and the offset range of the image to be aligned; After the image alignment is completed, before the alignment mapping relationship is converted to distortion correction based on the distortion coefficient, the alignment of the image to be aligned to the target image is realized. 7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor according to claim 1 is implemented when executing the program. - the step of the endoscopic image processing method described in any one of 6. 8. An endoscope system, comprising an endoscope body, further comprising the electronic device according to claim 7, wherein the processor is in communication connection with the camera module in the endoscope body.