JP7163155B2 - Image processing device - Google Patents

Description

The present invention relates to a technique for enhancing the visibility of a region of interest in an endoscopic image.

Patent Literature 1 discloses an endoscope apparatus that sets a region of interest for a captured image including a subject and performs local scaling processing to enlarge the region of interest relative to other regions. This endoscope apparatus performs variable magnification processing on an image while maintaining the angle of view, and enlarges a region of interest by a predetermined magnification.

JP 2012-245157 A

In recent years, neural networks have been developed for extracting lesions contained in captured endoscopic images for the purpose of supporting diagnosis by doctors in medical facilities. In addition, when observing an endoscopic image, since areas containing bubbles and residue are not suitable for observation, a neural network for extracting bubbles and residue contained in the endoscopic image is also attracting attention. When displaying endoscopic images, doctors can more efficiently interpret images by marking areas that contain lesions and areas that contain bubbles and debris.

However, even if the lesion area is marked, if the lesion area is small, the visibility of the lesion area is not high. Further, when displaying a plurality of endoscopic images side by side on the display device, each endoscopic image is displayed in a reduced size, which reduces the visibility of the lesion area. Also, if the screen size of the display device is originally small, the lesion area will be displayed small. Therefore, there is an urgent need to improve the visibility of the lesion area.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for enhancing the visibility of a region of interest including a lesion.

In order to solve the above problems, an image processing apparatus according to one aspect of the present invention divides an image region of an endoscopic image into a first region that is a region of interest, a second region that is an inappropriate region for observation, a first a region information acquisition unit that acquires a region and a result of classification into a third region different from the second region; a transformation processing unit that enlarges the first region and shrinks the second region with priority over the third region; and a display control unit that causes a display device to display the deformed image deformed by the deformation processing unit.

Any combination of the above constituent elements, and conversion of expressions of the present invention into methods, devices, systems, recording media, computer programs, etc. are also effective as aspects of the present invention.

ADVANTAGE OF THE INVENTION According to this invention, the technique which improves the visibility of the region of interest containing a lesion can be provided.

2 is a diagram showing functional blocks of an image processing system; FIG. It is a figure which shows the example of an endoscope image. FIG. 10 is a diagram showing an example of a deformed image obtained by deforming an endoscopic image; FIG. 10 is a diagram showing another example of an endoscopic image; FIG. 10 is a diagram showing an example of a deformed image obtained by deforming an endoscopic image;

FIG. 1 shows functional blocks of an image processing system 1 . The image processing system 1 includes an endoscope image storage unit 10 , a region classification device 20 , an image processing device 30 and a display device 40 . The endoscopic image storage unit 10 stores captured endoscopic images. The endoscopic image may be an image captured by a doctor during an endoscopy, or an image captured by a capsule endoscope.

The region classifier 20 may be a neural network that recognizes image regions contained in the endoscopic image and classifies categories for each image region. Here, the region classification device 20 includes at least a first function for extracting a first region, which is a region of interest in which lesions are photographed, and a second function for extracting a second region in which bubbles, residues, etc. are photographed. A second area is an area unsuitable for observation by a physician due to the presence of bubbles or debris, where unsuitable for observation includes impossible or unnecessary observation. The first function and the second function may each be realized by machine-learned models.

The region classifier 20 further has a third function of extracting a third region different from the first and second regions. The third area includes an imaged area of normal mucosa and the like. In an embodiment, the region classifying device 20 may extract a region that was not extracted as either the first region or the second region as the third region. That is, the third region may be defined as the image region that was not extracted as the first and second regions. Note that the third region may be extracted by a machine-learned model.

The region classification device 20 may classify the image regions of the endoscopic image into categories at arbitrary timing. For example, when an endoscopic image is stored in the endoscopic image storage unit 10, the region classification device 20 may perform image region classification processing. The classification processing result is stored in the endoscopic image storage unit 10 in association with the endoscopic image. Note that when a doctor observes an endoscopic image, the region classification device 20 reads the endoscopic image from the endoscopic image storage unit 10, performs classification processing, and outputs the classification processing result together with the endoscopic image to the image processing device. 30 may be provided. In any event, the region classifier 20 identifies categories of image regions of the endoscopic image prior to image processing by the image processor 30 .

FIG. 2 shows an example of an endoscopic image. In this example, the endoscopic image is divided into 8 in the horizontal direction and 8 in the vertical direction. The divided local regions have the same size and are composed of groups of a predetermined number of pixels that are one or more. The region classification device 20 performs image region classification processing by specifying the category of each of a total of 64 local regions.

The image regions of the endoscopic image of the embodiment are a first region 50 which is a region of interest, a second region 60 which is an inappropriate region for observation, and a third region 70 which is a region other than the first region and the second region. are categorized. Although the first region 50 is shown in FIG. 2 as a continuous region spanning multiple focal regions, lesions may vary in size and may be the size of a single cell.

The region classification device 20 divides the endoscopic image into a plurality of local regions and calculates the category of each local region. The category of each local region is neither a category indicating that it contains a lesion (category of interest), a category indicating that it contains bubbles or residue unsuitable for observation (category unsuitable for observation), nor a region of interest or unsuitable for observation. It is set to one of the categories (normal category) indicating that it is an area.

The region classification device 20 performs image analysis on the local region, calculates the probability of each category of the local region, and identifies the category with the highest probability. For example, if it is calculated that a certain local region has an 80% probability of being an interesting category, a 15% probability of being an unobservable category, and a 5% probability of being a normal category, the category of the local region is The most probable interest category is derived.

In FIG. 2, the horizontal axis is the X axis, the vertical axis is the Y axis, and the local areas are expressed by (X coordinate, Y coordinate), and the local areas classified into the interest categories are as follows.
(1,3), (2,3), (3,3), (4,3), (5,3), (6,3), (7,3), (8,3), (1 ,4), (2,4), (3,4), (4,4), (5,4), (6,4), (7,4), (8,4)
A set of local regions classified into interest categories constitutes the first region 50 .

Local areas classified into the unobservable category are as follows.
(1,6), (2,6), (3,6), (4,6), (5,6), (6,6), (7,6), (8,6), (1 ,7), (2,7), (3,7), (4,7), (5,7), (6,7), (7,7), (8,7), (1,8 ), (2,8), (3,8), (4,8), (5,8), (6,8), (7,8), (8,8)
A set of local regions classified into the unobservable category constitutes the second region 60 .

The locoregions classified into the normal category are:
(1,1), (2,1), (3,1), (4,1), (5,1), (6,1), (7,1), (8,1), (1 , 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (1, 5 ), (2,5), (3,5), (4,5), (5,5), (6,5), (7,5), (8,5)
A set of local regions classified into the normal category constitutes the third region 70 .

When a doctor observes an image, the image processing device 30 performs image deformation processing for enlarging a region of interest (first region) included in the endoscopic image based on the classification processing result of the region classification device 20, and deforms the region. The image is output to the display device 40 . This enhances the visibility of the region of interest, and the doctor observes the magnified region of interest.

The image processing device 30 includes an area information acquisition section 32 , a deformation processing section 34 and a display control section 36 . The area information acquisition unit 32 acquires the result of classifying the image area of the endoscopic image into the first area, the second area, and the third area. The region information acquisition unit 32 may acquire the classification result from the endoscopic image storage unit 10 together with the endoscopic image.

The deformation processing unit 34 enlarges the first area 50 and reduces the second area 60 with priority over the third area 70 . The display control unit 36 causes the display device 40 to display the deformed image deformed by the deformation processing unit 34 .

Each configuration of the image processing device 30 can be implemented by any processor, memory, or other LSI in terms of hardware, and implemented by programs loaded in the memory in terms of software. It depicts the functional blocks realized by cooperation. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof.

FIG. 3 shows an example of a deformed image obtained by deforming the endoscopic image shown in FIG. The deformation processing unit 34 enlarges the first area 50 and reduces the second area 60 by the area of the enlarged first area 50 . In this example, the deformation processing unit 34 doubles the area of each of the local regions classified into the interest category to improve the visibility of the first region 50 .

The deformation processing unit 34 calculates the increased area by enlarging the first region 50, and compares the calculated increased area with the total area of the second region 60 before deformation. If the increased area of the first region 50 after deformation is equal to or less than the total area of the second region 60 before deformation, the deformation processing unit 34 reduces the increased area of the first region 50 by reducing the second region 60. I will cover it by doing.

In this example, since the area of the local region forming the first region 50 is doubled, the increase in area is (the area of the local region before deformation×16). The total area of the second region 60 before deformation is (the area of the local region before deformation×24), and the total area of the second region 60 before deformation is larger. Therefore, the deformation processing unit 34 reduces the area of the second area 60 without reducing the area of the third area 70 , thereby allocating the area reduction of the second area 60 to the area increase of the first area 50 .

The visibility of the first area 50 can be improved by displaying the first area 50 in an enlarged manner. Further, since the display area of the third region 70 is not reduced, the visibility of the third region 70 can be maintained. In this way, the deformation processing unit 34 performs the deformation processing of shrinking the second region 60 with priority over the third region 70 , thereby maintaining the visibility of the third region 70 while reducing the size of the first region 50 . Visibility can be improved. Note that the deformation processing section 34 may change the shape of the third region 70 as long as the display area of the third region 70 is maintained.

The deformation processing unit 34 performs deformation processing of the local region by solving an optimization problem for finding the positions of the lattice points of the local region using the objective function and the constraint conditions. The objective function may be a function based on any one of the sum, average, variance, and standard deviation of the amount of movement of each lattice point. Constraints may be defined as follows.

(Constraint 1)
The area of the first region after deformation is a predetermined multiple of the area of the first region before deformation. The enlargement rate may be set as appropriate, but in the embodiment, the enlargement rate is set to 2 times.
(Constraint 2)
The area ratios of the local regions that constitute the second region are the same before and after the deformation. For example, in the example shown in FIG. 3, the (area after deformation/area before deformation) of the local region forming the second region is 1/3.
(Constraint 3)
The area of the local region forming the third region does not change before and after deformation.

ここでp_xyは、画像領域左下隅の原点から横方向にx番目、縦方向にy番目の格子点の移動後の位置を表す２次元ベクトルであり、o_xyは格子点の移動前の初期位置を表す２次元ベクトルである。 In the following example, the transformation processing unit 34 sets the objective function f to be the sum of the amounts of movement of each lattice point.

Here, p _xy is a two-dimensional vector representing the position after movement of the x-th grid point in the horizontal direction and the y-th grid point in the vertical direction from the origin at the lower left corner of the image area, and o _xy is the initial position before moving the grid point. A two-dimensional vector representing a position.

局所領域の面積S_ijに１からＮ（画像領域の分割数）の間で以下のように番号付けをする。
第２領域を構成する局所領域 (S₁, ... ,S_a)
第１領域を構成する局所領域 (Sa+1, ... ,S_b)
第３領域を構成する局所領域 (S_b+1, ... ,S_N) At this time, the area S _ij of the local region surrounded by the grid after the movement of the four points is expressed by the following formula.

The areas S _ij of the local regions are numbered from 1 to N (the number of divisions of the image region) as follows.
Local regions (S ₁ , ... ,S _a ) forming the second region
Local regions (Sa+1, ... ,S _b ) that make up the first region
Local regions (S _b+1 , ... ,S _N ) constituting the third region

ここでSOrig_nは、S_nの初期値であり、変形前の局所領域の面積である。第１領域の局所面積に乗算されている「２」は、第１領域の拡大率であり、第２領域の局所面積に乗算されている「ｒ」は、第１領域の増加面積分を吸収するための第２領域の拡大率（＜１）である。変形処理部３４は、制約条件を用いて、最急降下法等の最適化手法により、各格子点の移動後の座標を求める。 The constraint conditions at this time are shown.

Here, SOrig _n is the initial value of S _n and the area of the local region before deformation. "2" multiplied by the local area of the first region is the magnification of the first region, and "r" multiplied by the local area of the second region absorbs the increased area of the first region. is the enlargement ratio (<1) of the second region for The deformation processing unit 34 uses the constraint conditions and obtains the post-movement coordinates of each grid point by an optimization method such as the steepest descent method.

If the increase in area by enlarging the first region 50 in this way is equal to or less than the total area of the second region 60 before deformation, the deformation processing unit 34 does not reduce the third region 70, A deformation process for reducing the two regions 60 is performed.

Next, a description will be given of deformation processing in the case where the increase in area by enlarging the first region 50 is larger than the total area of the second region 60 before deformation.

FIG. 4 shows another example of an endoscopic image.
In this example, the local regions classified into categories of interest are:
(1,3), (2,3), (3,3), (4,3), (5,3), (6,3), (7,3), (8,3), (1 ,4), (2,4), (3,4), (4,4), (5,4), (6,4), (7,4), (8,4)

Local areas classified into the unobservable category are as follows.
(1,8), (2,8), (3,8), (4,8), (5,8), (6,8), (7,8), (8,8)
The local regions classified into the normal category are the remaining local regions.

The deformation processing unit 34 enlarges the first area 50 and reduces the second area 60 with priority over the third area 70 . Here, the deformation processing unit 34 compares the increased area of the first region 50 with the total area of the second region 60 before deformation. As described above, if the increased area of the first region 50 after deformation is equal to or less than the total area of the second region 60 before deformation, the deformation processing unit 34 changes the increased area of the first region 50 to the second region. This is covered by shrinking the region 60 .

In the example shown in FIG. 4, the area of the local area that constitutes the first area 50 is doubled, and (the area of the local area before deformation×16) is the increase in area. On the other hand, the total area of the second region 60 before deformation is (the area of the local region before deformation×8), and the increased area of the first region 50 is larger. Therefore, the deformation processing unit 34 determines that the increase in the area of the first region 50 cannot be compensated for by simply eliminating the second region 60, and that the third region 70 must be reduced.

FIG. 5 shows an example of a deformed image obtained by deforming the endoscopic image shown in FIG. In the deformed image shown in FIG. 5, the first area 50 is displayed enlarged, the second area 60 is deleted from the display, and the third area 70 is displayed reduced. At this time, it is preferable that the area ratios of the respective local regions forming the third region 70 before and after deformation are the same. In the example shown in FIG. 5, the area increase of the first region 50 minus the total area of the second region 60 (the area of the local region before deformation×8) is covered by the reduction of the third region 70 . Therefore, the (area after deformation/area before deformation) of all the local regions constituting the second region is 4/5.

The visibility of the first area 50 can be improved by displaying the first area 50 in an enlarged manner. Further, even though the display area of the third region 70 is reduced, the visibility of the third region 70 can be maintained by making the area ratios of the local regions before and after deformation the same. In this way, the deformation processing unit 34 performs the deformation processing of shrinking the second region 60 with priority over the third region 70 , thereby maintaining the visibility of the third region 70 while reducing the size of the first region 50 . Visibility can be improved.

第１領域の局所面積に乗算されている「２」は、第１領域の拡大率であり、第３領域の局所面積に乗算されている「ｒ」は、第３領域の拡大率（＜１）である。変形処理部３４は、制約条件を用いて、最急降下法等の最適化手法により、各格子点の移動後の座標を求める。 The constraint conditions at this time are shown.

“2” multiplied by the local area of the first region is the magnification of the first region, and “r” multiplied by the local area of the third region is the magnification of the third region (<1 ). The deformation processing unit 34 uses the constraint conditions and obtains the post-movement coordinates of each grid point by an optimization method such as the steepest descent method.

When the increase in area by enlarging the first region 50 in this way is larger than the total area of the second region 60 before deformation, the deformation processing unit 34 deletes the second region 60 while removing the third region. A deformation process is performed to reduce the local area of 70 at the same magnification.

Although the deformation processing unit 34 doubles the magnification of the first area 50 in the above embodiment, the magnification of the first area 50 may be determined according to the visibility of the first area 50 . The deformation processing unit 34 determines whether the visibility of the first region 50 before deformation is good or bad according to the display size of the first region 50 on the display device 40 . For example, when the display size (for example, vertically or horizontally) of first area 50 on display device 40 is less than a predetermined value, deformation processing unit 34 determines that visibility of first area 50 is low. At this time, the deformation processing unit 34 may enlarge the display size of the first area 50 on the display device 40 to a size equal to or larger than a predetermined value.

Note that the display size of the first area 50 on the display device 40 depends on the screen size of the display device 40 . In addition, the display size of the first area 50 is the same between when displaying on a terminal device such as a PC and when displaying on a portable terminal device such as a smartphone because the distance between the eyes and the display device is different. visibility is different. Therefore, the area information acquisition unit 32 may set a threshold value of the display size for determining the quality of visibility according to the type and screen size of the display device 40 .

It should be noted that the quality of visibility may be determined based on not only the display size but also the type of lesion, color, contrast, brightness, and the like. In any case, the deformation processing unit 34 enlarges and deforms the first region 50 included in the original endoscopic image when it is estimated that the visibility of the first region 50 is low. Note that the deformation processing unit 34 does not need to enlarge and deform the first region 50 when it is estimated that the visibility of the first region 50 is high.

The present invention has been described above based on the examples. Those skilled in the art will understand that the embodiments are illustrative, and that various modifications can be made to combinations of each component and each treatment process, and such modifications are also within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Image processing system 10... Endoscope image storage part 20... Region classification device 30... Image processing device 32... Region information acquisition part 34... Deformation process Section 36 Display control section 40 Display device 50 First area 60 Second area 70 Third area.

Claims (6)

Acquiring a result of classifying an image region of an endoscopic image into a first region that is a region of interest, a second region that is an inappropriate region for observation, and a third region that is different from the first region and the second region. a region information acquisition unit for
a transformation processing unit that enlarges the first area and shrinks the second area with priority over the third area;
a display control unit for displaying a deformed image deformed by the deformation processing unit on a display device;
An image processing device comprising: The deformation processing unit reduces the second area without reducing the third area.
2. The image processing apparatus according to claim 1, wherein: The deformation processing unit reduces the third region when the area increased by enlarging the first region is larger than the total area of the second region.
2. The image processing apparatus according to claim 1, wherein: The deformation processing unit determines the enlargement ratio of the first area according to the visibility of the first area.
2. The image processing apparatus according to claim 1, wherein: An endoscopic image is divided into a plurality of local regions,
The deformation processing unit generates the deformed image by deforming the local region.
2. The image processing apparatus according to claim 1, wherein: The deformation processing unit makes the area ratios of the local regions forming the third region the same after deformation and before deformation,
6. The image processing apparatus according to claim 5, characterized by:

Images