JP2011255006A - Image processor, endoscopic device, program and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor for preventing alert image from hindering diagnosis, treatment or the like, an endoscopic device, a program, an image processing method or the like.SOLUTION: The image processor includes: a white light image acquisition unit 320 and a special light image acquisition unit 330 which acquire an acquisition image that is at least one of a general light image and a specific light image; a target region detection unit 340 which detects a target region that is a region which should be observed based on the feature amount of pixels of the acquisition image; an alert image display setting unit 350 which determines, according to the detection result of the target region, whether to display an alert image corresponding to the target area; and a display mode control unit 360 which controls for displaying an alert image corresponding to a display object target region that is a target area for which it is determined that the alert image should be displayed.

Description

  The present invention relates to an image processing apparatus, an endoscope apparatus, a program, an image processing method, and the like.

  Conventionally, tissue in a body cavity is sequentially irradiated with light of three colors of RGB using a rotation filter, a white light image is acquired from the reflected light image, and diagnosis is performed using the white light image. Endoscopic endoscope devices are widely used. Further, the tissue in the body cavity is sequentially irradiated with two types of narrowband light G2 and B2 having different characteristics from the light of the three colors of RGB, and a narrowband light image is acquired from the reflected light image, and the narrowband light is obtained. An endoscope apparatus that performs diagnosis using a band light image has been proposed (for example, Patent Document 1). In addition, the tissue in the body cavity is irradiated with narrow-band excitation light, and the excitation light generates autofluorescence and drug fluorescence from the tissue in the body cavity to obtain a fluorescence image, and diagnosis is performed using the fluorescence image. An endoscope apparatus to be performed has been proposed (for example, Patent Document 2).

JP 2006-68113 A JP 2007-229053 A

  Now, in the diagnosis using the above-described white light image, it is known that a lesioned part such as a tumor is depicted as a region that is more reddish than a normal site. On the other hand, in the diagnosis using the narrow-band light image described above, a lesion such as squamous cell carcinoma that is difficult to visually recognize with a white light image is rendered as a brown region different from the normal site. Is known to be easier. In the diagnosis using the above-described fluorescence image, a fluorescent agent having a property of being specifically accumulated in a lesion such as a tumor is used, and fluorescence is generated only in the lesion so that the lesion can be found. It becomes easy.

  However, these narrow-band light images and fluorescent images (hereinafter referred to as special light images) generally have different colors compared to white light images. In addition, the special light image has a narrow band, so that the amount of light is insufficient and the image becomes very dark. Therefore, it is difficult to make a diagnosis using only the special light image.

  Therefore, in order to improve the diagnostic accuracy of the doctor, for example, a method of simultaneously acquiring and displaying a white light image and a special light image is conceivable. However, in this method, if two images are displayed side by side at the same time, the doctor always makes a diagnosis while paying attention to a plurality of images, and the burden on the doctor increases. It is also conceivable to miss a lesion by temporarily focusing on only one image.

  In order to suppress the oversight of the lesioned part, for example, only a white light image may be displayed, and an alert image may be displayed in an area where there is a possibility of a lesioned part in the white light image to call attention. However, in this method, the alert image interferes with the diagnosis and treatment of the lesion.

  According to some aspects of the present invention, it is possible to provide an image processing device, an endoscope device, a program, an image processing method, and the like that prevent an alert image from interfering with diagnosis or treatment.

  One aspect of the present invention is a normal light image including a subject image having information in a white wavelength band, and a special light image including a subject image having information in a specific wavelength band corresponding to the normal light image; An image acquisition unit that acquires an acquired image that is at least one of the image, a region of interest detection unit that detects a region of interest based on a feature amount of a pixel of the acquired image, and a region of the region of interest A determination unit that determines whether or not to display an alert image corresponding to the attention region according to a detection result, and a display target attention region that is the attention region that is determined to be displayed by the determination unit. A display mode control unit that performs control to display an alert image corresponding to the image processing apparatus.

  According to one aspect of the present invention, an acquired image that is at least one of a normal light image and a special light image is acquired, and a region of interest is detected from the acquired image. Then, it is determined whether or not the alert image corresponding to the detected attention area should be displayed, and control is performed to display the alert image corresponding to the attention area determined to display the alert image. This makes it possible to prevent the alert image from interfering with diagnosis and treatment.

  Another aspect of the present invention relates to an endoscope apparatus including the image processing apparatus described above.

  Still another aspect of the present invention includes a normal light image including a subject image having information in a white wavelength band, and a subject image having information in a specific wavelength band corresponding to the normal light image. An image acquisition unit that acquires an acquired image that is at least one of the special light image, an attention region detection unit that detects an attention region that is a region of interest based on the feature amount of a pixel of the acquired image, and A determination unit that determines whether or not to display an alert image corresponding to the attention region according to a detection result of the attention region; and an attention region that is determined by the determination unit to display an alert image The present invention relates to a program that causes a computer to function as a display mode control unit that performs control to display an alert image corresponding to a display target attention area.

  Still another aspect of the present invention includes a normal light image including a subject image having information in a white wavelength band, and a subject image having information in a specific wavelength band corresponding to the normal light image. Acquire an acquired image that is at least one of the special light image, detect an attention area that is an attention area based on a feature amount of a pixel of the acquired image, and according to a detection result of the attention area Determining whether or not to display an alert image corresponding to the attention area, and displaying an alert image corresponding to the display target attention area that is the attention area determined to be displayed by the determination unit. The present invention relates to an image processing method for performing control.

FIG. 1A and FIG. 1B are explanatory diagrams about the outline of the method of this embodiment. Explanatory drawing about the outline | summary of the method of this embodiment. 1 is a first configuration example of an endoscope apparatus. An example of spectral characteristics of a color filter for photographing a white light image. 6 is a configuration example of a color filter for capturing a special light image. An example of spectral characteristics of a color filter for photographing a special light image. Explanatory drawing about the outline | summary of the process which an image processing apparatus performs. The detailed structural example of a white light image acquisition part. The detailed structural example of a special light image acquisition part. Explanatory drawing about the method of producing | generating a narrow-band optical image. 3 shows a detailed configuration example of an attention area detection unit. Explanatory drawing about a local area | region. Explanatory drawing about the process which provides tag information. The detailed structural example of an alert image display setting part. The detailed structural example of a display mode control part. The detailed structural example of a display availability determination part. FIGS. 17A and 17B are explanatory diagrams of processing performed by the region size determination unit. 18A and 18B are explanatory diagrams of processing performed by the region size determination unit. An example of an image when a distant part is photographed from a distant view. An example of an image when a close-up image of a lesion is taken. Explanatory drawing about the method of calculating the detection elapsed time of an attention area group. The flowchart example showing the process sequence of an image process. The flowchart example showing the process sequence of a display mode change process. The 2nd structural example of an endoscope apparatus. 3 shows a detailed configuration example of an attention area detection unit. The flowchart example showing the process sequence of an image process. The 3rd structural example of an endoscope apparatus. Explanatory drawing about the outline | summary of the process which an image processing apparatus performs. 3 is a detailed configuration example of an attention area control unit. The detailed structural example of a display mode control part. The flowchart example showing the process sequence of an image process. The flowchart example showing the process sequence of attention area control processing. The 4th structural example of an endoscope apparatus. The flowchart example showing the process sequence of an image process. The system block diagram which shows the structure of a computer system. The block diagram which shows the structure of the main-body part in a computer system.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Method according to this embodiment An overview of the method according to this embodiment will be described with reference to FIGS.
As shown in FIG. 1A, when a doctor examines a patient using an endoscope, the doctor first checks whether there is an abnormal part in the body cavity to be examined while viewing the image. At this time, it is generally diagnosed from the color of the image, the blood vessel structure, and the like whether or not the lesion is a lesion (region of interest in a broad sense). However, in this visual diagnosis, there is a possibility of overlooking a lesion with low visibility. Therefore, in the present embodiment, when there is a region having a possibility of a lesion as indicated by E1, the region is detected using a special light image described later. And as shown to E2, an alert image is displayed on the area | region, a doctor's oversight is suppressed and a doctor's burden is reduced.

  Next, as shown in FIG. 1B, when a region suspected of being a lesion is found, an endoscope is brought close to the region, and an enlarged observation is performed for diagnosis and treatment. For example, when the lesioned part indicated by F1 is an object of diagnosis or treatment, an alert image is displayed on the lesioned part as indicated by F2. However, when the alert image is displayed in this way, the lesioned part cannot be observed directly, which hinders diagnosis and treatment. For example, as a technique for solving this problem, a technique for hiding all alert images when performing diagnosis or treatment can be considered. However, if all the alert images are not displayed, there is a possibility that the diagnosis and treatment may not be performed completely by overlooking a small lesion as shown in F3, for example.

  Therefore, in the present embodiment, as indicated by G1 in FIG. 2, for example, an alert image is not displayed on a lesion having a size larger than a predetermined threshold value, and it is easy to observe a lesion that is a target of diagnosis or treatment. . On the other hand, as shown in G2, for example, an alert image is displayed in a lesion part having a size smaller than a predetermined threshold value, and an oversight of a small lesion is suppressed.

2. First Configuration Example FIG. 3 shows a first configuration example of an endoscope apparatus that controls display / non-display of an alert image according to the size of a lesioned part. The endoscope apparatus (endoscope system) includes a light source unit 100, an imaging unit 200, a control device 300 (control unit), a display unit 400 (display device), and an external I / F unit 500.

  The light source unit 100 generates illumination light for irradiating a subject in the body cavity. Specifically, the light source unit 100 includes a white light source 110 that generates white light and a condenser lens 120 that condenses the white light on the light guide fiber 210.

  The imaging unit 200 is an insertion unit for imaging a subject in a body cavity. For example, the imaging unit 200 is formed to be elongated and bendable in order to enable insertion into a body cavity. Specifically, the imaging unit 200 includes a light guide fiber 210, an illumination lens 220, an objective lens 230, a half mirror 240, a first imaging element 250, and a second imaging element 260.

  The light guide fiber 210 guides the light collected by the light source unit 100 to the subject. The illumination lens 220 diffuses the light guided to the tip of the imaging unit 200 by the light guide fiber 210 and irradiates the subject. The objective lens 230 condenses the reflected light returning from the subject. The half mirror 240 separates the collected reflected light into two. The first image sensor 250 images the subject with one of the separated reflected lights. The second imaging element 260 images the subject with the other of the separated reflected light.

  More specifically, the first image sensor 250 is an image sensor having a Bayer array color filter for capturing a white light image (normal light image). For example, as shown in FIG. 4, the color filters r, g, and b of the first image sensor 250 have spectral characteristics that transmit red, green, and blue wavelength bands, respectively. As shown in FIG. 5, the second image sensor 260 is an image sensor in which, for example, two types of color filters g2 and b2 are arranged in a checkered pattern. As shown in FIG. 6, the color filter b2 has a spectral characteristic that transmits light in a wavelength band of, for example, 390 to 445 nm (nm: nanometer), and the color filter g2 transmits light in a wavelength band of 530 to 550 nm. It has spectral characteristics to transmit. For example, the first imaging element 250 and the second imaging element 260 have the same number of pixels.

  The control device 300 controls each component of the endoscope device and performs image processing on the captured image. The control device 300 includes a first A / D conversion unit 310, a second A / D conversion unit 311, an image processing device 301 (image processing unit), and a control unit 370.

  The image processing apparatus 301 detects a region of interest (for example, a lesion), determines whether an alert image can be displayed, and performs display control of the display image. The image processing apparatus 301 includes a white light image acquisition unit 320 (first image acquisition unit), a special light image acquisition unit 330 (second image acquisition unit), an attention area detection unit 340, and an alert image display setting unit 350 ( Alert image generation unit) and display mode control unit 360.

  The control unit 370 is connected to the white light image acquisition unit 320, the special light image acquisition unit 330, the attention area detection unit 340, the alert image display setting unit 350, and the display mode control unit 360, and controls these components.

  The external I / F unit 500 is an interface unit for performing input from the user to the endoscope apparatus. For example, the external I / F unit 500 includes a power switch for turning on / off the power, a shutter button for starting a photographing operation, a mode switching button for switching a photographing mode and various other modes. Further, the external I / F unit 500 outputs the input information to the control unit 370.

  The A / D conversion unit 310 converts the analog image signal output from the first image sensor 250 into a digital image signal, and outputs the digital image signal to the white light image acquisition unit 320. The A / D conversion unit 311 converts the analog image signal output from the second image sensor 260 into a digital image signal, and outputs the digital image signal to the special light image acquisition unit 330.

  The white light image acquisition unit 320 (ordinary light image acquisition unit) acquires a white light image based on the digital image signal from the A / D conversion unit 310, and uses the acquired white light image as an alert image display setting unit 350. The data is output to the display mode control unit 360. The special light image acquisition unit 330 acquires a special light image based on the digital image signal from the A / D conversion unit 311, and outputs the acquired special light image to the attention area detection unit 340. Details of the white light image acquisition unit 320 and the special light image acquisition unit 330 will be described later.

  The attention area detection unit 340 detects the attention area based on the special light image output from the special light image acquisition unit 330. Then, the attention area detection unit 340 outputs information on the detected attention area (for example, the coordinates of the area) to the alert image display setting unit 350 and the display mode control unit 360.

  The alert image display setting unit 350 creates an alert image corresponding to the attention area detected by the attention area detection unit 340. Specifically, the alert image display setting unit 350 performs enhancement processing on an area on the white light image corresponding to the attention area on the special light image. For example, the alert image display setting unit 350 creates an alert image by performing enhancement processing on all the pixels in the region. This alert image is created corresponding to each attention area (or each attention area group), and when a plurality of attention areas are detected, a plurality of alert images are also created. Then, the alert image display setting unit 350 outputs the created alert image to the display mode control unit 360.

  The display mode control unit 360 controls the display mode of the attention area. Specifically, the display mode control unit 360 first performs processing for determining whether or not an alert image can be displayed according to the information on the attention region output from the attention region detection unit 340. Then, the display mode control unit 360 superimposes only the alert image determined to be displayed as a result of the determination process on the white light image, and outputs the white light image to the display unit 400. Details of the alert image display setting unit 350 and the display mode control unit 360 will be described later.

3. Image processing in first configuration example 3.1. Outline of Image Processing Next, details of the image processing performed by the above configuration example will be described.
First, an outline of processing performed by the image processing apparatus 301 will be described with reference to FIG. As shown in H1 of FIG. 7, first, a special light image and a white light image are acquired. Next, as shown in H2, a region of interest (represented by an ellipse in FIG. 7) such as a lesion is detected from the special light image. At this time, a tag (represented by a number in FIG. 7) for distinguishing the attention area is assigned to each attention area. Next, as indicated by H3, an alert image (represented by a shaded square in FIG. 7) corresponding to each region of interest is generated based on the white light image. Next, as shown in H4, whether or not the alert image can be displayed is determined according to the size of the attention area and the like. For example, the alert image of the attention area of the tag “1” is determined not to be displayed, and the alert image of the attention area of the tag “2” is determined to be displayed. Then, as shown in H5, the alert image of the attention area determined to be displayed is superimposed on the white light image and displayed.

3.2. White Light Image Acquisition Unit, Special Light Image Acquisition Unit Next, a detailed configuration example and operation of each component will be described. FIG. 8 shows a detailed configuration example of the white light image acquisition unit 320. The white light image acquisition unit 320 includes a white light image generation unit 321 (normal light image generation unit) and a white light image storage unit 322 (normal light image storage unit). The white light image generation unit 321 performs image processing on the digital image signal output from the A / D conversion unit 310 to generate a white light image. Specifically, the white light image generation unit 321 performs known interpolation processing, white balance processing, color conversion processing, gradation conversion processing, and the like, and generates and outputs a white light image. For example, the white light image is an RGB image composed of pixel values of three primary colors. The white light image storage unit 322 stores the white light image output from the white light image generation unit 321.

  FIG. 9 shows a detailed configuration example of the special light image acquisition unit 330. The special light image acquisition unit 330 includes a special light image generation unit 331 and a special light image storage unit 332. The special light image generation unit 331 performs image processing on the digital image signal output from the A / D conversion unit 311 to generate a special light image. For example, the special light image is the above-described narrowband light image. As will be described later, this narrow-band light image is an RGB image composed of G2 and B2 pixel values. Then, the special light image generation unit 331 outputs the generated narrowband light image to the special light image storage unit 332. The special light image storage unit 332 stores the narrowband light image output from the special light image generation unit 331.

  A method in which the special light image generation unit 331 generates a narrowband light image will be described with reference to FIG. As described above in FIG. 3, the second image sensor 260 is an image sensor in which two types of color filters g2 and b2 are arranged in a checkered pattern. Therefore, as shown in FIG. 10, a checkered digital image signal is input to the special light image generation unit 331. Here, G2 (x, y) represents the signal value of the g2 filter, and B2 (x, y) represents the signal value of the b2 filter. X and y are the coordinates of the pixel.

  First, the special light image generation unit 331 performs an interpolation process on such an image signal, and a G2 image having a g2 filter signal value for all pixels and a b2 filter signal value for all pixels. A B2 image is generated. The signal value calculated by this interpolation processing is obtained, for example, from the average value of the signal values of the surrounding four pixels. For example, the signal value B2 (1,1) of the b2 filter at the position G2 (1,1) and the signal value G2 (1,2) of the g2 filter at the position B2 (1,2) shown in FIG. Are calculated by the following equations (1) and (2).

B2 (1,1) = {B2 (0,1) + B2 (1,0) +
B2 (1,2) + B2 (2,1)} / 4 (1)
G2 (1,2) = {G2 (0,2) + G2 (1,1) +
G2 (1,3) + G2 (2,2)} / 4 (2)

  Next, the special light image generation unit 331 generates a color image having three types of signal values of R, G, and B in the same manner as the white light image, from the G2 image and the B2 image generated by the above-described interpolation processing. Specifically, the special light image generation unit 331 inputs G2 (x, y) to the R signal of the pixel at the coordinate (x, y) of the color image, and B2 (x, y) to the G and B signals. Input an RGB color image. Then, the special light image generation unit 331 performs white balance processing, gradation conversion processing, and the like on the color image to generate a narrowband light image.

3.3. Attention Area Detection Unit FIG. 11 shows a detailed configuration example of the attention area detection unit 340. The attention area detection unit 340 includes a local area setting unit 341, a feature amount calculation unit 342, an area detection unit 343, and a tag information addition unit 344.

  The local region setting unit 341 sets a plurality of local regions for the narrowband light image output from the special light image acquisition unit 330. Specifically, as shown in FIG. 12, the local region setting unit 341 divides the narrowband light image into a plurality of rectangular regions, and sets each divided rectangular region as a local region. For example, one local area is a rectangular area of 20 × 20 pixels.

  More specifically, the narrowband light image is divided into M × N local regions. The coordinates of each local area are represented by (m, n), and the local area of coordinates (m, n) is represented by a (m, n) (m and n are integers of 0 or more). The coordinates of the local region located at the upper left of the image are defined as (0, 0), the right direction (horizontal direction) is defined as the positive direction of m, and the downward direction (vertical direction) is defined as the positive direction of n. The local area is not necessarily rectangular, and the narrowband light image may be divided into arbitrary polygons, and each divided area may be set as the local area. Moreover, you may enable it to set a local area | region arbitrarily according to a user's instruction | indication. Further, in this embodiment, in order to reduce the amount of calculation in the subsequent stage, an area composed of a plurality of adjacent pixel groups is set as one local area, but one pixel may be set as one local area. In this case as well, the subsequent processing can be performed in the same manner.

  The feature amount calculation unit 342 calculates feature amounts for all the local regions set by the local region setting unit 341. In the following, the process performed by the feature amount calculation unit 342 will be described by taking an example in which hue is used as the feature amount. Here, the hue corresponding to the local region a (m, n) is represented as H (m, n).

First, the feature amount calculation unit 342 calculates average values R_ave, G_ave, and B_ave of R, G, and B signals in each local region in order to calculate the hue H (m, n). This average value R_ave is an average value of R signals in all the pixels included in each local region. Similarly, G_ave and B_ave are average values of the G and B signals in all the pixels included in each local region. For example, each signal value is 8 bits (0 to 255). Next, the feature amount calculation unit 342 calculates the hue H (m, n) of each local region from the average values R_ave, G_ave, B_ave using, for example, the following equations (3) to (8). Here, the MAX function is a function that outputs a maximum value argument from among a plurality of arguments.
max = MAX (R_ave, G_ave, B_ave) (3)

When max = 0, the hue H (m, n) is obtained by the following equation (4).
H (m, n) = 0 (4)

When max is other than 0, first, the value d is obtained by the following equation (5). Here, the MIN function is a function that outputs an argument having a minimum value from a plurality of arguments.
d = MAX (R_ave, G_ave, B_ave) −
MIN (R_ave, G_ave, B_ave) (5)

When R_ave is the maximum among R_ave, G_ave, and B_ave, the hue H (m, n) is obtained by the following equation (6).
H (m, n) = 60 × (G_ave−B_ave) / d (6)

When G_ave is the maximum among R_ave, G_ave, and B_ave, the hue H (m, n) is obtained by the following equation (7).
H (m, n) = 60 × {2+ (B_ave−R_ave)} / d (7)

Further, when B_ave is maximum among R_ave, G_ave, and B_ave, the hue H (m, n) is obtained by the following equation (8).
H (m, n) = 60 × {4+ (R_ave−G_ave)} / d (8)

  If H <0 by the above calculation, 360 is added to H. When H = 360 is obtained by the above calculation, H = 0 is set.

  The region detection unit 343 detects a local region having a specific hue H from all the local regions as the attention region using the hue H (m, n) from the feature amount calculation unit 342. For example, the region detection unit 343 detects a local region in which the hue H (m, n) is in the range of 5 ≦ H ≦ 35 as the attention region. Then, the region detection unit 343 outputs the coordinates of the local region detected as the attention region (for example, the coordinates of all the pixels in the local region) to the tag information adding unit 344.

  Note that the narrow-band light image used in the present embodiment has a feature that a lesion such as squamous cell carcinoma is depicted as a brown region. The hue H range 5 ≦ H ≦ 35 is an example of the hue range representing the brown color, and a lesion area such as squamous cell carcinoma can be detected by detecting the area of the hue range.

  The tag information adding unit 344 performs a process of adding tag information to the attention area detected by the area detecting unit 343. Specifically, the tag information providing unit 344 performs processing for assigning the same tag information to the attention areas adjacent to each other among the detected attention areas. Then, the tag information providing unit 344 outputs the coordinates of all the attention areas (for example, the coordinates of all the pixels included in each attention area) and the tag information to the alert image display setting section 350 and the display mode control section 360.

  The process of assigning tag information will be described with reference to FIG. As shown in FIG. 13, it is assumed that there are a first lesion and a second lesion, and a local region represented by shading is detected as a region of interest. At this time, the tag information providing unit 344 integrates a plurality of adjacent attention areas, and selects a first attention area group corresponding to the first lesion area and a second attention area group corresponding to the second lesion area. Set. Then, the tag information giving unit 344 gives tag information “1” to all the local areas (attention areas) belonging to the first attention area group, and tag information to all the local areas belonging to the second attention area group. “2” is assigned. In the present embodiment, a set of attention areas to which the same tag information is assigned is referred to as an attention area group. By assigning tag information in this way, it is possible to identify different sets of attention areas.

3.4. Alert Image Display Setting Unit Next, FIG. 14 shows a detailed configuration example of the alert image display setting unit. The alert image display setting unit 350 includes a corresponding area selection unit 351 and an enhancement processing unit 352.

  The corresponding area selection unit 351 selects areas on the white light image corresponding to all the attention areas on the special light image output from the attention area detection unit 340, and information on all pixels included in the selected area. The tag information is output to the enhancement processing unit 352. Here, the pixel information is the coordinates of the pixel and the RGB signal value. Tag information is assigned to each pixel, and the tag information to be assigned is the same tag information as the tag information assigned to the attention area to which each pixel belongs.

  The enhancement processing unit 352 creates an alert image by performing color conversion processing on all the pixels output from the corresponding region selection unit 351 using the following formula (9), and displays the created alert image in a display mode. The data is output to the control unit 360. The enhancement processing unit 352 outputs information (coordinates, RGB pixel values) and tag information of all the pixels after the color conversion processing to the display mode control unit 360.

R_out (x, y) = gain × R (x, y) + (1−gain) × T_R,
G_out (x, y) = gain × G (x, y) + (1−gain) × T_G,
B_out (x, y) = gain × B (x, y) + (1−gain) × T_B
(9)

  Here, in the above equation (9), R (x, y), G (x, y), and B (x, y) are RGB signals at the coordinates (x, y) of the white light image before color conversion processing. Value. R_out (x, y), G_out (x, y), and B_out (x, y) are RGB signal values at the coordinates (x, y) of the white light image after the color conversion process. T_R, T_G, and T_B are arbitrary RGB signal values representing the target color. Further, gain is an arbitrary coefficient of 0 ≦ gain ≦ 1.

  For example, in the above equation (9), each RGB signal value is represented by 8 bits (0 to 255), (T_R, T_G, T_B) = (64, 64, 255), and gain = 0.5. Suppose there is. In this case, an alert image is obtained in which the region of the white light image corresponding to the attention region is depicted as a color region different from the normal region. Specifically, since T_B is 255, the B signal value of the white light image corresponding to the attention area is emphasized, and the area of the white light image corresponding to the attention area is depicted as a strong blue area. An alert image is obtained. Note that the T_R1, T_G1, T_B1, and gain parameters can be set by the user from the external I / F unit 500, or constant values may be set in advance.

  Here, in the above embodiment, the case where the alert image is created by the color conversion process using the above equation (9) has been described, but the present embodiment is not limited to this, and any luminance conversion process or color conversion process is performed. An alert image may be created using.

  In the above embodiment, a case has been described in which enhancement processing is performed on all pixels in the white light image region corresponding to the detected attention region using the above equation (9). It is not limited to this. For example, the alert image may be created by performing enhancement processing on the pixels in the white light image corresponding to the boundary between the attention area and the non-attention area not detected as the attention area. In this case, for example, a color conversion process represented by the following expression (10) is performed on all the pixels of the white light image corresponding to the boundary between the attention area and the non-attention area. By the color conversion process shown in the following equation (10), an alert image in which pixels in the white light image corresponding to the boundary between the attention area and the non-attention area are depicted in blue is obtained.

R_out (x, y) = 0,
G_out (x, y) = 0,
B_out (x, y) = 255 (10)

3.5. Display Mode Control Unit Now, it is assumed that all alert images created as described above are superimposed on a white light image. Then, since the lesion area is drawn as a color area different from the normal area, it is easy to find the lesion. However, in the state where the alert image is superimposed, the lesion is drawn as a color different from the normal color, so that there is a problem that the alert image hinders the diagnosis and treatment of the lesion.

  Therefore, in the present embodiment, the display mode of the alert image is controlled according to the information on the attention area detected by the attention area detection unit 340, and the alert image is prevented from obstructing the diagnosis or treatment of the lesion. Specifically, the diagnosis and treatment of a lesion is generally performed in a state where an endoscope is brought close to the lesion. Further, the treatment of the lesion is often performed in a state where the imaging unit 200 is fixed. Therefore, when diagnosing or treating a lesion, the lesion area occupies a large proportion on the white light image, and has a feature that it continues to exist on the white light image for a certain period of time. In addition, when the lesion area is composed of minute lesions, a large number of lesion areas are detected. Therefore, in the present embodiment, display / non-display of the alert image is controlled according to the size and number of the attention areas and the detection time.

  FIG. 15 shows a detailed configuration example of the display mode control unit 360 that controls the display mode of the alert image in accordance with the information on the attention area. The display mode control unit 360 includes a display availability determination unit 361 and an alert image superimposing unit 362.

  The display permission / inhibition determination unit 361 determines whether or not to display the alert image output from the alert image display setting unit 350, and displays the alert image determined to be displayed by the determination processing as the alert image superimposing unit 362. Output to. Note that the determination process is performed by a method described later using information on the region of interest output from the region of interest detection unit 340.

  The alert image superimposing unit 362 superimposes the alert image output from the display availability determination unit 361 on the white light image, and outputs the white light image to the display unit 400.

  FIG. 16 shows a detailed configuration example of the display availability determination unit 361. The display availability determination unit 361 includes a region number determination unit 3611, a region size determination unit 3612, a detection count calculation unit 3613, a memory 3614, an elapsed time determination unit 3615, and an alert image selection unit 3616.

  The region number determination unit 3611 calculates the number of attention region groups from the attention region information output from the attention region detection unit 340. Specifically, the number-of-regions determination unit 3611 compares the number of regions of interest described above with the threshold value Area_Num, and outputs the comparison result to the region size determination unit 3612 as flag information. More specifically, the area number determination unit 3611 outputs a flag “0” when the number of attention area groups is larger than the threshold value Area_Num, and the flag “0” when the number of attention area groups is equal to or less than the threshold value Area_Num. 1 "is output. For example, a predetermined value is set in the area number determination unit 3611 by the control unit 370 as the threshold value Area_Num. In addition, the number of attention area groups (number information in a broad sense) is calculated by the number-of-areas determination unit 3611 referring to all tag information assigned to each attention area and how many types of tag information exist. To decide.

  The area size determination unit 3612 determines whether or not to display an alert image corresponding to each attention area group according to the size information of the attention area group and the flag information output from the number-of-areas determination part 3611. This method is used. Then, the area size determination unit 3612 outputs the result of the determination process to the alert image selection unit 3616. Information output as a result of this determination processing is, for example, tag information given to the attention area group determined to display the alert image. Here, the size information of the attention area group is information indicating the size of the area occupied by the attention area group, for example, the number of local areas (attention areas) constituting each attention area group, or It may be a ratio between the number of local regions constituting and the number of local regions constituting the entire image.

  Next, the determination process performed by the region size determination unit 3612 will be described in detail. First, the area size determination unit 3612 calculates the size of the attention area group. For example, the size of the attention area group is the number of local areas constituting each attention area group. Next, the region size determination unit 3612 refers to the flag information output from the region number determination unit 3611.

  In the case of the flag “1” where the number of attention area groups is smaller than the threshold value, when an alert image corresponding to a large attention area group is displayed, there is a concern that the alert image may hinder a doctor's diagnosis or treatment. Therefore, the region size determination unit 3612 determines that the alert image of the attention region group that exceeds the threshold Size_Max is not displayed. Then, the region size determination unit 3612 outputs the tag information given to the attention region group whose size is equal to or smaller than the threshold Size_Max to the alert image selection unit 3616.

  On the other hand, in the case of the flag “0” in which the number of attention area groups is greater than the threshold, the number of attention area groups is large. Therefore, if alert images corresponding to all attention area groups are displayed, the alert image is displayed by the doctor. There is concern that it may interfere with diagnosis and treatment. However, if all the alert images are hidden, there is a risk of overlooking a minute lesion. Therefore, the region size determination unit 3612 determines that only the alert images of the attention region group having a smaller size among the attention region groups are displayed. Specifically, the region size determination unit 3612 determines that an alert image should be displayed only for the attention region group having a size equal to or smaller than the threshold Size_Min that is smaller than the threshold Size_Max, and the tag attached to the attention region group Only the information is output to the alert image selection unit 3616.

  With reference to FIGS. 17A to 18B, the process of the region size determination unit 3612 will be described with an example. In the following, it is assumed that the threshold values are Area_Num = 5, Size_Min = 1, and Size_Max = 3.

As shown in FIG. 17A, it is assumed that three attention area groups A1 to A3 are detected. Here, it is assumed that tag information “1”, “2”, and “3” are assigned to the attention area groups A1, A2, and A3, respectively. First, the area size determination unit 3612 calculates the number of attention areas constituting each attention area group as the size of each attention area group. The size of each attention area group is as follows.
Size of attention area group A1: 12
Size of attention area group A2: 3
Size of the attention area group A3: 1

  Next, since the number of attention area groups is 3 and the threshold value Area_Num = 5 or less, the area number determination unit 3611 outputs a flag “1”. Therefore, as illustrated in FIG. 17B, the region size determination unit 3612 determines that only the attention region group having the threshold Size_Max = 3 or less is the region where the alert is to be displayed. Therefore, in this case, the attention area groups A2 and A3 are determined to be areas where alerts are to be displayed, and tag information “2” and “3” are output to the alert image selection unit 3616.

Next, as shown in FIG. 18A, it is assumed that seven attention region groups B1 to B7 are detected. Here, it is assumed that tag information “1” to “7” is assigned to each of the attention area groups B1 to B7. The size of each attention area group calculated by the area size determination unit 3612 is as follows.
Size of attention area group B1, B2, B3, B5, B6: 3
Size of attention area group B4: 2
Size of attention area group B7: 1

  Next, since the number of attention area groups is 7, which is larger than the threshold value Area_Num = 5, the flag “0” is output from the area number determination unit 3611. Therefore, as illustrated in FIG. 18B, the region size determination unit 3612 determines that an alert image of only the attention region group having the threshold Size_Min = 1 or less is to be displayed. Therefore, in this case, it is determined that only the attention area group B7 is an area where an alert image should be displayed, and tag information “7” is output to the alert image selection unit 3616.

  According to the above determination processing, it is possible to prevent the alert image from interfering with diagnosis / treatment and oversight of minute lesions. Specifically, when a large lesion area is detected on a narrow-band optical image, it is determined that the alert image corresponding to the lesion area is not displayed, and the alert image hinders diagnosis / treatment. Can be suppressed. On the other hand, it is possible to display an alert image for a region that is easily overlooked by determining that an alert image is displayed in the attention region group having a size equal to or smaller than the threshold value.

  Further, according to the above determination processing, alert control according to the shooting situation can be performed by changing the size threshold according to the number of attention area groups. For example, as shown in FIG. 19, when a lesion is observed in a relatively distant view, it is assumed that a small number of large attention area groups are detected as shown in C1. In this case, since the size of the attention area group is large, the determination process is performed using the larger threshold Size_Max as described above with reference to FIG. On the other hand, as shown in FIG. 20, when observing a lesion part in a relatively close-up view, for example, as shown in D1 to D7, it is assumed that the structure of the lesion part is enlarged and a large number of small attention area groups are detected. The In this case, since the size of the attention area group is small, the alert image may not be hidden if the same threshold value is used. For this reason, as described above with reference to FIG. 18A and the like, the alert image in the region composed of minute lesions is hidden by performing determination processing using a smaller threshold Size_Min.

  Next, the detection frequency calculation unit 3613 illustrated in FIG. 16 will be described. The detection number calculation unit 3613 calculates the number of detections for all the attention regions output from the attention region detection unit 340. Here, the number of detections is information indicating the elapsed time when the attention area is continuously detected, and is, for example, the number of times (number of frames) in which each attention area is continuously detected as will be described later. In the following, a white light image and a narrowband light image are acquired at the same time, each image is acquired in time series, and the number of detections is calculated for the attention area detected from the narrowband light image acquired at a certain time t. An example of this will be described. Here, the narrowband optical image acquired at time t is Ft, and the narrowband optical image acquired N frames before Ft is Ft−N.

  First, the detection frequency calculation unit 3613 refers to the memory 3614 and acquires information (data) of the detection frequency stored in the memory 3614. As will be described later, the memory 3614 stores information on the coordinates of all the attention areas detected by Ft-1 and the number of times of detection of the attention area. Hereinafter, the attention area detected at Ft-1 is referred to as an old attention area.

Next, the number-of-detections calculation unit 3613 calculates the number of detections of all the attention areas detected by Ft output from the attention area detection section 340. Specifically, the detection count calculation unit 3613 compares the coordinates of the attention area with the coordinates of all the old attention areas acquired from the memory 3614, and the coordinates of the old attention area that match the coordinates of the attention area are the same. If it exists, the number of times of attention area detection is calculated by the following equation (11). Here, (m, n) is the coordinate of the attention area, Pt (m, n) is the number of times of detection of the old attention area (m, n) acquired from the memory 3614, and Pt ′ (m, n ) Is the number of times the attention area (m, n) is detected.
Pt ′ (m, n) = Pt (m, n) +1 (11)

  On the other hand, when there is no coordinate of the old region of interest that matches the coordinate of the region of interest, the detection number calculation unit 3613 sets the number of detections of the region of interest to Pt ′ (m, n) = 1. The number-of-detections calculation unit 3613 outputs the result to the memory 3614 and the elapsed time determination unit 3615 after the above processing is completed for all the attention areas. Specifically, the information output here is the coordinates and the number of detections of all the attention areas detected by Ft. In addition, when the information is output to the memory 3614, the information previously stored in the memory 3614 is deleted.

  Note that when acquiring images in time series, the position of the region of interest on the image changes between frames due to the subject or the minute movement of the imaging unit 200. However, in the present embodiment, since the size of the local area set by the local area setting unit 341 is 20 × 20, the influence of the subject and the minute movement of the imaging unit 200 can be ignored.

  The elapsed time determination unit 3615 calculates the elapsed time of each attention area group by a method described later based on the above-described number of detections of the attention area and the tag information. The elapsed time determination unit 3615 determines that an alert image corresponding to the attention area group should not be displayed when there is an attention area group whose elapsed time exceeds the threshold value Area_Time, and displays the determination result as an alert image. The data is output to the selection unit 3616. The information output by the elapsed time determination unit 3615 is tag information given to all the attention area groups determined to be equal to or less than the threshold value Area_Time by the determination process described above.

  A method for calculating the detection elapsed time of the attention area group will be described with reference to FIG. Here, the elapsed time (detected elapsed time information) of the attention area group is information indicating the length of time during which the attention area group is detected from the special light image, or the attention area group is a white light image. This is information indicating the length of time that the data continues to exist.

  For example, as shown in FIG. 21, the elapsed time of the attention area group is calculated from the number of times of detection of each attention area belonging to the attention area group. Specifically, as indicated by G1, when the attention area is detected, the detection count Pt of the local area detected as the attention area is sequentially incremented in each frame. And as shown to G2, the average value of the detection frequency Pt of all the local areas which belong to an attention area group is computed as elapsed time of an attention area group. For example, the threshold value for determining the elapsed time is set to 3. Then, as indicated by G3, the alert image is displayed when the average value of Pt is less than 3, and the alert image is not displayed when the average value of Pt is 3 or more. As the elapsed time of the attention area group, the average value of Pt may be used as described above, or a value obtained by another calculation method may be used.

  Here, the above-described threshold values Area_Num, Size_Min, Size_Max, and Area_Time can be set by the user from the external I / F unit 500, or may be set in advance.

  The alert image selection unit 3616 selects only the alert images corresponding to the tag information output in common from both the region size determination unit 3612 and the elapsed time determination unit 3615 from among the alert images output from the alert image display setting unit 350. The data is output to the alert image superimposing unit 362. Moreover, the alert image selection part 3616 outputs the trigger signal showing that there is no alert image which should be displayed to the alert image superimposition part 362, when the tag information output in common does not exist.

  The alert image superimposing unit 362 superimposes the alert image output from the display enable / disable determining unit 361 on the white light image output from the white light image acquiring unit 320 and outputs the superimposed image to the display unit 400. For example, as the superimposition process, as shown in the following formula (12), a process of replacing the attention area in the white light image with the alert image is performed. On the other hand, when a trigger signal is output from the display availability determination unit 361, the alert image superimposing unit 362 outputs the white light image to the display unit 400 as it is.

R_org (x, y) = R_out (x, y),
G_org (x, y) = G_out (x, y),
B_org (x, y) = B_out (x, y) (12)

  Here, in the above equation (12), R_org (x, y), G_org (x, y), and B_org (x, y) are RGB signal values at the coordinates (x, y) of the white light image. R_out (x, y), G_out (x, y), and B_out (x, y) are RGB signal values at the coordinates (x, y) of the alert image.

  By performing the above processing, when a diagnosis using a white light image and a special light image is performed, a lesion area existing in the white light image can be depicted as a color area different from a normal color area. As a result, it is possible to prevent oversight of the lesion without increasing the load on the doctor. Further, by controlling the display mode of the alert image according to the detected information on the attention area, it is possible to improve the problem that the alert display hinders the diagnosis and treatment of the doctor.

  In the above-described embodiment, the case has been described in which only the alert image corresponding to the attention area group smaller than the threshold Size_Min is displayed when the number of attention area groups is larger than the threshold value Area_Num. However, the present embodiment is not limited thereto. Not. For example, in this embodiment, when the number of attention area groups is larger than the threshold value Area_Num, alert images corresponding to all attention area groups may be hidden. In this case, when the flag “0” is output from the number-of-regions determination unit 3611, the region size determination unit 3612 determines to hide the alert images corresponding to all the attention region groups.

  Moreover, although the said embodiment demonstrated the case where alert display was controlled according to the size (the number of local areas, the number of pixels, etc.) of an attention area group, this embodiment is not limited to this. For example, in the present embodiment, the alert display may be controlled according to the ratio of the size (area) of the attention area group to the narrow-band light image instead of the size of the attention area group. In this case, for example, the ratio of the number of attention areas constituting the attention area group and the number of all local areas set by the local area setting unit 341 is the ratio of the size of the attention area group to the narrowband light image. And And when the ratio is larger than a threshold value, an alert display is controlled so that the alert image corresponding to an attention area group may not be displayed.

  Moreover, although the said embodiment demonstrated the case where alert display was controlled by the determination process using the number, size, and elapsed time of an attention area, this embodiment is not limited to this. For example, in the present embodiment, the alert image may be forcibly displayed regardless of the determination process described above. In this case, the user performs setting to forcibly display the alert image via the external I / F unit 500. This setting information is transmitted as a trigger signal to the control unit 370, and the control unit 370 outputs the trigger signal to the alert image selection unit 3616. Then, when a trigger signal is output from the control unit 370, the alert image selection unit 3616 outputs all the alert images output from the alert image display setting unit 350 to the alert image superimposing unit 362.

3.6. Software In the above-described embodiment, each unit configuring the image processing apparatus 301 is configured by hardware. However, the present embodiment is not limited to this, and may be realized as software by a CPU executing a program. . In this case, the control device 300 and the image processing device 301 are realized by a computer system described later with reference to FIG.

  As an example of a case where a part of the processing performed by each unit is configured by software, a processing procedure when the processing performed by the image processing apparatus 301 for an image acquired in advance is realized by software is described with reference to a flowchart illustrated in FIG. I will explain. For example, an image acquired in advance is an image obtained by recording an output image of the Bayer array output from the A / D conversion units 310 and 311 as a RAW file on a recording medium.

  As shown in FIG. 22, when this process is started, parameters used in the above-described enhancement process and determination process are read from the memory (step S0). Next, a narrow-band light image is read from the memory (step S1), and a white light image acquired simultaneously with the narrow-band light image is read from the memory (step S2). Next, a region of interest is detected from the read narrowband light image based on the hues shown in the above formulas (6) to (8) (step S3).

  Next, the enhancement processing shown in the above equation (9) is performed on all the pixels in the white light image region corresponding to the attention region detected in step S3, and an alert image is created (step S4). Next, the alert image created in step S4 is superimposed on the white light image by the method described above according to the information on the attention area detected in step S3. Then, the white light image on which the alert image is superimposed is output as a display image (step S5). Next, when a series of processing is completed for all the images, the processing ends (step S6, YES), and when an unprocessed image remains, the same processing is continued (step S6, NO). .

  A detailed processing procedure in the display mode changing process (step S5) will be described with reference to the flowchart shown in FIG.

  As shown in FIG. 23, when this process is started, the number of attention area groups detected in step S3 is calculated, the number of attention area groups is compared with a threshold Area_Num, and the comparison result is used as flag information. Output (step S51). Specifically, as flag information, a flag “0” is output when the number of attention area groups is larger than the threshold value Area_Num, and a flag “1” is output when the number is less than the threshold value.

  Next, the size of each attention area group detected in step S3 is calculated. Then, the flag information output in step S51 is referred to. When the flag is “0”, the result of comparing the size of the attention area group with the threshold Size_Min is output. On the other hand, if the flag is “1”, the result of comparing the size of the region of interest with the threshold Size_Max is output (step S52). Specifically, tag information is output as a comparison result. This tag information is all the tag information assigned to the attention area group whose size is determined to be equal to or smaller than the threshold value.

  Next, the detected elapsed time of each attention area group detected in step S3 is calculated by the method shown in the above equation (11), the detected elapsed time is compared with the threshold value Area_Time, and the comparison result is output (step S53). ). Specifically, tag information is output as a comparison result. This tag information is all tag information assigned to the attention area group determined that the detected elapsed time is equal to or less than the threshold value Area_Time.

  Next, an alert image is superimposed according to the determination results in step S52 and step S53 (step S54). Specifically, common tag information is extracted from the tag information output in steps S52 and S53, and only the alert image corresponding to the attention area group to which the tag information is assigned is superimposed on the white light image. The white light image is output as a display image. If there is no common tag information, the white light image is output as it is as a display image.

  With the above processing, the processing performed by the image processing apparatus 301 can be realized as software. For example, as in a capsule endoscope, image data is first stored, and then the stored image data can be processed in software by a computer system such as a PC.

  As described above, there is a problem that an alert image hinders diagnosis and treatment when an endoscope is brought close to a lesioned portion and diagnosis and treatment are performed. However, if all the alert images are hidden so as not to hinder diagnosis and treatment, there is a problem that a minute lesion may be missed.

  In this regard, according to the present embodiment, as illustrated in FIG. 3, the image processing device 301 includes an image acquisition unit (white light image acquisition unit 320, special light image acquisition unit 330), an attention area detection unit 340, A determination unit (display availability determination unit 361 shown in FIG. 15) and a display mode control unit 360 are included. Then, the image acquisition unit includes a normal light image (first image) including a subject image having information in the white wavelength band, and a subject image having information in a specific wavelength band corresponding to the normal light image. An acquired image that is at least one of the special light image (second image) including the image is acquired. The attention area detection unit 340 detects an attention area, which is an area to be noted, based on the feature amount of the pixel of the acquired image. The determination unit determines whether to display an alert image corresponding to the attention area (attention area group) according to the detection result of the attention area. The display mode control unit 360 performs control to display the alert image corresponding to the display target attention region (display target attention region group) that is the attention region (target region group) for which the determination unit determines that the alert image should be displayed. Do.

  As a result, it is possible to prevent the alert image from interfering with diagnosis and treatment by a doctor. Specifically, it is possible to display only the alert image corresponding to the display target attention area by determining whether or not the alert image corresponding to the attention area should be displayed. As a result, the alert image that hinders diagnosis and treatment can be hidden. Moreover, since the alert image is displayed in the display target attention area, it is possible to suppress oversight of the lesioned part.

  Note that the image processing apparatus 301 may include a white light image acquisition unit 320 and a special light image acquisition unit 330 as image acquisition units, and the image acquisition unit may acquire a white light image and a special light image as acquisition images. Alternatively, as will be described later with reference to FIG. 24 and the like, the image processing apparatus 301 may include only the white light image acquisition unit 320 as the image acquisition unit, and the image acquisition unit may acquire only the white light image as the acquisition image. Further, the image processing apparatus 301 may include a display availability determination unit 361 as a determination unit, and may include an attention area control unit 380 as a determination unit, as will be described later with reference to FIG.

  Here, the attention area is an area where the priority of observation for the user is higher than other areas. For example, if the user is a doctor and desires treatment, the mucous membrane area and the lesion area are copied. Refers to the area. As another example, if the object that the doctor desires to observe is a bubble or stool, the region of interest is a region in which the bubble or stool portion is copied. In other words, the object that the user should pay attention to depends on the purpose of observation, but in any case, in the observation, the region where the priority of observation for the user is relatively higher than other regions becomes the region of interest. . The attention area can be detected using the feature amount (hue, saturation, etc.) of the pixels of the special light image or the white light image. For example, the attention area can be detected by setting a threshold value for the feature amount, and the threshold value changes according to the type of the attention area. For example, the threshold value of the feature amount such as hue and saturation for the first type of attention region is different from the threshold value of the feature amount for the second type of attention region. Then, when the type of the attention area changes, it is only necessary to change the threshold value of the feature amount, and the processing after the detection of the attention area (determination processing, etc.) is the same as the processing described in the present embodiment. Can be realized.

  In addition, as shown in FIG. 3, the present embodiment includes an alert image generation unit (alert image display setting unit 350) that generates an alert image corresponding to the attention area detected by the attention area detection unit 340. And the display mode control part 360 performs control which displays the alert image corresponding to a display object attention area from the alert images produced | generated by the alert image production | generation part.

  In the present embodiment, the determination unit (display availability determination unit 361) should display the alert image generated by the alert image generation unit (alert image display setting unit 350) according to the detection result of the attention area. Judgment is made. And the display mode control part 360 performs control which hides the alert image determined as non-display by the determination part.

  If it does in this way, as shown to H3 of Drawing 7, the alert picture corresponding to all the attention fields detected will be generated, and the display / non-display of the alert picture will be judged as shown in H4, and it will be shown in H5 Thus, only the alert image determined to be displayed can be displayed. Thereby, display / non-display of the alert image can be controlled.

  In the present embodiment, as illustrated in FIG. 16 (or FIG. 29), the determination unit (display availability determination unit 361, attention area control unit 380) performs attention in the acquired image based on the detection result of the attention area. At least one or more pieces of information on the number of areas, the size information of the attention area, and the detection elapsed time information of the attention area are acquired. Then, the determination unit determines whether or not to display an alert image according to the acquired information.

  In this way, it is possible to determine whether or not to display an alert image corresponding to the attention area according to the detection result of the attention area, and to control display / non-display of the alert image.

  Further, in the present embodiment, the determination unit, when the number of attention areas represented by the number information (for example, the maximum value of the tag) is larger than a predetermined threshold Area_Num, a part of attention of the detected attention area It is determined that the alert image corresponding to the region is not displayed.

  Further, in the present embodiment, the determination unit, when the number of attention areas represented by the number information (for example, the maximum value of the tag) is larger than a predetermined threshold Area_Num, all the attention areas of the detected attention areas. It is determined that the alert image corresponding to is not displayed.

  In this way, when there are a large number of detected attention areas, the alert images corresponding to some or all of the attention areas are hidden, so that the alert images that hinder diagnosis and treatment are hidden. it can.

  In the present embodiment, the determination unit determines that an alert image corresponding to a region of interest whose size represented by the size information is larger than a predetermined threshold is not displayed.

  In this way, when the size of the detected attention area is large, the alert image corresponding to the attention area is hidden, so that the alert image that hinders diagnosis and treatment can be hidden.

  More specifically, as illustrated in FIG. 17A and the like, the determination unit determines that the size of the attention area is larger than the first threshold Size_Max when the number of attention areas is smaller than the predetermined threshold Area_Num. It is determined that the corresponding alert image is not displayed. On the other hand, as shown in FIG. 18B and the like, when the number of attention areas is larger than a predetermined threshold, the determination unit displays an alert image corresponding to the attention area whose size is larger than the second threshold Size_Min. Judgment to hide. The second threshold Size_Min is a threshold smaller than the first threshold Size_Max.

  In this way, the threshold value for determining the size of the attention area can be changed according to the number of the attention areas detected. Thereby, as described above with reference to FIGS. 19 and 20, the display / non-display of the alert image can be controlled in accordance with the structure of the lesioned part and the imaging state.

  In the present embodiment, as illustrated in FIG. 21, the determination unit determines to hide the alert image corresponding to the attention area whose detection elapsed time represented by the detection elapsed time information exceeds a predetermined threshold. . Specifically, the determination unit counts the number of frames in which the attention area is detected as the detection elapsed time information. More specifically, as illustrated in FIG. 13, the attention area detection unit 340 divides the acquired image into a plurality of local areas, and detects whether each local area corresponds to the attention area. Then, as indicated by G1 in FIG. 21, the determination unit counts the number of frames Pt (m, n) detected as corresponding to the region of interest in each local region.

  In this way, it is possible to control the display / non-display of the alert image by counting the number of frames in which the attention area is detected and using the count value as the detection elapsed time information. Thereby, the alert image can be displayed for a predetermined period to suppress oversight. Further, since the same area is photographed during diagnosis and treatment, it is not necessary to continue displaying the alert image, and hiding the alert image after a predetermined period can be prevented from hindering diagnosis and treatment.

  In the present embodiment, as shown by G1 in FIG. 21, the determination unit detects the previous frame (Ft−2) among the local regions detected as corresponding to the attention region in the current frame (for example, Ft−1). ) Is incremented for the count value of the local area (the area in which Pt (m, n) of the previous frame is stored in the memory 3614) detected as the attention area. In addition, the determination unit determines that the local region detected as not corresponding to the attention region in the previous frame (Ft-2) among the local regions detected as corresponding to the attention region in the current frame (Ft-1). The count value of the area where Pt (m, n) of the previous frame is not stored in the memory 3614 is set to an initial value (for example, “1”).

  In this way, it is possible to count the number of frames Pt (m, n) detected as corresponding to the region of interest continuously in each local region. Thereby, the detection elapsed time information in each local region can be acquired.

  In the present embodiment, as illustrated in FIG. 13, the attention area detection unit 340 sets adjacent local areas among the local areas detected as corresponding to the attention area as the attention area group. And as shown to G2 of FIG. 21, the determination part is based on the number of frames corresponding to the attention area in each local area in the attention area group, and the detection elapsed time information of the attention area group (for example, the average value of the count values). Is calculated to determine whether or not the alert image corresponding to the attention area group can be displayed.

  In this way, it is possible to determine whether or not to display the alert image corresponding to the attention area group based on the detection elapsed time information of the attention area group. As a result, as shown by G3 in FIG. 21, the display / non-display of the alert image can be controlled collectively by the attention area group, and the display / non-display of the alert image can be prevented from being performed separately for each local area.

  In the present embodiment, as shown in FIG. 3, an instruction to display / hide the alert image is input via the interface unit (external I / F unit 500). Then, when a display instruction is input to the interface unit, the display mode control unit 360 performs control to display an alert image corresponding to the attention area regardless of the determination result by the determination unit.

  In this way, when the doctor (user) wants to display the alert image in all the attention areas, the alert image can be forcibly displayed by giving an instruction via the interface unit.

  In the present embodiment, as shown in FIG. 15, the display mode control unit 360 includes a superimposition unit (alert image superimposition unit 362) that superimposes an alert image corresponding to the display target attention area on the output image. And the display mode control part 360 performs control which displays the output image on which the alert image corresponding to a display object attention area was superimposed. More specifically, the superimposing unit superimposes the alert image corresponding to the display target attention area on the normal light image (white light image) as the output image.

  In this way, it is possible to perform control to display the normal light image in which the alert image is superimposed on the display target attention area. Thereby, control which displays an alert picture can be performed.

  In the present embodiment, as shown in FIG. 14 (or FIG. 30), an area selection unit (corresponding area selection unit 351) is included. Then, the attention area detection unit 340 detects the attention area from the special light image. The area selection unit selects a corresponding attention area that is an area corresponding to the attention area from the normal light image. The alert image generation unit (alert image display setting unit 350 and enhancement processing unit 352) performs enhancement processing (for example, processing for assigning the target color described above) on pixels in the corresponding attention area included in the normal light image. To generate an alert image.

  Further, in the present embodiment, the alert image generation unit is a pixel located on the boundary line between the corresponding attention area in the normal light image and the non-corresponding attention area that is an area other than the corresponding attention area (region near the boundary line). The alert image may be generated by performing an emphasis process.

  In this way, an alert image corresponding to the attention area can be generated by performing enhancement processing on the pixels in the corresponding attention area or the pixels located on the boundary line. Here, the corresponding attention area is an attention area in the white light image, and corresponds to the attention area detected in the special light image on a one-to-one basis. For example, when the white light image and the special light image are images having the same number of pixels, the coordinates of the pixels in the corresponding attention area are the same as the coordinates of the pixels in the attention area in the special light image.

  In the present embodiment, as illustrated in FIG. 11, the attention area detection unit 340 detects the attention area based on a specific color feature amount (for example, hue) of a pixel in the special light image.

  In this embodiment, as will be described later with reference to FIG. 25 and the like, the attention area detection unit 340 is based on a specific color feature amount (for example, a ratio between the R pixel value and the G pixel value) of the pixel in the normal light image. Thus, the attention area may be detected.

  In this way, the attention area can be detected based on the feature amount of the pixel of the acquired image that is at least one of the normal light image and the special light image.

  In the present embodiment, the specific wavelength band is a narrower band (NBI: Narrow Band Imaging) than the white wavelength band (for example, 380 nm to 650 nm). For example, the normal light image and the special light image are in-vivo images of the living body, and a specific wavelength band included in the in-vivo image is a wavelength band of a wavelength that is absorbed by hemoglobin in blood. For example, the wavelength absorbed by this hemoglobin is 390 nm to 445 nm (first narrowband light, B2 component of narrowband light), or 530 nm to 550 nm (second narrowband light, G2 component of narrowband light). is there.

  Thereby, it becomes possible to observe the structure of the blood vessel located in the surface layer part and the deep part of the living body. In addition, by inputting the obtained signals to specific channels (G2 → R, B2 → G, B), lesions that are difficult to see with normal light such as squamous cell carcinoma can be displayed in brown, etc. Oversight of parts can be suppressed. Note that 390 nm to 445 nm or 530 nm to 550 nm are numbers obtained from the characteristic of being absorbed by hemoglobin and the characteristic of reaching the surface layer or the deep part of the living body, respectively. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% due to the variation factors such as the absorption by hemoglobin and the experimental results regarding the arrival of the living body on the surface layer or the deep part. It is also conceivable that the upper limit value increases by about 0 to 10%.

  In the present embodiment, the normal light image and the special light image are in-vivo images obtained by copying the inside of the living body, and the specific wavelength band included in the in-vivo image is a wavelength band of fluorescence emitted from the fluorescent substance. Also good. For example, the specific wavelength band may be a wavelength band of 490 nm to 625 nm.

  This enables fluorescence observation called AFI (Auto Fluorescence Imaging). In this fluorescence observation, autofluorescence (intrinsic fluorescence: 490 nm to 625 nm) from a fluorescent substance such as collagen can be observed by irradiating excitation light (390 nm to 470 nm). In such observation, the lesion can be highlighted with a color tone different from that of the normal mucous membrane, and the oversight of the lesion can be suppressed. Note that 490 nm to 625 nm indicates a wavelength band of autofluorescence emitted by a fluorescent substance such as collagen when irradiated with the excitation light described above. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% and the upper limit is 0 due to a variation factor such as an experimental result regarding the wavelength band of the fluorescence emitted by the fluorescent material. A rise of about 10% is also conceivable. Alternatively, a pseudo color image may be generated by simultaneously irradiating a wavelength band (540 nm to 560 nm) absorbed by hemoglobin.

  In the present embodiment, the specific wavelength band included in the in-vivo image may be a wavelength band of infrared light. For example, the specific wavelength band may be a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm.

  Thereby, infrared light observation called IRI (Infra Red Imaging) becomes possible. In this infrared light observation, by injecting intravenously ICG (Indocyanine Green), which is an infrared index drug that easily absorbs infrared light, and irradiating infrared light in the above-mentioned wavelength band, It is possible to highlight blood vessels and blood flow information in the deep mucosa, which is difficult to visually recognize, and it is possible to diagnose the depth of gastric cancer and determine the treatment policy. The numbers 790 nm to 820 nm are obtained from the characteristic that the absorption of the infrared index drug is strongest, and the numbers 905 nm to 970 nm are calculated from the characteristic that the absorption of the infrared index drug is the weakest. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% and the upper limit is 0 to 10 due to a variation factor such as an experimental result regarding absorption of the infrared index drug. It can also be expected to increase by about%.

  Moreover, in this embodiment, you may include the special light image acquisition part which produces | generates a special light image based on the acquired normal light image. For example, the special light image acquisition unit 330 illustrated in FIG. 3 may acquire the normal light image from the white light image acquisition unit 320 and generate the special light image from the acquired normal light image.

  Specifically, the special light image acquisition unit includes a signal extraction unit that extracts a signal in the white wavelength band from the acquired normal light image. Then, the special light image acquisition unit may generate a special light image including a signal in a specific wavelength band based on the extracted signal in the wavelength band of white light. For example, the signal extraction unit estimates the spectral reflectance characteristics of the subject in 10 nm increments from the RGB signal of the normal light image, and the special light image acquisition unit integrates the estimated signal component in the specific band and performs special processing. Generate a light image.

  More specifically, the special light image acquisition unit may include a matrix data setting unit that sets matrix data for calculating a signal in a specific wavelength band from a signal in the wavelength band of white light. Then, the special light image acquisition unit may generate a special light image by calculating a signal in a specific wavelength band from a signal in the white wavelength band using the set matrix data. For example, the matrix data setting unit sets table data in which spectral characteristics of irradiation light in a specific wavelength band are described in increments of 10 nm as matrix data. Then, the spectral characteristics (coefficients) described in the table data are multiplied by the spectral reflectance characteristics of the subject estimated in increments of 10 nm and integrated to generate a special light image.

  As a result, the special light image can be generated based on the normal light image, so that the system can be realized with only one light source that irradiates the normal light and one image sensor that images the normal light. . Therefore, the capsule endoscope and the insertion part of the scope endoscope can be made small, and the cost can be expected to be reduced because the number of parts is reduced.

  In the present embodiment, as shown in FIG. 3, the endoscope apparatus includes a light source (light source unit 100) and a white light filter that transmits light in a white wavelength band (color filter of the first image sensor 250). And a special optical filter that transmits light in a specific wavelength band (color filter of the second image sensor 260). Then, the image acquisition unit passes the light obtained by irradiating the light from the light source to the subject through the white light filter to acquire the normal light image, and passes through the special light filter to acquire the special light image. .

  In the present embodiment, a white light source that emits light in a white wavelength band and a special light source that emits light in a specific wavelength band may be included. The image acquisition unit acquires a normal light image obtained by irradiating the subject with light from the white light source, and acquires a special light image obtained by irradiating the subject with light from the special light source. May be. For example, the special light source includes a white light source and a special light filter, and generates special light by transmitting white light through the special light filter.

  In this way, it is possible to acquire a normal light image including a subject image having information in the white wavelength band and a special light image including a subject image having information in a specific wavelength band.

4). Second Configuration Example In the first configuration example, the case where the attention area is detected from the special light image has been described. However, in the present embodiment, the attention area may be detected from the white light image. FIG. 24 shows a second configuration example of the endoscope apparatus as a configuration example when detecting the attention area from the white light image.

  The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. In addition, the same code | symbol is attached | subjected to the component same as the component already demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably. For example, the operations of the components other than the imaging unit 200 and the control device 300 are the same as those in the first configuration example shown in FIG.

  The imaging unit 200 is formed to be elongated and bendable, for example, to enable insertion into a body cavity. The imaging unit 200 includes a light guide fiber 210 for guiding the light collected by the light source unit 100, an illumination lens 220 that irradiates the subject by diffusing the light guided to the tip by the light guide fiber 210, The objective lens 230 which condenses the reflected light which returns from a to-be-photographed object, and the 1st image sensor 250 for detecting the condensed reflected light are included. The first image sensor 250 is an image sensor having a Bayer array color filter for capturing a white light image. For example, the color filters r, g, and b of the first image sensor 250 have the spectral characteristics shown in FIG.

  The control device 300 includes an A / D conversion unit 310, an image processing device 301, and a control unit 370. The image processing apparatus 301 includes a white light image acquisition unit 320, an attention area detection unit 340, an alert image display setting unit 350, and a display mode control unit 360. The processing performed by the components other than the attention area detection unit 340 is the same as that in the first configuration example described above.

  FIG. 25 shows a detailed configuration example of the attention area detection unit 340. The attention area detection unit 340 detects the attention area from the white light image output from the white light image acquisition unit 320. Specifically, the attention area detection unit 340 includes a local area setting unit 341, a feature amount calculation unit 345, an area detection unit 346, and a tag information addition unit 344. Note that the processing performed by the local region setting unit 341 and the tag information adding unit 344 is the same as that in the first configuration example described above, and thus the description thereof is omitted.

The feature amount calculation unit 345 calculates average values R_ave and G_ave of R and G signal values in each local region for all local regions set by the local region setting unit 341. Then, the feature amount calculation unit 345 calculates the feature amount F using the following expression (13) for all local regions, and outputs the feature amount F to the region detection unit 346.

  The region detection unit 346 refers to the feature amount F of each local region calculated by the feature amount calculation unit 345, and detects a local region where the feature amount F is greater than the threshold value as the attention region. For example, the threshold value is a value greater than 0 and less than 1. The threshold value can be set by the user via the external I / F unit 500, or a predetermined value may be set in advance.

  According to the above-described embodiment, when a diagnosis using a white light image is performed, the lesion area existing in the white light image is rendered as a color area different from the normal, and thus it is possible to prevent oversight of the lesion. Become. Specifically, in a white light image, there is a feature in which a lesion such as a tumor is depicted as a region with strong redness. Therefore, a region where the feature amount F shown in the above equation (13) is larger than a threshold is detected as a region of interest This makes it possible to detect a lesion area such as a tumor.

  Moreover, according to the said embodiment, the problem which alert display obstructs a doctor's diagnosis and treatment can be improved by controlling the display mode of an alert image according to the information of the detected attention area.

  Further, according to the embodiment, since processing is performed using only the white light image, the imaging unit 200 and the control device 300 are compared with the case where the white light image and the special light image are used as in the first configuration example. Can be simplified.

  Here, in the above embodiment, each unit constituting the image processing apparatus 301 is configured by hardware. However, the present embodiment is not limited to this, and is realized as software by the CPU executing a program. Also good. In this case, the control device 300 and the image processing device 301 are realized by a computer system described later with reference to FIG.

  As an example of a case where a part of processing performed by each unit is configured by software, a processing procedure when the processing performed by the image processing apparatus 301 for an image acquired in advance is realized by software is illustrated in a flowchart illustrated in FIG. I will explain. For example, an image acquired in advance is an image obtained by recording an output image of the Bayer array output from the A / D conversion unit 310 on a recording medium as a RAW file.

  As shown in FIG. 26, when this process is started, parameters used in the above-described enhancement process and determination process are read from the memory (step S10). Next, a white light image is read from the memory (step S11). Next, a region of interest is detected from the read white light image based on the feature amount shown in the above equation (13) (step S12).

  Next, the enhancement process shown in the above equation (9) is performed on all the pixels in the white light image area corresponding to the attention area detected in step S12 to create an alert image (step S13). Next, the alert image created in step S13 is superimposed on the white light image according to the attention area information detected in step S12, and the white light image on which the alert image is superimposed is output as a display image (step S14). ). The alert image superimposing process is the same as the process described above with reference to FIG. Next, when a series of processing is completed for all the images, the processing ends (step S15, YES), and when an unprocessed image remains, the same processing is continued (step S15, NO).

5). Third Configuration Example In the above embodiment, a case has been described in which alert images corresponding to all detected attention areas are generated in advance. However, in this embodiment, an alert corresponding to an attention area determined to be displayed is displayed. Only an image may be generated. FIG. 27 shows a third configuration example of the endoscope apparatus as a configuration example when generating only an alert image corresponding to a region of interest determined to be displayed.

  The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. In addition, the same code | symbol is attached | subjected to the component same as the component already demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably. For example, the operations of the components other than the control device 300 are the same as those in the first configuration example shown in FIG.

  The control device 300 includes two A / D conversion units 310 and 311, an image processing device 301 (image processing unit), and a control unit 370. The image processing device 301 includes a white light image acquisition unit 320, a special light image acquisition unit 330, a region of interest detection unit 340, a region of interest control unit 380, and a display mode control unit 390.

  First, an outline of processing performed by the image processing apparatus 301 will be described with reference to FIG. As indicated by I1 and I2 in FIG. 28, a special light image and a white light image are acquired, a region of interest such as a lesion is detected from the special light image, and a tag is assigned to each region of interest. Up to this point, the processing is the same as that in the first configuration example described above. Next, as shown at I3, whether or not the alert image can be displayed is determined according to the size of the region of interest or the like. For example, the alert image of the attention area of the tag “1” is determined not to be displayed, and the alert image of the attention area of the tag “2” is determined to be displayed. Next, as indicated by I4, an alert image corresponding to the attention area determined to be displayed is generated based on the white light image. That is, the alert image of the attention area of the tag “1” is not generated, and the alert image of the attention area of the tag “2” is generated. And as shown to I5, the produced | generated alert image is superimposed and displayed on a white light image.

  Next, a detailed configuration example and operation of each component will be described. The processes of the A / D conversion units 310 and 311, the white light image acquisition unit 320, the special light image acquisition unit 330, and the attention area detection unit 340 are the same as those in the first configuration example described above, and thus the description thereof is omitted. To do.

  FIG. 29 shows a detailed configuration example of the attention area control unit 380. The attention area control section 380 includes an attention area selection section 381, an area number determination section 3611, an area size determination section 3612, a detection count calculation section 3613, a memory 3614, and an elapsed time determination section 3615. Note that the processing performed by the number-of-regions determination unit 3611, the region size determination unit 3612, the detection count calculation unit 3613, the memory 3614, and the elapsed time determination unit 3615 is the same as that in the first configuration example described above.

  The attention area selection unit 381 outputs only the information of the attention area group for which the alert should be displayed among the attention area groups output from the attention area detection unit 340 to the display mode control unit 390. Specifically, the attention area selection unit 381 has the coordinates (for example, attention area) of all attention areas to which tag information output in common from both the area size determination unit 3612 and the elapsed time determination unit 3615 is assigned. The coordinates of all the pixels) are output to the display mode control unit 390. When the tag information output in common does not exist, the attention area selection unit 381 outputs a trigger signal indicating that there is no alert image to be displayed to the display mode control unit 390.

  FIG. 30 shows a detailed configuration example of the display mode control unit 390. The display mode control unit 390 includes a corresponding area selection unit 351, an enhancement processing unit 352, and an alert image superimposing unit 362.

  The corresponding area selection unit 351 outputs information (such as coordinates and pixel values) of pixels included in the white light image area corresponding to the attention area output from the attention area control unit 380 to the enhancement processing unit 352. The enhancement processing unit 352 performs the color conversion processing shown in the above equation (9) on all the pixels output from the corresponding region selection unit 351, and alert information is superimposed using the pixel information after the color conversion processing as an alert image. To the unit 362. The alert image superimposing unit 362 superimposes the alert image on the white light image output from the white light image acquiring unit 320 and outputs the white light image to the display unit 400.

  According to the above embodiment, when a diagnosis using a white light image and a special light image is performed, the lesion area present in the white light image is rendered as a color area different from the normal, and thus a load on the doctor. It is possible to prevent oversight of the lesion without increasing the number of lesions. Further, by controlling the display mode of the alert image according to the detected information of the attention area, it is possible to improve the problem that the alert display hinders the diagnosis and treatment of the doctor.

  In the first configuration example, alert images corresponding to all detected attention area groups are created. In the third configuration example, the attention area selection unit 381 determines that an alert image should be displayed. Only the alert image corresponding to the attention area group is created. Therefore, according to the above embodiment, the processing time of the enhancement processing unit 352 can be shortened compared to the first configuration example.

  In the above embodiment, the case where the attention area is controlled by the determination process using the number, size, and elapsed time of the attention area has been described as an example. However, the present embodiment is not limited to this. For example, in the present embodiment, the alert image may be forcibly displayed regardless of the determination process. In this case, the user performs setting to forcibly display the alert image via the external I / F unit 500. The setting information is transmitted as a trigger signal to the control unit 370, and the control unit 370 outputs the trigger signal to the attention area selection unit 381. When the trigger signal is output from the control unit 370, the attention area selection unit 381 is output from the attention area detection unit 340 regardless of the determination results of the area size determination unit 3612 and the elapsed time determination unit 3615. The coordinates of all the attention areas are output to the display mode control unit 390.

  Moreover, although the said embodiment demonstrated the case where an alert image was produced by the color conversion process using said Formula (9), this embodiment is not limited to this, Arbitrary brightness conversion processes and color conversion processes are performed. An alert image may be created by using it.

  In the above embodiment, a case has been described in which enhancement processing is performed on all the pixels in the white light image region corresponding to the detected attention region using the above equation (9). Is not limited to this. For example, in the present embodiment, an alert image is created by performing enhancement processing on the pixels in the white light image corresponding to the boundary between the attention area and the non-attention area not detected as the attention area. Also good. In this case, for example, all the pixels in the white light image corresponding to the boundary (near the boundary) between the attention area and the non-attention area may be subjected to the color conversion processing represented by the above equation (10).

  Here, in the above embodiment, each unit constituting the image processing apparatus 301 is configured by hardware. However, the present embodiment is not limited to this, and is realized as software by the CPU executing a program. Also good. In this case, the control device 300 and the image processing device 301 are realized by a computer system described later with reference to FIG.

  As an example of a case where a part of the processing performed by each unit is configured by software, a processing procedure when the processing performed by the image processing apparatus 301 on an image acquired in advance is realized by software is described with reference to the flowchart illustrated in FIG. I will explain. For example, an image acquired in advance is an image obtained by recording an output image of the Bayer array output from the A / D conversion units 310 and 311 as a RAW file on a recording medium.

  As shown in FIG. 31, when this process is started, parameters used in the above-described enhancement process and determination process are read from the memory (step S20). Next, the narrow-band light image is read from the memory (step S21), and the white light image is read from the memory (step S22). Next, a region of interest is detected from the read narrowband light image by the same method as in the first configuration example (step S23).

  Next, determination processing is performed according to the number, size, and elapsed time of the attention area group detected in step S23, and information on the attention area determined to display the alert image is output (step S24). Next, an alert image is created by performing the enhancement processing shown in the above equation (9) on all pixels in the white light image region corresponding to the region of interest output from step S24 (step S25). Next, the alert image created in step S25 is superimposed on the white light image and output as a display image (step S26). If a trigger signal is output in step S24, the white light image is output as it is as a display image without performing the superimposition process. Next, if a series of processing is completed for all images, the processing ends (step S27, YES), and if an unprocessed image remains, the same processing is continued (step S27, NO). .

  Next, the processing procedure of the attention area control process (step S24) will be described using the flowchart shown in FIG.

  As shown in FIG. 32, when this process is started, the number of attention area groups detected in step S23 is calculated, and the result of comparing the number of attention area groups with a threshold Area_Num is output as flag information ( Step S241). Specifically, as flag information, a flag “0” is output when the number of attention area groups is larger than the threshold value Area_Num, and a flag “1” is output when the number is less than the threshold value.

  Next, the size of each attention area group detected in step S23 is calculated. Then, the flag information output in step S241 is referred to. When the flag is “0”, the result of comparing the size of the region of interest with the threshold Size_Min is output. On the other hand, if the flag is “1”, the result of comparing the size of the region of interest with the threshold Size_Max is output (step S242). Specifically, as the comparison result, all tag information assigned to the attention area group whose size is determined to be equal to or smaller than the threshold value is output.

  Next, the detected elapsed time of each region of interest detected in step S23 is calculated by the method shown in the above equation (11), and the result of comparing the detected elapsed time with the threshold value Area_Time is output (step S243). Specifically, tag information is output as a comparison result. This tag information is all tag information assigned to the attention area group determined that the detected elapsed time is equal to or less than the threshold value Area_Time.

  Next, the attention area where the alert image is to be displayed is selected according to the determination results of step S242 and step S243 (step S244). Specifically, common tag information is extracted from the tag information output in steps S242 and S243, and the coordinates of all the attention areas to which the common tag information is assigned are output. When common tag information does not exist, a trigger signal indicating that there is no alert to be displayed is output.

  In the above-described embodiment, the case has been described in which only the alert image corresponding to the attention area group smaller than the threshold Size_Min is displayed when the number of attention area groups is larger than the threshold value Area_Num. However, the present embodiment is not limited thereto. Not. For example, in this embodiment, when the number of attention area groups is larger than the threshold value Area_Num, alert images corresponding to all attention area groups may be hidden. In this case, when the flag “0” is output from the number-of-regions determination unit 3611, the region size determination unit 3612 determines to hide the alert images corresponding to all the attention region groups.

  Moreover, although the said embodiment demonstrated the case where alert display was controlled according to the size (the number of local areas, the number of pixels, etc.) of an attention area group, this embodiment is not limited to this. For example, in the present embodiment, the alert display may be controlled according to the ratio of the size (area) of the attention area group to the narrow-band light image instead of the size of the attention area group. In this case, for example, the ratio of the number of attention areas constituting the attention area group and the number of all local areas set by the local area setting unit 341 is the ratio of the size of the attention area group to the narrowband light image. And When the ratio is larger than the threshold value, the attention area is controlled so that the alert image corresponding to the attention area group is not displayed.

  According to the above embodiment, as shown in FIG. 27, the determination unit (attention region control unit 380) determines whether or not an alert image should be displayed, and the attention region detected by the attention region detection unit 340. A display target attention area is selected from the list. The alert image generation unit (enhancement processing unit 352 shown in FIG. 30) generates an alert image corresponding to the display target attention area selected by the determination unit. And the display mode control part 390 performs control which displays the alert image produced | generated by the alert image production | generation part.

  In this way, as shown in I3 of FIG. 28, the attention area where the alert image is to be displayed is selected from the detected attention areas, and only the selected attention area is supported as shown in I4. An alert image is generated, and only the alert image determined to be displayed can be displayed as indicated by I5. Thereby, display / non-display of the alert image can be controlled.

6). Fourth Configuration Example In the third configuration example, the case where the attention area is detected from the special light image has been described. However, in the present embodiment, the attention area may be detected from the white light image. FIG. 33 shows a fourth configuration example of the endoscope apparatus as a configuration example when detecting the attention area from the white light image.

  The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. In addition, the same code | symbol is attached | subjected to the component same as the component already demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably. For example, the operations of the components other than the control device 300 are the same as those in the second configuration example shown in FIG.

  The control device 300 includes an A / D conversion unit 310, an image processing device 301, and a control unit 370. The image processing device 301 includes a white light image acquisition unit 320, a region of interest detection unit 340, a region of interest control unit 380, and a display mode control unit 390. Note that the processing performed by the attention area detection unit 340 is the same as that in the above-described second configuration example, and the processing performed by components other than the attention area detection unit 340 is the same as that in the above-described third configuration example.

  According to the above configuration example, when a diagnosis using a white light image is performed, a lesion area existing in the white light image is rendered as a color area different from a normal one, so that the burden on the doctor is not increased. It becomes possible to prevent oversight of lesions. Further, by controlling the display mode of the alert image according to the detected information of the attention area, it is possible to improve the problem that the alert display hinders the diagnosis and treatment of the doctor.

  Further, according to the above embodiment, since processing is performed using only the white light image, the imaging unit 200 and the control device 300 are compared with the case where the white light image and the special light image are used as in the third configuration example. Can be simplified.

  Here, in the above embodiment, each unit constituting the image processing apparatus 301 is configured by hardware. However, the present embodiment is not limited to this, and is realized as software by the CPU executing a program. Also good. In this case, the control device 300 and the image processing device 301 are realized by a computer system described later with reference to FIG.

  As an example of a case where a part of the processing performed by each unit is configured by software, a processing procedure when the processing performed by the image processing apparatus 301 for an image acquired in advance is realized by software is illustrated in a flowchart illustrated in FIG. I will explain. For example, an image acquired in advance is an image obtained by recording an output image of the Bayer array output from the A / D conversion unit 310 on a recording medium as a RAW file.

  As shown in FIG. 34, when this process is started, parameters used in the above-described enhancement process and determination process are read from the memory (step S30). Next, a white light image is read from the memory (step S31). Next, the attention area is detected from the read white light image by the same method as in the second configuration example (step S32). Next, similarly to the above-described third configuration example, a process for determining a region of interest to be displayed is performed (step S33), an alert image is created by enhancement processing (step S34), and the alert image is converted into a white light image. (Step S35). Then, if a series of processing is completed for all the images, the processing ends (step S36, YES), and if an unprocessed image remains, the same processing is continued (step S36, NO).

7). Computer System As described above, the control device 300 and the image processing device 301 may be realized by hardware, or configured such that the CPU performs processing of each unit, and the CPU executes software to execute the program. It may be realized as. Or you may comprise a part of process which each part performs with software.

  When the imaging unit 200 is separated and the processing performed by each unit of the control device 300 and the image processing device 301 is realized as software, a known computer system such as a workstation or a personal computer is used as the control device or the image processing device. Can do. A program (control program, image processing program) for realizing processing performed by each unit of the control device 300 and the image processing device 301 is prepared in advance, and this program can be realized by the CPU of the computer system.

  FIG. 35 is a system configuration diagram showing a configuration of a computer system 600 in this modification, and FIG. 36 is a block diagram showing a configuration of a main body 610 in the computer system 600. As shown in FIG. 35, the computer system 600 includes a main body 610, a display 620 for displaying information such as an image on the display screen 621 according to instructions from the main body 610, and various information on the computer system 600. A keyboard 630 for inputting and a mouse 640 for designating an arbitrary position on the display screen 621 of the display 620 are provided.

  As shown in FIG. 36, the main body 610 in the computer system 600 includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, a CD-ROM drive 615 that accepts a CD-ROM 660, and a USB memory. USB port 616 to which 670 is detachably connected, I / O interface 617 to which display 620, keyboard 630 and mouse 640 are connected, and a LAN interface for connection to a local area network or wide area network (LAN / WAN) N1 618.

  Further, the computer system 600 is connected to a modem 650 for connecting to a public line N3 such as the Internet, and is another computer system via a LAN interface 618 and a local area network or a wide area network N1. A personal computer (PC) 681, a server 682, a printer 683, and the like are connected.

  The computer system 600 refers to a program (for example, FIGS. 22 and 23) recorded on a predetermined recording medium, reads out and executes a program for realizing the above-described processing procedure, An image processing apparatus is realized. Here, the predetermined recording medium is a “portable physical medium” including an MO disk, a DVD disk, a flexible disk (FD), a magneto-optical disk, an IC card, etc. in addition to the CD-ROM 660 and the USB memory 670, a computer A “fixed physical medium” such as HDD 614, RAM 612, and ROM 613 provided inside and outside the system 600, a public line N3 connected via the modem 650, and another computer system (PC) 681 or server 682 are connected. It includes any recording medium that records an image processing program readable by the computer system 600, such as a “communication medium” that stores the program in a short time when transmitting the program, such as a local area network or a wide area network N1.

  That is, the image processing program is recorded on a recording medium such as “portable physical medium”, “fixed physical medium”, and “communication medium” in a computer-readable manner. An image processing apparatus is realized by reading and executing an image processing program from a medium. Note that the image processing program is not limited to be executed by the computer system 600, and when the other computer system (PC) 681 or the server 682 executes the image processing program or in cooperation therewith. The present invention can be similarly applied to a case where an image processing program is executed.

  In addition, in this embodiment, each part (white light image acquisition unit, special light image acquisition unit, attention area detection unit, alert image display setting unit, display mode control unit, attention area control unit, control unit, etc.) of this embodiment is used. The present invention can also be applied to a computer program product in which a program code to be realized is recorded.

  For example, a computer program product includes an information storage medium (an optical disk medium such as a DVD, a hard disk medium, a memory medium, etc.) in which a program code is recorded, a computer in which the program code is recorded, and an Internet system in which the program code is recorded (for example, A system including a server and a client terminal), etc., such as an information storage medium, apparatus, device or system in which a program code is incorporated. In this case, each component and each processing process of this embodiment are mounted by each module, and the program code constituted by these mounted modules is recorded in the computer program product.

  As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in each embodiment or modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

  Further, in the specification or drawings, terms (capsule type endoscopes) described at least once together with different terms having a broader meaning or the same meaning (first endoscope device, second endoscope device, normal light image, etc.). Mirror, scope endoscope, white light image, etc.) may be replaced by the different terms anywhere in the specification or drawings.

100 light source unit, 110 white light source, 120 condenser lens, 200 imaging unit,
210 light guide fiber, 220 illumination lens, 230 objective lens,
240 half mirror, 250 first image sensor, 260 second image sensor,
300 control device, 301 image processing device, 310, 311 A / D conversion unit,
320 white light image acquisition unit, 321 white light image generation unit, 322 white light image storage unit,
330 special light image acquisition unit, 331 special light image generation unit, 332 special light image storage unit,
340 attention area detection section, 341 local area setting section, 342 feature amount calculation section,
343 area detection unit, 344 tag information addition unit, 345 feature amount calculation unit,
346 region detection unit, 350 alert image display setting unit, 351 corresponding region selection unit,
352 enhancement processing unit, 360 display mode control unit, 361 display availability determination unit,
362 alert image superimposing unit, 370 control unit, 380 attention area control unit,
381 region of interest selection unit, 390 display mode control unit, 400 display unit,
500 external I / F unit, 600 computer system, 610 main unit,
611 CPU, 612 RAM, 613 ROM, 614 HDD,
615 CD-ROM drive, 616 USB port,
617 I / O interface, 618 LAN interface,
620 display, 621 display screen, 630 keyboard, 640 mouse,
650 modem, 660 CD-ROM, 670 USB memory, 681 PC,
682 server, 683 printer, 3611 area number determination unit,
3612 area size determination unit, 3613 detection frequency calculation unit, 3614 memory,
3615 elapsed time determination unit, 3616 alert image selection unit,
a (m, n) local region,
Area_Num, Size_Min, Size_Max, Area_Time threshold,
r, g, b color filter, g2, b2 color filter, F feature quantity, H hue,
N1 wide area network, N3 public line, Pt detection count

Claims (41)

It is at least one image of a normal light image including a subject image having information in a white wavelength band and a special light image including a subject image having information in a specific wavelength band corresponding to the normal light image. An image acquisition unit for acquiring an acquired image;
An attention area detection unit that detects an attention area that is an attention area based on a feature amount of a pixel of the acquired image;
A determination unit configured to determine whether or not to display an alert image corresponding to the attention area according to the detection result of the attention area;
A display mode control unit that performs control to display an alert image corresponding to a display target attention area that is an attention area that is determined by the determination unit to display an alert image;
An image processing apparatus comprising: In claim 1,
Including an alert image generator,
The display mode control unit
An image processing apparatus that performs control to display an alert image corresponding to the display target attention area from among the alert images corresponding to the attention area generated by the alert image generation unit. In claim 2,
The determination unit
In accordance with the detection result of the attention area, it is determined whether or not to display the alert image generated by the alert image generation unit,
The display mode control unit
An image processing apparatus that performs control to hide an alert image determined to be non-displayed by the determination unit. In claim 1,
Including an alert image generator,
The determination unit
It is determined whether or not an alert image should be displayed, and the display target attention area is selected from the attention areas detected by the attention area detection section,
The alert image generator
Generating an alert image corresponding to the display target attention area selected by the determination unit;
The display mode control unit
An image processing apparatus that performs control to display an alert image generated by the alert image generation unit. In claim 1,
The determination unit
Based on the detection result of the attention area, acquiring at least one information of the number information of the attention area in the acquired image, the size information of the attention area, and the detection elapsed time information of the attention area, An image processing apparatus that determines whether or not to display the alert image according to the acquired information. In claim 5,
The determination unit
When the number of the attention areas represented by the number information is larger than a predetermined threshold, it is determined to hide the alert image corresponding to a part of the detected attention area of the attention area. A featured image processing apparatus. In claim 5,
The determination unit
When the number of the attention areas represented by the number information is larger than a predetermined threshold value, it is determined that alert images corresponding to all attention areas of the detected attention areas are not displayed. An image processing apparatus. In claim 5,
The determination unit
An image processing apparatus that determines that an alert image corresponding to a region of interest whose size represented by the size information is larger than a predetermined threshold is not displayed. In claim 8,
The determination unit
When the number of the attention areas represented by the number information is smaller than a predetermined threshold, the alert image corresponding to the attention area whose size is larger than the first threshold is determined to be hidden, An image processing apparatus, wherein when the number of attention areas is larger than a predetermined threshold, the alert image corresponding to the attention area whose size is larger than the second threshold is determined not to be displayed. In claim 9,
The second threshold is:
An image processing apparatus having a threshold value smaller than the first threshold value. In claim 5,
The determination unit
An image processing apparatus that performs a determination to hide an alert image corresponding to the region of interest whose detection elapsed time represented by the detection elapsed time information exceeds a predetermined threshold. In claim 11,
The determination unit
An image processing apparatus that counts the number of frames in which the region of interest is detected as the detection elapsed time information. In claim 11,
The attention area detector
Dividing the acquired image into a plurality of local regions, detecting whether each local region corresponds to the region of interest;
The determination unit
An image processing apparatus that counts the number of frames detected as corresponding to the region of interest in each of the local regions. In claim 13,
The determination unit
Among the local areas detected to correspond to the attention area, increment the count value of the local area detected to correspond to the attention area in the previous frame,
An image processing apparatus, wherein a count value of a local area detected as not corresponding to the attention area in a previous frame among local areas detected as corresponding to the attention area is set as an initial value. In claim 13,
The attention area detector
Of the local regions detected to correspond to the region of interest, an adjacent local region is a region of interest group,
The determination unit
The detection elapsed time information of the attention area group is calculated based on the number of frames corresponding to the attention area in each local area in the attention area group, and it is determined whether or not the alert image corresponding to the attention area group can be displayed. An image processing apparatus. In claim 1,
An instruction to display / hide the alert image is input via the interface unit,
The display mode control unit
An image processing apparatus that performs control to display an alert image corresponding to the region of interest when an instruction to display an alert image is input to the interface unit, regardless of a determination result by the determination unit. In claim 1,
The display mode control unit
A superimposing unit that superimposes an alert image corresponding to the display target attention area on an output image;
The display mode control unit
An image processing apparatus that controls to display the output image on which an alert image corresponding to the display target attention area is superimposed. In claim 17,
The superimposing unit is
An image processing apparatus, wherein an alert image corresponding to the display target attention area is superimposed on the normal light image as the output image. In claim 2,
Including an area selection section,
The attention area detector
Detecting the region of interest from the special light image;
The area selection unit
From the normal light image, a corresponding attention area that is an area corresponding to the attention area is selected,
The alert image generator
An image processing apparatus, wherein the alert image is generated by performing enhancement processing on pixels in the corresponding attention area included in the normal light image. In claim 2,
Including an area selection section,
The attention area detector
Detecting the region of interest from the special light image;
The area selection unit
From the normal light image, a corresponding attention area that is an area corresponding to the attention area is selected,
The alert image generator
Generating an alert image by performing enhancement processing on pixels located on a boundary line between the corresponding attention area in the normal light image and a non-corresponding attention area that is an area other than the corresponding attention area. A featured image processing apparatus. In claim 4,
Including an area selection section,
The attention area detector
Detecting the region of interest from the special light image;
The area selection unit
From the normal light image, a corresponding attention area that is an area corresponding to the attention area is selected,
The alert image generator
An image processing apparatus, wherein the alert image is generated by performing enhancement processing on pixels in the corresponding attention area corresponding to the attention area selected by the determination unit. In claim 4,
Including an area selection section,
The attention area detector
Detecting the region of interest from the special light image;
The area selection unit
From the normal light image, a corresponding attention area that is an area corresponding to the attention area is selected,
The alert image generator
Emphasis processing is performed on pixels located on a boundary line between the corresponding attention area corresponding to the attention area selected by the determination unit and the non-corresponding attention area that is an area other than the corresponding attention area, and the attention is performed. An image processing apparatus that generates an alert image corresponding to a region. In claim 1,
The attention area detector
An image processing apparatus that detects the region of interest based on a specific color feature amount of a pixel in the special light image. In claim 1,
The attention area detector
An image processing apparatus that detects the region of interest based on a specific color feature amount of a pixel in the normal light image. In claim 1,
The region of interest is
An image processing apparatus characterized by being an area representing a lesion. In claim 1,
The specific wavelength band is
An image processing apparatus having a band narrower than the white wavelength band. In claim 26,
The normal light image and the special light image are in-vivo images obtained by copying the inside of the living body,
The image processing apparatus according to claim 1, wherein the specific wavelength band included in the in-vivo image is a wavelength band of a wavelength absorbed by hemoglobin in blood. In claim 27,
The image processing apparatus according to claim 1, wherein the specific wavelength band is 390 nm to 445 nm or 530 nm to 550 nm. In claim 1,
The normal light image and the special light image are in-vivo images obtained by copying the inside of the living body,
The image processing apparatus according to claim 1, wherein the specific wavelength band included in the in-vivo image is a wavelength band of fluorescence emitted from a fluorescent material. In claim 29,
The image processing apparatus according to claim 1, wherein the specific wavelength band is a wavelength band of 490 nm to 625 nm. In claim 1,
The normal light image and the special light image are in-vivo images obtained by copying the inside of the living body,
The image processing apparatus, wherein the specific wavelength band included in the in-vivo image is a wavelength band of infrared light. In claim 31,
The specific wavelength band is a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm. In claim 1,
The image acquisition unit
An image processing apparatus comprising: a special light image acquisition unit that acquires the special light image based on the normal light image. In claim 33,
The special light image acquisition unit
A signal extraction unit that extracts a signal in the white wavelength band from the acquired normal light image;
The special light image acquisition unit
An image processing apparatus that generates the special light image including a signal in the specific wavelength band based on the extracted signal in the white wavelength band. In claim 34,
The special light image acquisition unit
A matrix data setting unit for setting matrix data for calculating a signal in the specific wavelength band from the signal in the white wavelength band;
The special light image acquisition unit
An image processing apparatus, wherein the special light image is generated by calculating a signal in the specific wavelength band from a signal in the white wavelength band using the set matrix data.   An endoscope apparatus comprising the image processing apparatus according to any one of claims 1 to 35. In claim 36,
An endoscope apparatus comprising: a display unit configured to display an alert image corresponding to the display target attention area under the control of the display mode control unit. In claim 36,
A light source;
A white light filter that transmits light in the white wavelength band;
A special optical filter that transmits light of the specific wavelength band;
Including
The image acquisition unit
The light obtained by irradiating the subject with light from the light source is passed through the white light filter to obtain the normal light image, and the light obtained by irradiating the subject with light from the light source is used as the special light. An endoscope apparatus, wherein the special light image is acquired by passing through an optical filter. In claim 36,
A white light source that emits light in the white wavelength band;
A special light source that emits light of the specific wavelength band;
Including
The image acquisition unit
Obtaining the normal light image obtained by irradiating the subject with light from the white light source, and obtaining the special light image obtained by irradiating the subject with light from the special light source. Endoscopic device characterized. It is at least one image of a normal light image including a subject image having information in a white wavelength band and a special light image including a subject image having information in a specific wavelength band corresponding to the normal light image. An image acquisition unit for acquiring an acquired image;
An attention area detection unit that detects an attention area that is an attention area based on a feature amount of a pixel of the acquired image;
A determination unit configured to determine whether or not to display an alert image corresponding to the attention area according to the detection result of the attention area;
As a display mode control unit that performs control to display an alert image corresponding to a display target attention area that is an attention area determined to display an alert image by the determination unit,
A program characterized by causing a computer to function. It is at least one image of a normal light image including a subject image having information in a white wavelength band and a special light image including a subject image having information in a specific wavelength band corresponding to the normal light image. Acquire the acquired image,
Based on the feature amount of the pixel of the acquired image, the attention area that is the attention area is detected,
In accordance with the detection result of the attention area, it is determined whether or not to display an alert image corresponding to the attention area,
An image processing method comprising: performing control to display an alert image corresponding to a display target attention area which is an attention area determined to display an alert image by the determination section.