JP6132901B2 - Endoscope device - Google Patents

Description

  The present invention relates to an endoscope apparatus and the like.

  2. Description of the Related Art Endoscope apparatuses that irradiate illumination light to a tissue in a body cavity and perform diagnosis / treatment using an image obtained by imaging the reflected light are widely used. An imaging element such as a CCD or CMOS and an objective lens that optically forms a subject image are provided at the distal end of the insertion portion. In general, a wide-angle objective lens is used as an objective lens for an endoscope in order to prevent oversight of a lesion. For example, an objective lens having a viewing angle of 170 ° is used.

  Now, it is conceivable that a wider field of view can be observed by using an objective lens (for example, Patent Document 1) that forms an image of a side field of view at the tip of the insertion portion.

JP 2010-169792 A JP 2002-263064 A JP-A-9-98943

  However, if the field of view is widened, the amount of information contained in the image is relatively larger than that of a conventional endoscope, and thus there is a problem that it is difficult to keep the entire image at a good brightness.

  Patent Document 2 discloses a light control processing method that calculates a luminance average value for pixels in an effective area from a mask shape of a scope and maintains the brightness of a subject image. Patent Document 3 discloses a technique for securing a good light distribution by disposing light guides on at least three sides of an imaging surface of an imaging element in an endoscope apparatus that acquires a wide-angle image.

  According to some aspects of the present invention, it is possible to provide an endoscope apparatus or the like that can maintain a region of interest at appropriate brightness.

  One aspect of the present embodiment includes an image acquisition unit that acquires a captured image including a subject image obtained by irradiating a subject with illumination light from a light source unit, and attention that sets a region of interest for the captured image An area setting unit; and a dimming control unit that performs dimming control of the amount of illumination light based on the set attention area, wherein the attention area setting unit calculates a feature amount based on the captured image. The present invention relates to an endoscope apparatus that calculates and sets the attention area based on an area having a predetermined feature amount.

  According to one aspect of the present embodiment, a feature amount is calculated based on a captured image, a region of interest is set for the captured image based on a region having a predetermined feature amount, and illumination light is generated based on the region of interest. Dimming control of the amount of light is performed. Thereby, it becomes possible to maintain the attention area which an operator pays attention to appropriate brightness.

Example of configuration of objective optical system. FIG. 2A to FIG. 2D are explanatory diagrams about the outline of the present embodiment. The structural example of the endoscope apparatus in 1st Embodiment. The detailed structural example of a rotation color filter. Examples of spectral characteristics of color filters. 2 is a detailed configuration example of an image processing unit according to the first embodiment. 3 is a detailed configuration example of a gradation conversion unit. Explanatory drawing about the process which an attention area setting part performs. FIGS. 9A and 9B are explanatory diagrams of processing performed by the attention area setting unit. Explanatory drawing about the process which an attention area setting part performs. Explanatory drawing about the process which an attention area setting part performs. Explanatory drawing about the process which an attention area setting part performs. Explanatory drawing about the process which an attention area setting part performs. The modification structural example of an image process part. Explanatory drawing about the process which a scaling process part performs. The structural example of the endoscope apparatus in 2nd Embodiment. 10 is a detailed configuration example of an image processing unit according to the second embodiment. Explanatory drawing about the process which the attention area setting part in 2nd Embodiment performs. The structural example of the endoscope apparatus in 3rd Embodiment. The detailed structural example of the state information acquisition part in 3rd Embodiment. FIG. 21A to FIG. 21C are explanatory diagrams of the bending operation. 14 is a detailed configuration example of an image processing unit according to the third embodiment. The structural example of the endoscope apparatus in 4th Embodiment. FIG. 24A to FIG. 24D are explanatory diagrams of a distance estimation method. The modification structural example of an endoscope apparatus. The detailed structural example of the attention area setting part in 5th Embodiment. Explanatory drawing about the process which the attention area setting part in 5th Embodiment performs. Explanatory drawing about the modification of 5th Embodiment. The structural example of the endoscope apparatus in 6th Embodiment. 6 is a configuration example of a color filter of a second image sensor. The transmittance | permeability characteristic example of the color filter of a 2nd image pick-up element. 14 is a detailed configuration example of an image processing unit according to the sixth embodiment. The example of the single-plate image signal imaged with the 2nd image sensor. The detailed structural example of the attention area setting part in 6th Embodiment. Example of setting a local area. Explanatory drawing about the process which the attention area setting part in 6th Embodiment performs. The structural example of the endoscope apparatus in 7th Embodiment. The detailed structural example of the image process part in 7th Embodiment. The detailed structural example of the attention area setting part in 8th Embodiment. Examples of images before and after distortion correction processing. Example of setting representative points. 42 (A) to 42 (D) are explanatory views of a method for determining the removal state. The structural example of the endoscope apparatus in 9th Embodiment. The detailed structural example of the state information acquisition part in 9th Embodiment. Explanatory drawing about the red ball area. FIG. 46A is an explanatory diagram of the red ball region. FIG. 46B is an explanatory diagram of the scaling process in the tenth embodiment. The detailed structural example of the image process part in 10th Embodiment. The detailed structural example of the attention area setting part in 10th Embodiment. Explanatory drawing about the process which the attention area setting part in 10th Embodiment performs. The detailed structural example of an akadama area | region candidate detection part. 3 shows a detailed configuration example of a defocus detection unit. Explanatory drawing about the process which the attention area setting part in 10th Embodiment performs. A detailed configuration example of a scaling parameter setting unit. An example of a correspondence curve of a plurality of normalized distances after scaling with respect to a normalized distance before scaling.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline of the present embodiment 1.1. First, the outline of the present embodiment will be described. Patent Document 1 discloses an objective optical system capable of observing the front field and the side field of the scope tip. FIG. 1 shows a configuration example of the objective optical system. As shown in FIG. 1, the light beam LC1 from the front visual field enters from the surface SF1, and the light beam LC2 from the side visual field enters from the surface SF3. These light rays LC1 and LC2 are refracted or reflected by the surfaces SF1 and SF2, thereby forming an image of the front field and the side field.

  By using such an objective optical system, it is possible to observe the inside of the body cavity with a wider field of view than the conventional endoscope apparatus. That is, as shown in FIG. 2A, when a lesion is screened in a luminal organ, observation is performed while the scope is parallel to the axial direction of the organ and the scope is inserted or removed. When there is a lateral visual field, it is possible to observe the lesioned part behind the folds, so that the oversight of the lesion can be reduced.

  However, when the field of view is thus widened, the amount of information contained in the image is relatively larger than that of a conventional endoscope, and thus there is a problem that it is difficult to keep the entire image at a good brightness. This point will be described with reference to FIGS.

  As shown in FIG. 2A, when the endoscope imaging unit is located at the center of the luminal organ, the distance from the endoscope imaging unit to the subject differs between the front visual field and the side visual field. Therefore, as shown in FIG. 2B, a difference in brightness occurs for each region in the image. Further, as shown in FIG. 2C, it is assumed that the distal end of the endoscope is brought close to the wall surface of the organ in order to observe a lesion portion existing on the wall surface of the luminal organ. Then, as shown in FIG. 2D, the attention area (for example, a lesion) causes halation. In such an image, it is difficult to perform appropriate observation and treatment on the region that the operator wants to pay attention to.

  In order to deal with such a problem, the endoscope apparatus performs a dimming process that appropriately maintains the brightness of the subject image. For example, Patent Document 2 discloses a dimming processing method that calculates a luminance average value for pixels in an effective region from a scope mask shape and maintains the brightness of a subject image.

  However, with this method, the average value of the entire effective area can take an appropriate value, but the light cannot be dimmed for each area in the image. Therefore, when a difference in brightness occurs in each region, the region that the operator wants to pay attention to cannot be adjusted to an appropriate brightness.

  Patent Document 3 discloses a technique for securing a good light distribution by disposing light guides on at least three sides of an imaging surface of an imaging element in an endoscope apparatus that acquires a wide-angle image. .

  However, with this method, as shown in FIG. 2C, when a subject is imaged from the oblique direction with respect to the lesioned part, the problem that halation occurs in the attention area cannot be solved.

  Therefore, in the present embodiment, as shown in FIG. 13, a region of interest is set in the captured image based on information (for example, a captured image) from the endoscope apparatus, and the light control is performed using the region of interest as a light control target region. I do. As a result, it is possible to maintain the area focused on by the operator at an appropriate brightness, and to reduce the load when the operator performs diagnosis and treatment.

  As will be described below, in the conventional dimming processing, the brightness average value is calculated using the entire image as a photometric area, and the light source light quantity is adjusted so that the brightness average value approaches the target value. Maintain the properness. In the present embodiment, by calculating the luminance average value using the attention area as the photometry area, even if the brightness is biased as shown in FIG. Brightness can be maintained.

1.2. Light Control Processing A conventional light control processing will be described in detail. In the light control processing, first, photometry processing for measuring the brightness of the subject is performed. Specifically, the average value of the luminance signals in the image signals for one frame or one field sequentially obtained from the image sensor is calculated. Here, the effective area of the image signal is generally used as the photometric area. The effective area of the image signal indicates a front area and a side area obtained by removing the mask area from the image signal shown in FIG.

  Next, the measured brightness of the subject is adjusted to the target brightness. Specifically, by controlling the aperture of the light source to adjust the light amount, the brightness of the photometric subject is matched with the target brightness.

  For example, when the entire subject is relatively far away from the objective lens at the distal end of the endoscope, the above-mentioned luminance average value is temporarily lower than the luminance reference value (target brightness). At this time, the light source aperture is widened to increase the amount of illumination light. On the other hand, when the entire subject approaches the objective lens at the endoscope tip, the above-mentioned luminance average value is higher than the luminance reference value (target brightness). Is narrowed and the amount of illumination light is reduced. When the average brightness value does not reach the brightness reference value (target brightness) due to the adjustment of the light source diaphragm, the brightness of the image signal is adjusted by applying a digital gain to the image signal. In this way, the image signal is maintained in an appropriate state corresponding to the luminance reference value regardless of the distance of the subject.

2. First embodiment 2.1. Endoscope Device An endoscope device according to this embodiment that sets an attention area in a captured image as described above and controls the brightness of the attention area by dimming will be described in detail.

  FIG. 3 shows a configuration example of the endoscope apparatus according to the first embodiment. The endoscope apparatus includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400 (processor unit), a display unit 500, and an external I / F unit 550.

  The light source unit 100 includes a white light source 101, a light source stop 102, a light source stop driving unit 103 that drives the light source stop 102, and a rotating color filter 104 having a plurality of spectral transmittance filters. In addition, the light source unit 100 includes a rotation driving unit 105 that drives the rotation color filter 104 and a condensing lens 106 that condenses the light transmitted through the rotation color filter 104 on the incident end face of the light guide fiber 301.

  The light source aperture driving unit 103 adjusts the light amount by opening and closing the light source aperture 102 based on a control signal from the control unit 420 of the control device 400.

  FIG. 4 shows a detailed configuration example of the rotation color filter 104. The rotary color filter 104 includes a primary color red (hereinafter abbreviated as R) filter 701, a green color (hereinafter abbreviated as G) filter 702, a blue color (hereinafter abbreviated as B) filter 703, and a rotary motor 704. Yes. FIG. 5 shows an example of spectral characteristics of these color filters 701 to 703.

  The rotation driving unit 105 rotates the rotation color filter 104 at a predetermined number of rotations in synchronization with the imaging period of the imaging element 304 based on a control signal from the control unit 420. For example, if the rotating color filter 104 is rotated 20 times per second, each color filter crosses incident white light at 1/60 second intervals. In this case, the image sensor 304 completes imaging and transfer of image signals at 1/60 second intervals.

  Here, the image sensor 304 is, for example, a monochrome image sensor. That is, in the present embodiment, frame sequential imaging is performed in which images of light of each of the three primary colors (R, G, or B) are captured at 1/60 second intervals.

  The operation unit 200 is provided with a bending operation lever 201 that performs a bending operation of the insertion unit 300 and an insertion port 202 into which a treatment instrument such as forceps is inserted.

  The bending operation lever 201 is connected to the bending operation wire 306. The bending operation wire 306 is inserted through the insertion portion 300 and fixed to the distal end of the insertion portion 300. The surgeon operates the bending operation lever 201 to pull or relax the bending operation wire 306 to bend the insertion portion 300. The insertion port 202 communicates with an insertion channel 307 through which the treatment tool is inserted.

  The insertion portion 300 is formed to be elongated and bendable, for example, to enable insertion into a body cavity. The insertion unit 300 diffuses the light guide fiber 301 for guiding the light collected by the light source unit 100 to the illumination lenses 314 and 315 and the light guided to the tip by the light guide fiber 301 to be an observation target. And illumination lenses 314 and 315 for irradiating. The illumination lens 314 is a lens for illuminating the observation object with the front visual field, and the illumination lens 315 is a lens for illuminating the observation object with the lateral visual field. The insertion unit 300 also includes an objective optical system 303 that collects reflected light returning from the observation target, an image sensor 304 that detects the collected reflected light, and an analog image obtained by photoelectric conversion of the image sensor 304. An A / D conversion unit 305 that converts the signal into a digital image signal. In addition, the insertion unit 300 includes a bending operation wire 306 that is inserted through the insertion unit 300 and is fixed to the distal end portion, and an insertion channel 307 through which the treatment instrument inserted by the operation unit 200 is inserted.

  The objective optical system 303 (object lens in a narrow sense) protrudes from the distal end portion of the insertion portion 300 and forms an image of the front field and the side field. For example, the objective optical system 303 has a viewing angle of 230 °.

  The imaging device 304 is, for example, a monochrome single-plate imaging device, and is configured by, for example, a CCD or a CMOS image sensor.

  The control device 400 controls each part of the endoscope device and performs image processing. The control device 400 includes an image processing unit 410 that performs setting of a region of interest and photometry processing, and a control unit 420.

  The image signal converted into a digital signal by the A / D conversion unit 305 is transferred to the image processing unit 410. The image signal processed by the image processing unit 410 is transferred to the display unit 500. The control unit 420 controls each unit of the endoscope apparatus. Specifically, the control unit 420 is connected to the light source aperture driving unit 103, the rotation driving unit 105, the image processing unit 410, and the external I / F unit 550, and performs these controls.

  The display unit 500 is a display device that can display a moving image, and includes, for example, a CRT or a liquid crystal monitor.

  The external I / F unit 550 is an interface for performing input from the user to the endoscope apparatus. The external I / F unit 550 includes, for example, a power switch for turning on / off the power, a mode switching button for switching a photographing mode and other various modes. The external I / F unit 550 transfers the input information to the control unit 420.

2.2. Image Processing Unit FIG. 6 shows a detailed configuration example of the image processing unit 410 in the first embodiment. The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a post-processing unit 415, a photometry unit 416, and a gradation conversion unit 419.

  The A / D conversion unit 305 is connected to the preprocessing unit 411. The preprocessing unit 411 is connected to the synchronization unit 412. The synchronization unit 412 is connected to the gradation conversion unit 419, the attention area setting unit 413, and the photometry unit 416. The tone conversion unit 419 is connected to the post-processing unit 415. The post-processing unit 415 is connected to the display unit 500. The attention area setting unit 413 is connected to the photometry unit 416. The control unit 420 is bidirectionally connected to the preprocessing unit 411, the synchronization unit 412, the gradation conversion unit 419, the post-processing unit 415, the attention area setting unit 413, and the photometry unit 416. These are designed to control.

  The preprocessing unit 411 uses the OB clamp value, the WB coefficient value, and the noise reduction coefficient value stored in advance in the control unit 420 for the image signal input from the A / D conversion unit 305 to perform OB clamp processing. , WB correction processing and noise reduction processing are performed. The preprocessing unit 411 transfers the preprocessed image signal to the synchronization unit 412.

  The synchronizer 412 synchronizes the frame sequential image signal with the image signal processed by the preprocessor 411 based on the control signal of the controller 420. Specifically, the synchronization unit 412 accumulates image signals of each color light (R, G, or B) input in frame order for each frame, and simultaneously reads the accumulated image signals of each color light. The synchronization unit 412 transfers the synchronized image signal to the attention area setting unit 413 and the photometry unit 416.

  The tone conversion unit 419 performs tone conversion processing on the synchronized image. The gradation conversion process is a process for flattening a histogram of pixel values of the entire image, for example. Specifically, the gradation conversion unit 419 divides an image into a plurality of areas, obtains gradation conversion characteristics of each divided area, and performs gradation conversion of each divided area based on the gradation conversion characteristics. The processing performed by the tone conversion unit 419 will be described later in detail.

  The attention area setting section 413 sets the attention area based on the image signal subjected to the synchronization processing. Specifically, the attention area setting unit 413 detects an area where the treatment instrument is imaged, and sets a peripheral area at the tip of the treatment instrument as the attention area. The processing performed by the attention area setting unit 413 will be described in detail later.

  The photometric unit 416 calculates the average luminance value (brightness information in a broad sense) of the subject in the set attention area, and transfers the calculated average luminance value to the control unit 420. The control unit 420 adjusts the aperture area of the light source stop 102 by controlling the light source stop drive unit 103 based on the average luminance value from the photometry unit 416. The light control performed by the photometry unit 416 and the control unit 420 will be described later in detail.

  The post-processing unit 415 performs color processing, contour enhancement processing, and enlargement processing on the image after the gradation conversion processing using the color conversion coefficient, the contour enhancement coefficient, and the enlargement factor that are stored in the control unit 420 in advance. Do. The post-processing unit 415 transfers the post-processed image signal to the display unit 500.

2.3. Gradation Conversion Unit Processing performed by the gradation conversion unit 419 will be described in detail. FIG. 7 shows a detailed configuration example of the gradation conversion unit 419. The gradation conversion unit 419 includes an area dividing unit 445, a histogram calculation unit 446, a gradation conversion characteristic calculation unit 447, and a conversion unit 448.

  The image signal from the synchronization unit 412 is input to the region division unit 445. The area dividing unit 445 is connected to the histogram calculating unit 446 and the converting unit 448. The histogram calculation unit 446 is connected to the gradation conversion characteristic calculation unit 447. The gradation conversion characteristic calculation unit 447 is connected to the conversion unit 448. The conversion unit 448 is connected to the post-processing unit 415. The control unit 420 is connected to the region dividing unit 445, the histogram calculation unit 446, the gradation conversion characteristic calculation unit 447, and the conversion unit 448, and performs these controls.

  The gradation conversion unit 419 performs gradation conversion processing for each area of the image signal. Specifically, the region dividing unit 445 divides the input image signal (maximum value image signal) into local regions. For example, the area dividing unit 445 divides the image signal into rectangular areas of a predetermined size, and sets each divided rectangular area as a local area. For example, the size of the rectangular area is 16 × 16 pixels, but is not limited thereto, and can be set as appropriate. The area dividing unit 445 outputs the divided local areas to the histogram calculating unit 446 and the converting unit 448. Note that the region dividing method is not limited to the above method. For example, a known area dividing method represented by texture analysis may be applied to divide the area.

  The histogram calculation unit 446 calculates a histogram (pixel value histogram) of each input local region, and outputs the calculated histogram of each local region to the gradation conversion characteristic calculation unit 447.

  The gradation conversion characteristic calculation unit 447 calculates a cumulative histogram of local regions based on the input local region histogram. In the following, processing for one local region will be described, but the same processing is performed for each local region. The gradation conversion characteristic calculation unit 447 normalizes the accumulated histogram so that the maximum value of the obtained accumulated histogram matches the output gradation width of the conversion unit 448. The gradation conversion characteristic calculation unit 447 outputs the normalized cumulative histogram to the conversion unit 448 as the local area gradation conversion characteristic.

  The conversion unit 448 performs gradation conversion processing by applying the gradation conversion characteristic input from the gradation conversion characteristic calculation unit 447 to the local region input from the region dividing unit 445. The conversion unit 448 returns each of the gradation-converted local regions to an image signal for one frame, and outputs the image signal to the post-processing unit 415.

2.4. Attention Area Setting Unit The process performed by the attention area setting unit 413 will be described in detail. The attention area setting unit 413 detects a treatment instrument area composed of pixels having an image of the treatment instrument from the image signal subjected to the synchronization processing, and sets the attention area based on the detected treatment instrument area. Hereinafter, a case where a highly reflective metal treatment tool such as forceps is detected will be described as an example.

Since the treatment tool is placed at a position very close to the image sensor 304, the luminance signal value of the treatment tool is sufficiently larger than that of the organ. Therefore, a high luminance area in the image signal is detected as a treatment instrument area. Specifically, the luminance signal value Y (x, y) of the pixel of interest is calculated by the following equation (1), where the coordinate of the pixel of interest (processing target pixel) is (x, y).

  Here, R (x, y), G (x, y), and B (x, y) are image signals of respective colors at coordinates (x, y).

Next, as shown in FIG. 8, the average value Yave (x, y) of luminance values from the coordinates (x−a, y) to the coordinates (x−1, y) on the left side of the target pixel is expressed by the following formula ( 2). In FIG. 8, the image signal is assumed to be N × M pixels. Also, the coordinate located at the upper left of the image signal is (0, 0), the right direction is the positive direction of the X axis, and the lower direction is the positive direction of the Y axis. For example, the X axis is an axis along the horizontal scanning line, and the Y axis is an axis orthogonal to the X axis.

  Here, a is a constant and is set according to the horizontal width N of the image signal. For example, the constant a is set to 3% of the horizontal width N of the image signal.

Next, it is detected whether the luminance signal value of the target pixel is sufficiently larger than the peripheral average luminance value using the following equation (3).

  Here, Yp is a value preset as a parameter. A pixel for which the above expression (3) is satisfied is a treatment tool candidate pixel. For example, as shown in FIG. 9A, in the case of an image signal including a treatment tool and an image of a bright spot, a treatment tool candidate pixel is detected as shown in FIG. 9B.

  Next, an area where a plurality of treatment tool candidate pixels are adjacent is extracted as a treatment tool candidate area. Specifically, the image signal is searched from the upper left, the value of the target pixel (x, y) is “treatment tool candidate pixel”, and the left (x−1, y) and upper left (x−1) of the target pixel. , Y−1), upper (x, y−1), and upper right (x + 1, y−1), the pixel of interest “not a treatment tool candidate pixel” is set as the start point pixel. As shown in FIG. 10, the hatched pixel corresponds to the start point pixel.

  Next, the treatment tool candidate pixel is searched counterclockwise from the lower left (x−1, y−1) of the start point pixel (x, y). When the “treatment tool candidate pixel is not detected” around the start point pixel, the treatment tool candidate pixel is searched from the next start point pixel. When the “treatment tool candidate pixel is detected” around the start point pixel, the treatment tool candidate pixel is searched again counterclockwise from the periphery of the detected treatment tool candidate pixel. The search is continued until the detected treatment tool candidate pixel returns to the start point pixel again. If the Y coordinate of the treatment tool candidate pixel becomes smaller than the start point pixel during the search, the search is terminated and the next start point pixel is searched. As shown in FIG. 11, when the treatment tool candidate pixel returns to the start point pixel again, a region surrounded by the detected treatment tool candidate pixels is set as a treatment tool candidate region. In FIG. 11, hatched pixels are treatment tool candidate pixels detected by the search.

Next, the number of pixels included in each region extracted as the treatment tool candidate region is counted, and the region having the largest number of pixels is extracted. When the number of pixels T _max included in the region having the largest number of pixels is larger than a preset threshold value TH _t (T _max > TH _t ), the region is set as a treatment tool region. On the other hand, if the number of pixels T _max included in the region having the largest number of pixels is equal to or less than a preset threshold value TH _t , the treatment tool region is set in the image signal as “there is no treatment tool” in the image signal. do not do. For example, as shown in FIG. 11, when the number of pixels included in the treatment tool candidate area 2 is the largest, it is determined whether or not the number of treatment tool candidate pixels included in the treatment tool candidate area 2 is larger than a preset threshold value. judge. When larger than the threshold value, the treatment tool candidate area 2 is set as the treatment tool area.

Next, an area corresponding to the distal end of the treatment instrument is extracted from the treatment instrument area, and the area is set as the treatment instrument distal end pixel. For example, as shown in FIG. 12, a pixel closest to the center of the image is extracted from the pixels included in the treatment instrument region as the treatment instrument tip pixel. In FIG. 12, the coordinates of the image center are (N / 2, M / 2), and the coordinates of the treatment instrument tip pixel are (x ₀ , y ₀ ). As another method for extracting the treatment instrument tip pixel, a plurality of pixels close to the coordinates (N / 2, M / 2) of the image center are selected from the pixels included in the treatment instrument region and selected. The center of gravity of the plurality of pixels may be used as the treatment instrument tip pixel.

Next, as shown in FIG. 13, a pixel inside a circle with a radius R centering on the treatment instrument tip pixel (x ₀ , y ₀ ) is set as a region of interest. Here, the radius R is a constant and is set according to the horizontal width N of the image signal. For example, the radius R is set to 10% of the horizontal width N of the image signal. Whether or not the pixel at the coordinates (x, y) is included inside the circle with the radius R is determined by the following equation (4).

The attention area setting section 413 transfers the set attention area and the central pixel (x ₀ , y ₀ ) to the photometry section 416.

In the above description, the radius R is a constant. However, the present embodiment is not limited to this, and for example, the radius R is determined between the region of interest (x ₀ , y ₀ ) and the image center (N / 2, M / 2). It is good also as a variable which changes according to the distance L. A distance L between the center (x ₀ , y ₀ ) of the attention area and the image center (N / 2, M / 2) is calculated by the following equation (5).

In the above description, the attention area is a circle centered on the coordinates (x ₀ , y ₀ ). However, the present embodiment is not limited to this. For example, the attention area is centered on the coordinates (x ₀ , y ₀ ). It may be a rectangle or an ellipse.

2.5. Dimming Control Next, the dimming control performed by the photometry unit 416 and the control unit 420 will be described in detail.

  The photometry unit 416 calculates the brightness of the subject from the image signal based on the control signal of the control unit 420. Specifically, when the attention area is set by the attention area setting section 413, the photometry section 416 calculates the average luminance value of the attention area from the image signal input from the synchronization section 412. On the other hand, when the attention area is not set by the attention area setting section 413, the photometry section 416 calculates the average brightness value of the effective pixel area of the image signal input from the synchronization section 412. The photometric unit 416 transfers the calculated average brightness value to the control unit 420 as the photometric value Yd.

  The method for calculating the brightness of the subject is not limited to the above method. For example, the luminance average value of the attention area and the luminance average value of the area other than the attention area are multiplied by different weighting factors, and the weighted average value is calculated. It may be calculated as the brightness of the subject. In this case, a larger coefficient is set as the weighting coefficient of the attention area than the weighting coefficients other than the attention area.

The control unit 420 calculates the light source aperture adjustment coefficient Lc by the following equation (6) using the photometric value Yd calculated by the photometric unit 416 and the preset target brightness Ybase.

  The control unit 420 controls the aperture of the light source aperture 102 by controlling the light source aperture driving unit 103 using the calculated light source aperture adjustment coefficient Lc.

  By performing the above processing, when a treatment tool is inserted, the brightness of a lesioned part existing around the treatment tool can be maintained at an optimum brightness. Thereby, the observation performance of a lesioned part can be improved and an image suitable for treatment can be obtained.

2.6. Modification In this embodiment, a scaling process for enlarging the attention area set as described above may be performed. Below, this modification is demonstrated in detail.

  FIG. 14 shows a modified configuration example of the image processing unit 410 when the scaling process is performed. The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a scaling processing unit 414, a post-processing unit 415, a photometry unit 416, and a gradation conversion unit 419. . In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 6 etc., and description is abbreviate | omitted suitably.

  The scaling processing unit 414 performs image scaling processing while maintaining (substantially maintaining) the angle of view, and enlarges the region of interest. Specifically, the scaling processing unit 414 performs a process of expanding the attention area and reducing other areas. Alternatively, only the process of enlarging the attention area at a predetermined magnification may be performed.

Specifically, the scaling processing unit 414 locally scales the image signal based on the set attention area and the center pixel (x ₀ , y ₀ ). When the attention area is not set by the attention area setting unit 413, the scaling process is not performed, and the input image signal is output to the post-processing unit 415 as it is.

As illustrated in FIG. 15, the scaling processing unit 414 enlarges the region of interest at an enlargement ratio α (magnification in a broad sense) and superimposes the enlarged image on the original image. Specifically, the center of the circle is enlarged in the radial direction at an enlargement ratio α, and the center of the enlarged circle area is overlapped with the center pixel (x ₀ , y ₀ ) of the original attention area. Hereinafter, the attention area after enlargement is referred to as an enlargement area. A coordinate (x ′, y ′) obtained by converting the pixel of the coordinate (x, y) by α times is calculated by the following equation (7).

Here, α is a preset constant larger than 1, and is set to α = 1.5, for example. Further, the enlargement ratio α is not limited to a fixed constant, and may be freely input by an operator from the outside. As shown in FIG. 14, the enlargement ratio α may be a variable that changes according to the distance L between the center (x ₀ , y ₀ ) of the region of interest and the image center (N / 2, M / 2). .

  The scaling processing unit 414 generates an interpolation pixel at a pixel position included in the enlargement area in the enlargement process. Specifically, linear interpolation is performed on the pixel positions in the enlarged region using the four neighboring pixels after the coordinate conversion by the above equation (7). The interpolation processing is not limited to linear interpolation, and for example, an interpolation method such as a nearest neighbor method or a cubic convolution interpolation method (bicubic) may be used.

The scaling processing unit 414 provides a reduced area outside the above-described enlarged area, and performs a process of reducing and converting the reduced area. Specifically, the reduced area is an area inside the radius R _s with the coordinates (x ₀ , y ₀ ) as the center, and is an area outside the enlarged area. Scaling section 414, reduced by the circle and the radius R _s reduction rate region between a circle of radius R beta (magnification in a broad sense). The reduction ratio β is a variable that changes according to the distance rd between the processing target pixel (x, y) and the center coordinates (x ₀ , y ₀ ). The distance rd is calculated by the following equation (8). Reduction factor beta is the distance rd is a coefficient that decreases the larger the distance rd is when R beta = alpha becomes, the beta = 1 when R _s.

  The scaling processing unit 414 generates an interpolation pixel at a pixel position included in the reduced area in the reduction process. Specifically, linear interpolation is performed on the pixel position in the reduced region using the pixels that have been reduced at the reduction rate β. The interpolation processing is not limited to linear interpolation, and for example, an interpolation method such as a nearest neighbor method or a cubic convolution interpolation method (bicubic) may be used.

  According to the above modification, since the attention area can be enlarged while maintaining the angle of view, the visibility of the attention area can be improved while ensuring a wide field of view necessary for scope operation and the like. In addition, by providing the reduced area outside the enlarged area, the continuity of the image is maintained at the boundary portion of the enlarged area, so that it is possible to obtain an image signal that has been subjected to the scaling process without any sense of incongruity.

In the above has been the pixels included the reduction area than the inner radius R _s, the embodiment is not limited thereto, for example, all outer expansion region may be reduced region. In that case, the reduction ratio β decreases toward the edge of the image, and the enlargement ratio is 1 at the edge of the image.

  In the above description, the reduced area is provided. However, the present embodiment is not limited to this. For example, only the enlarged area may be provided and enlarged at the magnification α, and the enlarged area may be superimposed on the original image.

  According to the above embodiment, as shown in FIG. 6, the endoscope apparatus includes an image acquisition unit (for example, A / D conversion unit 305), an attention area setting unit 413, and a dimming control unit (in a broad sense, a light amount). Control unit). The image acquisition unit acquires a captured image including a subject image. The attention area setting unit 413 sets an attention area for the captured image based on information from the endoscope apparatus. The dimming control unit performs dimming control of the amount of illumination light based on the set attention area.

  In this way, an attention area can be set based on information from the endoscope apparatus, and the brightness of the attention area can be controlled by dimming. Thereby, the surgeon can observe the region of interest with appropriate brightness, and can perform appropriate diagnosis and treatment.

  For example, in the present embodiment, the photometry unit 416 and the control unit 420 correspond to the dimming control unit. As described above, the photometry unit 416 performs photometry processing of the set attention area, and the control unit 420 controls the dimming control by controlling the amount of light emitted from the light source unit 100 based on the result of the photometry processing. Do.

  Here, the information from the endoscope apparatus is information acquired by each unit of the endoscope apparatus, and controls, for example, a captured image by the imaging unit, a signal obtained by processing the captured image, and each unit. These are control signals and sensing signals from various sensors provided in the endoscope apparatus.

  The attention area is an area where the priority of observation for the user is relatively higher than other areas. For example, if the user is a doctor and desires treatment, a mucous membrane part or a lesion part is copied. Refers to an area. As another example, if the object that the doctor desires to observe is a bubble or stool, the region of interest is a region in which the bubble or stool portion is copied. In other words, the object that the user should pay attention to depends on the purpose of observation, but in any case, in the observation, the region where the priority of observation for the user is relatively higher than other regions becomes the region of interest. .

  Further, in the present embodiment, as described in the above equation (6) and the like, the dimming control unit calculates brightness information (photometric value Yd, for example, luminance average value) indicating the brightness of the attention area. Dimming control is performed based on the brightness information.

  Specifically, the dimming control unit performs dimming control so that the brightness of the attention area becomes a predetermined brightness (target brightness information Ybase, for example, a target luminance value). For example, the dimming control unit performs dimming control so that the brightness of the attention area matches a predetermined brightness.

  In this way, the brightness of the attention area can be brought close to the target brightness by dimming control, and the attention area can be maintained at an appropriate constant brightness.

  In the present embodiment, the attention area setting unit 413 sets the attention area using the captured image as information from the endoscope apparatus.

  Specifically, the attention area setting unit 413 includes a treatment instrument detection unit (not shown). As described with reference to FIG. 13 and the like, the treatment instrument detection unit detects a treatment instrument region in which a treatment instrument for performing treatment on a subject is imaged based on the captured image. The attention area setting unit 413 sets an attention area based on the detected treatment instrument area.

  In this way, it is possible to maintain the periphery of the treatment tool, which is considered to be an area that the operator is paying attention, at an appropriate brightness. Thereby, since it becomes possible to treat a lesioned part without halation of a lesioned part when a surgeon performs treatment, the burden on the surgeon can be reduced. In addition, since the treatment instrument region is detected from the captured image, it is not necessary to provide a new treatment instrument detector in the insertion portion 300 of the endoscope.

  In this embodiment, the treatment instrument detection unit detects the tip of the treatment instrument region. The attention area setting unit 413 sets an attention area within a circle having a predetermined radius from the detected tip.

Specifically, as described with reference to FIG. 12, the treatment instrument detection unit detects the pixel closest to the center (N / 2, M / 2) of the captured image among the pixels included in the treatment instrument region (the treatment instrument). Set to the tool tip pixel (x ₀ , y ₀ )).

  In this way, it is possible to set a treatment target area that is considered to be present at the distal end of the treatment tool as the attention area, and to control the brightness of the attention area.

  Note that the circle (circular region) set as the attention region is not limited to a perfect circle, and may be a circular region. For example, an ellipse or the like may be used.

  In this embodiment, the treatment instrument detection unit includes a luminance feature amount calculation unit (not shown). The luminance feature amount calculation unit calculates a luminance feature amount (for example, luminance signal value Y (x, y)) related to the luminance of the pixel of the captured image. The treatment tool detection unit detects a treatment tool region based on the calculated luminance feature amount.

  Specifically, as described in FIG. 9B and the like, the treatment instrument detection unit detects a treatment instrument candidate pixel that is a candidate for the treatment instrument region based on the luminance feature amount, and detects the detected treatment instrument candidate pixel. The treatment tool region is determined based on the above.

  More specifically, the treatment instrument detection unit includes a comparison unit (not shown). As described with reference to FIG. 8 and the like, the comparison unit performs the luminance feature amount (luminance signal value Y (x, y)) of the processing target pixel and the average luminance feature amount (average value Yave (x) of the peripheral pixels of the processing target pixel. , Y)). The treatment instrument detection unit sets a pixel whose luminance feature amount is larger than the average luminance feature amount as a treatment instrument candidate pixel.

  Since the treatment tool is made of metal or the like and has a high reflectance of illumination light, the treatment tool is picked up as an image with higher brightness than other subjects. Therefore, the treatment tool can be detected from the image by setting the treatment tool candidate pixel based on the luminance feature amount.

  In this embodiment, as described with reference to FIG. 11 and the like, the treatment instrument detection unit determines that the treatment instrument candidate pixel is one or more treatment instrument candidate areas (for example, a treatment instrument candidate area) based on the position information of the treatment instrument candidate pixels. 1 and 2) (for example, by a classifying unit (not shown)). The treatment tool detection unit selects a treatment tool region from the one or more treatment tool candidate regions.

  Specifically, the treatment tool detection unit extracts a pixel that is a boundary between the treatment tool candidate region and another region from the treatment tool candidate pixels, and the treatment tool candidate pixel surrounded by the boundary is extracted from the treatment tool candidate region. 1 or a plurality of treatment tool candidate areas are set.

  The treatment instrument detection unit determines the treatment instrument area based on the number of treatment instrument candidate pixels included in each treatment instrument candidate area.

Specifically, the treatment tool detection unit is an area including the largest number of treatment tool candidate pixels in one or a plurality of treatment tool candidate areas, and has more treatment tool candidate pixels than a predetermined threshold (TH _t ). A region including the treatment tool candidate region 2 is determined as a treatment tool region.

  In this way, even when a plurality of treatment tool candidate areas are generated due to a high-luminance area such as a bright spot generated when the illumination light is reflected by the mucous membrane, the area where the treatment tool is imaged is selected from among the treatment tool candidate areas. You can choose.

  In the present embodiment, as shown in FIG. 14, the endoscope apparatus may include a scaling unit 414. The scaling processing unit 414 performs local scaling processing that enlarges the region of interest relatively than other regions.

  In this way, the region of interest can be enlarged relative to other regions, so that more detailed information can be obtained even for minute lesions, and appropriate diagnosis and treatment can be performed on the region of interest. Is possible.

  In this embodiment, as described with reference to FIG. 15 and the like, the scaling processing unit 414 performs local scaling processing while maintaining the angle of view of the captured image.

  In this way, the surgeon can observe the region of interest with an appropriate size while grasping the overall image of the subject. Further, since the entire image of the subject and the region to be focused can be displayed on the same image, the burden on the operator can be reduced as compared with the case where a plurality of images are observed. In addition, since it is not necessary to switch between the wide-angle image and the enlarged image, the operator does not need to perform complicated work.

  Here, maintaining the angle of view means that the range of the subject displayed in the image is not changed by the scaling process. The angle of view does not need to be strictly maintained, and may be maintained substantially.

  Moreover, in this embodiment, as shown in FIG. 3, the one light source part 100, the 1st irradiation part (illumination lens 314), and the 2nd irradiation part (illumination lens 315) are included. The first irradiating unit irradiates light emitted from one light source unit 100 in a first direction (for example, a direction along the optical axis of the objective optical system). The second irradiation unit irradiates light emitted from one light source unit 100 in a second direction (for example, a direction at a predetermined angle with respect to the optical axis of the objective optical system). The dimming control unit performs dimming control by controlling the amount of light emitted from one light source unit 100.

  Specifically, the first direction is the direction of the forward visual field of the scope (insertion unit 300), and the second direction is the direction of the lateral visual field of the scope. The image acquisition unit acquires a captured image including subject images of a front visual field and a side visual field.

  As described with reference to FIGS. 2A to 2D, when one light source unit 100 is shared as illumination for the front visual field and the side visual field, brightness unevenness occurs in the image. The attention area may not be appropriately dimmed. In this regard, according to the present embodiment, since the light control is performed based on the brightness of the attention area, the attention area can be adjusted to an appropriate brightness.

  In the present embodiment, as described with reference to FIGS. 1, 3, and the like, the image acquisition unit acquires a captured image in which the front visual field and the lateral visual field at the distal end of the scope (insertion unit 300) are captured.

  Specifically, the captured image is an image obtained by an objective optical system that forms an image of the front field and the side field.

  The objective optical system has a viewing angle greater than 180 ° (eg, a viewing angle of 230 °).

  This makes it possible to observe the subject on the side of the scope simultaneously with the observation of the subject in front of the scope, as described with reference to FIG. Thereby, oversight of the lesioned part of the wall surface in the tubular organ can be prevented.

  Here, the objective optical system is not limited to the objective lens but may be a reflective optical system, or may be an optical system in which a lens and a reflective optical system are combined as shown in FIG. In addition, it is not limited to the objective optical system of FIG. 1, You may use the fisheye lens which a viewing angle exceeds 180 degrees.

  The front visual field (front visual field range) is a visual field range including the optical axis direction of the objective optical system, and is, for example, a range of 0 to 45 degrees with respect to the optical axis. A side view (side view range) is a view range including a direction orthogonal to the optical axis, and is, for example, a range of 45 degrees to 135 degrees with respect to the optical axis. The objective optical system 303 of this embodiment has a visual field range of 0 degree to 115 degrees with respect to the optical axis, for example.

3. Second Embodiment A second embodiment that senses insertion of a treatment tool and sets a region of interest will be described.

  FIG. 16 shows a configuration example of the endoscope apparatus according to the second embodiment. The endoscope apparatus includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The insertion unit 300 includes a treatment instrument sensor 308. The treatment instrument sensor 308 is provided in the insertion channel 307. The treatment instrument sensor 308 detects a treatment instrument that has passed through the insertion channel 307 and protruded forward of the insertion portion 300. The treatment instrument sensor 308 is connected to a state information acquisition unit 430 described later.

  The control device 400 includes a state information acquisition unit 430. The state information acquisition unit 430 is connected to the image processing unit 410. In addition, the control unit 420 is bidirectionally connected to the state information acquisition unit 430.

  When the treatment instrument sensor 308 detects that the treatment instrument protrudes from the distal end of the insertion section 300, the state information acquisition unit 430 sends a control signal indicating that “the treatment instrument is present in the image signal” to the image processing unit. Forward to 410.

  FIG. 17 shows a detailed configuration example of the image processing unit 410 in the second embodiment. The state information acquisition unit 430 is connected to the attention area setting unit 413.

  In the second embodiment, the processing performed by the attention area setting unit 413 is different from that in the first embodiment. Specifically, the attention area setting unit 413 sets the attention area in the image signal when receiving the control signal indicating that “the treatment tool is present in the image signal” from the state information acquisition unit 430. For example, as shown in FIG. 18, an image center (N / 2, M / 2) is set as the center of the attention area, and an area within a radius R from the center of the attention area is set as the attention area. The attention area setting section 413 transfers the set attention area to the photometry section 416.

  The center of the attention area is not limited to the image center (N / 2, M / 2), and may be set in advance according to the relative positional relationship between the image sensor and the insertion channel. For example, when the insertion channel is provided in the Y axis positive direction when viewed from the center of the image, the region of interest may be set on the Y axis positive direction side of the image center.

  According to the above embodiment, as shown in FIG. 16, the endoscope apparatus includes the state information acquisition unit 430 that acquires the state information of the endoscope apparatus. The attention area setting unit 413 sets the attention area using the state information as information from the endoscope apparatus.

  Specifically, the state information acquisition unit 430 detects whether or not a treatment tool for performing treatment on the subject protrudes from the distal end of the scope (detected based on the sensing signal of the treatment tool sensor 308), The detection result is acquired as state information. The attention area setting unit 413 sets the attention area based on the detection result.

  More specifically, as described with reference to FIG. 18 and the like, the attention area setting unit 413 inserts the treatment tool from the center area of the captured image or the center of the captured image when the protrusion of the treatment tool is detected. A region on the side (for example, a region on the Y axis positive direction side of the image center) is set as a region of interest.

  In this way, when the surgeon operates the treatment tool, the attention area can be set, and the brightness of the attention area can be dimmed. Further, the image center or the lower part of the image where the surgeon normally positions the lesion can be set as the attention area. As a result, it becomes possible to display the lesion area to be treated with appropriate brightness regardless of the image quality of the image signal, and an image suitable for the surgeon's treatment can be obtained.

  Here, the state information of the endoscope apparatus is information representing the state of each part of the endoscope apparatus. For example, information such as a traveling direction of a scope obtained by processing a captured image and the state of each part are controlled. And a sensing signal by various sensors that sense the state of each part.

4). Third Embodiment 4.1. Endoscope Device A third embodiment in which a region of interest is set based on a bending angle or a bending movement amount of a scope will be described.

  FIG. 19 shows a configuration example of an endoscope apparatus according to the third embodiment. The endoscope apparatus includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The light source unit 100 includes a white light source 101, a light source aperture 102, a light source aperture drive unit 103, and a condenser lens 106.

  The operation unit 200 includes a bending operation lever 201 that performs a bending operation of the insertion unit 300, an insertion port 202 into which a treatment instrument such as a forceps is inserted, and operation information of a bending operation such as a bending angle and a bending movement amount of a distal end portion And an operation amount detection sensor 203 for detecting. The operation amount detection sensor 203 is connected to the state information acquisition unit 430 and transmits operation information of the bending operation to the state information acquisition unit 430.

  The imaging device 304 of the insertion unit 300 is a Bayer-type array imaging device, and is configured by, for example, a CCD or a CMOS image sensor.

  The control device 400 includes an image processing unit 410, a control unit 420, and a state information acquisition unit 430. The components of the image processing unit 410 are the same as those in the configuration example described above with reference to FIG.

  The state information acquisition unit 430 acquires information related to the operation of the endoscope in the operation unit 200. Specifically, the state information acquisition unit 430 acquires at least one of the bending angle and the bending movement amount of the scope tip as state information.

4.2. Bending Angle Detection Unit FIG. 20 shows a detailed configuration example of the state information acquisition unit 430 in the third embodiment. The state information acquisition unit 430 includes at least one of a bending angle detection unit 431 and a bending movement amount detection unit 432.

  The bending angle detection unit 431 detects the current bending angle of the insertion unit 300 that has been bent by the operation unit 200. The bending movement amount detection unit 432 detects the bending movement amount of the insertion unit 300 that is bent by the operation unit 200. The bending movement amount is a change amount per unit time of the bending angle.

  Specifically, the bending operation lever 201 is of a dial type, and when the operator rotates the dial, the bending operation wire 306 connected to the dial is pulled or pushed and inserted accordingly. The tip of the part 300 is curved. The operation amount detection sensor 203 detects the “length” in which the bending operation wire 306 is pulled or pushed as an operation amount. The state information acquisition unit 430 obtains a bending angle from the detected operation amount. For example, the state information acquisition unit 430 obtains the bending angle from the operation amount using a lookup table, a polynomial, or the like.

  For example, as shown in FIG. 21A, when the operation amount of the wire 306 is zero, the bending angle θ = 0 degrees. As shown in FIG. 21B, when the wire 306 is pulled LWcm, the bending angle θ is 45 degrees. As shown in FIG. 21C, when the wire 306 is pushed 2 LW cm, the bending angle θ is −90 degrees.

  The bending angle only needs to be obtained as information representing the bending angle, and does not have to be the bending angle itself. For example, the bending angle information may be the length by which the wire 306 is operated. That is, the operation amount in the rotation direction of the bending operation lever 201 and the length of the bending operation wire 306 being pulled or pushed are 1: 1, and if it is determined, the bending angle is also obtained.

  Further, the operation amount detected by the operation amount detection sensor 203 is not limited to the length, and may be, for example, the operation amount in the rotation direction of the bending operation lever 201. Also in this case, the bending angle information may be a bending angle obtained from the rotational operation amount or a rotational operation amount.

  As described above with reference to FIG. 2A and the like, when screening is performed using a wide-field endoscope, a lesion on the back of a fold can be found with a lateral visual field. When the surgeon finds the lesioned part, the front visual field is easier to observe. Therefore, the bending operation lever 201 is operated to bend the insertion part 300 so that the lesioned part enters the front visual field.

  Therefore, in the present embodiment, the attention area setting unit 413 sets the region of the side field of view as the attention area according to the bending angle acquired by the bending angle detection unit 431. For example, when the bending angle is equal to or greater than a certain threshold value, the lateral visual field region is set as the attention region.

4.3. Next, the processing performed by the bending motion amount detection unit 432 will be described in detail. The operation amount detection sensor 203 detects the rotation operation amount per unit time of the bending operation lever 201. The bending motion amount detection unit 432 obtains a bending motion amount from the detected rotational operation amount.

  Note that the present embodiment is not limited to this, and the bending motion amount may be obtained based on the length of the bending operation wire 306 that is pulled or pushed per unit time. Further, the amount of bending motion is not limited to the amount of change in the bending angle per unit time, but may be obtained as information representing the amount of bending motion. For example, the bending movement amount information may be a rotation operation amount per unit time of the bending operation lever 201 or a length that the bending operation wire 306 is pulled or pushed per unit time. .

  As described above with reference to FIG. 2A and the like, when screening is performed using a wide-field endoscope, a lesion on the back of a fold can be found with a lateral visual field. When the surgeon finds the lesioned part, the front visual field is easier to observe. Therefore, the bending operation lever 201 is operated to bend the insertion part 300 so that the lesioned part enters the front visual field. That is, it is considered that there is a lesion in the lateral visual field during the bending operation. When the lesion is in the anterior visual field, stop the curving operation.

  Therefore, in this embodiment, the side field of view is set as a region of interest when the curve starts, and the front field of view is set as the region of interest when the curve stops. When the bending angle stops at the maximum angle, it can be assumed that the region of interest of the operator is in the side view without entering the front view, so the side view is set as the view region.

  Specifically, the attention area setting unit 413 sets the attention area according to the bending motion amount acquired by the bending motion amount detection unit 432. The attention area setting unit 413 sets the area of the side field of view as the attention area when the amount of bending movement is large (when it is bent sharply). For example, the region of interest is set when the amount of bending motion is greater than or equal to a threshold value.

4.4. Image Processing Unit Next, the image processing unit 410 that performs attention area setting processing and photometry processing will be described in detail. FIG. 22 shows a detailed configuration example of the image processing unit 410 according to the third embodiment. The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a post-processing unit 415, a photometry unit 416, and a gradation conversion unit 419. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 6 etc., and description is abbreviate | omitted suitably.

  The state information acquisition unit 430 is connected to the attention area setting unit 413 and the photometry unit 416. The synchronization unit 412 is connected to the gradation conversion unit 419 and the photometry unit 416.

  The synchronization unit 412 converts the Bayer-type single-plate image into a three-plate image (RGB image) by interpolation processing. The synchronization unit 412 transfers the three-plate image signal to the gradation conversion unit 419 and the photometry unit 416.

  The attention area setting section 413 sets the attention area according to the bending angle acquired by the bending angle detection section 431 and the bending movement amount acquired by the bending movement amount detection section 432. Specifically, the attention area setting unit 413 determines the region of the lateral visual field as the attention area when the bending angle is equal to or larger than a predetermined threshold value or when the bending motion amount (curving speed) is equal to or larger than the predetermined threshold value. Set as. The attention area setting section 413 transfers the set attention area to the photometry section 416.

The photometry unit 416 calculates the brightness of the subject from the image signal. Specifically, when the attention area is set by the attention area setting section 413, the photometry section 416 multiplies the attention area and the area other than the attention area by different weighting factors, and performs weighted addition represented by the following equation (9). An average value Ywa is calculated.

  Here, Yc (x, y) is a luminance signal value of a pixel included in the front visual field, and Ya (x, y) is a luminance signal value of a pixel included in the lateral visual field. Nc is the number of pixels included in the front view, and Na is the number of pixels included in the side view. A is a weighting factor for the front field of view, and B is a weighting factor for the side field of view. Rc is a region corresponding to the front visual field, and Ra is a region corresponding to the lateral visual field. The sum (Σ) is the sum for the pixels belonging to each region.

  The photometry unit 416 sets the weighting coefficient of the attention area to a coefficient that is relatively larger than the weighting coefficients other than the attention area. For example, when the side area is the attention area, A <B. Further, the photometry unit 416 sets the weighting coefficient according to the bending angle (or bending movement amount). Specifically, the larger the bending angle (or the amount of bending movement), the larger the side view weighting coefficient B compared to the front viewing weighting coefficient A.

  When the attention area is not set by the attention area setting section 413, the photometry section 416 calculates the average luminance value of the effective pixel area of the image signal input from the synchronization section 412. The photometry unit 416 transfers the calculated luminance addition average value Ywa of the attention area to the control unit 420.

  As described in the above equation (9), the dimming control unit includes the first brightness information (luminance signal value Ya (x, y)) indicating the brightness of the attention area and the brightness of the area other than the attention area. 2nd brightness information (luminance signal value Yc (x, y)) showing the brightness is calculated. The dimming control unit weights and adds the first brightness information and the second brightness information with the first weighting factor B and the second weighting factor A, respectively, and adjusts the brightness based on the obtained weighted addition value. Perform light control. The dimming control unit uses a coefficient larger than the second weighting coefficient A as the first weighting coefficient B.

  In this way, the brightness of the entire image can be controlled by dimming while mainly controlling the brightness of the attention area by dimming. Thereby, the visibility of the whole image can be improved while improving the visibility of the attention area.

  According to the above embodiment, as shown in FIG. 20, the state information acquisition unit 430 includes the bending angle detection unit 431. The bending angle detection unit 431 acquires bending angle information representing the bending angle of the scope tip (angle θ in FIGS. 21A to 21C) as state information. The attention area setting unit 413 sets the attention area based on the acquired bending angle information.

  In this way, an observation region in which the operator is interested can be estimated from the bending angle, and an image that can be easily observed by the operator can be provided by dimming control of the brightness of the region.

  In the present embodiment, the attention area setting unit 413 sets, as the attention area, an area corresponding to the side field of view in the captured image when it is determined that the bending angle is larger than the threshold value.

  In this way, when there is a lesion in the lateral visual field, the operator is assumed to perform a bending operation on the distal end of the scope, so that the lateral visual field can be set as a region of interest. A slight bending operation with a small bending angle (below the threshold) can be estimated as an error or a fine adjustment of the front visual field. Therefore, unnecessary thresholding of the photometric target area can be prevented by performing threshold determination.

  Here, the region corresponding to the front visual field is a region where a subject in the front visual field is imaged, for example, a region where a subject in the optical axis direction of the objective optical system is imaged. For example, when the optical axis coincides with the image center, the region corresponding to the front visual field is a central region including the image center.

  Further, the region corresponding to the side field is a region where the subject in the side field is imaged, for example, a region where the subject in the direction orthogonal to the optical axis of the objective optical system is imaged. For example, when the optical axis coincides with the center of the image, the region corresponding to the side field of view is a peripheral region of the central region.

  In the present embodiment, the attention area setting unit 413 sets an area corresponding to the side field of view in the captured image as the attention area. As described in the above equation (9), the dimming control unit makes the first weighting factor B larger than the second weighting factor A as the bending angle increases.

  In this way, it can be assumed that the greater the angle of curvature, the more the operator is paying attention to the side field of view, so by performing photometric processing with a larger weighting factor for the region, an image that is easy for the operator to observe is obtained. Can be provided.

  In the present embodiment, as illustrated in FIG. 20, the state information acquisition unit 430 includes a bending motion amount detection unit 432. The bending motion amount detection unit 432 acquires, as state information, bending motion amount information that represents the bending motion amount of the scope tip (change in angle θ in FIGS. 21A to 21C per unit time). The attention area setting unit 413 sets the attention area based on the acquired bending motion amount information.

  In this way, an observation region of interest to the operator can be estimated from the amount of bending motion, and an image that can be easily observed by the operator can be provided by dimming control of the brightness of the region.

  In the present embodiment, the attention area setting unit sets, as the attention area, an area corresponding to the lateral visual field in the captured image when it is determined that the amount of bending motion is larger than the threshold value.

  In this way, when there is a lesion in the lateral visual field, the operator is assumed to perform a bending operation on the distal end of the scope, so that the lateral visual field can be set as a region of interest. A slight bending operation with a small bending motion amount (below the threshold value) can be estimated as an error or a fine adjustment of the front visual field. Therefore, unnecessary thresholding of the photometric target area can be prevented by performing threshold determination.

  In the present embodiment, the attention area setting unit 413 sets an area corresponding to the side field of view in the captured image as the attention area. As described in the above equation (9), the dimming control unit makes the first weighting factor B larger than the second weighting factor A as the bending motion amount increases.

  In this way, it can be assumed that the greater the amount of curve movement, the more the operator is paying attention to the lateral field of view. Can be provided.

5. Fourth embodiment 5.1. Endoscope Device A fourth embodiment in which a region of interest is set based on distance information to a subject estimated from the amount of light emitted from a light source will be described.

  FIG. 23 shows a configuration example of an endoscope apparatus according to the fourth embodiment. The endoscope apparatus includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The control device 400 includes an image processing unit 410, a control unit 420, and a state information acquisition unit 430. The state information acquisition unit 430 is connected to the image processing unit 410. The control unit 420 is connected to the light source aperture driving unit 103, the rotation driving unit 105, the image processing unit 410, the state information acquisition unit 430, and the external I / F unit 550, and performs these controls.

  In the present embodiment, information on the amount of light emitted from the light source unit 100 is acquired in a state where the dimming process is in operation, distance information on the subject is estimated based on the information on the amount of emitted light, and based on the distance information. Adaptively controls the photometric processing of the attention area and other areas.

5.2. State Information Acquisition Unit A method for estimating a distance to a subject and setting a region of interest based on the distance will be described in detail.

  When the surgeon finds a region that is assumed to be a lesion in observation with the endoscope apparatus, the surgeon brings the distal end of the insertion portion 300 close to the region. In this case, the amount of reflected light from the subject becomes strong, and the captured image signal becomes bright. In the present embodiment, the state information acquisition unit 430 performs dimming processing, controls the light source diaphragm driving unit 103 via the control unit 420, and reduces the aperture area of the light source diaphragm 102 until the image reaches the target observation brightness. To do.

  On the other hand, in a state where the distal end of the insertion unit 300 is away from the observation area, the amount of reflected light from the subject is weak and the captured image signal is dark. The state information acquisition unit 430 controls the light source aperture driving unit 103 via the control unit 420 and expands the aperture area of the light source aperture 102 until the image reaches the target observation brightness. That is, in the dimming process, when the opening area of the light source stop 102 is small, the tip of the insertion unit 300 is close to the subject, and when the opening area of the light source stop 102 is large, the tip of the insertion unit 300 is The subject is away.

  In this embodiment, in order to observe a wider range of a subject, objective optics that can observe a lateral (including substantially lateral) object simultaneously with the observation of the front object as described above with reference to FIG. Use the system. Therefore, as described above with reference to FIG. 2B, the front field of view (center part) and the side field of view (peripheral part) centered on the coordinates (N / 2, M / 2) of the image center are displayed.

  As shown in FIGS. 24A and 24B, when the surgeon finds a region that is assumed to be a lesioned part, the lesioned part is usually matched with the central part (front visual field) of the image signal. However, as shown in FIG. 24C and FIG. 24D, when examining an elongated tubular portion such as the intestine, it is difficult to match the lesion to the front field of view of the image signal due to the limitation of the spatial region. That is, since the space is narrow, it is difficult to rotate the distal end of the insertion portion 300. Therefore, as shown in FIG. 24D, there is no choice but to adjust the lesioned part to the lateral visual field of the image signal.

  In view of this, in the present embodiment, whether or not the distal end of the scope is close to the subject is determined based on the aperture information of the light source diaphragm 102. When it is determined that they are close to each other, the front view area or the side view area is set as the attention area according to the photometry mode, and the light control of the attention area is performed.

  Specifically, the state information acquisition unit 430 acquires the opening area (opening information in a broad sense) of the light source diaphragm 102 via the control unit 420, and when the opening area is equal to or smaller than a predetermined threshold, It is determined that the tip is “close to” the attention area. The state information acquisition unit 430 transfers the determination result to the attention area setting unit 413 as state information.

  In the present embodiment, a plurality of photometry modes are prepared in accordance with the observation situation described above with reference to FIG. The operator sets the photometry mode via the external I / F unit 550. For example, when the surgeon wants to pay attention to the front visual field of the image signal, the photometry mode is set to “front mode”.

  The attention area setting unit 413 sets the attention area based on the state information transferred from the state information acquisition unit 430 and the photometry mode. Specifically, the attention area setting unit 413 determines that the photometric mode is “front mode” and the scope tip is “close” to the subject, the coordinates of the image center (N / 2, A front visual field centered on M / 2) is set as a region of interest. The attention area setting unit 413 sets the side field of view as the attention area when it is determined that the photometry mode is “side mode” and the distal end of the scope is “close to” the subject. If it is not determined that the scope tip is “close to” the subject, the attention area is not set. The attention area setting section 413 transfers the set attention area to the photometry section 416.

  In this way, the region of interest (front field of view or side field of view) is adaptively set according to the diaphragm condition of the light source, and the dimming control of the region of interest is performed, which improves the lesion site visibility for the surgeon. Connected.

5.3. Modified Example In the above embodiment, the photometric processing is adaptively performed according to the opening state of the light source stop 102, but the present embodiment is not limited to this. For example, a photometric process may be performed using an LED light source based on light amount control information (for example, drive current) of the LED light source.

  FIG. 25 shows a modified configuration example of the endoscope apparatus when such processing is performed. The endoscope apparatus includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The light source unit 100 includes a white light source 101, a rotating color filter 104, a rotation driving unit 105, a condenser lens 106, and a light source control unit 107 that controls the amount of light emitted from the light source. The white light source 101 is constituted by an LED light source. The light source control unit 107 controls the intensity of the emitted light amount of the LED light source based on the control signal from the control unit 420. The control unit 420 adjusts the amount of emitted light based on the adjustment coefficient Lc described in the above equation (6).

  The attention area setting unit 413 sets the attention area based on the control signal of the amount of emitted light from the control unit 420. That is, when the amount of emitted light is smaller than the threshold value, it is determined that the scope tip is close to the subject, and an attention area corresponding to the photometry mode is set. The photometry unit 416 performs photometry processing on the set attention area.

  According to the embodiment described above, the state information acquisition unit 430 acquires distance information indicating the distance between the subject and the scope distal end based on the emitted light amount of the light source unit 100 that illuminates the subject. The attention area setting unit 413 sets the attention area using the distance information as state information.

  Specifically, the attention area setting unit 413 sets the attention area when it is determined that the distance is closer than the threshold.

  More specifically, the endoscope apparatus includes a mode setting unit (control unit 420 in FIG. 23) and a dimming control unit (state information acquisition unit 430 and control unit 420 in FIG. 23). The mode setting unit sets the first photometry mode (front mode) or the second photometry mode (side mode). The dimming control unit controls the amount of emitted light by controlling the aperture area of the diaphragm of the light source unit 100 (the light source diaphragm 102 in FIG. 23). The distance information is an opening area controlled by the dimming control unit. When it is determined that the aperture area is smaller than the threshold value in the first photometry mode, the attention area setting unit 413 sets an area corresponding to the front visual field in the captured image as the attention area. When it is determined that the aperture area is smaller than the threshold value in the second photometry mode, the attention area setting unit 413 sets an area corresponding to the side field of view in the captured image as the attention area. The dimming control unit controls the opening area based on the brightness of the set attention area.

  In this way, since the region where the scope tip is close is considered to be the region that the operator wants to observe, by setting the region of interest according to the distance, the region that the operator wants to observe is appropriately dimmed and controlled it can. Further, since the brightness of the image is kept constant by the light control, it is possible to determine whether or not the scope tip has approached the subject by estimating the distance based on the light control instead of the image.

6). Fifth embodiment 6.1. Endoscope Device A fifth embodiment in which a region of interest is set based on distance information to a subject estimated from the brightness of an image will be described.

  The endoscope apparatus in the fifth embodiment includes a light source unit 100, an operation unit 200, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. The configuration is the same as that of the first embodiment described in FIG. In the following, description of components having the same operations and processes as those of the first embodiment will be omitted as appropriate.

  The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a post-processing unit 415, a photometry unit 416, and a gradation conversion unit 419. The configuration is the same as that of the first embodiment described in FIG.

  A relatively bright area in the image signal is considered to be an area where the distance between the tip of the insertion unit 300 and the subject is short. Therefore, in the present embodiment, brightness is calculated for each local region in the image signal, and a relatively bright local region among the plurality of local regions is set as a region of interest.

6.2. Attention Area Setting Unit FIG. 26 shows a detailed configuration example of the attention area setting unit 413 according to the fifth embodiment. The attention area setting unit 413 includes a luminance conversion unit 441, an addition unit 442, and a specification unit 443. The synchronization unit 412 is connected to the luminance conversion unit 441. The luminance conversion unit 441 is connected to the addition unit 442. The adding unit 442 is connected to the specifying unit 443. The specifying unit 443 is connected to the photometry unit 416. The control unit 420 is bi-directionally connected to the luminance conversion unit 441, the addition unit 442, and the specification unit 443.

  As shown in FIG. 27, the luminance conversion unit 441 calculates the luminance signal value Yi (x, y) of the sampling pixel of the image signal after synchronization by the above equation (1), and the luminance signal value Yi of the sampling pixel. Transfer (x, y) to the adder 442.

The adder 442 adds and averages the luminance signals Yi (x, y) of the sampling pixels in the forward visual field and the lateral visual field according to the following formula (10), and measures the brightness value Ydf of the forward visual field and the brightness of the lateral visual field. The measured value Yds is calculated. Here, the center coordinates (N / 2, M / 2) of the image signal are set as the origin, pixels within a predetermined radius are set as the front field of view, and other pixels are set as the side field of view.

  Here, m is the number of sampling pixels (constant) in the front field of view, and n is the number of sampling pixels (constant) in the side field of view. These are set according to the angle of view of the image signal. a (x, y) and b (x, y) are weighting factors.

  In the above-described embodiment, the image is divided into two regions of the front visual field and the side visual field, and the brightness measurement value of each visual field is calculated. However, the present embodiment is not limited to this. For example, when the side of the distal end of the insertion unit 300 is observed close to the subject, the possibility that the side field of view is all bright is low. As shown in FIG. 28, assuming such a case, for example, the side field of view is further divided into a plurality of regions, and the luminance signals Yi (x, y) of the sampling pixels included in each region are added and averaged. An area brightness measurement value may be calculated. In this case, the maximum value among the calculated brightness measurement values of the plurality of side regions is set as the brightness measurement value Yds of the side region. For example, in FIG. 28, the brightness measurement value of the side visual field 3 is the maximum value. In this case, the brightness measurement value of the side visual field 3 is transferred to the specifying unit 443 as the brightness measurement value Yds of the side area.

  The specifying unit 443 sets a region of interest based on the brightness measurement values Ydf and Yds calculated by the adding unit 442. Specifically, when the brightness measurement value Ydf of the front visual field is larger than a predetermined threshold, the specifying unit 443 sets the region corresponding to the front visual field as the attention region. In addition, when the brightness measurement value Yds of the side field of view is larger than a predetermined threshold, the specifying unit 443 sets a region corresponding to the side field of view as the attention region. When the brightness measurement values Ydf and Yds are both greater than a predetermined threshold, a brighter area is set as the attention area. The specifying unit 443 transfers the set attention area information to the photometry unit 416.

  Note that when the side field of view is divided as described with reference to FIG. 28, the divided area is set as the attention area. Specifically, when the measurement value Yds of the region with the largest measurement value Yds in the divided side field of view is larger than a predetermined threshold, the specifying unit 443 sets the region as the attention region.

  According to the above embodiment, the distance between the distal end of the insertion unit 300 and the subject is estimated from the brightness of the image signal, and the front visual field and the lateral visual field are adaptively dimmed based on the distance, It is possible for a person to improve the visibility of a lesioned part.

  According to the above embodiment, the attention area setting unit 413 includes the distance information acquisition unit (the luminance conversion unit 441 and the addition unit 442 in FIG. 26). The distance information acquisition unit acquires distance information representing the distance between the subject and the scope tip based on the brightness of the captured image (for example, the luminance signal Yi (x, y)). The attention area setting unit 413 sets the attention area based on the acquired distance information.

  Specifically, the distance information acquisition unit calculates a luminance feature amount related to the luminance of the pixel of the captured image (for example, a luminance conversion unit 441 (brightness feature amount calculation unit in a broad sense)) and calculates the calculated luminance feature amount. Get distance information based on.

  More specifically, the distance information acquisition unit divides the captured image into a plurality of regions (for example, a region corresponding to the front visual field and a region corresponding to the side visual field), and brightness (brightness) of each divided region. Measurement values Ydf, Yds) are acquired as distance information. The attention area setting unit 413 sets the brightest area among the plurality of areas as the attention area as the area closest to the distal end portion of the scope.

  In this way, the attention area can be set based on the brightness of the image, and the brightness of the attention area can be dimmed. That is, when the scope tip is brought close to an area that the operator wants to observe, the area is brightened by illumination, so that the attention area can be set according to the brightness of the image.

7). Sixth Embodiment 7.1. Endoscope Device A sixth embodiment in which a lesion area is detected with special light and the detected lesion area is set as a region of interest will be described.

  FIG. 29 shows a configuration example of an endoscope apparatus according to the sixth embodiment. The endoscope apparatus includes a light source unit 100, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The light source unit 100 includes a white light source 101, a light source aperture 102, a light source aperture drive unit 103, and a condenser lens 106. Compared to the first embodiment, the rotation color filter 104 and the rotation drive unit 105 are omitted.

  The insertion unit 300 includes a light guide fiber 301, illumination lenses 314 and 315, and an objective optical system 303. The insertion unit 300 also includes a half mirror 309 that separates the reflected light collected by the objective optical system 303 into two parts, a first imaging element 304 and a second imaging element 310 that detect the separated reflected light, and The first A / D conversion unit 305 and the second A / D conversion unit 311 are included.

  The A / D converter 311 converts the analog image signal detected by the second image sensor 310 into a digital image signal.

  The first image sensor 304 is a Bayer-type array image sensor. As shown in FIG. 30, the second image sensor 310 is an image sensor in which two types of color filters nB and nG are arranged on a checkered pattern. As shown in FIG. 31, each color filter nB, nG has a characteristic of transmitting light in a narrow band. For example, the color filter nB has a characteristic of transmitting light of 390 to 445 nm, and the color filter nG has a characteristic of transmitting light of 530 to 550 nm. For example, the first imaging element 304 and the second imaging element 310 have the same number of pixels.

  The control device 400 includes an image processing unit 410 and a control unit 420. The A / D conversion units 305 and 311 transfer the digitally converted image signal to the image processing unit 410. The image processing unit 410 transfers the processed image signal to the display unit 500. The control unit 420 is connected to the light source aperture driving unit 103, the image processing unit 410, and the external I / F unit 550, and performs these controls.

7.2. Image Processing Unit FIG. 32 shows a detailed configuration example of the image processing unit 410 in the sixth embodiment. The image processing unit 410 includes a first preprocessing unit 411, a second preprocessing unit 418, a first synchronization unit 412, a second synchronization unit 417, a region of interest setting unit 413, a post processing unit 415, and photometry. Part 416 and a gradation conversion part 419. Note that the processes of the pre-processing unit 411, the gradation conversion unit 419, the post-processing unit 415, and the photometry unit 416 are the same as those in the first embodiment, and a description thereof will be omitted.

  The synchronization unit 412 is connected to the gradation conversion unit 419 and the photometry unit 416. The preprocessing unit 418 is connected to the synchronization unit 417. The synchronization unit 417 is connected to the attention area setting unit 413. The attention area setting unit 413 is connected to the photometry unit 416. The control unit 420 is bi-directionally connected to the preprocessing unit 418 and the synchronization unit 417, and performs these controls.

  The synchronization unit 412 performs synchronization processing on the image signal processed by the preprocessing unit 411. As described above, the image signal acquired by the first image sensor 304 is a single-plate image signal in a Bayer type array. The synchronization unit 412 generates an RGB three-plate image signal from the single-plate image signal using interpolation processing. For example, a known bicubic interpolation process may be used as the interpolation process. Hereinafter, the image signal output from the synchronization unit 412 is referred to as a normal light image.

  The pre-processing unit 418 uses the OB clamp value, gain correction value, and WB coefficient value stored in advance in the control unit 420 for the image signal input from the A / D conversion unit 311, Gain correction processing and WB correction processing are performed. The preprocessing unit 418 transfers the preprocessed image signal to the synchronization unit 417.

  The synchronization unit 417 performs synchronization processing on the image signal processed by the preprocessing unit 418. As described above with reference to FIG. 30, the second image sensor 310 is an image sensor in which two types of color filters nB and nG are arranged on a checkered pattern. Therefore, the image signal processed by the preprocessing unit 418 becomes a single-plate image signal as shown in FIG. In FIG. 33, the image signal acquired by the color filter nB is represented as B2, and the image signal acquired by the color filter nG is represented as G2.

The synchronization unit 417 generates a B2 image signal and a G2 image signal from the single plate image signal by the following equations (11) and (12). The B2 image signal is an image signal having a B2 signal at all pixels, and the G2 image signal is an image signal having a G2 signal at all pixels. For example, the image signal B2 (1,1) of the nB filter at the position G2 (1,1) in FIG. 33 may be calculated using the following equation (11). Further, the image signal G2 (1,2) of the nG filter at the position B2 (1,2) in FIG. 33 may be calculated using the following equation (12).

  The synchronization unit 417 generates an RGB three-plate image signal using the B2 image signal and the G2 image signal generated by the above equations (11) and (12). Specifically, the synchronization unit 417 generates a three-plate image signal by using the G2 image signal as the R image signal and using the B2 image signal as the G and B image signals. Hereinafter, the image signal output from the synchronization unit 417 is referred to as a narrow-band light image (special light image in a broad sense).

  The attention area setting unit 413 detects a lesion area from the narrow-band light image by a method described later, and sets the attention area based on the lesion area. On the narrow-band light image, there is a feature that a lesion such as squamous cell carcinoma is depicted as a brown region. Therefore, a lesion area can be detected by detecting an area having a specific hue (brown area) from a narrow-band light image.

7.3. Attention Area Setting Unit FIG. 34 shows a detailed configuration example of the attention area setting unit 413 according to the sixth embodiment. The attention area setting section 413 includes a local area setting section 451, a feature amount calculation section 452, a lesion area detection section 453, a labeling processing section 454, an area selection section 455, a coordinate calculation section 456, and an association section 457. And a photometric processing condition setting unit 459.

  The synchronization unit 417 is connected to the local region setting unit 451. The local region setting unit 451 is connected to the feature amount calculation unit 452 and the association unit 457. The feature amount calculation unit 452 is connected to the lesion area detection unit 453. The lesion area detection unit 453 is connected to the labeling processing unit 454. The labeling processing unit 454 is connected to the area selection unit 455. The area selection unit 455 is connected to the coordinate calculation unit 456. The coordinate calculation unit 456 is connected to the association unit 457. The associating unit 457 is connected to the photometric processing condition setting unit 459. The photometric processing condition setting unit 459 is connected to the photometric unit 416.

  The local region setting unit 451 sets a plurality of local regions for the narrowband light image output from the synchronization unit 417. In the following, a case where a narrowband light image is divided into rectangular areas and each divided area is set as a local area will be described as an example.

  As shown in FIG. 35, for example, 5 × 5 pixels are defined as one local region. The narrowband image signal is composed of M × N local regions, and the coordinates of each region are represented by (m, n). In addition, a local area of coordinates (m, n) is represented by a (m, n). Also, the coordinates of the local region located at the upper left of the image are defined as (0, 0), the right direction is defined as the positive direction of m, and the downward direction is defined as the positive direction of n. In order to reduce the amount of calculation, an area composed of a plurality of adjacent pixel groups is set as one local area, but one pixel can be set as one local area. Also in this case, the subsequent processing is the same.

  The local region setting unit 451 outputs the size of the local region and the coordinates of all the local regions to the feature amount calculating unit 452. The local region setting unit 451 outputs the coordinates of all the local regions and the coordinates on the narrowband light image corresponding to the coordinates to the associating unit 457. Here, the coordinates on the narrow-band light image are the coordinates of the pixel existing at the center of the local region.

  The feature amount calculation unit 452 calculates a feature amount from all the local regions set by the local region setting unit 451. Hereinafter, a case where hue is used as the feature amount will be described as an example.

  The hue of the local region a (m, n) is expressed as H (m, n). In order to calculate H (m, n), the feature amount calculation unit 452 first calculates average values R_ave, G_ave, and B_ave of R, G, and B signals in each local region. Here, R_ave is an average value of R signals of all pixels included in each local region. The same applies to G_ave and B_ave. For example, each signal value is 8 bits (0 to 255).

The feature amount calculation unit 452 calculates the hue H (m, n) of each local region from R_ave, G_ave, B_ave, using, for example, the following equations (13) to (18). First, max is obtained by the following equation (13).

  Here, the MAX () function is a function that outputs the maximum value of a plurality of arguments in parentheses.

When max is 0, the hue H is calculated by the following equation (14).

When max is other than 0, d is calculated by the following equation (15).

  Here, the MIN () function is a function that outputs minimum values of a plurality of arguments in parentheses.

When R_ave is the maximum among R_ave, G_ave, and B_ave, hue H is calculated by the following equation (16).

When G_ave is maximum among R_ave, G_ave, and B_ave, hue H is calculated by the following equation (17).
When B_ave is the maximum among R_ave, G_ave, and B_ave, the hue H is calculated by the following equation (18).

  When H <0, 360 is added to H. When H = 360, H = 0.

  The lesion area detection unit 453 detects a local area having a specific hue H as a lesion area, and outputs the coordinates of all the local areas detected as the lesion area to the labeling processing unit 454. For example, the lesion area detection unit 453 detects an area (corresponding to a brown area) having a hue H of 5 to 35 as a lesion area.

  The labeling processing unit 454 gives the same label to the adjacent lesion regions among the lesion regions output from the lesion region detection unit 453. Hereinafter, a set of lesion areas assigned the same label is referred to as a lesion area group. The labeling processing unit 454 calculates the size of the lesion area group given the same label. Here, the size of the lesion area group may be the number of lesion areas to which the same label is assigned. However, the present invention is not limited to this, and the size of the lesion area group may be information indicating the area of the lesion area group.

  The process performed by the labeling processing unit 454 will be described in detail with reference to FIG. For example, when the lesion area shown in FIG. 36 is detected, the label 1 is assigned to the lesion area belonging to the area A1. Similarly, the label 2 is assigned to the lesion area belonging to the area indicated by A2, and the label 3 is assigned to the lesion area belonging to the area indicated by A3. The labeling processing unit 454 calculates the size of the lesion area group given the same label. The size of lesion area group 1 (A1) to which label 1 is assigned is 7. Similarly, the size of the lesion area group 2 (A2) is 3, and the size of the lesion area group 3 (A3) is 2.

  The region selection unit 455 selects a lesion region group having the maximum size as a region of interest among a plurality of lesion region groups to which labels are assigned by the labeling processing unit 454. Then, the coordinates of all the local areas included in the attention area are output to the coordinate calculation unit 456. In the case of FIG. 36, the lesion area group 1 indicated by A1 is selected as the attention area.

The coordinate calculation unit 456 calculates and calculates the maximum value (m _MAX , n _MAX ) and the minimum value (m _MIN , n _MIN ) of the local region coordinates from all the local region coordinates output from the region selection unit 455. The value is output to the associating unit 457.

Specifically, the number of all local areas output from the area selection unit 455 is K, and all the local areas are a (m ₁ , n ₁ ) to a (m _K , n _K ) for convenience. Represent. In this case, the coordinate calculation unit 456 calculates the maximum value (m _MAX , n _MAX ) and the minimum value (m _MIN , n _MIN ) of the local region coordinates using the following equation (19).

The associating unit 457 calculates the coordinates of the narrowband light image corresponding to the maximum value (m _MAX , n _MAX ) and the minimum value (m _MIN , n _MIN ). Specifically, the coordinates are calculated based on the correspondence between the local area output from the local area setting unit 451 and the coordinates of the narrowband light image. Hereinafter, the coordinates of the narrowband light image corresponding to (m _MAX , n _MAX ), (m _MIN , n _MIN ) are represented as (x _MAX , y _MAX ), (x _MIN , y _MIN ). The associating unit 457 outputs the coordinates (x _MAX , y _MAX ) and (x _MIN , y _MIN ) to the photometric processing condition setting unit 459.

The photometric processing condition setting unit 459 determines a photometric processing condition that is a condition of an attention area that is a target of the photometric processing, and outputs the photometric processing condition to the photometric unit 416. Specifically, the photometric processing condition setting unit 459 calculates the center coordinates (x ₀ , y ₀ ) of the attention area by the following expression (20), calculates the radius R of the attention area by the following expression (21), The calculated center coordinates (x ₀ , y ₀ ) and radius R are used as photometric processing conditions. In the example described with reference to FIG. 36, a circle with a radius R is set as a region of interest for photometry as shown.

  Here, int () is a function that returns a real integer value in parentheses.

  According to the above embodiment, the attention area setting unit 413 calculates a feature amount (for example, a hue value) based on the captured image, and based on an area having a predetermined feature amount (for example, a range of the hue value of 5 to 35). To set the attention area.

  In this way, it is possible to set a region of interest based on the feature amount of the image that appears characteristically in the lesion, and to control the brightness of the region of interest. Thereby, the surgeon can observe the lesioned part with appropriate brightness.

  In this embodiment, as described with reference to FIG. 36 and the like, the attention area setting unit 413 selects an area having the largest area among areas having a predetermined feature amount, and selects a circular area including the selected area as the attention area. Set as.

  Since the surgeon brings the scope close to the lesion to be observed, it is considered that the lesion to be observed is displayed large. Therefore, by setting the lesion area group having the maximum size as the attention area, it is possible to display the area that is considered to be noticed by the operator with appropriate brightness.

  In the above embodiment, the case where the feature amount is a hue value has been described as an example, but the present embodiment is not limited to this. For example, the feature amount may be anything that can identify a lesion and another region. For example, when identifying a bleeding site, the R pixel value may be the feature amount.

  In the above embodiment, an example in which the attention area is set based on the lesion area group having the maximum size among the lesion area groups labeled by the labeling processing unit 454 has been described. It is not limited to this. For example, the attention area may be set based on all the lesion areas detected by the lesion area detection unit 453. In this case, the region selection unit 455 may output the coordinates of the local regions included in all the lesion region groups output from the labeling processing unit 454 to the coordinate calculation unit 456.

  As a result, even when there are a plurality of lesion area groups, all the lesion area groups can be displayed with appropriate brightness.

  In the above embodiment, an example in which a normal light image is used as a display image has been described. However, the present embodiment is not limited to this. For example, a narrow band light image may be used as the display image.

  In this embodiment, the captured image includes a special light image in which a subject image having information in a specific wavelength band is captured and a normal light image (white light) in which a subject image having information in the wavelength band of white light is captured. Image). The attention area setting unit 413 sets the attention area based on the special light image.

  The dimming control unit performs dimming control of the normal light image based on the set attention area.

  In this way, by acquiring a special light image corresponding to the lesion part to be detected, it becomes easy to extract the feature amount of the lesion part, and the attention area can be set based on the extracted feature amount. Further, dimming control of a normal light image that is normally used in observation can be performed according to the attention area set by the special light image.

  In the present embodiment, the specific wavelength band is a band narrower than the white wavelength band (for example, 380 nm to 650 nm) (NBI: Narrow Band Imaging). For example, the normal light image and the special light image are in-vivo images of the living body, and a specific wavelength band included in the in-vivo image is a wavelength band of a wavelength that is absorbed by hemoglobin in blood. For example, the wavelength absorbed by this hemoglobin is 390 nm to 445 nm (first narrowband light nB) or 530 nm to 550 nm (second narrowband light nG).

  Thereby, it becomes possible to observe the structure of the blood vessel located in the surface layer part and the deep part of the living body. In addition, by inputting the obtained signals to specific channels (G2 → R, B2 → G, B), lesions that are difficult to see with normal light such as squamous cell carcinoma can be displayed in brown, etc. Oversight of parts can be suppressed. Note that 390 nm to 445 nm or 530 nm to 550 nm are numbers obtained from the characteristic of being absorbed by hemoglobin and the characteristic of reaching the surface layer part or the deep part of the living body, respectively. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% due to the variation factors such as the absorption by hemoglobin and the experimental results regarding the arrival of the living body on the surface layer or the deep part. It is also conceivable that the upper limit value increases by about 0 to 10%.

  In the present embodiment, the normal light image and the special light image are in-vivo images obtained by copying the inside of the living body, and the specific wavelength band included in the in-vivo image is a wavelength band of fluorescence emitted from the fluorescent substance. Also good. For example, the specific wavelength band may be a wavelength band of 490 nm to 625 nm.

  This enables fluorescence observation called AFI (Auto Fluorescence Imaging). In this fluorescence observation, autofluorescence (intrinsic fluorescence: 490 nm to 625 nm) from a fluorescent substance such as collagen can be observed by irradiating excitation light (390 nm to 470 nm). In such observation, the lesion can be highlighted with a color tone different from that of the normal mucous membrane, and the oversight of the lesion can be suppressed. Note that 490 nm to 625 nm indicates a wavelength band of autofluorescence emitted by a fluorescent substance such as collagen when irradiated with the excitation light described above. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% and the upper limit is 0 due to a variation factor such as an experimental result regarding the wavelength band of the fluorescence emitted by the fluorescent material. A rise of about 10% is also conceivable. Alternatively, a pseudo color image may be generated by simultaneously irradiating a wavelength band (540 nm to 560 nm) absorbed by hemoglobin.

  In the present embodiment, the specific wavelength band included in the in-vivo image may be a wavelength band of infrared light. For example, the specific wavelength band may be a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm.

  Thereby, infrared light observation called IRI (Infra Red Imaging) becomes possible. In this infrared light observation, by injecting intravenously ICG (Indocyanine Green), which is an infrared index drug that easily absorbs infrared light, and irradiating infrared light in the above-mentioned wavelength band, It is possible to highlight blood vessels and blood flow information in the deep mucosa, which is difficult to visually recognize, and it is possible to diagnose the depth of gastric cancer and determine the treatment policy. The numbers 790 nm to 820 nm are obtained from the characteristic that the absorption of the infrared index drug is strongest, and the numbers 905 nm to 970 nm are calculated from the characteristic that the absorption of the infrared index drug is the weakest. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% and the upper limit is 0 to 10 due to a variation factor such as an experimental result regarding absorption of the infrared index drug. It can also be expected to increase by about%.

  Moreover, in this embodiment, you may include the special light image acquisition part which produces | generates a special light image based on the acquired normal light image. For example, in FIG. 29, the half mirror 309, the image sensor 310, and the A / D conversion unit 311 are omitted, and the preprocessing unit 418 and the synchronization unit 417 in FIG. 32 serve as the special light image acquisition unit from the normal light image to the special light image. May be generated.

  Specifically, the special light image acquisition unit includes a signal extraction unit that extracts a signal in the white wavelength band from the acquired normal light image. The special light image acquisition unit may generate a special light image including a signal in a specific wavelength band based on the extracted signal in the wavelength band of normal light. For example, the signal extraction unit estimates the spectral reflectance characteristics of the subject in 10 nm increments from the RGB signal of the normal light image, and the special light image acquisition unit integrates the estimated signal component in the specific band and performs special processing. Generate a light image.

  More specifically, the special light image acquisition unit may include a matrix data setting unit that sets matrix data for calculating a signal in a specific wavelength band from a signal in the wavelength band of normal light. Then, the special light image acquisition unit may generate a special light image by calculating a signal in a specific wavelength band from a signal in the white wavelength band using the set matrix data. For example, the matrix data setting unit sets table data in which spectral characteristics of irradiation light in a specific wavelength band are described in increments of 10 nm as matrix data. Then, the spectral characteristics (coefficients) described in the table data are multiplied by the spectral reflectance characteristics of the subject estimated in increments of 10 nm and integrated to generate a special light image.

  As a result, the special light image can be generated based on the normal light image, so that the system can be realized with only one light source that irradiates the normal light and one image sensor that images the normal light. . Therefore, the capsule endoscope and the insertion part of the scope endoscope can be made small, and the cost can be expected to be reduced because the number of parts is reduced.

8). Seventh embodiment 8.1. Endoscope Device A seventh embodiment in which a region of interest is set based on a scope ID will be described.

  FIG. 37 shows a configuration example of the endoscope apparatus according to the seventh embodiment. The endoscope apparatus includes a light source unit 100, an insertion unit 300, a control device 400, a display unit 500, and an external I / F unit 550. In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The insertion unit 300 includes a light guide fiber 301, illumination lenses 314 and 315, an objective optical system 303, an image sensor 304, an A / D conversion unit 305, and a memory 313. Since the configuration of each unit other than the memory 313 is the same as that of the first embodiment, description thereof is omitted.

  The insertion unit 300 is generally called a scope. Therefore, in the following, the insertion unit 300 is appropriately referred to as a scope. In endoscopic diagnosis, different scopes are used depending on the site to be diagnosed. For example, the upper gastrointestinal scope is used for diagnosis of the esophagus and stomach, and the lower gastrointestinal scope is used for diagnosis of the large intestine. On the other hand, the scope memory 313 holds an identification number (scope ID) unique to each scope.

  The control device 400 includes an image processing unit 410, a control unit 420, and a state information acquisition unit 430. The status information acquisition unit 430 refers to the identification number unique to each scope held in the memory 313 and identifies the type of scope to be connected. The type of scope is, for example, either the upper digestive scope or the lower digestive scope. The state information acquisition unit 430 outputs the identified scope type to the attention area setting unit 413 described later.

8.2. Image Processing Unit FIG. 38 shows a detailed configuration example of the image processing unit 410 in the seventh embodiment. The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a post-processing unit 415, a photometry unit 416, and a gradation conversion unit 419. Since the processing other than the attention area setting unit 413 is the same as that of the first embodiment, the description thereof is omitted.

  The attention area setting unit 413 sets the attention area according to the type of scope output from the state information acquisition unit 430. The attention area setting method is the same as that of the third embodiment. Specifically, when the scope type is the lower digestive organ scope, the region of the side field of view is set as the attention region. When the scope type is the upper digestive scope, the area of the front visual field is set as the attention area.

  According to the above embodiment, as described in FIG. 37 and the like, the scope (insertion unit 300) can be attached to and detached from the endoscope apparatus. The state information acquisition unit 430 acquires identification information (scope ID) for identifying the attached scope. The attention area setting unit 413 sets the attention area based on the acquired identification information.

  In this way, by performing appropriate light control according to the connected scope, the operator can observe the lesion area with appropriate brightness.

  Specifically, the scope has an objective optical system that forms an image of the front field and the side field. When the identification information is identification information representing a scope for the lower digestive tract, the attention area setting unit 413 sets an area corresponding to the lateral visual field in the captured image as the attention area.

  In the large intestine to be diagnosed by the lower gastrointestinal scope, it is assumed that lesions behind the folds are displayed in the lateral region of the image. Therefore, the risk of oversight of a lesion can be improved by dimming control of the brightness of the region corresponding to the side visual field.

  In the present embodiment, the attention area setting unit 413 determines the area corresponding to the front visual field in the captured image as the attention area when the identification information is identification information indicating a scope for the upper digestive tract (for example, stomach or esophagus). Set to.

  Since the esophagus and stomach to be diagnosed by the upper digestive tract have few folds, the front visual field is more important than the side visual field for diagnosis. Therefore, by controlling the brightness of the area corresponding to the front visual field, the important visual field can be easily seen.

9. Eighth Embodiment 9.1. Attention Area Setting Unit An eighth embodiment in which the amount of motion of a subject (movement information in a broad sense) is calculated from an image and the attention area is set based on the amount of motion will be described.

  Since the configurations of the endoscope apparatus and the image processing unit 410 are the same as those in the first embodiment described above with reference to FIGS. 3 and 6, description thereof will be omitted as appropriate. Hereinafter, the attention area setting unit 413 having a configuration different from that of the first embodiment will be described in detail.

  FIG. 39 shows a detailed configuration example of the attention area setting unit 413 in the eighth embodiment. The attention area setting unit 413 includes a distortion correction unit 471, an image memory 472, a motion detection unit 473 (a motion information acquisition unit in a broad sense), and a setting unit 474.

  The synchronization unit 412 outputs the synchronized image signal to the distortion correction unit 471. The distortion correction unit 471 is connected to the image memory 472, the motion detection unit 473, and the setting unit 474. The motion detection unit 473 is connected to the setting unit 474. The setting unit 474 is connected to the photometry unit 416. The control unit 420 is bidirectionally connected to the distortion correction unit 471, the image memory 472, the motion detection unit 473, and the setting unit 474, and performs these controls.

  The distortion correction unit 471 corrects distortion aberration (hereinafter, referred to as distortion as appropriate), which is a kind of aberration, with respect to the synchronized image signal. FIG. 40 shows examples of images before and after the distortion correction process.

Specifically, the distortion correction unit 471 acquires pixel coordinates of the image after the distortion correction processing. It is assumed that the image size after the distortion correction processing is acquired in advance based on the distortion of the optical system. The distortion correction unit 471 converts the acquired pixel coordinates (x, y) into coordinates (x ′, y ′) having the optical axis center as the origin using the following expression (22).

  Here, (center_x, center_y) is the coordinates of the optical axis center after the distortion correction processing. For example, the center of the optical axis after the distortion correction process is the center of the image after the distortion correction process.

Next, the distortion correction unit 471 calculates the object height r using the following equation (23) based on the converted pixel coordinates (x ′, y ′).

  Here, max_r is the maximum object height in the image after distortion correction processing.

Next, the distortion correction unit 471 calculates a ratio (R / r) between the image height and the object height based on the calculated object height r. Specifically, the distortion correction unit 471 holds a relationship between the ratio R / r and the object height r as a table, and acquires the ratio R / r corresponding to the object height r by referring to the table. . Next, the distortion correction unit 471 acquires the pixel coordinates (X, Y) before the distortion correction processing corresponding to the pixel coordinates (x, y) after the distortion correction processing using the following expression (24).

  Here, (center_X, center_Y) is the coordinates of the optical axis center before the distortion correction processing. For example, the center of the optical axis before the distortion correction process is the center of the image before the distortion correction process.

  Next, the distortion correction unit 471 calculates a pixel value at the pixel coordinates (x, y) after the distortion correction processing based on the calculated pixel coordinates (X, Y) before the distortion correction processing. When (X, Y) is not an integer, the pixel value is calculated by linear interpolation based on surrounding pixel values. The distortion correction unit 471 performs the above process on all the pixels of the image after the distortion correction process. The distortion correction unit 471 outputs the image corrected in this way to the image memory 472 and the motion detection unit 473.

  The image memory 472 stores the image after distortion correction corrected by the distortion correction unit 471. The stored image after distortion correction is output to the motion detection unit 473 in synchronization with the timing at which a new image after distortion correction is output from the distortion correction unit 471.

  The motion detection unit 473 detects a local motion of the image based on the distortion-corrected image corrected by the distortion correction unit 471 and the distortion-corrected image stored in the image memory 472. In the following, for the sake of simplicity, the former image after distortion correction is referred to as a current frame image, and the latter image after distortion correction is referred to as a previous frame image.

  Specifically, the motion detection unit 473 sets a representative point, which is a point for detecting local motion, on the current frame image. For example, as shown in FIG. 41, representative points (● shown in FIG. 41) are set at regular intervals and in a grid pattern in the pixels of the current frame image. The motion detection unit 473 calculates a motion vector of a representative point between the current frame image and the previous frame image as a local motion amount. For example, a motion vector is detected using block matching which is a known technique. The motion detection unit 473 outputs the set coordinates of the representative points and the motion vector detected at each representative point to the setting unit 474.

  The setting unit 474 sets a region of interest based on the current frame image from the distortion correction unit 471, the coordinates of the representative points from the motion detection unit 473, and the motion vector of each representative point. Specifically, the setting unit 474 determines whether or not the extracted state is based on the current frame image and the motion vector. If the extracted state, the side field of view is set as the attention area. Is set as the attention area. The setting unit 474 outputs the set attention area to the photometry unit 416.

  Here, the extracted state refers to a state in which observation is performed while the endoscope is pulled out after the endoscope has been inserted deep into the hollow organ. Since this type of observation is generally performed on the large intestine, the side field of view is set as a region of interest so that it is easy to observe the back of the folds when it is in the removed state. The front field of view is set as a region of interest so that

9.2. Extraction State Determination Method The extraction state determination method will be described with reference to FIGS. 42 (A) to 42 (D). First, the vanishing point of the motion vector is detected based on the motion vector at the representative point.

Specifically, in FIG. 42A, ● represents a representative point, a solid arrow represents a motion vector, and x represents a vanishing point. The vanishing point is an intersection of straight lines that start from each representative point and extend in the direction along the motion vector at each representative point. For example, if the observation target is a cylindrical shape with a constant inner diameter and the motion vector is generated only by the movement of the endoscope, in the removed state, straight lines extending from the representative points intersect at one point at the vanishing point. Actually, the inner diameter of the luminal organ to be observed is not constant, and the motion vector is also generated by the pulsation of the living body. Therefore, even in the removed state, the straight lines do not intersect at the vanishing point. Therefore, the sum of squares of the distances from all the straight lines is set as the first evaluation value, and the point with the smallest first evaluation value is set as the vanishing point candidate. When the straight line is ax + by + c = 0 and the point is (p, q), the first evaluation value D is obtained by the following equation (25).

  Here, the sum in the above equation (25) represents the sum of the squares of the distances for all straight lines extended from the representative point.

By the least square method, the coordinates (xsk, ysk) of the vanishing point candidate that minimizes the first evaluation value D are calculated using the following equation (26).

  Here, assuming that the coordinates of the representative point are (Px, Py) and the motion vector at the representative point is (Mx, My), a = My, b = −Mx, c = MxPy−MyPx.

  Next, when the first evaluation value D in the vanishing point candidate is equal to or less than a predetermined first threshold and the vanishing point candidate exists in the image, the vanishing point candidate is set as the vanishing point. If the vanishing point candidate does not satisfy this condition, it is determined that the vanishing point cannot be detected.

  The reason why the removal state can be detected under these conditions will be described in detail. Here, as shown in FIG. 42B, the moving direction of the endoscope tip is defined. The x direction, the y direction, and the z direction correspond to the horizontal direction, the vertical direction, the horizontal direction, and the depth direction orthogonal to the vertical direction on the image, respectively. In addition, the movement along the arrow in the x direction, the y direction, and the z direction is defined as a positive movement, and the opposite movement is defined as a negative movement. In the extracted state, the endoscope tip moves negatively in the z direction. For example, when the movement of the tip of the endoscope is positive in the x direction, as shown in FIG. 42C, the motion vectors are almost parallel, so that the vanishing point candidate is outside the image. Exists. Alternatively, even if a vanishing point candidate exists in the image, the first evaluation value D in the vanishing point candidate is a large value. That is, the motion in the z direction can be detected on the condition that the vanishing point candidate exists in the image and the first evaluation value D is smaller than the first threshold value.

  Next, it is determined whether or not it is in the removal state. As shown in FIG. 42D, the vanishing point is detected even when the endoscope is inserted. Therefore, a determination is made based on a vector (broken arrow) from each representative point to the vanishing point and a motion vector (solid arrow). Specifically, when there are a predetermined number or more of representative points where the inner product of the vector to the vanishing point and the motion vector is negative, it is determined that no vanishing point is detected. When the vanishing point is not detected, it is determined that the removal state is not established.

  Next, the pixel value of the current frame image at the detected vanishing point coordinates is set as a second evaluation value, and it is determined based on the second evaluation value whether the observation target is a luminal organ. If the vanishing point coordinates are not an integer, the second evaluation value is calculated by linear interpolation based on the surrounding pixel values. When the second evaluation value is equal to or less than a predetermined second threshold value, it is determined as a luminal organ. Even when the observation target is flat rather than luminal, the vanishing point is detected when the endoscope tip moves negatively in the z direction, so such a determination is necessary.

  Specifically, when the observation target is a luminal shape, the second evaluation value is small because the vanishing point is located behind the luminal organ that is not irradiated with much illumination light on the current frame image. On the other hand, when the observation target is flat, the second evaluation value is large because the vanishing point is located in a region where the illumination light is relatively irradiated on the current frame image. Therefore, by determining whether or not the second evaluation value is equal to or less than the second threshold value, it is possible to determine an extraction state that is an observation performed while removing the endoscope from the luminal observation target.

  In the above embodiment, when the determination is switched from the extracted state to the non-extracted state, the front visual field is immediately set as the attention area, but the present embodiment is not limited to this. For example, when the determination of the non-extraction state continues for a predetermined frame, the front view may be set as the attention area. Similarly, when the determination is switched from the non-extraction state to the extraction state, the side field of view may be set as the attention area when the determination of the extraction state continues for a predetermined frame. By setting the attention area in this manner, when the insertion and removal are repeated in small increments, it is possible to prevent the attention area from frequently switching and the display screen from becoming unstable.

  According to the above embodiment, as shown in FIG. 39, the attention area setting unit 413 includes the motion detection unit 473. The motion detection unit 473 acquires the amount of motion of the subject based on the captured image. The attention area setting unit sets the attention area based on the acquired amount of motion.

  Specifically, as described with reference to FIG. 42A and the like, the attention area setting unit 413 determines that the scope has been removed based on the amount of motion, An endoscope apparatus characterized in that an area corresponding to a side visual field is set as an attention area.

  Usually, when observing the large intestine, observation is performed while removing the scope. Therefore, information useful for diagnosis, such as the back of the folds of the large intestine, can be suitably acquired by setting the region corresponding to the lateral visual field as the region of interest at the time of extraction and performing dimming control.

  In this embodiment, the motion detection unit 473 calculates the motion vectors of a plurality of representative points as the motion amount, and obtains the vanishing point of the calculated motion vector. The attention area setting unit 413 is in a state where the scope has been removed when the obtained vanishing point exists in the captured image and the inner product of the vector from the representative point to the vanishing point and the motion vector is positive. Is determined.

  In this way, it can be determined that the scope has been inserted or removed by determining the position of the vanishing point, and whether the scope has been inserted or removed can be determined by determining the direction of the motion vector.

  In the present embodiment, the attention area setting unit 413 determines that the scope has been removed when it is determined that the pixel value of the vanishing point is smaller than the threshold (second threshold).

  In this way, as described above, since it can be determined whether the scope is in a state of facing the wall surface or in a direction along the lumen, the extraction is a movement in the direction along the lumen. The state can be determined.

  In the present embodiment, the attention area setting unit 413 sets the area corresponding to the front visual field in the captured image as the attention area when it is determined that the scope is not removed based on the amount of movement. .

  In this way, dimming control can be performed by setting the front field of view considered to be mainly used at times other than the time of removal as the attention area. For example, since the endoscope is operated using the front visual field at the time of insertion, it is possible to present forward information including a lot of information useful for the operation.

  In this embodiment, as shown in FIG. 39, the motion detection unit 473 includes a distortion correction unit 471. The distortion correction unit 471 corrects the distortion of the captured image based on the distortion aberration of the optical system of the scope. The motion detection unit 473 acquires a motion amount based on the captured image after distortion correction.

  In this way, for example, when an optical system having a wide field of view such as the objective optical system of FIG. 1 is used, image distortion due to the distortion can be corrected. Thereby, the influence of distortion on the motion amount detection can be reduced.

10. Ninth Embodiment 10.1. Endoscope Device A ninth embodiment in which shape information of a subject is acquired by a shape detection unit, and a region of interest is set based on the shape information will be described.

  FIG. 43 shows a configuration example of the endoscope apparatus according to the ninth embodiment. The endoscope apparatus includes a light source unit 100, an insertion unit 300, a control device 400, a display unit 500, an external I / F unit 550, and a shape detection unit 700 (for example, UPD (Position Detecting Unit)). In addition, the same code | symbol is attached | subjected about the component same as the component demonstrated in FIG. 3 etc., and description is abbreviate | omitted suitably.

  The insertion unit 300 includes a light guide fiber 301, illumination lenses 314 and 315, an objective optical system 303, an image sensor 304, an A / D conversion unit 305, and a magnetic coil 312. Except for the magnetic coil 312, the second embodiment is the same as the first embodiment, and a description thereof is omitted.

  For example, ten or more magnetic coils 312 are provided in the insertion unit 300, and each transmits magnetism to the shape detection unit 700 (shape information acquisition unit in a broad sense). The plurality of magnetic coils 312 are arranged along the length direction of the insertion portion 300, for example.

  The shape detection unit 700 receives magnetism transmitted from the magnetic coils 312 provided in the respective units of the insertion unit 300 by an antenna (not shown), and acquires three-dimensional position information of each magnetic coil in real time. The shape detection unit 700 acquires endoscope shape information representing the shape of the insertion unit 300 in real time from the acquired three-dimensional position information of the magnetic coil.

  The control device 400 includes an image processing unit 410, a control unit 420, and a state information acquisition unit 430. Since the control unit 420 is the same as that of the first embodiment, description thereof is omitted.

10.2. Status Information Acquisition Unit FIG. 44 shows a detailed configuration example of the status information acquisition unit 430 in the ninth embodiment. The state information acquisition unit 430 includes a subject information acquisition unit 436 and an observation information acquisition unit 437. The shape detection unit 700 outputs the detected endoscope shape information to the subject information acquisition unit 436 and the observation information acquisition unit 437. The subject information acquisition unit 436 is connected to the attention area setting unit 413. The observation information acquisition unit 437 is connected to the attention area setting unit 413.

  The subject information acquisition unit 436 determines whether or not the subject is the lower digestive tract based on the endoscope shape information acquired by the shape detection unit 700. Specifically, the subject information acquisition unit 436 determines, based on the endoscope shape information, the length of the endoscope inserted into the body and whether or not the endoscope shape forms a loop. And whether or not it is the lower digestive tract is determined based on the length and the determination result.

  More specifically, in general, when endoscopy of the lower digestive tract is performed, the endoscope is inserted deeper than when endoscopy of other organs is performed. Therefore, the subject information acquisition unit 436 determines that the subject is the lower digestive tract if the endoscope is inserted into the body for a predetermined length or more. In addition, there is an insertion method in which a loop is formed with an endoscope to facilitate insertion only when the endoscope is inserted into the lower digestive tract. Therefore, the subject information acquisition unit 436 determines that the subject is the lower digestive tract if a loop is formed in the endoscope.

  The subject information acquisition unit 436 determines that the subject is not the lower digestive tract until the endoscope inserted in the body exceeds a predetermined length or the endoscope forms a loop. If either condition is satisfied, it is determined that the subject is the lower digestive tract, and the determination result is not changed thereafter. The subject information acquisition unit 436 outputs the determination result to the attention area setting unit 413 as subject information.

  The observation information acquisition unit 437 determines whether or not the endoscope is in the removal state based on the endoscope shape information acquired by the shape detection unit 700. Specifically, the observation information acquisition unit 437 determines whether or not the length of the endoscope inserted into the body is shortened over time based on the endoscope shape information. When it is shortened, it is determined that it is in the removed state. The observation information acquisition unit 437 outputs the determination result of the removal state to the attention area setting unit 413 as observation information.

10.3. Image Processing Unit The image processing unit 410 according to the ninth embodiment includes a preprocessing unit 411, a synchronization unit 412, a region of interest setting unit 413, a post processing unit 415, a photometry unit 416, and a gradation conversion unit 419. including. The configuration is the same as that of the second embodiment described in FIG. Since operations and processes other than the attention area setting unit 413 are the same as those in the second embodiment, description thereof will be omitted.

  The attention area setting section 413 sets the attention area based on the subject information and the observation information output from the state information acquisition section 430. Specifically, the attention area setting unit 413 determines that the subject is the lower digestive tract based on the subject information, and determines that the endoscope is in the removed state based on the observation information. Is set as the attention area. In other cases, the front visual field is set as the attention area. The attention area setting section 413 outputs the set attention area to the photometry section 416.

  According to the above embodiment, as shown in FIG. 44, the endoscope apparatus acquires shape information of the scope insertion unit (insertion unit 300) (for example, three-dimensional position information of the magnetic coil 312 in FIG. 43). A shape detection unit 700 is included. The state information acquisition unit 430 includes a subject information acquisition unit 436. The subject information acquisition unit 436 acquires the acquired shape information as state information, and acquires subject information based on the shape information. The attention area setting unit 413 sets the attention area based on the acquired subject information.

  Specifically, the subject information acquisition unit 436 determines a part of the subject based on the shape information, and acquires the determined part as subject information. When it is determined that the subject part is the lower digestive tract, the attention area setting unit 413 sets an area corresponding to the lateral visual field in the captured image as the attention area.

  More specifically, the subject information acquisition unit 436 determines that the region of the subject is the lower digestive tract when it is determined that the scope insertion unit has a loop shape.

  In this way, light control can be performed by setting a region of interest according to the part determined based on the shape information of the subject. In addition, in the case of the lower gastrointestinal tract, by controlling the brightness of the lateral visual field, information useful for diagnosis, such as the back of the fold of the large intestine, can be suitably presented to the surgeon during removal.

  Here, the subject information is information about the subject. The subject information is not limited to the part of the subject, and may be information estimated from the shape information of the subject. For example, the subject information may be the size or length of the subject.

  In the present embodiment, the state information acquisition unit 430 includes an observation information acquisition unit 437. The observation information acquisition unit 437 acquires the acquired shape information as state information, and acquires observation state information based on the shape information. The attention area setting unit 413 sets the attention area based on the acquired observation state information.

  Specifically, the observation information acquisition unit 437 determines the traveling direction of the scope based on the shape information, and acquires the determined traveling direction as observation state information. When it is determined that the scope has been removed, the attention area setting unit 413 sets an area corresponding to the lateral visual field in the captured image as the attention area.

  In this way, the brightness of the side field of view can be dimmed when the scope is being removed. For example, when the subject is the lower digestive tract and the scope is being removed, the brightness of the side visual field is dimmed and controlled so that the operator can easily observe the back of the folds.

  In the present embodiment, the attention area setting unit 413 determines the area corresponding to the front visual field in the captured image as the attention area, except when the subject is determined to be the lower digestive tract and is determined to be in the removed state. Set as.

  In this way, it is possible to set the front field of view important in the observation of the upper digestive tract and the insertion operation in the lower digestive tract as the attention area, and to control the brightness of the attention area.

11. Tenth Embodiment 11.1. Outline A tenth embodiment in which a red ball region is detected from an image and a region other than the red ball region is set as a region of interest will be described.

  The reason why it is difficult to insert an endoscope into the large intestine is that the large intestine is a narrow lumen, is bent and twisted, and is further stretched and deformed depending on the insertion of the endoscope. In such a situation, the operator performs a technique of guiding in the insertion direction by making full use of the operation of the endoscope while observing the endoscope image to determine the insertion direction. The operation of the endoscope includes an up / down / left / right angle operation of the endoscope tip, a push / pull twist of an endoscope insertion portion (referred to as a scope as appropriate), and the like.

  Such an insertion procedure is difficult for an unskilled doctor. Especially during insertion, the endoscope insertion part is often pressed, so the endoscope tip frequently comes into contact with the intestinal wall, and the entire endoscopic image called a red ball is in a red defocused state, making it impossible to determine the insertion direction at all. . In such a state, it is necessary to return to the state where the insertion direction can be confirmed from the red ball state by slightly pulling back the endoscope insertion portion, and it is necessary to perform the insertion operation again, which is one of the causes of prolonged insertion time. .

  In an endoscope having an optical system with a wide field of view of 180 ° or more at the tip, even if the tip of the endoscope contacts the intestinal wall, the entire field of view is not blocked. It does not occur at all, and a part of the field of view can be secured. That is, there is a possibility that the insertion direction can be confirmed even when the endoscope tip is in contact with the intestinal wall.

  Therefore, in this embodiment, a red ball region is detected in an endoscopic image of an optical system of 180 degrees or more, a region other than the red ball region is set as a region of interest, and the doctor is in a state in which the visibility of the region of interest is improved. By presenting it, the insertion time can be shortened. The state in which the visibility is improved is a state in which the attention area is adjusted to appropriate exposure by light control for the attention area.

  FIG. 45 shows a state in which the distal end of the endoscope insertion portion is in contact with the wall surface of the lumen modeling the large intestine from an oblique direction. As shown in FIG. 46A, in the endoscopic image in this state, the red ball region occupies most of the endoscopic image. Even in this state, when the lumen is straight and the mouth is open, the insertion direction can be determined as a dark part. On the other hand, in a state where the lumen is bent or the mouth of the lumen is closed, the insertion direction is not a clear dark part and it is difficult to distinguish. I want to observe in a more magnified state. Therefore, as shown in FIG. 46B, a small region of interest other than the red ball region is enlarged and displayed. In this way, it is possible to determine an insertion direction that is difficult to identify.

  In the red ball region of FIG. 46 (A), since the illumination light hits the intestinal wall with the tip of the insertion portion in contact with the intestinal wall, a relative light / dark difference occurs between the red ball region and the other region of interest. . In particular, in an endoscope, the amount of illumination light is adjusted (light adjustment processing), and the endoscopic image is controlled to an appropriate exposure state. For this reason, when the red ball region occupies a large amount, the light control is performed so that the brightness of the red ball region is in an appropriate exposure state. In this case, even if a region of interest other than the red ball region is in the field of view, there is a possibility that the region of interest is dark and does not have proper exposure. Cannot be secured. Therefore, a dimming process is performed by designating a region of interest other than the red ball region as the dimming target region. As a result, the attention area can be observed in the proper exposure state, and the insertion direction can be easily identified.

  FIG. 46 (A) is an example in which the intestinal wall is contacted at an angle, so that it can be inserted to some extent even if the insertion direction is roughly determined as an area other than the red ball area. However, at the bent portion of the large intestine (lumen), the endoscope tip may come into contact with the intestinal wall while facing the intestinal wall. In this case, the central portion of the endoscopic image is a red ball region, and the peripheral portion is a region of interest other than the red ball region. In this case, the insertion direction cannot be recognized only by finding an area other than the red ball area. Therefore, similarly to the above, the peripheral area which is the attention area is enlarged and displayed, and the peripheral area is designated as the dimming target area to perform the dimming process. As a result, the direction to be inserted can be more easily recognized from the 360 ° directions of the enlarged peripheral region.

11.2. Image Processing Unit Next, a method for detecting the red ball region and a method for setting a region other than the red ball region as the attention region will be described in detail.

  FIG. 47 shows a detailed configuration example of the image processing unit 410 in the tenth embodiment. The image processing unit 410 includes a preprocessing unit 411, a synchronization unit 412, an attention area setting unit 413, a scaling processing unit 414, a post processing unit 415, and a photometry unit 416. In the following, the same components as those described in FIG. 6 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate. Note that the endoscope apparatus according to the tenth embodiment is the same as that of the first embodiment described above with reference to FIG.

  The preprocessing unit 411 is connected to the synchronization unit 412. The synchronization unit 412 is connected to the attention area setting unit 413 and the scaling processing unit 414. The attention area setting unit 413 is connected to the scaling processing unit 414 and the photometry unit 416. The scaling processing unit 414 is connected to the post-processing unit 415. The post-processing unit 415 is connected to the display unit 500. The control unit 420 is connected to the pre-processing unit 411, the synchronization unit 412, the attention area setting unit 413, the scaling processing unit 414, the post-processing unit 415, and the photometry unit 416.

  The attention area setting unit 413 sets an area other than the red ball area as the attention area. The scaling processing unit 414 performs processing for enlarging the set attention area. These processes will be described later in detail. The photometry unit 416 calculates the brightness (for example, luminance) of the image. The photometry unit 416 calculates the brightness of the attention area when the attention area is set. The photometry unit 416 outputs the calculated brightness to the control unit 420. The control unit 420 controls the light source aperture driving unit 103 based on the brightness and performs a light control process.

11.3. Attention Area Setting Unit FIG. 48 shows a detailed configuration example of the attention area setting unit 413 in the tenth embodiment. The attention area setting unit 413 includes a color conversion unit 461, a red ball region candidate detection unit 462, a defocus detection unit 463, a red ball region determination unit 464, and a scaling parameter setting unit 465.

  The color conversion unit 461 is connected to the red ball region candidate detection unit 462 and the defocus detection unit 463. The red ball region candidate detection unit 462 is connected to the red ball region determination unit 464. The defocus detection unit 463 is connected to the red ball region determination unit 464. The red ball region determination unit 464 is connected to the zoom parameter setting unit 465 and the photometry unit 416. The scaling parameter setting unit 465 is connected to the scaling processing unit 414. The control unit 420 is connected to the red ball region candidate detection unit 462, the defocus detection unit 463, and the scaling parameter setting unit 465.

  The color conversion unit 461 converts the RGB signal of the color image output from the synchronization unit 412 into a luminance signal and a color signal. In the following description, an example in which an RGB signal is converted into a YCbCr signal will be described. The color conversion unit 461 outputs the calculated luminance signal Y to the defocus detection unit 463 and the photometry unit 416, and outputs the color difference signals Cb and Cr to the red ball region candidate detection unit 462.

  The red ball region candidate detection unit 462 divides the input color difference signals Cb and Cr into block regions based on block size information (for example, N × N blocks) from the control unit 420. Then, the red ball region candidate detection unit 462 determines a red ball region candidate from the statistic of the color difference signal in the block region, and outputs determination information for each block region to the red ball region determination unit 464.

  The defocus detection unit 463 divides the input luminance signal Y into block areas based on the block size information from the control unit 420. Then, the defocus detection unit 463 determines whether or not the block region is in a defocus state from the presence or absence of a high frequency component in the block region, and outputs determination information for each block region to the red ball region determination unit 464.

  The red ball region determination unit 464 determines that the block region determined to be a red ball candidate region by the red ball region candidate detection unit 462 and determined to be in the defocused state by the defocus detection unit 463 is a red ball region. The red ball region determination unit 464 outputs the determination result to the magnification parameter setting unit 465 and the photometry unit 416.

  As shown in FIG. 49, the scaling parameter setting unit 465 sets scaling parameters for each divided block area (each block of N × N blocks). Specifically, the scaling parameter setting unit 465 sets, as a region of interest, a block region that is determined not to be a red ball region among block regions in the imaging region based on the determination result of the red ball region determination unit 464. . The scaling parameter setting unit 465 sets a scaling parameter for each block area based on the relationship between the attention area and the red ball area, and outputs the set scaling parameter to the scaling processing unit 414.

  The scaling processing unit 414 resets the display magnification with respect to the non-target region so that the target region in the imaging region is displayed in an enlarged manner, and the target region is enlarged relative to the red ball region. Is generated. The resetting is to create a conversion table of the coordinate position (real number accuracy) before the scaling process with respect to the pixel position after the scaling process (integer precision) with the change of the display magnification.

  The photometry unit 416 calculates the average luminance level of the attention area based on the attention area information other than the red ball area output from the attention area setting section 413 and the luminance signal output from the color conversion section 461.

11.4. Red Ball Region Candidate Detection Unit FIG. 50 shows a detailed configuration example of the red ball region candidate detection unit 462. The red ball region candidate detection unit 462 includes a region division unit 481, a statistic calculation unit 482, a specific color region determination unit 483, and a memory 484.

  The area dividing unit 481 is connected to the statistic calculating unit 482. The statistic calculation unit 482 is connected to the specific color region determination unit 483. The specific color area determination unit 483 is connected to the memory 484 and the red ball area determination unit 464. The control unit 420 is connected to the region dividing unit 481 and the specific color region determining unit 483.

  The area dividing unit 481 divides the Cb signal and the Cr signal output from the color conversion unit 461 into block areas based on the block size from the control unit 420, and outputs the block areas to the statistic calculation unit 482. To do.

  The statistic calculation unit 482 calculates the average value Cba (x, y) and the standard deviation value Cbs (x, y) of the Cb signal in the block region of the Cb signal, and the average value Cra of the Cr signal in the block region of the Cr signal. (X, y) and the standard deviation value Crs (x, y) are calculated. The statistic calculation unit 482 outputs these calculated values to the specific color region determination unit 483. Here, x represents the coordinates of the block area in the horizontal direction, and y represents the coordinates of the block area in the vertical direction. The upper left of the image is the origin (0, 0). The value of x increases as the image moves to the right (horizontal scanning direction). The value of y increases toward the bottom of the image (the direction orthogonal to the horizontal scanning direction).

  The specific color region determination unit 483 receives the above average value and standard deviation value. Also, the specific color area determination unit 483 receives, from the control unit 420, specific color area information that defines a red ball area candidate in a two-dimensional plane (hereinafter referred to as a hue plane) having the Cb signal and the Cr signal as coordinate axes. The

  Here, the specific color area information is information for specifying an area in the hue plane, and may be specified by, for example, a hue angle and saturation, or may be specified by CbCr coordinates.

  The specific color area determination unit 483 receives normal color area information defining the color areas of the mucous membrane and blood vessels when the large intestine is generally observed from the control unit 420. The normal color area information is information for specifying an area in the hue plane, and may be specified in the same format as the specific color area information. Hereinafter, the hue plane area represented by the specific color area information is referred to as a specific color area, and the hue plane area represented by the normal color area information is referred to as a normal color area.

  As shown in FIG. 49, when the average values Cba (x, y) and Cra (x, y) fall within the specific color area, the specific color area determination unit 483 has a red ball area candidate flag Frc ( x, y) is set to “ON”, and the flag Frc (x, y) is recorded in the memory 484. The red ball region candidate flag Frc (x, y) is assigned to each block region as a flag representing the red ball region candidate. If the block area is not determined to be a red ball area candidate, the specific color area determination unit 483 sets the red ball area candidate flag Frc (x, y) of the block area to “OFF” and the flag Frc (x, y). Is stored in the memory 484.

  When the standard deviation value Cbs (x, y) is smaller than the threshold value ThCb and the standard deviation value Crs (x, y) is smaller than the threshold value ThCr, the specific color area determination unit 483 determines the red ball area candidate flag of the block area. Leave Frc (x, y) unchanged. The threshold values ThCb and ThCr are stored in the memory 484. On the other hand, if the standard deviation value Cbs (x, y) is greater than or equal to the threshold ThCb, or if the standard deviation value Crs (x, y) is greater than or equal to the threshold ThCr, the red region candidate flag Frc ( Set x, y) to 'OFF'. The specific color region determination unit 483 outputs the set red ball region candidate flag Frc (x, y) to the red ball region determination unit 464.

  Here, the threshold value ThCb and the threshold value ThCr are the cases where the average value Cba (x, y) and Cra (x, y) of the block area in the color image captured in the past are included in the normal color area, and The maximum deviations MaxCbss and MaxCrss of the standard deviation values Cbs (x, y) and Crs (x, y) in the block area are multiplied by a predetermined coefficient (less than 1). The predetermined coefficient is stored in advance in the control unit 420 and acquired from the control unit 420.

  Based on the maximum standard deviations MaxCbss and MaxCrss of the past standard mucous membrane area, the block area where the color variation contained in the specific color area is smaller than the normal observed mucosal area is detected as the red ball area candidate. It is to do. That is, in the defocused red ball region, blood vessels in the mucous membrane are not decomposed due to blurring, and color variation between the mucosa and the blood vessels is reduced.

11.5. Defocus Detection Unit FIG. 51 shows a detailed configuration example of the defocus detection unit 463. The defocus detection unit 463 includes a region division unit 491, a frequency component generation unit 492, and a high frequency presence / absence determination unit 493.

  The area dividing unit 491 is connected to the frequency component generation unit 492. The frequency component generation unit 492 is connected to the high frequency presence / absence determination unit 493. The control unit 420 is connected to the region dividing unit 491 and the high frequency presence / absence determining unit 493.

  The region dividing unit 491 divides the luminance signal output from the color conversion unit 461 into block regions based on the block size from the control unit 420, and outputs the divided block cool air to the frequency component generation unit 492.

  The frequency component generation unit 492 performs, for example, DCT (Discrete Cosine Transform) or FFT (Fast Fourier Transform) on the block region, and converts the block region into frequency components. The frequency component generation unit 492 outputs the generated frequency component of each block region to the high frequency presence / absence determination unit 493.

  The high frequency presence / absence determination unit 493 receives the frequency component of each block area, the high frequency determination threshold Thf and the high frequency presence / absence determination threshold Thp from the control unit 420. The high frequency presence / absence determination unit 493 determines that the block region is in the focus state when the maximum frequency of the frequency component having an amplitude value larger than the predetermined value is greater than the high frequency determination threshold Thf and the sum of frequency components is greater than the high frequency presence / absence determination threshold Thp. The defocus state flag Fdf (x, y) is set to “OFF”. When the sum of the frequency components is equal to or lower than the high frequency presence / absence determination threshold Thp, the high frequency presence / absence determination unit 493 determines that the state is the defocused state and sets the defocus state flag Fdf (x, y) to ‘ON’. The high frequency presence / absence determining unit 493 outputs the set defocus state flag Fdf (x, y) to the red ball region determining unit 464.

  Note that the defocus detection unit 463 is not limited to the above configuration. For example, the defocus detection unit 463 processes the luminance signal by a simple high-pass filter or band-pass filter, adds the absolute value of the extracted high-frequency component signal in the block region, and determines the threshold value using a predetermined threshold value. Thus, the defocus state may be determined.

  The red ball region determination unit 464 obtains a red ball region flag Fr (x, y) by taking the logical product of the red ball region candidate flag Frc (x, y) and the defocus state flag Fdf (x, y). As shown in FIG. 52, the red ball region determination unit 464 sets a boundary line including a plurality of block regions whose red ball region flag Fr (x, y) is “ON”, for example, by approximating it with a closed curve, and this boundary line The inside area is taken as the final determination result of the red ball area. Then, the red ball region determination unit 464 sets all red ball region flags Fr (x, y) in this region to “ON” and outputs the red ball region flag Fr (x, y) to the scaling parameter setting unit 465. To do.

11.6. Scaling Parameter Setting Unit FIG. 53 shows a detailed configuration example of the scaling parameter setting unit 465. The scaling parameter setting unit 465 includes an akadama area parameter calculation unit 495 and a block scaling parameter calculation unit 496.

  The red ball region parameter calculation unit 495 is connected to the block scaling parameter calculation unit 496. The block scaling parameter calculation unit 496 is connected to the scaling processing unit 414. The control unit 420 is connected to the red ball region parameter calculation unit 495 and the block scaling parameter calculation unit 496.

As shown in FIG. 52, the red ball region parameter calculation unit 495 displays the center of gravity position R ₀ of the region where the input red ball region flag Fr (x, y) is “ON”, and the red ball region boundary from the center of gravity position R _0. The maximum distance Rr is calculated. Specifically, based on the block size from the control unit 420, the center coordinates of all the block areas whose red ball area flag Fr (x, y) is “ON” are calculated, and the average of the calculated center coordinates in the red ball area is calculated. Let the value be the center of gravity position _R0 . Further, the maximum distance between the center coordinates of the block area in the red ball area and the gravity center position R ₀ is searched to obtain the maximum distance Rr. The red ball region parameter calculation unit 495 outputs the calculated center of gravity position R ₀ and the maximum distance Rr to the block scaling parameter calculation unit 496.

The block scaling parameter calculation unit 496 defines a red ball region as a circle based on the input center-of-gravity position R ₀ and the maximum distance Rr (hereinafter, this circle is referred to as a red ball region). The center of the red ball region is the gravity center position _R0 , and the radius of the red ball region is the maximum distance Rr. Further, the entire imaging region is also defined by a circle (hereinafter, this circle is called an imaging region). The center of the imaging region is the center of the optical axis and is determined in advance based on the optical system together with the radius of the imaging region. The center and radius of the imaging region are input from the control unit 420 to the block scaling parameter calculation unit 496.

The block scaling parameter calculation unit 496 determines a scaling parameter for each block region from the red ball region and the imaging region. Specifically, the center position of the red-out area as _{R 0,} when the center position of the block area _B 0 (x, y) and determines a straight line passing through the _{R 0} and _B 0 (x, y). Then, an intersection point IS _b (x, y) between the straight line and the boundary of the imaging region is calculated, and an intersection point Rb (x, y) between the straight line and the boundary of the red ball region is calculated.

The block scaling parameter calculation unit 496 calculates the distance DS (R) between the line segment | R ₀ −R _b (x, y) | and the line segment | IS _b (x, y) −R _b (x, y) | The ratio DRatio (x, y) with x, y) is calculated by the following equation (27). Further, the distance DB (x, y) between R ₀ and B ₀ (x, y) is calculated by the following equation (28). DRatio (x, y) and DB (x, y) are calculated for all block areas.
DRatio (x, y)
_{= | IS b (x, y} ) -R b (x, y) | / | R 0 -R b (x, y) |
= | IS _b (x, y) −R _b (x, y) | / Rr (27)
DB (x, y)
= | R ₀ -B ₀ (x, y) | (28)

The block scaling parameter calculation unit 496 performs scaling processing on the ratio DRatio (x, y), the distance DB (x, y), the center R _{0 of} the red ball region, and the radius Rr of the red ball region calculated corresponding to each block. To the unit 414.

  The scaling processing unit 414 accumulates the parameters for each block area input from the block scaling parameter calculation unit 496 for one screen, and creates a magnification conversion table for each pixel. As shown in FIG. 54, the distance ratio DRatio (x, y) between the red ball region and the non-red ball region is used to select one from the correspondence curves of the normalized distances after scaling with respect to the normalized distance before scaling. Used. This curve may be described by a look-up table (LUT) or may be described by a polynomial coefficient. This curve is stored in a ROM table (not shown) in the scaling unit 414, and the address of the ROM table corresponds to the distance ratio DRatio (x, y).

The scaling processing unit 414 calculates the normalized distance ND (x, y) of each block area by the following equation (29).
ND (x, y) = DB (x, y) / (1 + DRatio (x, y)) × Rr
(29)

  Based on the calculated normalized distance ND (x, y), the scaling processing unit 414 determines a correspondence relationship between normalized distances before and after scaling that represents each block region (hereinafter, referred to as a representative correspondence relationship). To do. The representative correspondence is a normalized distance before scaling with respect to the normalized distance after scaling shown in FIG. The scaling processing unit 414 calculates the correspondence between the normalized distances before and after scaling for each pixel in the target block area by interpolation using the representative correspondence relation of the target block area and the representative correspondence relation of the surrounding block areas. To do. For example, linear interpolation can be used as the interpolation process.

The scaling processing unit 414 determines the correspondence between the actual distance from the center position R ₀ of the red ball area to the pixel position P (i, j) in the target block area and the normalized distance before and after scaling at the pixel position. Multiply and convert the correspondence at the normalized distance to the correspondence at the actual distance. Here, i and j indicate the horizontal coordinate and vertical coordinate of pixel pitch accuracy, respectively. The pixel position (real number accuracy) before scaling corresponding to the pixel position after scaling (integer precision) is a line connecting the pixel position P ′ (i, j) after scaling and the center position R ₀ of the red ball area. Since it is on the minute, it is uniquely determined as a coordinate value corresponding to the actual distance.

  The scaling processing unit 414 calculates the pixel value at the determined coordinate value before scaling by interpolation using a plurality of peripheral pixel values of the coordinate value (the pixel value is at an integer precision pixel position). For example, linear interpolation or BiCubic can be used as the interpolation process. In this way, it is possible to generate a zoomed image in which a region of interest that is not a red ball region is relatively enlarged with respect to the red ball region.

  The determination process of the red ball area may be always performed, or the determination process of the red ball area may be manually controlled ON / OFF by a switch (not illustrated) included in the external I / F unit 550. That is, when the user determines that a red ball region has occurred, the red ball region determination process may be performed by turning on the switch.

  According to the above embodiment, the captured image is an image in which a living body is captured. The attention area setting section 413 includes a contact state information acquisition section (color conversion section 461, red ball area candidate detection section 462, defocus detection section 463, red ball area determination section 464 in FIG. 48). The contact state information acquisition unit acquires contact state information (red ball region information) between the scope tip and the living body based on the captured image. The attention area setting unit 413 sets the attention area based on the contact state information.

  Specifically, the contact state information acquisition unit detects a specific color region (a color region representing a red ball region on the hue plane) having a color in a specific color range in the captured image, and determines the contact region based on the specific color region. Set. The attention area setting unit 413 sets an area other than the contact area as the attention area.

  For example, in the present embodiment, the red ball region candidate detection unit 462 (specific color region detection unit in a broad sense) detects a specific color region, and the red ball region determination unit 464 (contact region setting unit in a broad sense) detects the contact region. Set.

  In this way, it is possible to determine the contact area with the living body (red ball area; observation area in a broad sense), set the area other than the contact area as the attention area, and perform dimming control. As described above, in a wide-field optical system in which image distortion occurs due to distortion, even if there is a contact area, the insertion direction of the lumen may be reflected in another area. Therefore, by controlling the brightness of the area other than the contact area, the visibility in the insertion direction can be improved, and the insertion time can be shortened.

  In the present embodiment, the contact state information acquisition unit converts the image of the specific color region into a spatial frequency component, and sets the specific color region as the contact region when the converted spatial frequency component is equal to or less than a predetermined threshold. . For example, when it is determined that the maximum frequency at which the amplitude value of the converted spatial frequency component is greater than or equal to a predetermined value is less than the first threshold value, or the sum of the amplitude values of the spatial frequency components is less than or equal to the second threshold value, the specific color region Set as the contact area

  For example, in the present embodiment, the defocus detection unit 463 (spatial frequency conversion unit in a broad sense) converts an image of a specific color region into a spatial frequency component, and whether or not the converted spatial frequency component is equal to or less than a predetermined threshold value. Determine whether.

  In this way, since the high frequency component of the image is considered to be small due to defocus in the contact area, the contact area can be detected by determining the threshold value of the high frequency component of the specific color area.

  In the present embodiment, as described with reference to FIG. 52, the contact state information acquisition unit sets a circle including the specific color area, and sets the inside of the circle as the contact area. The dimming control unit performs dimming control on a region outside the circle of the contact region.

The contact state information is information indicating the position of the contact region (the center position R _{0 of the} circle) and the size (radius Rr).

  In this way, by setting the contact area as a circular area and controlling the brightness of the area outside the circle by dimming, it is possible to display areas other than the contact area with appropriate exposure.

  In the present embodiment, as shown in FIG. 47, the endoscope apparatus may include a scaling unit 414. As described with reference to FIG. 52, the scaling processing unit 414 determines the distance Rr from the center of the circle to the circumference and the circle on the straight line connecting the center of the circle of the contact area and the outer circumference of the captured image (the outer circumference of the imaging area). Based on the ratio DRatio (x, y) of the distance DS (x, y) from the circumference to the circumference, the enlargement ratio of the attention area (corresponding relationship between the normalized distances before and after zooming in FIG. 54) is set.

  In this way, the region of interest can be enlarged and displayed by scaling processing along a straight line connecting the outer periphery of the captured image from the center of the circle of the contact region.

  As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in each embodiment or modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

  In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings.

100 light source unit, 101 white light source, 102 light source aperture, 103 drive unit,
104 rotation color filter, 105 rotation drive unit, 106 condenser lens,
107 light source control unit, 200 operation unit, 201 bending operation lever,
202 insertion port, 203 manipulated variable detection sensor,
300 insertion section, 301 light guide fiber, 303 objective optical system,
304 imaging device, 305 A / D converter, 306 bending operation wire,
307 insertion channel, 308 treatment instrument sensor, 309 half mirror,
310 imaging device, 311 A / D converter, 312 magnetic coil,
313 Memory, 314, 315 Illumination lens 400 Control device, 410 Image processing unit, 411 Preprocessing unit,
412 synchronization unit, 413 attention area setting unit, 414 scaling processing unit,
415 post-processing unit, 416 photometry unit, 417 synchronization unit, 418 pre-processing unit,
419 gradation conversion unit, 420 control unit, 430 state information acquisition unit,
431 bending angle detection unit, 432 bending movement amount detection unit,
436 subject information acquisition unit, 437 observation information acquisition unit, 441 luminance conversion unit,
442 addition unit, 443 identification unit, 445 region division unit,
446 histogram calculation unit, 447 gradation conversion characteristic calculation unit,
448 conversion unit, 451 local region setting unit, 452 feature amount calculation unit,
453 lesion region detection unit, 454 labeling processing unit, 455 region selection unit,
456 coordinate calculation unit, 457 association unit, 459 photometric processing condition setting unit,
461 color conversion unit, 462 red ball region candidate detection unit,
463 defocus detection unit, 464 red ball region determination unit,
465 zoom parameter setting unit, 471 distortion correction unit,
472 image memory, 473 motion detection unit, 474 setting unit,
481 area division unit, 482 statistic calculation unit, 483 specific color area determination unit,
484 memory, 491 region division unit, 492 frequency component generation unit,
493 high-frequency presence determination unit, 495 red ball region parameter calculation unit,
496 block scaling parameter calculation unit, 500 display unit,
550 External I / F unit, 700 shape detection unit, 701 to 703 color filter,
704 rotary motor,
A, B weighting factor, a (m, n) local region, B ₀ (x, y) block center,
B2, G2 image signal, DB (x, y) distance, DRatio distance ratio,
DS (x, y) distance, IS _b (x, y) intersection, LC1, LC2 rays,
LW length, N, M Number of horizontal pixels, (N / 2, M / 2) Image center coordinates,
nB, nG color filter, R radius, rd distance, R ₀ center position,
R _b (x, y) intersection, Rr radius, R _s radius, SF1 to SF3 plane,
(X, y) coordinates, (x ₀ , y ₀ ) coordinates of the center pixel of the region of interest,
α enlargement ratio, β reduction ratio, θ bending angle

Claims (5)

An image acquisition unit for acquiring a captured image including a subject image obtained by irradiating the subject with illumination light from the light source unit;
An attention area setting section for setting an attention area for the captured image;
Based on the set region of interest, a dimming control unit that performs dimming control on the amount of illumination light; and
Including
The captured image is a special light image in which a subject image having information in a specific wavelength band is captured, and a normal light image in which a subject image having information in a wavelength band of white light is captured,
The attention area setting unit calculates a feature amount based on the special light image , sets the attention area based on an area having a predetermined feature amount ,
The dimming control unit performs the dimming control of the normal light image based on the set attention area so that the brightness of the attention area is maintained at a predetermined brightness. Endoscopic device. In claim 1 ,
The endoscope apparatus according to claim 1, wherein the specific wavelength band is a band narrower than the wavelength band of the white light. In claim 2 ,
The special light image and the normal light image are in-vivo images obtained by imaging the inside of the living body,
The endoscope apparatus, wherein the specific wavelength band included in the in-vivo image is a wavelength band of a wavelength that is absorbed by hemoglobin in blood. In claim 1 ,
The special light image and the normal light image are in-vivo images obtained by copying the inside of the living body,
The endoscope apparatus, wherein the specific wavelength band included in the in-vivo image is a wavelength band of fluorescence emitted from a fluorescent material. In claim 1 ,
The special light image and the normal light image are in-vivo images obtained by copying the inside of the living body,
The endoscope apparatus, wherein the specific wavelength band included in the in-vivo image is an infrared light wavelength band.