JP2012135345A - Endoscope apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an endoscope apparatus and program, etc. which make it possible in enlarged observation to time sequentially display a display image with a small blur quantity equivalent to a focused state, and with suppressed noise increase.SOLUTION: The endoscope apparatus includes: an image acquisition unit (image configuration processing unit 310) which time sequentially acquires a captured image in an enlarged observation state in which magnification of an optical system is a higher than that of a regular observation state; a blur quantity information extraction unit (texture extraction unit 320) which extracts blur quantity information from the captured image in the enlarged observation state; and a blur quantity correction unit (texture correction quantity calculation unit 340, texture correction unit 360, and compositing unit 370) which correct the captured image on the basis of the extracted blur quantity information.

Description

  The present invention relates to an endoscope apparatus, a program, and the like.

  Conventionally, there has been a demand for improving the accuracy of detection of lesions in body cavities in endoscopic diagnosis, and enlargement that achieves improved detection accuracy by magnifying the difference in tissue between the lesioned part and the normal part at a magnification equivalent to that of a microscope. An endoscope provided with an optical system (hereinafter referred to as a magnifying endoscope) is generally known.

  Some of these magnifying endoscopes have magnifications of several tens to several hundreds of times. By using in combination with staining and spreading and blood vessel-enhanced images (also referred to as NBI images) using narrow-band illumination light, A fine structure and a pattern of blood vessel running can be observed. It is known that there is a difference in these patterns between the lesioned part and the normal part, which is one criterion for lesion diagnosis.

Japanese Patent Laid-Open No. 10-165365 JP 2003-140030 A

  However, when such magnified observation is performed, the depth of field becomes extremely narrow as compared with the normal observation as the magnification is increased. In that case, it is difficult to keep the distal end position of the endoscope insertion part (hereinafter referred to as a scope or an imaging part) with respect to the subject within the in-focus range, and it is considerable to keep observing an always focused image. The current situation requires skill.

  If the scope alignment work time associated with focusing during such magnified observation increases, the overall diagnosis time will take a long time. As a result, there arises a problem that the burden on the patient increases with the fatigue of the doctor.

  As a method for solving such a problem, Japanese Patent Application Laid-Open No. 2004-228867 has a method of setting a region assumed to be outside the depth range in one image and performing blur recovery processing to realize depth expansion. In this method, distance detection is performed while observing the inside of a lumen with an endoscope, and blur recovery processing is performed on a predetermined distance range region. The disadvantage of this method is that distance detection is required, and if the distance detection accuracy is low, an error occurs in the determination of the blur area, and the blur recovery process that is used uses deconvolution, etc. Concerns over-correction and noise amplification.

  As another method for solving the problem, as disclosed in Patent Document 2, autofocusing is performed during magnified observation. In this autofocus process, since focus control is performed using a contrast value, focus is not always achieved, and focused images and blurred images are alternately and repeatedly displayed in a short time series. Such autofocus processing is not problematic because it is hardly discernable even when the focused image and blurred image are repeated periodically when the frame rate is sufficiently high, but it becomes noticeable when the frame rate is slow. There is.

  According to some aspects of the present invention, an endoscope apparatus and a program capable of time-series display of a display image with a small amount of blur equivalent to the in-focus state and suppressed noise increase in magnified observation Etc. can be provided.

  In addition, according to some aspects of the present invention, since a high-definition image with a small change in blur amount in time series can be displayed, the lesion diagnosis time can be shortened by improving the visibility and the burden on the patient can be reduced. An endoscope apparatus and a program that can be provided can be provided.

  One aspect of the present invention is an image acquisition unit that acquires, in time series, a captured image in an enlarged observation state in which the magnification of the optical system is higher than that in a normal observation state, and the captured image in the enlarged observation state. The present invention relates to an endoscope apparatus that includes a blur amount information extraction unit that extracts blur amount information and a blur amount correction unit that corrects the captured image based on the extracted blur amount information.

  In one aspect of the present invention, the captured image is corrected based on blur amount information of the captured image in the magnified observation state. Because the process is based on the amount of blur information, correction processing is efficiently performed on areas that require blur recovery processing, such as areas that are strongly affected by blur, and correction processing is not performed on other areas. it can. Therefore, it is possible to provide a blur correction image that does not give the user a sense of incongruity.

  Another aspect of the present invention includes an image acquisition unit that acquires a captured image in a time series in an enlarged observation state in which the magnification of the optical system is higher than that in the normal observation state, and the captured image in the enlarged observation state. The present invention relates to a program that causes a computer to function as a blur amount information extraction unit that extracts blur amount information from a blur amount correction unit that corrects the captured image based on the extracted blur amount information.

The system configuration example of the endoscope apparatus of this embodiment. 2 is a configuration example of an image processing unit. The structural example of a texture extraction part. 4A to 4C are diagrams for explaining that the frequency range to be extracted changes according to the amplitude range. FIG. The structural example of a texture correction amount calculation part. The figure explaining the switching process of the observation state by a zoom lever, and the update process of a reference | standard amplitude value. FIG. 7A to FIG. 7C are diagrams for explaining the relationship between the deviation in the Z direction and the correction amount. Filter transmittance of the color filter. The structural example of a rotation filter. The other system structural example of the endoscope apparatus of this embodiment. The schematic diagram of the auto-focus process by a contrast method. The other example of a structure of an image process part. The other structural example of a texture extraction part. 14A and 14B show examples of subband image division. The other structural example of a texture correction amount calculation part. The structural example of a texture correction part. 3 is a configuration example of a focus position calculation unit. The figure explaining the relationship of the corrected amount with respect to an amplitude gain amount. FIGS. 19A to 19C are diagrams illustrating the relationship between the observation magnification of the optical system and the frequency range to be extracted.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. As described above, according to the method of the present embodiment, it is possible to acquire an image with a small amount of blur equivalent to the in-focus state without increasing noise.

  In the magnification observation mode using an observation magnification of several tens to several hundreds of times, the depth of field becomes very narrow, and it is difficult to acquire a focused image. Therefore, an image with a small amount of blur equivalent to the in-focus state is obtained by performing blur recovery processing on the image that is out of focus and has been blurred.

  The method of this embodiment will be described. The image acquired by the image acquisition unit (image configuration processing unit 310) of the present embodiment is a blurred image because it is difficult to focus by magnified observation. At this time, the blurring effect appears greatly in a region showing a minute gradation change such as a mucous membrane surface or a blood vessel structure of a living body, and the blurring effect is small in a region showing a large gradation change such as a fold. . In other words, it is largely due to the influence of the amplitude reduction of the fine structure image of the mucous membrane and blood vessels due to out-of-focus. Therefore, a fine structure image (such as a texture image or a predetermined frequency band image in the present embodiment) such as a mucous membrane or a blood vessel is extracted from the captured image.

  Then, a correction process is performed on the extracted texture image so as to enhance the reduced amplitude value (for example, amplify the signal value). Then, the texture that is useful for diagnosis is brought into focus by synthesizing the captured image that is the image (blurred image) acquired by the image acquisition unit and the texture image that has been subjected to the correction process. Acquire the corrected image. By doing this, blur recovery processing is performed for areas that are strongly affected by blur, and blur recovery processing is not performed for areas that are less affected by blur and noise components, etc. A blur correction image can be acquired. That is, it is possible to acquire an image in which blur recovery processing is performed on a texture that is useful for diagnosis.

  Hereinafter, in the first embodiment, a method of extracting a texture image by using a two-stage bilateral filter will be described. In this embodiment, it is necessary to extract images of mucosal surfaces, blood vessels, etc., except for structures such as folds. Therefore, the applied filter has not only frequency characteristics (for example, high pass and band pass) but also amplitude characteristics. Here, the amplitude is obtained from the pixel value of the target pixel and the pixel values of the peripheral pixels, and the amplitude value of the fold is large, and the amplitude value of the mucous membrane and blood vessel is small. Note that it is sufficient that an image having an amplitude value corresponding to a small amplitude to a small amplitude and in a predetermined frequency range can be acquired, and a filter therefor is not limited to a bilateral filter.

  In the second embodiment, texture images are extracted using subband images as shown in FIGS. 14A and 14B. Further, the second embodiment is characterized in that an image with a small amount of blur is obtained by performing not only the image processing-based blur recovery processing but also the optical autofocus processing.

2. 1. First Embodiment 2.1 Configuration FIG. 1 is a block diagram illustrating an overall configuration of an endoscope apparatus according to a first embodiment. The endoscope apparatus constituting the present invention includes a light source unit 100, an imaging unit 200, a processor unit 300, a display unit 400, and an external I / F unit 500.

  The light source unit 100 includes a white light source 101 and a condenser lens 104 that condenses illumination light from the white light source 101 on the incident end face of the light guide fiber 201.

  The imaging unit 200 is formed to be elongated and bendable, for example, to enable insertion into a body cavity. The imaging unit 200 includes a light guide fiber 201 for guiding the light collected by the light source unit 100, an illumination lens 202 for diffusing the light guided to the tip by the light guide fiber 201 and irradiating the observation target, The objective lens 203 that condenses the reflected light returning from the observation target, the focal position control unit 206, the focal position adjustment lens 207, the imaging element 209 for detecting the focused imaging light, and the imaging element 209. An A / D conversion unit 210 that converts the photoelectrically converted analog signal into a digital signal is provided. Here, the image pickup device 209 is a single plate image pickup device (complementary color or primary color) in which color filters are arranged in each pixel in a predetermined arrangement, and a CCD, a CMOS, or the like can be used.

  The processor unit 300 includes an image processing unit 301 and a control unit 302. The display unit 400 is a display device capable of displaying a moving image such as a CRT or a liquid crystal monitor.

  The external I / F unit 500 is an interface for performing input from the user to the endoscope apparatus, a power switch for turning on / off the power, a shutter button for starting a photographing operation, and photographing. A mode switching knob for switching between modes and various other modes (for example, moving the zoom lever shown in FIG. 6 to the magnification observation mode side moves the focus position adjustment lens, and obtains a focusing state at a closer position. ) And the like. The external I / F unit 500 outputs input information to the control unit 302.

  Next, the details of the relationship among the focal position control unit 206, the focal position adjustment lens 207, and the control unit 302, which are components of the imaging unit 200, will be described. First, two observation modes having different observation magnifications of the endoscope apparatus of the present embodiment will be described.

  The two observation modes are a normal observation mode and a magnification observation mode. The normal observation mode is a mode in which a screening inspection is mainly performed with a pan-focus wide-field image. The magnification observation mode is a scrutinization mode for closely examining whether or not the lesion is malignant by magnifying the mucosal structure, the blood vessel running state, etc. in the vicinity of the lesion found by the screening test.

  In order to switch between these two modes, the control unit 302 transmits a focus position adjustment control signal to the focus position control unit 206 based on an operation of the external I / F unit 500 on the zoom lever shown in FIG. The focal position control unit 206 controls the movement of the focal position adjustment lens 207 based on the received control signal.

  Next, specific control and user operations in the magnification observation mode of the focal position control unit 206 will be described.

  When the magnification observation mode is selected, the focus position adjustment lens 207 is moved via the focus position control unit 206 so as to be a predetermined focus position determined in advance in the proximity state. As an example of an operation procedure for smoothly shifting to magnified observation, first, as close as possible to the observation target while maintaining the in-focus state in the normal observation mode. At this point, the user operates the zoom lever of the external I / F unit 500 to switch to the magnification observation mode. From this state, the zoom lever is operated to change the magnification continuously (or stepwise), and at the same time, the depth is narrowed, so the scope is brought closer to the observation object to focus. By such an operation, a desired magnification and in-focus state can be acquired.

  In other words, if the user does not perform an advanced operation of bringing the scope tip closer to the observation target with a delicate sense of distance as the magnification increases, the user will observe the blurred image, and the enlarged image of the observation target to be examined will always be in focus on the screen. It is very difficult to continue framing in a wet state.

  On the other hand, when the normal observation mode is selected, the focal position adjustment lens 207 is moved via the focal position control unit 206 so that the focal position is obtained to obtain a sufficient angle of view and depth of field for performing the screening examination. Will be allowed to. In this case, since it is always in focus, the user can observe only by moving the scope tip to the observation object, and does not impose complicated and delicate operations.

  Next, details of the image processing unit 301 constituting the processor unit 300 will be described based on the block diagram of FIG.

  The image processing unit 301 includes an image configuration processing unit 310, a texture extraction unit 320, a texture correction amount calculation unit 340, a texture correction unit 360, and a synthesis unit 370.

  Next, the data flow between the components will be described. The time-series captured image output from the A / D conversion unit 210 is input to the image configuration processing unit 310. A control unit 302 is further connected to the image configuration processing unit 310, and processing parameters stored in advance in the control unit 302 (OB clamp value, WB coefficient, color correction coefficient, gradation conversion table, contour enhancement level, etc.). Is input to the image configuration processing unit 310. Based on these processing parameters, the image configuration processing unit 310 generates a time-series display image that can be observed by the display device from the input time-series captured image, and outputs the time-series display image to the texture extraction unit 320 and the synthesis unit 370.

  The texture extraction unit 320 receives the time-series display image output from the image configuration processing unit 310, and the control unit 302 receives an observation mode signal (enlarged observation mode or normal observation mode) specified by the external I / F unit 500. And the magnification information of the imaging unit 200 are input. The texture extraction unit 320 first operates when the observation mode becomes the magnification observation mode, and extracts a texture image for each frame of the input time-series display image. Here, the texture image is a minute gradation change image corresponding to the fine mucosal structure or blood vessels on the surface of the living body to be observed, and a large gradation change such as a region where illumination light is regularly reflected on the mucous membrane surface layer or folds. The portion exhibiting is an image that has been removed as much as possible. Details of the texture extraction unit 320 will be described later.

  The texture image extracted by the texture extraction unit 320 is input to the texture correction amount calculation unit 340, and a correction amount for the texture image is calculated based on the observation mode signal from the control unit 302, the texture reference threshold value, and the amplitude value of the texture image. The The calculated correction amount is output to the texture correction unit 360. Here, the texture reference threshold is a determination threshold for selecting a reference amplitude value from the amplitude values of a plurality of texture images calculated in time series by the texture correction amount calculation unit 340. The correction amount is calculated based on the amplitude gain amount that makes the amplitude value of the texture image extracted at the present time the same as the reference amplitude value. However, when the amplitude gain amount becomes larger than a predetermined threshold value, the correction amount is corrected to 0 because there is no significant information in the extracted texture image and the image is out of focus. Here, as shown in FIG. 18, the correction amount may be provided with two threshold values Th1 and Th2 with respect to the amplitude gain amount, and may be continuously changed so as to gradually reduce the correction amount between the two threshold values. Details of the texture correction amount calculation unit 340 will be described later.

  The texture correction unit 360 multiplies the input texture image and the correction amount to generate a corrected texture image and outputs it to the synthesis unit 370.

  The synthesizing unit 370 receives the time-series display image output from the image configuration processing unit 310 and the corrected texture image corresponding to the time-series display image output from the texture correction unit 360, and the observation mode signal of the control unit 302 is input. Is in the magnified observation mode, the two images are added to generate a blur-corrected image.

  The blur correction image is output to the display unit 400, and the blur corrected image is displayed. When the observation mode is the normal observation mode, the time-series display image is output to the display unit 400 as it is without being added to the corrected texture image.

2.2 Detailed configuration of the blur amount information extracting unit Next, details of the blur amount information extracting unit will be described. In the present embodiment, the blur amount information extraction unit corresponds to the texture extraction unit 320 in FIG.

  Details of the texture extraction unit 320 will be described with reference to the block diagram of FIG. The texture extraction unit 320 includes a region extraction unit 321, a filter coefficient calculation unit 322, a filter processing unit 323, a noise amount estimation unit 324, a subtraction processing unit 325, a region extraction unit 326, a filter coefficient calculation unit 327, a filter processing unit 328, a filter parameter. The control unit 329 is configured.

  Next, the data flow between the components will be described. The observation mode signal and magnification information output from the control unit 302 are input to the filter parameter control unit 329 and converted into filter coefficient generation parameters. The converted filter coefficient generation parameter is output to the filter coefficient calculation unit 322 and the noise amount estimation unit 324.

  The time-series display image output from the image configuration processing unit 310 is input to the region extraction unit 321, the target pixel and its surrounding p × q pixel region (p and q are arbitrary positive integers) are extracted, and the filter coefficient is calculated. Output to the unit 322 and the filter processing unit 323. The time of the time-series display image captured here is t, and the pixel value at the three-dimensional coordinate including the two-dimensional coordinate (x, y) is P (x, y, t), and the target pixel to be filtered Let (x0, y0, t) be the three-dimensional coordinate. Since the time-series display image is a color image, the pixel value P (x, y, t) is a three-dimensional vector composed of three channels of R, G, and B.

  The filter coefficient calculation unit 322 calculates a filter coefficient based on the filter coefficient generation parameter output from the filter parameter control unit 329. The filter coefficient generation parameter input here is an amount for converting and controlling the absolute value of the difference between the pixel values of the pixel of interest and the surrounding pixels in the p × q pixel region into a filter coefficient. For example, when a bilateral filter configured by the product of two Gaussian functions shown in Expression (1) is used, the index value σ1s with respect to the absolute value of the difference between the pixel values of the target pixel and its surrounding pixels is the filter coefficient generation parameter. It becomes.

Here, the filter coefficient F1 (x, y, t) is expressed by the following equation (1), Σxy is the sum in the p × q pixel region, and Pf (x0, y0, t) is the pixel value after filtering. Show.

The filter processing unit 323 uses the pixel value P (x, y, t) of the p × q pixel region output from the region extraction unit 321 and the filter coefficient F1 (x, y, t) output from the filter coefficient calculation unit 322. Filter processing is performed, and the pixel value Pf (x0, y0, t) after the filter processing is calculated by the following equation (2). The calculation result is output to the subtraction processing unit 325 and the noise amount estimation unit 324.

  The subtraction processing unit 325 includes the pixel value P (x0, y0, t) of the time-series display image output from the image configuration processing unit 310 and the pixel value Pf (x0, y0) after the filter processing output from the filter processing unit 323. , t) is input, a difference value D (x0, y0, t) obtained by subtracting Pf (x0, y0, t) from P (x0, y0, t) is calculated, and the calculated difference value D (x0, y0) is calculated. , t) is temporarily stored. The stored difference image is output to the region extraction unit 326.

  The noise amount estimation unit 324 includes a noise amount estimation model (a function for converting a pixel value into a noise amount) output as a filter coefficient generation parameter from the filter parameter control unit 329, and a pixel value after filter processing from the filter processing unit 323. Pf (x0, y0, t) is input, and a noise amount N (x0, y0, t) corresponding to Pf (x0, y0, t) is estimated. The estimated noise amount N (x0, y0, t) is output to the filter coefficient calculation unit 327.

  Here, the noise amount estimation model captures a gray chart having a plurality of reflectances having uniform brightness in a predetermined area, and the predetermined area in the gray chart of the time-series display image output from the image configuration processing unit 310 is captured. The average value and the standard deviation σ of the pixel value P (x, y, t) are calculated and obtained. From these multiple average values and standard deviations, a more detailed correspondence between each pixel value that can be taken as the definition area and the noise amount (for example, the standard deviation is the noise amount) is obtained by a function fitted with a polynomial approximation curve. For example, a function obtained by converting the function into a polygonal line approximation or a look-up table is used.

  The region extraction unit 326 extracts a v × w pixel region (v and w are arbitrary positive integers) around the target pixel from the difference image output from the subtraction processing unit 325, a filter coefficient calculation unit 327, and filter processing To the unit 328.

Based on the noise amount N (x0, y0, t) output from the noise amount estimation unit 324, the filter coefficient calculation unit 327 and the difference pixel value D (x0, y0, t) of the v × w pixel region and the peripheral difference A filter coefficient is calculated based on the absolute value of the difference between the pixel values D (x, y, t). Here, the noise amount N (x0, y0, t) is a conversion control amount to the filter coefficient. When the bilateral filter shown in Expression (3) is used, the index value σ2s with respect to the absolute value of the difference between the target difference pixel value and the peripheral difference pixel value is expressed as a noise amount N (x0, y0, t).

The filter processing unit 328 outputs the difference pixel value D (x, y, t) of the v × w pixel region output from the region extraction unit 321 and the filter coefficient F 2 (x, y, t) output from the filter coefficient calculation unit 327. The filter process is performed by the above, and the difference pixel value Df (x0, y0, t) after the filter process is calculated by the following expression (4). The calculation result is output to the texture correction amount calculation unit 340.

  The characteristics of the two bilateral filters F1 (x, y, t) and F2 (x, y, t) and the concept of texture extraction will be described with reference to FIGS. 4 (A) to 4 (C).

  First, the index value σ1s for controlling the filter coefficient of the filter F1 (x, y, t) is set to a value sufficiently larger than the index value σ2s of the filter F2 (x, y, t).

  Pixel value difference | P (x, y, t) −P (x0, y0, t) | in the second term that mainly determines the size of the filter coefficient, or difference pixel value difference | D (x, y, t ) -D (x0, y0, t) | is equal to or lower than the index value σ1s and the pixel value difference is larger than the index value σ2s, the frequency characteristic of the filter F1 (x, y, t) is As indicated by reference numeral 804 in FIG. 4B, the low-pass characteristic is strong. Further, the frequency characteristic of the filter F2 (x, y, t) is weak in the low-pass characteristic as indicated by 803 in FIG. 4B. As a result, a great difference occurs between the two filter characteristics.

  On the other hand, in a region where a pixel value difference larger than the above pixel value difference occurs, the frequency characteristic of the filter F1 (x, y, t) is weak as shown by 802 in FIG. 4A, and the filter F2 (x , y, t), the frequency characteristic becomes even weaker as indicated by 801 in FIG. 4A, but the difference is small.

  Furthermore, in the region where the pixel value difference is smaller than the index value σ2s, the frequency characteristics of the filter F1 (x, y, t) are strong as shown by 806 in FIG. Further, the frequency characteristic of the filter F2 (x, y, t) is also strong in the low-pass characteristic as indicated by reference numeral 805 in FIG. 4C. In this case as well, the difference between the two filter characteristics is small.

  In this way, the texture of the region having a predetermined small pixel value difference is extracted by using the difference in frequency characteristics accompanying the magnitude of the pixel value difference (amplitude value) of the two bilateral filters.

  That is, in the texture region where the pixel value difference is small, the frequency band of the difference image output from the subtraction processing unit 325 has a high-pass characteristic other than the frequency characteristic 804. The high-pass characteristic signal is cut by the filter processing unit 328, and only the high frequency is cut as in the frequency characteristic 803. Therefore, the signal output from the filter processing unit 328 is a wide band sandwiched between the frequency characteristic 803 and the frequency characteristic 804. It has a frequency characteristic of a band pass characteristic. By matching this characteristic to the frequency band of the mucosal structure to be observed and the blood vessel running pattern, it is possible to extract the target texture.

  On the other hand, the band of the signal output from the subtraction processing unit 325 has a high-pass characteristic other than the frequency characteristic 802 at a specular reflection region having a large pixel value difference or a large structural boundary such as a fold. Since the high-pass characteristic signal is cut by the filter processing unit 328 by the frequency characteristic 801, the signal output from the filter processing unit 328 has almost no band to pass through and has a large specular reflection area or folds. The edge structure can be cut.

  Further, FIG. 4C shows a noise component in a region where the pixel value change is very small (here, the change accompanying the amount of noise is sufficiently smaller than the pixel value change in the texture region). Thus, since there is no difference between the two filter characteristics, the signal output from the filter processing unit 328 has almost no band to pass, and the noise component can be cut.

  As a result, only the texture image with a small pixel value difference is output from the texture extraction unit 320. The frequency characteristic and amplitude characteristic of the texture image to be extracted can be controlled using the two index values σ1s and σ2s.

  Particularly, the zoom information of the time series display image is changed continuously (or stepwise) by operating the zoom lever of the external I / F unit 500, and the magnification information is controlled by the filter parameter via the control unit 302. By being input to the unit 329, the filter parameter control unit 329 can control the two index values σ1s and σ2s, and more accurately extract the anatomy according to each magnification as a texture image.

  For example, as shown in FIG. 19A, when the magnification is small, σ1s and σ2s are designated as large values (where σ1s> σ2s), and the frequency characteristics of the corresponding filters F1 and F2 are 1502 and 1501, respectively. A texture image from which the low frequency band 1507 is extracted is used. As shown in FIG. 19B, σ1s and σ2s are made smaller in the state where the magnification is medium than in the case where the magnification is small (where σ1s> σ2s), and the frequency characteristics of the corresponding filters F1 and F2 are 1504 respectively. , 1503 are texture images obtained by extracting the medium frequency band 1508. Further, as shown in FIG. 19C, when the magnification is large, σ1s and σ2s are made smaller than the medium magnification (where σ1s> σ2s), and the frequency characteristics of the corresponding filters F1 and F2 are obtained. The texture images are extracted from the high frequency band 1509 as 1506 and 1505, respectively.

  The reason for changing the frequency band of the texture image to be extracted according to the magnification as described above is that the area of reddening and fading that shows subtle color changes is important at low magnification, and the gland duct and blood vessel running pattern at medium magnification This is because a fine pit pattern and a fine blood vessel structure on the surface of the mucous membrane are important for diagnosis at a high magnification.

2.3 Detailed Configuration of Blur Amount Correction Unit Next, the blur amount correction unit will be described. In the present embodiment, the blur amount correction unit corresponds to the texture correction amount calculation unit 340, the texture correction unit 360, and the synthesis unit 370.

  The texture correction amount calculation unit 340 will be described based on the block diagram of FIG. The texture correction amount calculation unit 340 includes an amplitude calculation unit 341, a reference amplitude update determination unit 342, a reference amplitude storage unit 343, a correction amount calculation unit 344, and a correction amount control unit 345.

  Next, the data flow between the components will be described. The correction amount control unit 345 receives the observation mode signal output from the control unit 302, the texture reference threshold value, the correction amount upper limit value, and the current reference amplitude value from the reference amplitude storage unit 343, and the observation mode signal, the texture reference threshold value, A reference amplitude determination threshold is set based on the reference amplitude value, and the correction amount upper limit value is output to the reference amplitude update determination unit 342 and further to the correction amount calculation unit 344.

  Here, the texture reference threshold is a reference value for determining whether or not the captured time-series display image is in a blurred state, and varies depending on the difference in imaging performance, so that a different value is set depending on the type of scope. The reference value is recorded in a ROM (not shown) held by the imaging unit 200 as the set value, and the reference value is taken into the control unit 302 of the processor unit 300 when the imaging unit 200 is connected to the processor unit 300. It is. When the amplitude value of the texture image is smaller than the texture reference threshold, it is determined that the degree of blur is strong and blur recovery processing cannot be performed.

  The reference amplitude value is a value that is a correction target. Basically, the blur is corrected by performing a correction process for making the amplitude value of the texture image the same as the reference amplitude value. However, since there is exceptional processing, details will be described later.

  The reference amplitude value determination threshold is a threshold for selecting the above-described reference amplitude value. When the observation mode signal changes to the magnification observation mode, the reference amplitude value determination threshold is set to the texture reference threshold value. Sets the average value of the reference amplitude value. One of the conditions for updating the reference amplitude value is that the amplitude value of the texture image exceeds the reference amplitude value determination threshold.

  The amplitude calculation unit 341 calculates an amplitude value MA (t) for the texture image extracted at time t output from the texture extraction unit 320. Here, the amplitude value may be defined by the maximum value with respect to the absolute value of the pixel value of the texture image constituting the texture image, or may be defined by the difference between the maximum value and the minimum value of the pixel value of the texture image. The calculated amplitude value MA (t) is output to the reference amplitude update determination unit 342.

  The reference amplitude update determination unit 342 receives the amplitude value MA (t) output from the amplitude calculation unit 341 and the reference amplitude determination threshold output from the correction amount control unit 345, and is stored in the reference amplitude update determination unit 342. Based on the previous amplitude values MA (t-1) and MA (t-2), it is determined whether or not the amplitude value MA (t) becomes a new reference amplitude value.

  This will be described in more detail with reference to the schematic diagram of FIG. First, immediately after switching to the magnification observation mode, the correction amount control unit 345 outputs the texture reference threshold as the reference amplitude determination threshold to the reference amplitude update determination unit 342. Therefore, the texture image amplitude value at the time of switching to the observation mode is selected as the reference amplitude value except for exceptions such as large movements and blurring, or too close to the observation target and out of focus. And output to the reference amplitude storage unit 343.

  In the state in which the subsequent magnified observation mode is maintained, the reference amplitude determination threshold is, for example, the average value of the texture reference threshold and the reference amplitude value as described above. Therefore, the reference amplitude determination threshold for updating the reference amplitude value is more A large value is set. Therefore, frequent updating of the reference amplitude value is suppressed. The reason for performing such control is to suppress frequent fluctuations in the reference amplitude value, thereby suppressing temporal fluctuations in resolution of the blur-corrected image output from the synthesizing unit 370 that changes in conjunction with the reference amplitude value. There is. As a result, the in-focus state can be maintained apparently.

  On the other hand, the user moves the zoom lever while maintaining the tip of the scope close to the observation target during magnified observation, and operates the endoscope to maintain the desired magnification and in-focus state. Therefore, the focused state cannot be maintained, and the focused state and the non-focused state are repeated as in the time and amplitude value graph of FIG.

  In such a state, the reference amplitude update determination unit 342 determines the amplitude value during the period t−2 to t from the three amplitude values MA (t), MA (t−1), and MA (t−2). Determine whether has a local maximum. This is performed by fitting, for example, three amplitude values with a quadratic expression to determine whether a maximum value exists within this period. When the local maximum value exists, the local maximum value calculated by the quadratic expression becomes a reference amplitude value candidate, and is then compared with the reference amplitude determination threshold value. When the maximum value is larger than the reference amplitude determination threshold value, it is output to the reference amplitude storage unit 343 as a reference amplitude value. On the other hand, when the maximum value is equal to or less than the reference amplitude determination threshold value, it is not regarded as the reference amplitude value and is not output to the reference amplitude storage unit 343. On the other hand, when the minimum value is detected, comparison with the reference amplitude determination threshold value is not performed, and the reference amplitude value is not changed.

  The reason for updating the reference amplitude value in this way is that the observation mucosal structure and blood vessels that cannot be resolved at low magnification appear due to the difference in observation magnification, or the observation position itself is changed in an enlarged state. This is to cope with changes in the temporal characteristics of the texture image.

  The reference amplitude storage unit 343 receives the reference amplitude value output from the reference amplitude update determination unit 342, and updates the reference amplitude value recorded in the past to the latest value. The latest reference amplitude value is output to the correction amount control unit 345 and the correction amount calculation unit 344.

  In the correction amount calculation unit 344, the amplitude value MA (t) at time t output from the amplitude calculation unit 341, the reference amplitude value RMA output from the reference amplitude storage unit 343, and the correction amount output from the correction amount control unit 345 The upper limit value is input, and the correction amount is calculated from the amplitude gain amount at which the amplitude value MA (t) is the same as the reference amplitude value RMA and the two threshold values Th1 and Th2. The calculated correction amount is output to the texture correction unit 360.

Correction amount = Amplitude gain amount-1 (5)
However, amplitude gain amount ≧ 1 and amplitude gain amount <Th1
Correction amount = correction amount upper limit value−α × amplitude gain amount + β (6)
However, the amplitude gain amount ≧ Th1 and the amplitude gain amount <Th2.
Correction amount = 0 (7)
However, the amplitude gain amount <1 or the amplitude gain amount ≧ Th2.
Th1 = correction amount upper limit value + 1, Th2 = Th1 + reference amplitude value / texture reference threshold α = amplitude upper limit value / (Th2-Th1)
β = α × Th2−correction amount upper limit amplitude gain amount = RMA / MA (t)

  Here, the relationship between the amplitude value of the texture image and the correction amount will be described with reference to the schematic diagrams of FIGS. As shown in FIGS. 7A and 7B, the scope tip and the observation target exceed the depth range from the in-focus position in the Z direction (where the Z direction is the optical axis direction of the optical system provided at the scope tip). In addition, when there is ±± blurring, the amplitude value of the texture image takes the largest value at the in-focus position. In other words, during magnification observation, if there is a blur of ± Δ, an image that is in focus and an image that is in focus are continuously observed, and a time-series display image that fully draws out the resolution of the optical system is observed. It will not be done.

  Therefore, as shown in FIG. 7C, a corrected texture image obtained by correcting a decrease in the amplitude value of the texture image due to the blur in the Z direction is obtained from the texture correction unit 360 so that the amplitude of the texture image is aligned with the size at the in-focus position. Is output. Since the corrected texture image is added to the time-series display image output from the image configuration processing unit 310 by the synthesis unit 370, mucosal structures, blood vessels, and the like on the surface of the living body are not completely blurred even when they are not in focus. An image holding the rendered texture can be provided to the user.

  In the above embodiment, the texture extraction process is described as being performed on the final display image. However, this is not necessary, and this process can be incorporated into the enhancement process in the image configuration processing unit 310.

  In the above-described embodiment, an example in which texture extraction processing is performed using two bilateral filters has been described, but the present invention is not limited to this. For example, a texture image may be simply extracted using a high-pass filter or a band-pass filter that extracts a specific frequency or more. However, in this case, processing for removing steep edge portions that change with time, such as a bright spot region due to regular reflection of illumination light from a subject (for example, a threshold value in which the luminance level is set based on the average value of the image as a simple one) Stable control can be achieved by cutting the bright spot region and suppressing the influence of the correction amount on the texture image as much as possible by the determination processing and the deletion processing of the determination region (for example, morphological processing).

  As described above, according to the first embodiment, blurring due to blurring in the Z direction is improved at a shallow depth during magnified observation, and a lesion can be examined in a short time without stress, and the burden on the patient is also increased. Can be reduced.

  In the above-described embodiment, the endoscope apparatus includes an image acquisition unit that acquires captured images in time series in a magnified observation state, and a blur amount information extraction unit that extracts blur amount information from the captured images in the magnified observation state. A blur amount correction unit that corrects the captured image based on the extracted blur amount information.

  Here, the image acquisition unit corresponds to the image configuration processing unit 310 in FIG. The blur amount information extraction unit corresponds to the texture extraction unit 320 in FIG. The blur amount correction unit corresponds to the texture correction amount calculation unit 340, the texture correction unit 360, and the synthesis unit 370 in FIG.

  The magnified observation state is an observation state in which the magnification of the optical system is higher than that in the normal observation state, for example, an observation state at a magnification of several tens to several hundreds. In the present embodiment, a state where the magnification of the optical system is equal to or greater than a predetermined magnification is defined as a magnified observation state.

  As a result, in an endoscope apparatus capable of magnifying observation, blur amount information that is information indicating how much the captured image is blurred is extracted from a captured image at the time of magnifying observation, and based on the extracted blur amount information The captured image can be corrected. As described above, the higher the imaging magnification of the optical system of the endoscope apparatus, the narrower the depth of field, and it becomes difficult to acquire a focused image. However, in view of the fact that the endoscope apparatus is used for diagnosis and treatment by a user (doctor), it is desirable to always obtain a focused image. Therefore, it is very useful to perform the blur recovery process by the method of this embodiment. In addition, since the correction process is performed based on the blur amount information that is information indicating how much the captured image is blurred, an excessive correction process (for example, a noise component is increased) is not performed, It is possible to acquire a blur-corrected image without a sense of incongruity.

  The blur amount information extraction unit includes a predetermined frequency band image extraction unit that extracts a predetermined frequency band image (texture image) that is an image of a predetermined frequency band component from the captured image.

  Here, the predetermined frequency band image extracting unit corresponds to the texture extracting unit 320 in FIG.

  This makes it possible to extract an image of a predetermined frequency band component of the captured image as the blur amount information. By appropriately setting the predetermined frequency at the time of extraction, a region that is strongly affected by blur (for example, a minute gradation structure such as the surface of a mucous membrane of a living body or a blood vessel structure) is extracted and is not significantly affected by blur. A process of not extracting a region or a region corresponding to a noise component can be performed. For this reason, it is possible to appropriately extract only a region where correction processing is desired.

  Further, the frequency band image extraction unit may include a captured image amplitude calculation unit that calculates an amplitude value of the captured image.

  Here, the amplitude value of the captured image is obtained by the difference between the pixel value of the pixel of interest (the pixel for which the amplitude value is to be obtained) and the pixel value of the peripheral pixel that is a pixel around the pixel of interest.

  The captured image amplitude calculation unit corresponds to the region extraction unit 321 and the filter coefficient calculation unit 322 in FIG. Further, it also corresponds to the region extraction unit 326 and the filter coefficient calculation unit 327 in FIG. Actually, the region extraction unit 321 or the region extraction unit 326 obtains the region of the peripheral pixel with respect to the target pixel. Then, the filter coefficient calculation unit 322 or the filter coefficient calculation unit 327 obtains the difference between the pixel values of the target pixel and the surrounding pixels when calculating the filter coefficient. Specifically, | P (x, y, t) −P (x0, y0, t) | in the above formula (1) or | D (x, y, t) −D (x0) in the formula (3). , y0, t) |.

  Thereby, the amplitude value of the captured image can be obtained. Therefore, processing according to the amplitude value is possible, and a structure such as a fold having a large amplitude value is excluded, and an image corresponding to a structure such as a mucous membrane or blood vessel having a small amplitude value can be extracted.

  The predetermined frequency band image extraction unit calculates the captured image amplitude calculation unit without extracting the predetermined frequency band image when the amplitude value calculated by the captured image amplitude calculation unit is in the first amplitude range. When the determined amplitude value is in the second amplitude range, a predetermined frequency band image is extracted.

  Here, for example, the first amplitude range corresponds to FIG. 4A, and the second amplitude range corresponds to FIG. 4B. That is, the amplitude value in the first amplitude range is larger than the amplitude value in the second amplitude range. Specifically, the image in the first amplitude range is an image corresponding to a fold on the living body surface, and the image in the second amplitude range is an image corresponding to the mucosal surface and blood vessels.

  Thereby, the process according to the amplitude value of a captured image is attained. Specifically, the first amplitude range image having a large amplitude value is not extracted as a predetermined frequency band image, and the second amplitude range image having an amplitude value smaller than the first amplitude range is extracted as a predetermined frequency band image. To do. Thereby, it is possible to extract only structures such as mucous membranes and blood vessels having a small amplitude value without extracting a structure such as a fold having a large amplitude value. That is, since only a region that is strongly affected by blur can be extracted, efficient blur correction can be performed.

  The predetermined frequency band image extraction unit does not extract the predetermined frequency band image when the amplitude value calculated by the captured image amplitude calculation unit is in the third amplitude range.

  Here, the third amplitude range corresponds to, for example, FIG. That is, the amplitude value in the third amplitude range is smaller than the amplitude value in the second amplitude range. Specifically, the image in the third amplitude range is an image corresponding to noise.

  Thereby, the process according to the amplitude value of a captured image is attained. Specifically, an image in the third amplitude range whose amplitude value is smaller than the second amplitude range is not extracted as a predetermined frequency band image. As a result, an image corresponding to a noise component having an amplitude value smaller than that of a mucous membrane, blood vessel, or the like can be prevented from being extracted. That is, it is possible to perform the blur recovery process while suppressing the enhancement to the noise component.

  The predetermined frequency band image is an image whose amplitude value is in the second amplitude range and whose frequency component is in the first frequency range.

  Here, the first frequency range corresponds to the range of C1 in FIG. 4B, and the second frequency range corresponds to the range of C2 in FIG. 4B. That is, the frequency in the first frequency range is higher than the frequency in the second frequency range.

  As a result, it is possible to extract not only the amplitude range but also a specific range in frequency to obtain a predetermined frequency band image. At this time, effective blur correction processing can be performed by matching the first frequency range to be extracted with a region to be focused on (and a region where blur correction is to be appropriately applied).

  The predetermined frequency band image extraction unit may include an extraction condition control unit that controls extraction conditions of the predetermined frequency band image based on magnification information of the optical system of the endoscope apparatus.

  Here, the extraction condition control unit corresponds to the filter parameter control unit 329 in FIG.

  Accordingly, it is possible to control the extraction condition of the predetermined frequency band image based on the magnification information of the optical system. Therefore, even when the anatomy to be observed changes according to the magnification, it is possible to appropriately extract the predetermined frequency band image in accordance with the change.

  In addition, as shown in C1 and C2 of FIG. 4B, when the frequency in the first frequency range is higher than the frequency in the second frequency range, the first frequency range is the first frequency range. It is assumed that the frequency range is larger than the first frequency threshold and smaller than the second frequency threshold. For example, a case in which the first frequency threshold corresponds to the cutoff frequency of the filter 804 in FIG. 4B and the second frequency threshold corresponds to the cutoff frequency of the filter 803 can be considered (of course, the frequency threshold). Is not limited to the filter cutoff frequency). Further, consider a case where the predetermined frequency band image has a frequency component in the first frequency range. At this time, the predetermined frequency band image extraction unit may increase the first frequency threshold and the second frequency threshold as the magnification represented by the magnification information of the optical system of the endoscope apparatus increases.

  Thereby, the processing as shown in FIGS. 19A to 19C is possible. As shown in FIGS. 19A to 19C, the higher the magnification of the optical system is, the more the first frequency range that is the frequency range to be extracted can be shifted in the higher frequency direction. Accordingly, it is possible to appropriately extract a frequency range corresponding to an important observation target according to the magnification. Specifically, the area of redness and fading that shows subtle color changes is important at low magnification, the structure of the gland duct and blood vessel running pattern at medium magnification, and finer pit patterns and The structure of microvessels on the surface of the mucosa is important.

  Moreover, the predetermined frequency band image extraction unit includes a first filter processing unit, a subtraction processing unit 325, a noise amount estimation unit 324, and a second filter processing unit, as shown in FIG. Then, the first filter processing unit performs the first filter processing, and the subtraction processing unit 325 generates a structure image by taking the difference between the image obtained by the first filter processing and the captured image. The noise amount estimation unit estimates the amount of noise included in the structure image based on the extraction condition, and the second filter processing unit separates noise from the structure image and acquires a predetermined frequency band image.

  Here, the structure image is an image generated intermediately when a predetermined frequency band image (texture image) is extracted from the captured image. Specifically, in FIG. 3, an image output from the image construction unit 310 and input to the first filter processing unit (region extraction unit 321) and the subtraction processing unit 325 is a captured image, and the first filter processing is performed. An image obtained by obtaining the difference between the output from the unit and the captured image by the subtraction processing unit 325 becomes the structure image. Further, the structure image is input to the second filter processing unit (region extraction unit 326), and as a result of the second filter processing being performed, it is output from the second filter processing unit (filter processing unit 328). It becomes a predetermined frequency band image (texture image).

  As a result, it is possible to extract a predetermined frequency band image by two-stage filter processing. Specifically, the first-stage filter performs the first filter processing (more specific examples will be described later), and the second-stage filter performs noise removal.

  The blur amount information extraction unit includes a captured image amplitude calculation unit that calculates an amplitude value of the captured image. And the 1st filter process part is the frequency corresponding to the 1st frequency range and the 2nd frequency range with respect to the image in which the amplitude value computed by the picked-up image amplitude calculation part is in the 1st amplitude range. A low-pass filter process that transmits the component is performed. The first filter processing unit transmits a frequency component corresponding to the second frequency range to an image in the second amplitude range and cuts a frequency component corresponding to the first frequency range. Perform filtering.

  Accordingly, processing corresponding to 802 in FIG. 4A and 804 in FIG. 4B can be performed. That is, for an image having a large amplitude value (corresponding to a fold or the like), a high-frequency component is left, and for an image having a small amplitude value (corresponding to a mucous membrane surface or a blood vessel), the high-frequency component is cut. Since the difference from the captured image is taken by the subtraction processing unit as described above after the first filter processing, an image with a large amplitude value remains only a frequency component higher than the first frequency range, whereas the amplitude value In an image having a small value, frequency components in the first frequency range also remain.

  Further, as shown in FIG. 3, the first filter processing unit includes a region extraction unit 321 that extracts a predetermined region around the target pixel from the captured image. Also included is a filter coefficient calculation unit 322 that calculates the spatial distance of the surrounding pixels with respect to the target pixel and the distance between pixel values (pixel value difference) and calculates the filter coefficient based on the extraction condition. Furthermore, a filter processing unit 323 that performs filter processing based on the predetermined region extracted by the region extraction unit 321 and the filter coefficient calculated by the filter coefficient calculation unit 322 is included.

As a result, it is possible to perform the filtering process with the structure as shown in FIG. Specifically, the peripheral region of the target pixel is extracted and the peripheral pixel is set. Then, the spatial distance and pixel value difference between the target pixel and the surrounding pixels are calculated, and the filter coefficient is calculated based on the extraction condition. This corresponds to obtaining {(x−x0) ² + (y−y0) ² }, | P (x, y, t) −P (x0, y0, t) | in Equation (1). . And based on the filter coefficient calculated | required by Formula (1), a filter process is performed by Formula (2). Therefore, the amplitude value | P (x, y, t) −P (x0, y0, t) | is greatly related to the coefficient of the filter, so that extraction processing according to the amplitude range is possible. Moreover, since σ1s is also adjusted, the amplitude range can be appropriately set according to the conditions.

  The blur amount information extraction unit includes a captured image amplitude calculation unit that calculates an amplitude value of the captured image. Then, the second filter processing unit has a frequency corresponding to the first frequency range and the second frequency range for an image in which the amplitude value calculated by the captured image amplitude calculation unit is in the second amplitude range. A low-pass filter process that transmits the component is performed. In addition, the second filter processing unit transmits a frequency component corresponding to the second frequency range to an image in the third amplitude range and cuts a frequency component corresponding to the first frequency range. Perform filtering.

  Accordingly, processing corresponding to 803 in FIG. 4B and 805 in FIG. 4C can be performed. That is, for an image having a small amplitude value (corresponding to a mucous membrane surface or blood vessel), the high frequency component is left, and for an image having a smaller amplitude value (corresponding to noise), the high frequency component is cut. Since such filter processing is performed after the first filter processing, the first frequency range remains in the image in the second amplitude range, whereas most components are cut in the image in the third amplitude range. Will be. Therefore, noise can be removed.

  Further, as shown in FIG. 3, the second filter processing unit includes a region extraction unit 326 that extracts a predetermined region around the target pixel from the captured image. Also included is a filter coefficient calculation unit 327 that calculates a spatial distance and a distance between pixel values (pixel value difference) with respect to the target pixel and calculates a filter coefficient based on the extraction condition. Further, a noise separation unit (corresponding to the filter processing unit 328 in FIG. 3) that performs filter processing based on the predetermined region extracted by the region extraction unit 326 and the filter coefficient calculated by the filter coefficient calculation unit 327 and separates noise. )including.

As a result, it is possible to perform the filtering process with the structure as shown in FIG. Specifically, the peripheral region of the target pixel is extracted and the peripheral pixel is set. Then, the spatial distance and pixel value difference between the target pixel and the surrounding pixels are calculated, and the filter coefficient is calculated based on the extraction condition. This corresponds to obtaining {(x−x0) ² + (y−y0) ² }, | D (x, y, t) −D (x0, y0, t) | in Equation (3). . And based on the filter coefficient calculated | required by Formula (3), a filter process is performed by Formula (4). Therefore, the amplitude value | D (x, y, t) −D (x0, y0, t) | is greatly related to the coefficient of the filter, so that extraction processing according to the amplitude range is possible. Further, since σ2s is also adjusted, the amplitude range can be appropriately set according to the conditions.

  The blur amount correction unit corrects the captured image acquired in time series based on the blur amount of the focused captured image that is the captured image determined to be in focus.

  Here, the blur amount correction unit corresponds to the texture correction amount calculation unit 340, the texture correction unit 360, and the synthesis unit 370 of FIG.

  This makes it possible to set a correction reference. Since the in-focus captured image is an image determined to be in focus, if the correction process (for example, amplification of the signal value) is performed up to the level of the in-focus captured image, an image with a sufficiently small amount of blur is acquired. be able to.

  The blur amount correction unit includes a predetermined frequency band image amplitude calculation unit that calculates an amplitude value of the predetermined frequency band image extracted by the predetermined frequency band image extraction unit.

  Here, the predetermined frequency band image amplitude calculation unit corresponds to the amplitude calculation unit 341 in FIG. Further, the amplitude value of the predetermined frequency band image may be defined as the maximum value with respect to the absolute value of the pixel value of the predetermined frequency band image, or may be defined by the difference between the maximum value and the minimum value of the pixel value of the predetermined frequency band image. May be.

  As a result, the amplitude value of the predetermined frequency band image can be calculated. Therefore, the amplitude value of the image can be used as a correction reference. Specifically, the correction may be performed by amplifying the amplitude value in the blurred image to be equal to the amplitude value in the focused image.

  Further, as shown in FIG. 2, the blur amount correcting unit corrects the predetermined frequency band image based on the amplitude value of the predetermined frequency band image calculated by the predetermined frequency band image amplitude calculating unit. Unit (corresponding to the texture correction unit 360), a captured image, and a combining unit 370 that combines the corrected predetermined frequency band image.

  As a result, it is possible to perform the correction process by combining the corrected predetermined frequency band image and the captured image after correcting the predetermined frequency band image. In view of the fact that the predetermined frequency band image is strongly influenced by blur and corresponds to an important observation target, only the correction target is extracted, corrected, and returned to the captured image in the form of synthesis. Therefore, it is possible to realize the correction that stably maintains the observation target with the same resolution as that at the time of focusing.

  The predetermined frequency band image correction unit selects an amplitude value determined to be in focus as a reference amplitude value from the amplitude values calculated in time series by the predetermined frequency band image amplitude calculation unit. A reference amplitude value selection unit (corresponding to the reference amplitude update determination unit 342 in FIG. 5) and a correction amount multiplication unit (correction amount calculation unit in FIG. 5) that calculates and multiplies a correction amount for a predetermined frequency band image based on the reference amplitude value. 344 and the texture correction unit 360).

  Thereby, it becomes possible to calculate the reference amplitude value and perform correction based on the calculated reference amplitude value. Specifically, for example, as shown in 1 <amplitude gain amount <Th1 in FIG. 18, a correction process may be performed so that the amplitude value of the blurred image becomes the same value as the reference amplitude value. In FIG. 18, when the amplitude gain amount is larger than a certain level (that is, when the amplitude value of the blurred image is smaller than the reference amplitude value), the degree of blur is too strong and the correction does not work well. Judgment is made and the degree of correction is weakened.

  The focused captured image may be a captured image acquired at the time when the optical system is switched to the enlarged observation state.

  As a result, it is possible to efficiently obtain a focused captured image. That is, since it is considered that there is a high possibility that the subject is in focus at the time of switching to the magnified observation state, there are few problems even if the image at the time of switching is used as a focused captured image. Therefore, as described later, it is not necessary to make a determination based on conditions, and a focused captured image can be acquired with simple processing.

  Further, the focused captured image may be a captured image acquired after the time point when the optical system is switched to the magnified observation state, and the amplitude value satisfies a predetermined condition within a predetermined period. . Specifically, for example, the predetermined condition may be a condition that the amplitude value is larger than a reference amplitude determination threshold value and takes a maximum value.

  Thereby, even after switching to the magnified observation state, it is possible to update the focused captured image that is the reference image (that is, update the reference amplitude value). In observation with an endoscope, the observation magnification may change or the subject to be observed may change (for example, when there is a large lateral movement), in which case the amplitude value of the image changes. there is a possibility. Even in such a case, an appropriate reference can be set by updating the focused captured image. In addition, by setting a condition for updating the focused captured image, it is possible to prevent the update process from being performed too frequently.

  In addition, the present embodiment described above includes an image acquisition unit that acquires captured images in a magnified observation state in time series, a blur amount information extraction unit that extracts blur amount information from the captured images in the enlarged observation state, and the extracted blur. The present invention relates to a program that causes a computer to function as a blur amount correction unit that corrects a captured image based on amount information.

  Thereby, this embodiment is not limited to what acquires an image and performs image processing in a system like an imaging device or an endoscope system, for example, image data is first accumulated and then accumulated. The present invention can also be applied to image data processed by a computer system such as a PC. The program is recorded on an information storage medium. Here, as the information recording medium, various recording media that can be read by an optical detection system, such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk (HDD), a memory such as a nonvolatile memory or a RAM, can be assumed. .

3. Second Embodiment 3.1 Configuration FIG. 10 is a block diagram illustrating an overall configuration of an imaging apparatus according to a second embodiment. The imaging apparatus constituting the present embodiment includes a light source unit 100, an imaging unit 200, a processor unit 300, a display unit 400, and an external I / F unit 500, and has the same basic configuration as that of the first embodiment. The configuration will be mainly described.

  First, for the light source unit 100, the white light source 101, the rotating color filter 102 having a plurality of spectral transmittances, the rotation driving unit 103 for driving the rotating color filter 102, and the light having the spectral characteristics from the rotating color filter 102 are emitted. Condensing lens 104 which condenses the light guide fiber 201 on the incident end face.

  For example, as shown in FIG. 9, the rotary color filter 102 includes three primary color red color filters 601, a green color filter 602, a blue color filter 603, and a rotary motor 604. The spectral characteristics of these three color filters are as shown in FIG.

  The rotation driving unit 103 rotates the rotating color filter 102 at a predetermined number of rotations in synchronization with the imaging period of the imaging device 209 based on a control signal from the control unit 302 of the processor unit 300. For example, when the color filter is rotated 20 times per second, each color filter crosses the incident white light at intervals of 1/60 second, and the image sensor 209 has the light of each of the three primary colors (R or R) at intervals of 1/60 second. The imaging and transfer of the reflected light image in G or (B) is completed, where the image sensor 209 is for monochrome use, that is, in this embodiment, the R image, G image, and B image are 1 / 60th. This is an example of an endoscope apparatus that is imaged in a frame sequential manner at intervals of seconds, and the actual frame rate is 20 fps.

  Subsequently, the imaging device 209 is different from the imaging unit 200 as described above, and a monochrome single-plate imaging device is used. Furthermore, the focus position control unit 206 and the focus position adjustment lens 207 can be driven at higher speed, and in the second embodiment, autofocus can be performed in real time.

  Further, an in-focus position calculation unit 303 is added to the processor unit 300. The control content of the control unit 302 is modified with the addition of the in-focus position calculation unit 303 and the addition of the rotation color filter 102 and the rotation drive unit 103.

  The in-focus position calculation unit 303 added to the processor unit 300 receives the time-series display image output from the image processing unit 301 and the information of the currently captured color filter output from the control unit 302, and the corresponding color filter. The focus position is calculated on the basis of the image, and the obstruction amount in the Z direction (the optical axis direction of the scope tip) is output to the control unit 302 together with the calculated focus position.

  The control unit 302 outputs the movement position of the focus position adjustment lens 207 to the focus position control unit 206 based on the focus position output from the focus position calculation unit 303, and the color filter to be photographed next selects the white light source 101. The focal position adjustment lens 207 is moved by the timing of crossing.

  The texture image is extracted by the texture extraction unit 320 from the time-series display image output from the image configuration processing unit 310, and the extracted texture image is output to the in-focus position calculation unit 303.

  Further, based on the texture image extracted by the texture extraction unit 320, the texture correction amount calculation unit 340 calculates a correction amount for the texture image. Then, the texture correction unit 360 corrects the texture image based on the calculated correction amount, and the synthesis unit 370 synthesizes the corrected texture image and the captured image from the image configuration processing unit, thereby correcting the blur correction image. Is generated.

3.2 Detailed Configuration of Blur Amount Information Extraction Unit The configuration of the texture extraction unit 320 is different from that of the first embodiment, and is configured as shown in the block diagram of FIG.

  The texture extraction unit 320 is a low-pass processing unit 1321, 1326, 1331, a downsampling unit 1322, 1327, a subtraction processing unit 1323, 1328, 1332, a noise amount estimation unit 1324, 1329, 1333, and a coring unit 1325, 1330, 1334. A parameter control unit 1335.

  In this configuration, sub-band images obtained by dividing each time-series display image into a plurality of frequency bands are generated, and independent noise reduction processes are incorporated. The feature of the texture extraction by the frequency band division is that the frequency band to be divided is changed according to the frequency band of the texture (mucosal structure or blood vessel running pattern) to be observed according to the magnification of magnification observation.

  Next, the data flow between the components will be described. The processing parameter control unit 1335 tabulates the correspondence relationship between the magnification information and the bandwidth so that the bandwidth divided into a plurality of frequency bands (subbands) is different based on the magnification information output from the control unit 302. An example is shown in FIGS. 14A and 14B. FIG. 14A shows an example of frequency band division when the magnification observation magnification is small, and FIG. 14B shows an example of frequency band division when the magnification observation magnification is large. Both are composed of four frequency bands (1) to (4).

  That is, the magnification information from the control unit 302 and a predetermined threshold value are determined, and which division method is used to divide the band information is determined. When the division method is determined, filter coefficients having a low-pass characteristic suitable for each frequency band and a reduction rate at the time of downsampling are determined. Is output. Further, the noise amount estimation model for each frequency band is also changed based on the division method. The noise estimation model corresponding to each frequency band is output to the noise amount estimation units 1324, 1239, and 1333, respectively.

  Hereinafter, since the processing of a plurality of frequency bands is the same, the processing of the subband image of the frequency band (1) will be described. The low-pass processing unit 1321 receives the time-series display image output from the image configuration processing unit 310 and the filter coefficient output from the processing parameter control unit 1329. Then, the low-pass time series display image obtained by filtering the time series display image is output to the subtraction processing unit 1323, the down sampling unit 1322, and the noise amount estimation unit 1324.

  The time-series display image output from the image configuration processing unit 310 and the low-pass time-series display image output from the low-pass processing unit 1321 are input to the subtraction processing unit 1323. Then, a high-pass time-series display image is generated by subtracting the low-pass time-series display image from the time-series display image, and is output to the coring unit 1325.

  The noise amount estimation unit 1324 receives the low-pass time-series display image output from the low-pass processing unit 1321 and the noise estimation model output from the processing parameter control unit 1335, and the pixel value of the low-pass time-series display image is determined by the noise estimation model. It is converted into a noise amount and output to the coring unit 1325.

  The coring unit 1325 receives the high-pass time-series display image output from the subtraction processing unit 1323 and the noise amount output from the noise amount estimation unit 1324, and performs coring processing with a coring width corresponding to the noise amount. Is done. The high-pass time-series display image that has undergone the coring process is output to the texture correction amount calculation unit 340, the in-focus position calculation unit 303, and the texture correction unit 360.

  Up to here is the extraction of the frequency band of (1) in FIG. 14 (A) or FIG. 14 (B). Since the extraction of the frequency bands of (2) and (3) is basically the same, the description thereof is omitted.

3.3 Detailed Configuration of Blur Amount Correction Unit Next, the texture correction amount calculation unit 340 will be described in detail with reference to FIG.

  The texture correction amount calculation unit 340 includes an amplitude calculation unit 1341, a reference amplitude update determination unit 1342, a reference amplitude storage unit 1343, a correction amount calculation unit 1344, and a correction amount control unit 1353. The texture correction amount calculation unit 340 receives the frequency bands (1), (2), and (3) shown in FIG. 14A or FIG. 14B, and calculates each correction amount.

  Only the correction amount calculation for the frequency band (1) in FIG. 14A or FIG. 14B will be described below. An observation mode signal, a texture reference threshold value, a correction amount upper limit value output from the control unit 302, and a current reference amplitude value from the reference amplitude storage unit 1343 are input to the correction amount control unit 1353. Then, a reference amplitude determination threshold is set based on the observation mode signal, the texture reference threshold, and the reference amplitude value, and is output to the reference amplitude update determination unit 1342, and further, the correction amount upper limit value is output to the correction amount calculation unit 1344.

  Here, the texture reference threshold is a reference value for determining whether or not the captured time-series display image is in a blurred state, and varies depending on the difference in imaging performance, so that a different value is set depending on the type of scope. The reference amplitude value determination threshold value is a threshold value for selecting a reference amplitude value that is a correction target for the texture image, and is set to the texture reference threshold value when the observation mode signal is switched to the magnification observation mode. And the average value of the reference amplitude values is set as a reference amplitude value determination threshold.

  The amplitude calculator 1341 calculates an amplitude value MA (t) for the texture image extracted at time t output from the texture extractor 320. Here, the amplitude value may be defined by the maximum value with respect to the absolute value of the pixel value of the texture image, or may be defined by the difference between the maximum value and the minimum value of the pixel value of the texture image. The calculated amplitude value MA (t) is output to the reference amplitude update determination unit 1342.

  The reference amplitude update determination unit 1342 receives the amplitude value MA (t) output from the amplitude calculation unit 1341 and the reference amplitude determination threshold value output from the correction amount control unit 1353, and is stored in the reference amplitude update determination unit 1342. Amplitude values MA (t-1), MA (t-2), MA (t-3), MA (t-4), MA (t-5), MA (t-6), MA before the field period Based on (t-7) and MA (t-8), it is determined whether or not the amplitude value MA (t) becomes the reference amplitude value. Here, one field period is a period (1/60 second) in which R image, B image, and G image taken corresponding to the R, G, and B color filters are acquired.

  Details of the reference amplitude update determination will be described with reference to the schematic diagram of FIG. The user operates the scope distal end position of the endoscope so as to obtain a desired magnification by moving the zoom lever while bringing the distal end of the scope closer to the observation target during magnified observation. Since the present embodiment has an autofocus function, the user need not care about the in-focus state. However, since it is assumed that the autofocus is controlled based on the contrast value, the focal position is obfuscated in the A → B → C and Z axis directions (optical axis direction of the optical system) as shown in the schematic diagram of FIG. It is necessary to estimate the in-focus position by shooting with a ring and generating a contrast change. Such obling in the Z-axis direction creates a focused state and a state that is not in focus (a state in which the focus is not good).

  In particular, when the frame rate is low, the period during which an image that is not in focus is displayed becomes longer and the focus change becomes a concern. However, the frame sequential endoscope apparatus of the second embodiment has a frame rate of 20 fps. And low.

  When the contrast change is used in the field sequential endoscope apparatus of the second embodiment, imaging with R, G, and B filters is performed at the respective focal positions of A → B → C in FIG. 11. That is, the period of transition from A to B corresponds to one frame period (1/20 seconds).

  The time-series display image output from the image configuration processing unit 310 according to the present embodiment is updated only in the image captured by the R, G, and B filters (R image, B image, or G image) in the field period. In order to reliably extract three different texture images corresponding to the R image, the G image, or the B image from the plurality of texture images, nine texture images are required.

  Therefore, the reference amplitude update determination unit 1342 has nine amplitude values MA (t), MA (t−1), MA (t−t) corresponding to nine texture images in a period of t−8 to t continuous in time series. 2), the amplitude value of MA (t-3), MA (t-4), MA (t-5), MA (t-6), MA (t-7), MA (t-8) is a maximum value. To determine whether or not Therefore, three groups {MA (t), MA (t-3), MA (t-6)} and {MA (t-1), MA (t-4), MA (t− 7)} and {MA (t-2), MA (t-5), MA (t-8)}. Then, it is fitted to a quadratic expression for the three amplitude values of each group, and it is determined whether any one of the three groups has a maximum value.

  When a local maximum exists, the local maximum calculated by the quadratic expression is a candidate for a reference amplitude value. When a plurality of local maximum values are detected, one maximum local maximum value is selected as a reference amplitude value candidate. The candidate for the reference amplitude value is further compared with the reference amplitude determination threshold value, and when the maximum value is larger than the reference amplitude determination threshold value, it is output to the reference amplitude storage unit 1343 as the reference amplitude value. When the maximum value is less than or equal to the reference amplitude determination threshold, it is not regarded as the reference amplitude value and is not output to the reference amplitude storage unit 1343. On the other hand, when only the minimum value is detected in all three groups, the comparison with the reference amplitude determination threshold value is not performed, and the reference amplitude is not changed.

  The reason for updating the reference amplitude value is the same as in the first embodiment.

  The reference amplitude storage unit 1343 receives the reference amplitude value output from the reference amplitude update determination unit 1342 and updates the reference amplitude value recorded in the past to the latest value. The latest reference amplitude value is output to the correction amount control unit 1353 and the correction amount calculation unit 1344.

  In the correction amount calculation unit 1344, the amplitude value MA (t) at time t output from the amplitude calculation unit 1341, the reference amplitude value RMA output from the reference amplitude storage unit 1343, and the correction amount output from the correction amount control unit 1353. An upper limit value is input, a gain amount with which the amplitude value MA (t) is the same as the reference amplitude value RMA is calculated, and the equations (5), (6), and (7) shown in the first embodiment are calculated. The correction amount is calculated in the same manner as described above.

  Similarly, the frequency bands (2) and (3) are omitted because they are the same processing, but the correction amounts are calculated independently in each frequency band, and the three correction amounts are output to the texture correction unit 360.

  Next, the texture correction unit 360 will be described in detail with reference to FIG. The texture correction unit 360 includes a processing parameter control unit 1368, multiplication processing units 1361, 1362, 1263, upsampling units 1364, 1366, and addition processing units 1365, 1367.

  Next, the data flow between the components will be described. The processing parameter control unit 1368 outputs the reduction ratio applied by the texture extraction unit 320 corresponding to the frequency bandwidth divided based on the magnification information output from the control unit 302 to the upsampling units 1364 and 1366. .

  The multiplication processing unit 1361 includes a subband image corresponding to the frequency band (3) of FIG. 14A or 14B output from the texture extraction unit 320 and the frequency band output from the texture correction amount calculation unit 340. The correction amount of the subband image corresponding to (3) is input, and the correction subband image corresponding to the frequency band (3) is generated by multiplying the correction amount by the subband image corresponding to the frequency band (3). Output to the upsampling unit 1364.

  The multiplication processing unit 1362 includes a subband image corresponding to the frequency band (2) output from the texture extraction unit 320 and the frequency band output from the texture correction amount calculation unit 340. The correction amount of the subband image corresponding to (2) is input, and a correction subband image corresponding to the frequency band (2) is generated by multiplying the correction amount by the subband image corresponding to the frequency band (2). To the addition processing unit 1365.

  The multiplication processing unit 1363 includes a subband image corresponding to the frequency band (1) of FIG. 14A or 14B output from the texture extraction unit 320, and a frequency band output from the texture correction amount calculation unit 340. The correction amount of the subband image corresponding to (1) is input, and the correction subband image corresponding to the frequency band (1) is generated by multiplying the correction amount by the subband image corresponding to the frequency band (1). To the addition processing unit 1367.

  The upsampling unit 1364 receives the correction subband image corresponding to the frequency band (3) output from the multiplication processing unit 1361 and the reduction rate of the frequency band (3) output from the processing parameter control unit 1368, and the reduction rate. Then, an enlarged corrected subband image corresponding to the frequency band (3) obtained by enlarging the corrected subband image corresponding to the frequency band (3) is generated at a magnification that is the reciprocal of the frequency band (3), and is output to the addition processing unit 1365.

  The addition processing unit 1365 has an enlarged corrected subband image corresponding to the frequency band (3) expanded by the upsampling unit 1364 and a corrected subband image corresponding to the frequency band (2) output from the multiplication processing unit 1362. The corrected subband image corresponding to the frequency band ((3) + (2)) is generated by inputting the two subband images and output to the upsampling unit 1366.

  The upsampling unit 1366 includes a correction subband image corresponding to the frequency band ((3) + (2)) output from the addition processing unit 1365 and a reduction rate of the frequency band (2) output from the processing parameter control unit 1368. Is input, and the frequency band ((3) + (2)) obtained by enlarging the corrected subband image corresponding to the frequency band ((3) + (2)) at a magnification that is the reciprocal of the reduction ratio of the frequency band (2). ) Is generated and output to the addition processing unit 1367.

  The addition processing unit 1367 corresponds to the enlarged corrected subband image corresponding to the frequency band ((3) + (2)) expanded by the upsampling unit 1366 and the frequency band (1) output from the multiplication processing unit 1363. The corrected subband image to be input is input, and the two subband images are added to generate a corrected subband image corresponding to the frequency band ((3) + (2) + (1)) and output to the combining unit 370. To do.

  The processing after the combining unit 370 is the same as that in the first embodiment.

3.4 Details of Autofocus Processing Next, the in-focus position calculation unit 303 will be described in detail with reference to FIG. The focus position calculation unit 303 includes area extraction units 1371, 1375, 1379, average amplitude calculation units 1372, 1376, 1380, average amplitude storage units 1373, 1377, 1381, focus position calculation units 1374, 1378, 1382, The focus position determination unit 1384 and the focus position calculation control unit 1383 are configured.

  The in-focus position calculation unit 303 is premised on using the contrast method, and it is necessary to set a detection region for detecting the contrast. For this area, for example, focus position information designated by the user from the external I / F unit 500 (for example, an identification number of an area selected from a plurality of preset rectangular areas) is input to the control unit 302. In 302, it is assumed that the information is converted into contrast detection area information (coordinates indicating a rectangular area in the time-series display image).

  Next, the data flow between the components will be described. The contrast detection area information and magnification information output from the control unit 302 and the current focus position set in the focus position control unit 206 are input to the focus position calculation control unit 1383. The focus position calculation control unit 1383 calculates the contrast detection region coordinates corresponding to the three subband images output from the texture extraction unit 320 based on the contrast detection region information and the magnification information, and the calculated contrast detection region coordinates are The data are output to the area extraction units 1371, 1375, and 1379, respectively. The current focal position is output to the average amplitude storage units 1373, 1377, and 1381. Here, the contrast detection area coordinates are the coordinates from the control unit 302 as they are for the frequency band (1), and the frequency band (2) is the frequency band output from the processing parameter control unit 1368 for the frequency band (1). The coordinates reduced by the reduction ratio of (2) are calculated, and the frequency band (3) is the coordinates reduced by the reduction ratio of the frequency band (3) output from the processing parameter control unit 1368 with respect to the frequency band (2). calculate.

  The region extraction units 1371, 1375, and 1379 extract contrast detection regions based on the contrast detection region coordinates output from the in-focus position calculation control unit 1383 for each of the three subband images output from the texture extraction unit 320, It outputs to average amplitude calculation parts 1372, 1376, and 1380, respectively.

  The average amplitude calculation units 1372, 1376, and 1380 calculate the average value of the absolute values of the pixel values in the contrast detection region extracted from the subband images output from the region extraction units 1371, 1375, and 1379 as the average amplitude value. And output to the average amplitude storage units 1373, 1377, and 1381, respectively.

  In the average amplitude storage units 1373, 1377, and 1381, at least nine focal positions corresponding to the average amplitude values calculated in time series are recorded in the form of a ring buffer, and the average amplitude calculation units 1372, 1376 are recorded. , 1380 and the focus position output from the in-focus position calculation control unit 1383 are always stored as the latest average amplitude value and the focus position.

  The in-focus position calculation units 1374, 1378, and 1382 include the average amplitude storage units 1373, 1377, and 1381 for every three average amplitude values including the latest average amplitude value from nine average amplitude values and the corresponding focal position. Are read from each ring buffer, and the in-focus position at which the maximum amplitude value is obtained is calculated by Lagrange interpolation or the like. The calculated three in-focus positions are output to the in-focus position determination unit 1384 together with the corresponding maximum amplitude value.

  The in-focus position determination unit 1384 receives three in-focus positions corresponding to the three subband images output from the in-focus position calculation units 1374, 1378, and 1382 and the three calculated maximum amplitude values. The focus position determination unit 1384 selects the maximum amplitude value among the three input maximum amplitude values, and controls the focus position calculation control with the focus position corresponding to the selected amplitude value as the true focus position. Output to the unit 1383. As a method for determining the in-focus position that further reduces the influence of noise, a weighted average with weights corresponding to the maximum amplitude values corresponding to the three in-focus positions may be calculated to obtain a true in-focus position.

  The focus position calculation control unit 1383 outputs the focus position output from the focus position determination unit 1384 to the control unit 302 when the focus position is input.

  In response to this, the control unit 302 outputs a focus position to the focus position control unit 206, and performs focus control by moving the focus position adjustment lens 207 to the focus position.

  As described above, according to the second embodiment, since the autofocus process at the time of magnifying observation of the endoscope apparatus having a low frame rate can be controlled by the contrast value based on the texture extracted corresponding to the observation target, the focusing accuracy can be controlled. As a result, it is possible to suppress the time fluctuation of the amount of blurring caused by the obstruction in the Z direction due to autofocus, so that the user can perform a detailed examination of the lesion in a short time without stress. Can also reduce the burden.

  In the present embodiment described above, the predetermined frequency band image extraction unit selects a predetermined frequency band image by selecting a frequency band separation unit that generates a plurality of subband images, and one or a plurality of subband images from the plurality of subband images. A subband selection unit to be generated.

  Here, the sub-band image is an image obtained by separating a captured image into a plurality of predetermined frequency bands determined based on magnification information. In the examples of FIGS. 14A and 14B, (1) to (4) are subband images, respectively. Further, the frequency band separation unit corresponds to the texture extraction unit 320 in FIG. 12, and the subband selection unit corresponds to the texture correction amount calculation unit 340 and the like in FIG.

  This makes it possible to extract a predetermined frequency band image based on the subband image. Specifically, the captured image is divided into a plurality of subband images, and an appropriate subband image is selected from the divided subband images to generate a predetermined frequency band image. At that time, as shown in FIGS. 14A and 14B, the width of the subband image is changed in accordance with the imaging magnification of the optical system, so that the control according to the magnification (in the first embodiment, FIG. Equivalent to control).

  The correction amount multiplication unit calculates a correction amount corresponding to each subband image based on the amplitude value corresponding to each subband image calculated by the predetermined frequency band image amplitude calculation unit and the magnification information of the optical system. And a subband image correction amount multiplication unit for multiplying each subband image by the calculated correction amount.

  Here, the subband image correction amount multiplication unit corresponds to the multiplication processing units 1361 to 1363 in FIG.

  As a result, the predetermined frequency band image can be corrected by multiplying each subband image by the correction amount. Specifically, as in the first embodiment, correction is performed so that the amplitude value of the image to be corrected is the same value as the reference amplitude value. In the present embodiment, the bright spot region removing process by removing the sharp edge portion that varies with time is inserted before the sub-band image is divided, so that the control according to the amplitude can be handled. Here, the removal processing of the steep edge portion that fluctuates with time is, for example, as simple processing that determines the luminance level based on a threshold set based on the average image value, and processing for deleting the determination region (Specifically, for example, morphological treatment). In the present embodiment, after selecting a plurality of subband images, the correction amount may be multiplied only for the selected subband images, or the correction amounts may be applied to all the plurality of subband images. Multiplication may be performed and 0 may be multiplied for unnecessary subband images. The latter configuration is used in the block diagrams of FIGS.

  The optical system of the endoscope apparatus includes a focus position control unit that controls the focus position, and a target focus position calculation unit that calculates the target focus position based on the amount of blur of the captured image. The focus position control unit controls the focus position based on the target focus position calculated by the target focus position calculation unit.

  Here, the blur amount information of the captured image is calculated by a blur amount information extraction unit for a plurality of captured images at a plurality of focus positions. Further, the focus position control unit corresponds to the focus position control unit 206 in FIG. 12, and the target focus position calculation unit corresponds to the in-focus position calculation unit 303 in FIG.

  Thereby, an autofocus process can be performed. However, as shown in FIG. 11, since autofocus based on the contrast method for obtaining the in-focus position after driving the lens to a plurality of focus positions is premised, timing out of the depth of field range may also occur. is there. Therefore, it does not mean that blur recovery processing by image processing is unnecessary. Rather, since the in-focus state and the out-of-focus state appear in a short time, if the frame rate is low as in the present embodiment, image processing is performed in order to provide a user with an uncomfortable image. An effective blur recovery process is important.

  The blur amount correction unit determines that the captured image acquired at the focus position determined to be in focus by the focus position control unit is the focused image. Then, correction is performed based on the blur amount information of the in-focus captured image.

  As a result, it is possible to use an image acquired at a focus position determined to be in focus in autofocus as a focused captured image serving as a reference for calculating the correction amount. Therefore, in addition to the reference amplitude value update processing similar to the first embodiment, the reference amplitude value update processing by autofocus is also performed, so that the reference amplitude value can be updated with high accuracy. Become.

  As described above, the two embodiments 1 and 2 to which the present invention is applied and the modified examples thereof have been described. However, the present invention is not limited to the embodiments 1 and 2 and the modified examples as they are. The constituent elements can be modified and embodied without departing from the spirit of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described first and second embodiments and modifications. For example, you may delete a some component from all the components described in each Embodiment 1-2 or the modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 100 Light source part, 101 White light source, 102 Rotation color filter, 103 Rotation drive part,
104 condensing lens, 200 imaging unit, 201 light guide fiber,
202 illumination lens, 203 objective lens, 206 focal position control unit,
207 focal position adjustment lens, 209 imaging device, 210 A / D conversion unit,
300 processor unit, 301 image processing unit, 302 control unit,
303 in-focus position calculation unit, 310 image configuration processing unit, 320 texture extraction unit,
321 region extraction unit, 322 filter coefficient calculation unit, 323 filter processing unit,
324 noise amount estimation unit, 325 subtraction processing unit, 326 region extraction unit,
327 filter coefficient calculation unit, 328 filter processing unit,
329 filter parameter control unit, 340 texture correction amount calculation unit,
341 amplitude calculation unit, 342 reference amplitude update determination unit, 343 reference amplitude storage unit,
344 correction amount calculation unit, 345 correction amount control unit, 360 texture correction unit,
370 synthesis unit, 400 display unit, 500 external I / F unit,
601 color filter, 602 color filter, 603 color filter, 604 rotary motor,
1321 Low-pass processing unit, 1322 down-sampling unit, 1323 subtraction processing unit,
1324 Noise amount estimation unit, 1325 coring unit,
1335 processing parameter control unit, 1341 amplitude calculation unit,
1342 reference amplitude update determination unit, 1343 reference amplitude storage unit, 1344 correction amount calculation unit,
1353 Correction amount control unit, 1361 multiplication processing unit, 1362 multiplication processing unit,
1363 multiplication processing unit, 1364 upsampling unit, 1365 addition processing unit,
1366 Upsampling unit, 1367 addition processing unit,
1368 processing parameter control unit, 1371 region extraction unit, 1372 average amplitude calculation unit,
1373 average amplitude storage unit, 1374 in-focus position calculation unit,
1383 In-focus position calculation control unit, 1384 In-focus position determination unit

Claims (28)

An image acquisition unit for acquiring captured images in a time-series manner in an enlarged observation state in which the magnification of the optical system is high compared to the normal observation state;
A blur amount information extraction unit that extracts blur amount information from the captured image in the magnified observation state;
A blur amount correction unit that corrects the captured image based on the extracted blur amount information;
An endoscopic device comprising: In claim 1,
The blur amount information extraction unit
An endoscope apparatus comprising: a predetermined frequency band image extracting unit that extracts a predetermined frequency band image that is an image of a predetermined frequency band component from the captured image. In claim 2,
The predetermined frequency band image extraction unit includes:
An endoscope apparatus comprising: a captured image amplitude calculation unit that calculates an amplitude value of the captured image. In claim 3,
The predetermined frequency band image extraction unit includes:
When the amplitude value calculated by the captured image amplitude calculation unit is in the first amplitude range, the predetermined frequency band image is not extracted,
When the amplitude value calculated by the captured image amplitude calculation unit is in a second amplitude range that corresponds to an amplitude value smaller than the amplitude value in the first amplitude range, the predetermined frequency band image An endoscope apparatus characterized by performing extraction. In claim 4,
The predetermined frequency band image extraction unit includes:
When the amplitude value calculated by the captured image amplitude calculation unit is in a third amplitude range that corresponds to an amplitude value smaller than the amplitude value in the second amplitude range, the predetermined frequency band image An endoscope apparatus characterized by not performing extraction. In claim 4,
The predetermined frequency band image is:
The amplitude value calculated by the captured image amplitude calculation unit is in the second amplitude range,
An endoscope apparatus, wherein the frequency component is an image in a first frequency range that is a range corresponding to a frequency higher than a frequency in the second frequency range. In claim 5,
The predetermined frequency band image in which the amplitude value calculated by the captured image amplitude calculation unit is the second amplitude range,
An endoscope apparatus characterized by being an image corresponding to a mucosal surface and a blood vessel on a living body surface. In claim 5,
An image whose amplitude value calculated by the captured image amplitude calculation unit is the first amplitude range is:
It is an image corresponding to folds on the surface of a living body
An image in which the amplitude value calculated by the captured image amplitude calculation unit is the third amplitude range is:
An endoscope apparatus characterized by being an image corresponding to noise. In claim 2,
The predetermined frequency band image extraction unit includes:
An endoscope apparatus comprising: an extraction condition control unit that controls an extraction condition of a predetermined frequency band image based on magnification information of the optical system of the endoscope apparatus. In claim 9,
The frequency in the first frequency range is higher than the frequency in the second frequency range,
The first frequency range is a range that is larger than the first frequency threshold and smaller than the second frequency threshold;
When the predetermined frequency band image is an image whose frequency component is in the first frequency range,
The predetermined frequency band image extraction unit includes:
The endoscope apparatus, wherein the first frequency threshold and the second frequency threshold are increased as the magnification represented by the magnification information of the optical system of the endoscope apparatus increases. In claim 9,
The predetermined frequency band image extraction unit includes:
A frequency band separation unit that generates a plurality of subband images that are images obtained by separating the captured image into a plurality of predetermined frequency bands determined based on the magnification information;
A subband selection unit that selects one or a plurality of subband images from the plurality of subband images and generates the predetermined frequency band image;
An endoscopic device comprising: In claim 9,
The predetermined frequency band image extraction unit includes:
A first filter processing unit;
A subtraction processing unit;
A noise estimation unit;
A second filter processing unit;
Including
The subtraction processing unit
A structure image is generated by taking the difference between the image acquired by the first filter processing by the first filter processing unit and the captured image,
The noise amount estimation unit
Estimating the amount of noise contained in the structure image based on the extraction condition,
The second filter processing unit includes:
An endoscope apparatus, wherein noise is separated from the structure image based on a noise amount estimated by the noise amount estimation unit, and the predetermined frequency band image is acquired. In claim 12,
The blur amount information extraction unit
A captured image amplitude calculation unit that calculates an amplitude value of the captured image;
The first filter processing unit includes:
For an image in the first amplitude range, a low-pass filter process that transmits frequency components corresponding to the first frequency range and the second frequency range is performed,
The image of the second amplitude range is subjected to low-pass filter processing that transmits a frequency component corresponding to the second frequency range and cuts the frequency component corresponding to the first frequency range. Endoscope device. In claim 12,
The first filter processing unit includes:
An area extracting unit that extracts a predetermined area around the target pixel from the captured image;
A filter coefficient calculation unit that calculates a spatial distance between neighboring pixels with respect to a pixel of interest in the predetermined area, and a distance between pixel values, and calculates a filter coefficient for pixels of each predetermined area based on the extraction condition;
A filter processing unit that performs a filter process using the predetermined region and the filter coefficient;
An endoscopic device comprising: In claim 12,
The blur amount information extraction unit
A captured image amplitude calculation unit that calculates an amplitude value of the captured image;
The second filter processing unit includes:
For the image of the second amplitude range, low pass filter processing that transmits the frequency components corresponding to the first frequency range and the second frequency range is performed,
The image of the third amplitude range is subjected to a low-pass filter process that transmits a frequency component corresponding to the second frequency range and cuts a frequency component corresponding to the first frequency range. Endoscope device. In claim 12,
The second filter processing unit includes:
A region extracting unit that extracts a predetermined region around the target pixel from the structure image;
A filter coefficient calculation unit that calculates a spatial distance between neighboring pixels with respect to a pixel of interest in the predetermined area, and a distance between pixel values, and calculates a filter coefficient for pixels of each predetermined area based on the amount of noise;
A noise separation unit that separates noise from the target pixel by filtering with the extraction region and the filter coefficient;
An endoscopic device comprising: In claim 1,
The blur amount correction unit
An endoscope apparatus, wherein correction is performed on the captured images acquired in time series based on a blur amount of a focused captured image that is a captured image determined to be in focus. In claim 17,
The blur amount correction unit
An endoscope apparatus comprising: a predetermined frequency band image amplitude calculating unit that calculates an amplitude value of the predetermined frequency band image extracted by the predetermined frequency band image extracting unit. In claim 18,
The blur amount correction unit
A predetermined frequency band image correcting unit that corrects the predetermined frequency band image based on the amplitude value of the predetermined frequency band image calculated by the predetermined frequency band image amplitude calculating unit;
A combining unit that combines the captured image and the corrected predetermined frequency band image;
An endoscopic device comprising: In claim 19,
The predetermined frequency band image correction unit,
A reference amplitude value selection unit that selects an amplitude value determined to be in focus as a reference amplitude value from amplitude values of the predetermined frequency band image calculated in time series by the predetermined frequency band image amplitude calculation unit;
A correction amount multiplication unit that calculates and multiplies a correction amount for the predetermined frequency band image based on the reference amplitude value;
An endoscopic device comprising: In claim 20,
The predetermined frequency band image extracted by the predetermined frequency band image extraction unit is:
One or a plurality of subband images composed of one or a plurality of frequency bands;
The correction amount multiplication unit
Correction corresponding to each subband image based on one or a plurality of amplitude values corresponding to each subband image calculated by the predetermined frequency band image amplitude calculator and magnification information of the optical system of the endoscope apparatus An endoscope apparatus comprising: a subband image correction amount multiplication unit that calculates an amount and multiplies each subband image. In claim 17,
The focused captured image is
An endoscope apparatus characterized in that it is a captured image extracted at the time when the optical system of the endoscope apparatus is switched to an enlarged observation state. In claim 17,
The focused captured image is
It is a captured image extracted after the time point when the optical system of the endoscope apparatus is switched to a magnified observation state, and the captured image satisfies a predetermined condition within a predetermined period. Endoscopic device. In claim 23,
The focused captured image is
Imaging images extracted after the time point when the optical system of the endoscope apparatus is switched to the enlarged observation state, the amplitude value is larger than a reference amplitude determination threshold value, and the amplitude value is a maximum value. An endoscope apparatus characterized by being an image. In claim 1,
A focus position control unit for controlling the focus position;
A target focus position calculation unit that calculates a target focus position based on blur amount information calculated by the blur amount information extraction unit for a captured image based on a plurality of focus positions moved in time series by the focus position control unit; ,
Including
The focus position control unit
An endoscope apparatus, wherein a focus position is controlled based on the target focus position. In claim 25,
The blur amount correction unit
It is determined that a captured image at a focus position determined to be in focus by the focus position control unit is the focused captured image, and correction is performed based on blur amount information of the focused captured image. Endoscope device. In claim 1,
The magnified observation state is
An endoscope apparatus characterized in that an enlargement magnification of an optical system of the endoscope apparatus is a predetermined magnification or more. An image acquisition unit for acquiring captured images in a time-series manner in an enlarged observation state in which the magnification of the optical system is high compared to the normal observation state;
A blur amount information extraction unit that extracts blur amount information from the captured image in the magnified observation state;
As a blur amount correction unit that corrects the captured image based on the extracted blur amount information,
A program characterized by causing a computer to function.

Images