JP6242230B2 - Image processing apparatus, endoscope apparatus, operation method of image processing apparatus, and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an endoscope apparatus, an operation method of the image processing apparatus, an image processing program, and the like.

  As a moving image noise reduction method, there are a noise reduction process in a spatial direction using one frame of a moving image and a noise reduction process in a time direction using a plurality of frames having different shooting times. The noise reduction process in the time direction provides an image with a higher resolution than the noise reduction process in the spatial direction, but an afterimage occurs when the subject is not stationary.

  As a technique for performing high-resolution noise reduction while reducing such afterimages, Patent Document 1 discloses a technique for blending the result of noise reduction processing in the spatial direction and the result of noise reduction processing in the time direction. Yes. In this method, the SAD (absolute difference value) of the pixel value at the same position in the current frame and the previous frame is calculated, and the blend rate and the cyclic coefficient used in the time-direction noise reduction processing are determined according to the SAD. To do.

  Since it is considered that the subject is not stationary in the region where the SAD is large in the image, the blend rate as a result obtained from the noise reduction processing in the time direction is lowered in the region where the SAD is large. Increase the blend ratio of the results obtained from the noise reduction process. As a result, a high-resolution noise reduction processing result can be obtained in an area where the subject is stationary.

  Further, the cyclic coefficient is determined so that the coefficient becomes larger as the SAD is smaller. As a result, in an area where the subject is not stationary, noise reduction is performed based on a small cyclic coefficient, so that afterimage generation is suppressed. On the other hand, since noise reduction processing is performed based on a large cyclic coefficient in a region where the subject is stationary, the noise reduction effect can be improved.

JP 2005-150903 A

  However, even when the SAD is small, an afterimage may be generated in the noise reduction process in the time direction, or an artifact may be generated by blending the results of the noise reduction process in the time direction. In this case, since the blend ratio is determined using only SAD in Patent Document 1, there is a problem that the blend ratio of the noise reduction process in the time direction is increased, and afterimages and artifacts are generated.

  According to some aspects of the present invention, it is possible to provide an image processing device, an endoscope device, an image processing method, an image processing program, and the like that can reduce afterimages and artifacts by noise reduction processing in consideration of shooting conditions. .

  One aspect of the present invention is an image acquisition unit that acquires a time-series captured image by an imaging unit, and a first noise reduction process that generates a first image by performing noise reduction processing in a time direction on the captured image. A second noise reduction processing unit that performs at least spatial noise reduction processing on the captured image to generate a second image, and a similarity detection unit that detects a similarity between frames of the captured image An image composition unit that performs composition processing of the first image and the second image so that the degree of composition of the first image increases and the degree of composition of the second image decreases as the similarity increases. A shooting condition acquisition unit that acquires shooting conditions of the captured image, and the image synthesis unit relates to an image processing apparatus that changes the degree of synthesis in the synthesis process based on the shooting conditions.

  According to one aspect of the present invention, the first image and the second image are synthesized such that the higher the similarity between frames of the captured image, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image. It is processed. At this time, the degree of synthesis is changed depending on the shooting conditions. As a result, noise reduction processing in consideration of shooting conditions is performed, and afterimages and artifacts can be reduced.

  Another aspect of the present invention relates to an endoscope apparatus including the image processing apparatus described above.

  According to still another aspect of the present invention, a first noise reduction process for acquiring a time-series captured image by an imaging unit and generating a first image by performing a noise reduction process in a time direction on the captured image is performed. , Performing a second noise reduction process for generating a second image by performing at least a spatial noise reduction process on the captured image, detecting a similarity between frames of the captured image, and capturing conditions of the captured image The first image and the second image are combined so that the higher the similarity is, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image are. This relates to the image processing method for changing the degree of synthesis.

  According to still another aspect of the present invention, a first noise reduction process for acquiring a time-series captured image by an imaging unit and generating a first image by performing a noise reduction process in a time direction on the captured image is performed. , Performing a second noise reduction process for generating a second image by performing at least a spatial noise reduction process on the captured image, detecting a similarity between frames of the captured image, and capturing conditions of the captured image The first image and the second image are combined so that the higher the similarity is, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image are. This relates to an image processing program for causing a computer to execute the step of changing the degree of synthesis.

1 is a configuration example of an image processing apparatus. The structural example of the endoscope apparatus in 1st Embodiment. An arrangement configuration example of a color filter of an image sensor. An example of transmittance characteristics of a color filter. The detailed structural example of the noise reduction process part in 1st Embodiment. FIG. 6A and FIG. 6B are explanatory diagrams of the difference average value mSAD. FIG. 7A to FIG. 7C are explanatory diagrams of the difference average value mSAD. FIG. 8A to FIG. 8C are explanatory diagrams of the difference average value mSAD. 3 is a detailed configuration example of an imaging condition acquisition unit in the first embodiment. Explanatory drawing about the process which detects a motion vector. 3 is a detailed configuration example of an image composition unit in the first embodiment. Explanatory drawing about the blend rate regarding similarity. Explanatory drawing about the blend rate regarding a motion amount. The detailed structural example of the imaging condition acquisition part in 2nd Embodiment. Explanatory drawing about the blend rate regarding the variation | change_quantity of brightness. Explanatory drawing about the image synthetic | combination part in 3rd Embodiment. The structural example of the endoscope apparatus in 5th Embodiment. An example of transmittance characteristics of a narrow band filter. The detailed structural example of the imaging condition acquisition part in 5th Embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline of this Embodiment The SAD described above is a comparison of pixel values of the current frame and the previous frame at the same position, and it can be said that it represents the degree of local motion (degree of stillness). In a region where the SAD is small, the subject is considered to be stationary, and thus it is considered that no afterimage has occurred in the noise reduction processing in the time direction. For this reason, in a region where the SAD is small, it is possible to reduce noise with high resolution while suppressing afterimages by blending the results of noise reduction processing in the time direction with a large blend rate.

  However, in the above method, the blend ratio is determined only by SAD, and imaging conditions are not taken into consideration, so there is a possibility that the image quality is reduced in noise reduction. That is, the shooting conditions are conditions that affect the image quality of the image when the result of the noise reduction process in the time direction and the result of the noise reduction process in the spatial direction are combined in the noise reduction process. In other words, there is a possibility that the image quality of the synthesized image may deteriorate under a specific shooting condition.

  As the deterioration of the image quality, for example, the above-mentioned afterimage may remain or an artifact may occur. For example, the following situations can be considered as a situation in which these decreases occur.

  The first is a case where an afterimage is generated in the noise reduction processing in the time direction even though the local motion (SAD) is small. The occurrence of an afterimage means that the subject has movement, but the SAD becomes small in a low-contrast image, for example. Endoscopic devices often observe subjects with relatively few structures, such as the mucous membrane of the digestive tract, and this situation is likely to occur.

  As described in the following equation (2), in the noise reduction process in the time direction, a motion vector between frames is detected and alignment is performed. If this alignment is successful, no afterimage occurs, but in reality, the detection of the motion vector may fail. For example, this is a case where a relatively large movement occurs in the subject. For example, a motion exceeding the search range for motion detection cannot be detected in the first place. In such a case, if the blend rate is determined only by the small SAD, the result of noise reduction processing in the time direction is blended at a large blend rate, and an afterimage remains.

  The second is a case where the brightness of the current frame and the previous frame are significantly different. Such a situation can occur, for example, when the imaging unit is moved greatly. In the endoscope apparatus, since the illumination light is irradiated from the tip of the imaging unit, the brightness varies depending on the distance to the subject. Therefore, the brightness changes greatly when the imaging unit is moved in the depth (optical axis) direction, or when the distance to the subject changes due to peristalsis of the digestive tract or the like.

  Since noise reduction in the spatial direction is basically performed on the current frame, the image has a brightness close to that of the current frame. On the other hand, since the current frame and the previous frame are added in the noise reduction process in the time direction, if the brightness of the current frame and the previous frame is greatly different, the brightness of the image after the addition is different from the brightness of the current frame. . Since SAD represents a local motion, it generally has a different value for each pixel or area, and the blend rate determined from that SAD differs for each pixel or area. Then, the result of the noise reduction process in the time direction and the result of the noise reduction process in the spatial direction with different brightness are blended at different ratios for each pixel or area, and the difference in brightness becomes an artifact.

  Note that the shooting conditions for reducing the image quality of the noise reduction processing are not limited to the above example. For example, if there is a lot of noise in the image, there is a high possibility that the detection of the motion vector will fail, and an afterimage tends to occur in the noise reduction processing in the time direction. As a case where noise increases, for example, a narrow-band optical image described in the fifth embodiment can be considered.

  As described above, when the result of the noise reduction process in the time direction and the result of the noise reduction process in the spatial direction are combined only with the SAD that is the degree of local motion, there is a problem that the image quality is deteriorated in the noise reduction process.

  FIG. 1 shows a configuration example of an image processing apparatus according to this embodiment that can solve such a problem. The image processing apparatus includes an image acquisition unit 315, a first noise reduction processing unit 323, a second noise reduction processing unit 324, a similarity detection unit 321, an imaging condition acquisition unit 322, and an image composition unit 325.

  The image acquisition unit 315 acquires time-series captured images captured by the imaging unit. The image acquisition unit 315 corresponds to, for example, the interpolation processing unit 310 described later with reference to FIG. That is, an interpolation process is performed on the Bayer array image, and an RGB image in which each pixel has an RGB pixel value is acquired as a captured image. Alternatively, in the case of a frame sequential method that sequentially captures an R image, a G image, and a B image, the image acquisition unit 315 combines the R image, the G image, and the B image into one RGB image. A captured image may be acquired.

  The first noise reduction processing unit 323 performs a time-direction noise reduction process on the captured image to generate a first image. The noise reduction process in the time direction is a process of performing smoothing using pixel values of a plurality of frames. For example, as shown in the following equation (2), the pixel values of the current frame and the previous frame are weighted and averaged in units of pixels.

  The second noise reduction processing unit 324 performs at least spatial noise reduction processing on the captured image to generate a second image. The noise reduction process in the spatial direction is a process of performing smoothing using pixel values of the same frame. Since it is sufficient to perform noise reduction processing in at least the spatial direction, smoothing may be performed using only pixel values of the same frame, or pixel values of a plurality of frames may be further reduced as shown in the following equations (3) and (4). The smoothing used may be combined.

  The similarity detection unit 321 detects the similarity between frames of a captured image. That is, the similarity between the current frame and the previous frame is detected. The similarity is an index that is calculated for each pixel or each local region and represents the degree of local correlation between frames. For example, the difference average value mSAD shown in the following formula (1) is used as the similarity. This difference average value mSAD is calculated for each pixel position (x, y).

  The shooting condition acquisition unit 322 acquires shooting conditions of a captured image. The shooting conditions are conditions that may cause a reduction in image quality as described above. The shooting conditions may be acquired, for example, by extracting a predetermined amount from the captured image, may be acquired by a sensor, or may be acquired by a control signal or operation input.

  The image synthesizing unit 325 synthesizes the first image and the second image so that the higher the similarity is, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image. In other words, the result of the noise reduction process in the time direction is synthesized with a high degree of synthesis in a pixel or local region with a high degree of similarity. At this time, the image composition unit 325 changes the composition degree according to the shooting conditions. When the shooting condition is a condition for generating an afterimage or an artifact, the degree of synthesis of the first image is lowered. That is, even with the same similarity, the degree of synthesis of the results of noise reduction processing in the time direction is reduced, and the occurrence of afterimages and artifacts can be suppressed.

  Here, the synthesis process is a blend process or a selection process. The blend process is a process of performing a weighted average of the first image at a first blend ratio (for example, 1-α described later) and the second image at a second blend ratio (for example, α described later). In this case, the degree of synthesis is a blend rate. On the other hand, the selection process is a process of selecting either the first image or the second image. In this case, the degree of synthesis is the degree of selection (ease of selection). For example, when the first image is selected when the similarity is greater than or equal to a threshold, the degree to which the first image is selected increases if the similarity is high, and the degree to which the first image is selected if the similarity is low Go down. In addition, the degree to which the first image is selected can be changed by changing the threshold value.

  Specifically, the shooting condition acquisition unit 322 acquires motion information that changes due to relative movement between the imaging unit and the subject as a shooting condition. Then, the image composition unit 325 lowers the degree of synthesis of the first image and increases the degree of synthesis of the second image as the relative motion represented by the motion information is larger.

  The motion information includes, for example, the above-described subject motion amount and brightness change amount. The amount of movement of the subject in the image is large, for example, when the imaging unit and the subject move relatively horizontally (in the direction intersecting the optical axis). The brightness of the image greatly changes when, for example, the imaging unit and the subject move relatively vertically (in the optical axis direction).

  When the motion information is a motion amount, the larger the motion amount, the greater the relative motion. In addition, when the motion information is a brightness change amount, the greater the brightness change amount, the greater the relative motion. In the present embodiment, the greater the motion represented by such motion information, the lower the degree of synthesis of the results of noise reduction processing in the time direction, thereby suppressing the occurrence of afterimages and artifacts.

  Further, the similarity detection unit 321 detects similarity in the first region of the captured image, and the imaging condition acquisition unit 322 acquires motion information in a second region wider than the first region of the captured image. .

  For example, when the similarity is the difference average value mSAD shown in the following equation (1), the similarity is detected in an area (first area) of (2k + 1) × (2k + 1) pixels. The motion information such as the motion amount and the brightness change amount is detected in a region (second region) wider than the (2k + 1) × (2k + 1) pixel region. For example, as described in the first and second embodiments, the amount of motion and the amount of change in brightness are detected for the entire image.

  The above-described shooting conditions that lower the image quality are expected to occur in a global (global) area rather than a similarity detection area. For example, it is considered that movements caused by operation of the endoscope scope or peristalsis of the digestive tract extend over a wide range of images. Further, even if there is no difference between frames in a small area where the similarity is detected in a low-contrast image or the like, it is possible to detect the amount of movement and the amount of change in brightness when viewed in a wide area.

  For this reason, by acquiring global motion information, it is possible to reduce the degree to which the results of noise reduction processing in the time direction are combined even when the degree of similarity is small, reducing afterimages and artifacts. it can.

2. First embodiment 2.1. Endoscope Device Hereinafter, detailed embodiments will be described. FIG. 2 shows a configuration example of the endoscope apparatus according to the first embodiment. The endoscope system includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. The control device 300 includes an interpolation processing unit 310, a noise reduction processing unit 320, a frame memory 330, a display image generation unit 340, and a control unit 390.

  The light source unit 100 includes a white light source 110 that generates white light and a lens 120 that collects the white light on the light guide fiber 210.

  The imaging unit 200 is formed to be elongate and bendable, for example, so as to be inserted into a body cavity. In addition, since different imaging units are used depending on the site to be observed, the imaging unit 200 has a structure that can be attached to and detached from the control device 300. In the following description, the imaging unit 200 is appropriately referred to as a scope.

  The imaging unit 200 includes a light guide fiber 210 for guiding the light collected by the light source unit 100, and an illumination lens 220 that diffuses the light guided by the light guide fiber 210 and irradiates the subject. The imaging unit 200 includes a condenser lens 230 that collects reflected light from the subject, an imaging element 240 for detecting the reflected light collected by the condenser lens 230, and a memory 250. The memory 250 is connected to the control unit 390.

  The image sensor 240 is an image sensor having a Bayer array color filter as shown in FIG. There are three types of color filters: r filters, g filters, and b filters. As shown in FIG. 4, the r filter transmits light of 580 to 700 nm, the g filter transmits light of 480 to 600 nm, and the b filter transmits light of 390 to 500 nm.

  The external I / F unit 500 is an interface for performing input from the user to the endoscope apparatus. For example, the external I / F unit 500 is configured to include a power switch for turning on / off the power, a mode switching button for switching a photographing mode and various other modes. The external I / F unit 500 outputs the input information to the control unit 390.

  The control device 300 performs control of each part of the endoscope device, image processing on the captured image, and the like. The interpolation processing unit 310 is connected to the noise reduction processing unit 320. The noise reduction processing unit 320 is connected to the display image generation unit 340. In addition, the noise reduction processing unit 320 is bidirectionally connected to the frame memory 330. The display image generation unit 340 is connected to the display unit 400. The control unit 390 is connected to the interpolation processing unit 310, the noise reduction processing unit 320, the frame memory 330, and the display image generation unit 340, and performs these controls.

  The interpolation processing unit 310 performs an interpolation process on the image acquired by the image sensor 240. Since the image sensor 240 has a Bayer array as described above, each pixel of an image acquired by the image sensor 240 has a signal value of one color of R, G, and B signals, and the other two. The color signal value is missing. The interpolation processing unit 310 interpolates the missing signal value by performing interpolation processing on each pixel of this image, and generates an image having all the signal values of R, G, and B signals at each pixel. To do. For example, a known bicubic interpolation process may be used as the interpolation process. Hereinafter, the image after the interpolation processing is referred to as an RGB image. The interpolation processing unit 310 outputs the generated RGB image to the noise reduction processing unit 320.

  The noise reduction processing unit 320 performs noise reduction processing on the RGB image output from the interpolation processing unit 310. Details of the noise reduction processing unit 320 will be described later. Hereinafter, the image after the noise reduction process is referred to as an NR image (noise reduced image).

  The display image generation unit 340 performs, for example, existing white balance, color conversion processing, and gradation conversion processing on the NR image output from the noise reduction processing unit 320 to generate a display image. The display image generation unit 340 outputs the generated display image to the display unit 400. The display unit 400 displays a display image and is configured by a display device such as a liquid crystal display device.

2.2. Noise Reduction Processing Unit FIG. 5 shows a detailed configuration example of the noise reduction processing unit 320. The noise reduction processing unit 320 includes a similarity detection unit 321, an imaging condition acquisition unit 322, a first noise reduction processing unit 323, a second noise reduction processing unit 324, and an image composition unit 325.

  The similarity detection unit 321 is connected to the interpolation processing unit 310, the frame memory 330, and the image composition unit 325. The similarity detection unit 321 detects the similarity using the current RGB image obtained from the interpolation processing unit 310 and the past image obtained from the frame memory 330, and transfers the similarity to the image composition unit 325. The past image is an NR image output from the noise reduction processing unit 320 at a time one frame before the current RGB image. Hereinafter, the current RGB image is called the current frame, and the past image is called the previous frame. The similarity detection unit 321 detects a difference value in a predetermined rectangular area between the current frame and the previous frame as the similarity using the following formula (1) described later.

  The imaging condition acquisition unit 322 is connected to the interpolation processing unit 310, the frame memory 330, and the image composition unit 325. The shooting condition acquisition unit 322 detects the amount of movement of the subject from the current frame and the previous frame, and transfers the amount of movement to the image composition unit 325 as the shooting condition. Specifically, the imaging condition acquisition unit 322 detects the motion vector of the subject between the current frame and the previous frame for each local region, and arranges the detected motion vectors in ascending order of size. Then, a plurality of lower-order motion vectors are extracted from the motion vectors arranged in ascending order, an average value of the extracted motion vectors is obtained, and the average value is used as a motion amount.

  The first noise reduction processing unit 323 is connected to the interpolation processing unit 310, the frame memory 330, and the image composition unit 325. The first noise reduction processing unit 323 performs noise reduction processing in the time direction using the current frame and the previous frame, and transfers the resulting image (hereinafter referred to as a time direction NR image) to the image synthesis unit 325. Details of the noise reduction processing in the time direction will be described later with the following equation (2).

  The second noise reduction processing unit 324 is connected to the interpolation processing unit 310 and the image synthesis unit 325. The second noise reduction processing unit 324 performs noise reduction processing in the spatial direction using the current frame, and transfers the resulting image (hereinafter referred to as spatial direction NR image) to the image synthesis unit 325. Details of the noise reduction processing in the spatial direction will be described later with the following equations (3) and (4).

  The image synthesis unit 325 performs a process of synthesizing the time direction NR image and the spatial direction NR image based on the similarity and the motion amount. The image composition unit 325 transfers the image resulting from the composition process to the frame memory 330 and the display image generation unit 340.

  The frame memory 330 stores the NR image transferred from the image composition unit 325. Then, the NR image is transferred as a previous frame to the similarity detection unit 321, the imaging condition acquisition unit 322, and the first noise reduction processing unit 323.

2.3. Similarity Detection Unit Next, details of the similarity detection unit will be described. The similarity detection unit 321 calculates a difference average value mSAD, which is a similarity, using the current frame and the previous frame. The difference average value mSAD will be described in detail below.

Assuming that the coordinates of the pixel of interest, which is a target pixel for noise reduction processing, are (x, y), the difference average value mSAD (inter-frame difference value) is calculated using the following equation (1).

  The similarity detection unit 321 outputs the value of the difference average value mSAD that minimizes the SAD (m, n) of the above formula (1) to the image synthesis unit 325. In the following, the processing for the G signal will be described, but the same processing is performed for the R signal and the B signal.

Here, in the above formula (1), min {} represents a process of acquiring the minimum value in the parentheses. Further, m = -1, 0, 1, and n = -1, 0, 1. _{Further, F G_cur (x,} y) is a G signal value at the coordinates (x, y) of the current _{frame, F G_pre (x,} y) is a G signal value at the coordinates (x, y) of the previous frame. Also, k is a natural number, and (2k + 1) corresponds to the kernel size when calculating the difference average value mSAD. A constant value can be set for k in advance, or the user can set an arbitrary value from the external I / F unit 500. || A || represents a process of acquiring the absolute value of the real value A.

  The difference average value mSAD differs depending on whether the subject is in a stationary state or in an operating state (a state where there is motion). This feature will be described below. In order to simplify the description, the image is handled as a one-dimensional signal. In addition, it is assumed that the component of the image signal includes a structural component illustrated in FIG. 6A and a noise component illustrated in FIG.

  Consider a case where the image shown in FIG. 7A is acquired at a certain time t and the image shown in FIG. 7B is acquired at the time t-1 immediately before that. In this case, the position of the structural component does not change at time t and time t-1 (corresponding to a stationary state). Therefore, when the difference value (absolute value of the difference value) between the image at time t and the image at time t−1 is obtained, only the noise component is included in the difference value as shown in FIG. become. On the other hand, consider a case where the image shown in FIG. 8A is acquired at a certain time t, and the image shown in FIG. 8B is acquired at the time t-1 immediately before that. D shown in FIG. 8B corresponds to the amount of movement of the subject between frames (corresponding to the operating state). In this case, when the difference value between the image at time t and the image at time t−1 is obtained, the difference value includes both a noise component and a structural component as shown in FIG.

  Therefore, the difference average value mSAD in the operating state is characterized by being larger than the difference average value mSAD in the stationary state. The difference average value mSAD is used to determine the blend ratio, as will be described in the following equations (9) to (12).

2.4. First Noise Reduction Processing Unit Next, details of noise reduction processing in the time direction will be described. The following formula (2) is used for the noise reduction process in the time direction.

Here, in the above equation (2), _{FG_TimeNR} (x, y) is a G signal value at the coordinates (x, y) of the time direction NR image. Further, we_cur and we_pre are weighting coefficients at the time of the weighted average process. By making the weighting coefficient we_pre larger than the weighting coefficient we_cur, the amount of noise reduction is increased. The weighting factors we_cur and we_pre may be set to certain values in advance, or the user may set arbitrary values from the external I / F unit 500. Further, (u, v) is a motion vector. That is, the motion vector (u, v) of the subject at each coordinate (x, y) is detected in the noise reduction process in the time direction, and alignment is performed based on the motion vector (u, v).

2.5. Second Noise Reduction Processing Unit Next, details of the noise reduction processing in the spatial direction will be described. In the present embodiment, the noise reduction process in the spatial direction is a weighted average process using a pixel (target pixel) that is a target of the noise reduction process and surrounding pixels. Specifically, noise is reduced using the following equation (3).

Here, in the above equation (3), we_diff_cur (x + i, y + j) and we_diff_pre (x + i, y + j) correspond to the weighting coefficients in the weighted average process. This coefficient is given by a Gaussian distribution as shown in the following equation (4). I is a natural number. M and n are m and n of SAD (m, n) selected as the difference average value mSAD in the above equation (1).

  As shown in the above equation (4), in the spatial noise reduction processing used in the present embodiment, the weighting factor is adaptively set according to the difference between the signal value of the target pixel and the signal value of the surrounding pixels. Is set. Specifically, when the difference is large, the weight during the weighted average process is small. Therefore, pixels in regions where signal values change suddenly, such as edge portions, do not contribute to the weighted average processing, and there is an advantage that only noise components can be reduced while retaining edge portions.

2.6. Imaging condition acquisition unit As described above, the noise reduction processing unit 320 performs the noise reduction processing in the time direction and the noise reduction processing in the spatial direction. While the noise reduction process in the time direction can maintain a sense of resolution, there are side effects of afterimages. The noise reduction process in the time direction has no side effects, but has a side effect of reducing resolution.

  Therefore, in the present embodiment, by blending the spatial direction NR image and the temporal direction NR image according to the similarity (difference average value mSAD), each advantage of the temporal direction noise reduction processing and the spatial direction noise reduction processing. Realizing high-performance noise reduction by utilizing

  Specifically, when the difference average value mSAD is small, it can be determined that the subject is stationary between the current frame and the previous frame. Therefore, since the similarity increases when the difference average value mSAD is small, control is performed to increase the blend ratio of the time-direction NR image. Conversely, when the value of the difference average value mSAD is large, it can be determined that the subject is in an operating state between the current frame and the previous frame. Therefore, when the difference average value mSAD is large, the degree of similarity is low, so control is performed to increase the blend ratio of the spatial direction NR image.

  By such control, the resolution can be improved by increasing the blend ratio of the time direction NR image in a still state where no afterimage is generated, and the afterimage can be suppressed by increasing the blend ratio of the spatial direction NR image in an operation state where the afterimage is generated.

  However, there is a case where the reliability of the difference average value mSAD is low as an index for determining the stationary state. In such a case, even if the difference average value mSAD is small, there is a possibility that it is not actually in the stationary state. is there. For example, this is the first situation described in “1. Overview of the present embodiment”. That is, the difference average value mSAD is erroneously determined due to the influence of the contrast and noise of the subject, and the degree of synthesis increases even though there is an afterimage in the time direction NR image.

  Therefore, in the present embodiment, the blend rate is determined using a global motion amount that represents the degree of motion of the subject in a wide range of the image. That is, when the global motion amount is large, the noise reduction processing with reduced afterimage is realized by adjusting the blend rate so as to reduce the time-direction NR image.

  Hereinafter, this configuration will be described in detail. FIG. 9 shows a detailed configuration example of the imaging condition acquisition unit 322. The imaging condition acquisition unit 322 includes a motion vector detection unit 601 that calculates a motion vector from a plurality of local regions using a known block matching process, a motion vector alignment unit 602 that arranges the obtained motion vectors in ascending order, A motion vector averaging unit 603 that employs lower-order motion vectors in proportion and calculates an average value thereof.

  Specifically, the motion vector detection unit 601 sets a plurality of local regions centered on the representative point as shown in FIG. 10 for the current frame. x is a coordinate in the horizontal scanning direction of the image, and y is a coordinate in the vertical scanning direction of the image. The size and number of local areas to be set can be set in advance, or can be configured so that the user can set arbitrary values using the external I / F unit 500.

  Next, the motion vector detection unit 601 calculates a motion vector (mx, my) using a known block matching process for all the set local regions. In the block matching process, the position of a block having a high correlation with respect to each block of the current frame (reference image) is searched in the previous frame (target image). The relative shift amount between the blocks corresponds to the motion vector (mx, my) of the block.

As a method of searching for a highly correlated block by block matching, for example, a square error SSD or an absolute error SAD may be used. This is a method for obtaining the position of I ′ having a high correlation with I, where I is the block area in the reference image and I ′ is the block area in the target image. Assuming that the pixel position in each block region is pεI and qεI ′ and the signal values of each pixel are Lp and Lq, SSD and SAD are defined by the following equations (5) and (6), The smaller the value, the higher the correlation.

  Here, p and q each have a two-dimensional value, I and I 'have a two-dimensional area, and pεI indicates that the coordinate p is included in the area I. || A || represents a process of acquiring the absolute value of the real value A.

The motion vector alignment unit 602 aligns the motion vectors (mx, my) calculated in all local regions in ascending order based on the size (size). The size of the motion vector is an absolute value of mx, which is the horizontal component of the motion vector, and my, which is the vertical component of the motion vector. As shown in the following formula (7), the motion amount mv in each representative pixel to be calculated is obtained by comparing | mx | and | my | and selecting the larger one.

The motion vector averaging unit 603 employs a lower predetermined ratio (for example, several percent) of the motion amount mv from the aligned motion amounts mv. The following equation (8) shows an equation for averaging the plurality of employed motion amounts mv.

Here, num is the number of motion vectors, mv _i is the motion amount mv of the i-th representative point, the results calculated by they are global motion amount MV.

  In the above example, the motion amount MV is calculated for the entire frame. However, the present invention is not limited to this, and the motion amount MV may be calculated for each predetermined region or each pixel.

2.7. Image Composition Unit FIG. 11 shows a detailed configuration example of the image composition unit 325. The image composition unit 325 includes a blend rate determination unit 611 that determines a blend rate based on the difference average value mSAD that is the similarity and the global motion amount MV, and a temporal direction NR image and a spatial direction NR based on the blend rate. A blend processing unit 612 for blending images.

Specifically, the blend rate determination unit 611 acquires a parameter for determining the blend rate from the control unit 390, acquires the similarity (difference average value mSAD) from the similarity detection unit 321, and captures an imaging condition acquisition unit. The amount of movement MV is acquired from 322. Then, the blend rate α is determined by the following equation (9). The blend rate α is the blend rate of the spatial direction NR image, and the blend rate of the time direction NR image is (1−α).

  α is normalized so as to be a value between 0 and 1. BLEND_mSAD is a blend rate related to the similarity, and BLEND_CONDITION is a blend rate related to the observation condition.

  First, the blend rate BLEND_mSAD related to the similarity will be described. The memory 250 holds a threshold value mSAD_TH and a gradient mSAD_SLP, which are parameters used to determine the blend ratio BLEND_mSAD. These parameters are registered in the memory 250 in advance. The control unit 390 refers to the memory 250 and transfers the parameters to the image composition unit 325.

As shown in FIG. 12, the horizontal axis is the difference average value mSAD, and the vertical axis is the blend rate BLEND_mSAD. First, BLEND_mSAD ′ is calculated by the following equation (10) using the threshold value mSAD_TH and the slope mSAD_SLP.

Next, BLEND_mSAD ′ is clipped with the lower limit value 0 and the upper limit value 64 as shown in the following equation (11). The lower limit value 0 and the upper limit value 64 can be set to arbitrary values.

Next, BLEND_mSAD ′ is normalized as shown in the following equation (12) to obtain BLEND_mSAD.

  Since the difference average value mSAD is calculated at each position (x, y), the blend rate BLEND_mSAD becomes a different value at each position (x, y). That is, the final blend rate α also has a different value at each position (x, y).

Next, the blend rate BLEND_CONDITION related to the observation conditions will be described. As shown in the following formula (13), the blend rate BLEND_CONDITION is the blend rate BLEND_MV related to the motion amount MV.

  The memory 250 holds parameters used for determining the blend ratio BLEND_MV, a threshold MV_TH and a slope MV_SLP. These parameters are registered in the memory 250 in advance. The control unit 390 refers to the memory 250 and transfers the parameters to the image composition unit 325.

As shown in FIG. 13, the horizontal axis is the amount of movement MV, and the vertical axis is the blend rate BLEND_MV. First, BLEND_MV ′ is obtained by the following equation (14) using the threshold MV_TH and the gradient MV_SLP. Then, BLEND_MV ′ is clipped by the following equation (15), and BLEND_MV ′ is normalized as shown in the following equation (16) to obtain the blend ratio BLEND_MV.

  The thresholds mSAD_TH and mSAD_TH are provided for the following reason. That is, due to the presence of the threshold value, the blend ratio α of the spatial direction NR image is lowered until the difference average value mSAD and the motion amount MV become a certain level. This is because the blend ratio (1-α) of the time-direction NR image can be increased and the resolution can be maintained under conditions where afterimages are unlikely to occur.

  Based on the blend rate α determined as described above, the blend processing unit 612 performs the temporal direction NR image from the first noise reduction processing unit 323 and the spatial direction NR image from the second noise reduction processing unit 324. And blend.

Specifically, the blended image _{FG_NR} (x, y) is _obtained by the following equation (17).

Here, _{FG_TimeNR} (x, y) is a pixel value of the time direction NR image, and _{FG_SpaceNR} is a pixel value of the spatial direction NR image.

  In the above description, the example of calculating the blend ratio α of the spatial direction NR image using the blend ratios BLEND_mSAD and BLEND_MV has been described. However, the present invention is not limited to this, and the blend ratio (1-α) of the time direction NR image is calculated. You may do it. In that case, the graphs of FIG. 12 and FIG.

  According to the first embodiment described above, the imaging condition acquisition unit 322 acquires the relative motion amount MV between the imaging unit 200 and the subject as motion information. Then, the image composition unit 325 increases the degree of composition of the first image (time direction NR image) and increases the degree of composition of the second image (space direction NR image) as the motion amount MV increases.

  Specifically, the imaging condition acquisition unit 322 acquires the magnitude of the motion vector between the frames of the subject on the captured image as the motion amount MV. The amount of motion is not limited to detecting a motion vector from an image. For example, a motion sensor (displacement amount detection unit) such as an acceleration sensor is provided in the imaging unit 200, and the amount of motion is detected based on the sensing signal. Also good.

  In this way, even if the degree of similarity is low, if the amount of motion MV is detected, the degree of synthesis of the time-direction NR image can be lowered, so that afterimages can be suppressed in the final NR image.

  Specifically, as shown in FIG. 12, the blend ratio BLEND_mSAD increases as the local similarity decreases (as the difference average value mSAD increases), so that the blend ratio α of the spatial direction NR image increases. growing. Also, as shown in FIG. 13, as the global motion amount MV increases, the blend rate BLEND_MV increases, so the blend rate α of the spatial direction NR image increases.

  Thereby, when at least one of the local similarity or the global motion amount MV is large, the blend ratio (1-α) of the time direction NR image becomes small, and the afterimage can be reduced. In particular, even when the local similarity is high (the difference average value mSAD is small), if the global motion amount MV is large, an afterimage is likely to occur in the time-direction NR image. However, since the global motion amount MV is taken into consideration, the blend ratio (1-α) of the time-direction NR image is reduced, and the afterimage can be suppressed.

  In the first embodiment, the synthesis process is a blend process, but as described in the third embodiment, the synthesis process may be a selection process. In the case of the selection process, the same effect as described above can be obtained.

  In the first embodiment, the imaging condition acquisition unit 322 detects motion vectors for a plurality of areas of the captured image, and arranges the motion vectors for the plurality of areas in the order of magnitude mv (the above formula (7)). Then, the imaging condition acquisition unit 322 acquires the average value of the motion vectors at a predetermined ratio (the above equation (8)) as the motion amount MV from the smaller size mv.

  For example, when a treatment is performed with forceps, the background is stationary, but a subject (forceps) in a relatively small area in the entire screen moves greatly. In such a situation, it is desirable to increase the resolution by adopting the time direction NR image as much as possible.

  However, in such a case, when the final global motion amount MV is obtained by adopting all the motion amounts mv, the global motion amount MV increases due to the motion amount mv of the moving subject (forceps). End up. Then, although the background is stationary and there is no global motion, the motion amount MV as if there is motion is calculated, and the degree of synthesis of the time direction NR image is reduced.

  In this regard, in the present embodiment, since a predetermined percentage of motion vectors is adopted from the smaller size mv, even if there is motion in a part of the image such as the above-mentioned forceps, the magnitude mv of the motion vector is Excluded. Thereby, when there is no motion in the background, a small value can be calculated as the motion amount MV, and the degree of synthesis of the time direction NR image can be increased.

3. Second Embodiment Next, a second embodiment will be described. As described in “1. Overview of the present embodiment”, there is a case where a large change occurs in the brightness of the image as a second situation in which the image quality is deteriorated. For example, assume that the current frame is a dark image and the previous frame is a bright image. In this case, the spatial direction NR image is a dark image like the current frame, and the temporal direction NR image is an intermediate brightness image obtained by adding the current frame and the previous frame. Since these images are blended at different blend ratios α depending on places, pixels or regions having different brightness are adjacent to each other, and the difference in brightness appears as a false contour, resulting in an artifact.

  Therefore, in the second embodiment, the blend rate α is determined in consideration of the amount of change in brightness between frames. In the following, a case where the motion amount MV and the brightness change amount are combined as an imaging condition will be described as an example. However, the present invention is not limited to this, and only the brightness change amount may be used as the imaging condition.

  FIG. 14 shows a detailed configuration example of the imaging condition acquisition unit 322 in the second embodiment. The imaging condition acquisition unit 322 includes a motion vector detection unit 601, a motion vector alignment unit 602, a motion vector average unit 603, and a brightness change amount detection unit 604. Note that the configurations of the endoscope apparatus and the noise reduction processing unit 320 are the same as those in the first embodiment. Further, the same components as those already described are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The brightness change amount detection unit 604 detects a change in brightness between the current frame and the previous frame as a brightness change amount, and transfers the brightness change amount to the image composition unit 325.

Specifically, as shown in the following equation (18) obtains an average value _{Ave All_cur} pixel values or luminance values of the current frame and an average value _{Ave All_pre} pixel value or luminance value of the previous frame, the difference between them The value is _obtained as a brightness change amount AVE _{all_diff} .

The average values Ave _{all_cur and} Ave _{all_pre} represent the brightness of the entire image. For example, the pixel values of pixels within a predetermined rectangle centered at the center position (x, y) = (W / 2, L / 2) of the image are obtained by averaging. W is the image size in the horizontal scanning direction, and L is the image size in the vertical scanning direction.

Based on the brightness change amount AVE _{all_diff} obtained in the above, the similarity (difference average value mSAD) and the global motion amount MV described in the first embodiment, the image composition unit 325 determines the temporal direction NR image and the space. Blend direction NR images.

  Specifically, as in FIG. 11 of the first embodiment, the image composition unit 325 includes a blend rate determination unit 611 and a blend processing unit 612.

The blend rate determination unit 611 acquires a parameter for determining the blend rate from the control unit 390, acquires the average difference value mSAD from the similarity detection unit 321, and acquires the motion amount MV and the brightness from the imaging condition acquisition unit 322. The change amount AVE _{all_diff} is acquired.

The formula for determining the blend rate α is the same as that in the above formula (9), but the blend rate BLEND_CONDITION relating to the observation condition is expressed by the following formula (19). BLEND_LN is a blend rate related to the brightness change amount AVE _{all_diff} .

  The blend rate BLEND_LN is obtained as follows. In other words, the memory 250 holds the threshold value LN_TH and the gradient LN_SLP as parameters used to determine the blend ratio BLEND_LN. These parameters are registered in the memory 250 in advance. The control unit 390 refers to the memory 250 and transfers the parameters to the image composition unit 325.

As shown in FIG. 15, the horizontal axis represents the brightness change amount AVE _{all_diff} for each frame, and the vertical axis represents the blend rate BLEND_LN. First, BLEND_LN ′ is obtained by the following equation (20) using the threshold value LN_TH and the gradient LN_SLP. Then, BLEND_LN ′ is clipped by the following equation (21), and BLEND_LN ′ is normalized as shown in the following equation (22) to obtain the blend ratio BLEND_LN.

The reason why the threshold value LN_TH is provided is as follows. That is, due to the presence of the threshold, the blend ratio α of the spatial direction NR image is lowered until the brightness change amount AVE _{all_diff} becomes a certain level. This is because the blend ratio (1-α) of the time-direction NR image can be increased and the resolution can be maintained under conditions where artifacts are unlikely to occur.

  In the above description, the example of calculating the blend ratio α of the spatial direction NR image using the blend ratios BLEND_mSAD and BLEND_CONDITION has been described. You may do it. In that case, the graphs of FIG. 12, FIG. 13, and FIG.

According to the second embodiment described above, the imaging condition acquisition unit 322 acquires the brightness change amount AVE _{all_diff} between frames of the captured image as motion information. The image composition unit 325 lowers the degree of synthesis of the first image (time direction NR image) and increases the degree of synthesis of the second image (space direction NR image) as the brightness change amount AVE _{all_diff} is larger.

In this way, even if the degree of similarity is low, if the brightness change amount AVE _{all_diff} is detected, the degree of synthesis of the time-direction NR image can be lowered, and thus artifacts can be suppressed.

Specifically, as shown in FIG. 15, as the brightness change amount AVE _{all_diff} increases, the value of the blend rate BLEND_LN increases, so the blend rate α of the spatial direction NR image increases. That is, even when the local similarity is high (the difference average value mSAD is small), if the brightness change amount AVE _{all_diff} of the entire image is large, the blend ratio (1− α) becomes smaller. Thereby, the spatial direction NR image and the time direction NR image having different brightness are not mixed and blended, and the artifact can be suppressed.

  In the second embodiment, the synthesis process is a blend process, but as described in the third embodiment, the synthesis process may be a selection process. In the case of the selection process, the same effect as described above can be obtained.

4). Third Embodiment Next, a third embodiment will be described. In the third embodiment, a threshold value is determined using a difference average value mSAD that is a similarity, a motion amount MV, and a brightness change amount AVE _{all_diff} , and based on the threshold value, either a time direction NR image or a spatial direction NR image Select.

In the following description, the threshold value is determined using both the motion amount MV and the brightness change amount AVE _{all_diff.} However, the present invention is not limited to this, and any one of the motion amount MV and the brightness change amount AVE _{all_diff} is described. One may be used to determine the threshold value.

  FIG. 16 shows a detailed configuration example of the image composition unit 325 in the third embodiment. The image composition unit 325 includes a threshold value determination unit 621 and a selection unit 622. The configuration of the endoscope apparatus and noise reduction processing unit 320 is the same as that of the first embodiment, and the configuration of the imaging condition acquisition unit 322 is the same as that of the second embodiment. Further, the same components as those already described are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The memory 250 holds a threshold mSAD_TH related to the similarity as a parameter. The control unit 390 refers to the parameters held in the memory 250 and sends the parameters to the image composition unit 325.

The threshold determination unit 621 acquires a parameter for determining a threshold from the control unit 390, acquires a similarity (difference average value mSAD) from the similarity detection unit 321, and acquires the motion amount MV and the brightness from the imaging condition acquisition unit 322. The change amount AVE _{all_diff} of the _length is acquired. The threshold determination unit 621 determines a threshold based on the obtained similarity, motion amount, and brightness change amount.

Specifically, the threshold value determination unit 621 resets the threshold value mSAD_TH ′ as shown in the following equation (23) using the threshold value mSAD_TH and the value CONDITION_VAL related to the imaging condition.

Here, CONDITION_VAL is calculated by the following equation (24) using the motion amount MV and the brightness change amount AVE _{all_diff} . The lower limit value 0 and the upper limit value 64 of CONDITION_VAL can be set to arbitrary values. In the above formula (23), the denominator 64 of CONDITION_VAL / 64 is the same value as the upper limit value of CONDITION_VAL, which means normalization.

According to the above equations (23) and (24), the threshold value mSAD_TH ′ can be set such that the larger the motion amount MV or the brightness change amount AVE _{all_diff, the} smaller the threshold value mSAD_TH.

  The selection unit 622 selects either the time direction NR image or the space direction NR image based on the threshold value mSAD_TH ′, and transfers the selected image to the frame memory 330 and the display image generation unit 340.

Specifically, the selection unit 622 performs a determination process as shown in the following equation (25) using the reset threshold value mSAD_TH ′ and the difference average value mSAD that is the similarity.

F _{G_TimeNR} (x, y) is a pixel value of the time direction NR image, F _{G_SpaceNR} is a pixel value of the spatial direction NR image, and F _{G_NR} (x, y) is the selected image.

  When the difference average value mSAD is smaller than mSAD_TH ′, it can be determined that the local motion detection based on the difference average value mSAD is successful, or that there is little change in brightness between the current frame and the previous frame. Select the result of noise reduction processing. On the other hand, when the difference average value mSAD is larger than mSAD_TH ′, it can be determined that the local motion detection based on the difference average value mSAD has failed, or the change in brightness between the current frame and the previous frame is large. Select the result of noise reduction processing in the spatial direction. In this way, afterimages and artifacts can be suppressed by taking the imaging conditions into consideration.

5. Fourth Embodiment Next, a fourth embodiment will be described. In the fourth embodiment, by correcting the brightness of the image based on the brightness change amount AVE _{all_diff} , the brightness of the spatial direction NR image and the time direction NR image are matched, and artifacts are suppressed. Specifically, the brightness change amount AVE _{all_diff} is added to the time direction NR image, and then blended or selected with the spatial direction NR image.

  First, a case where blending is performed will be described. The configuration of the image composition unit 325 is the same as that of the first embodiment of FIG. Further, the configuration of the endoscope apparatus and noise reduction processing unit 320 is the same as that of the first embodiment of FIGS. 1 and 5, and the configuration of the imaging condition acquisition unit 322 is the same as that of the second embodiment of FIG.

As shown in the following equation (26), the brightness change amount detection unit 604 calculates the average value AVE _cur (x, y) of the pixel values of the current frame within a predetermined rectangle centered on the target pixel (x, y). An average value AVE _pre (x + u, x + v) of the pixel values of the previous frame is taken, and a brightness change amount AVE _diff (x, y) between the current frame and the previous frame is calculated. (U, v) is a motion vector (u, v) of the above equation (2). That is, the brightness change amount detection unit 604 receives the motion vector (u, v) detected by the first noise reduction processing unit 323 and performs alignment. If no motion is detected, (u, v) = (0, 0).

The image composition unit 325 adds the brightness change amount AVE _diff (x, y) to the time direction NR image _{FG_TimeNR} (x, y) as shown in the following expression (27), and the image and the spatial direction NR. The image _{FG_SpaceNR} (x, y) is blended.

  Next, a case where selection is performed will be described. The configuration of the image composition unit 325 is the same as that of the third embodiment in FIG. Further, the configuration of the endoscope apparatus and noise reduction processing unit 320 is the same as that of the first embodiment of FIGS. 1 and 5, and the configuration of the imaging condition acquisition unit 322 is the same as that of the second embodiment of FIG.

The brightness change amount detection unit 604 calculates the brightness change amount AVE _diff (x, y) by the above equation (26). Then, the image composition unit 325 adds the brightness change amount AVE _diff (x, y) to the time direction NR image _{FG_TimeNR} (x, y) as shown in the following equation (28), and the image and space One of the direction NR images _{FG_SpaceNR} (x, y) is selected.

As an effect of the above processing, the brightness change amount AVE _diff (x, y) is added to the time direction NR image and then blended, thereby preventing artifacts due to the difference in brightness.

6). Fifth Embodiment Next, a fifth embodiment will be described. In the fifth embodiment, the blend rate and selection are controlled using the type of image as a shooting condition.

  FIG. 17 shows a configuration example of an endoscope apparatus according to the fifth embodiment. The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. In the following description, the same components as those already described are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 200 includes a light guide fiber 210, an illumination lens 220, a condenser lens 230, an imaging device 240, a memory 250, a filter driving unit 280, and a narrow band filter 290. The filter driving unit 280 is connected to the narrow band filter 290 and further connected to the control unit 390 in both directions.

  As shown in FIG. 18, the narrow band filter 290 transmits light of 380 to 450 nm and 530 to 550 nm. The narrow band filter 290 can be inserted into the optical path between the condenser lens 230 and the image sensor 240. The insertion / non-insertion of the narrow band filter 290 is controlled by the user from the external I / F unit 500, for example. In this case, a user instruction is sent from the external I / F unit 500 to the filter driving unit 280 via the control unit 390, and the filter driving unit 280 drives the narrow band filter 290. When the narrowband filter 290 is inserted in the optical path, the control unit 390 outputs a trigger signal to the interpolation processing unit 310 and the noise reduction processing unit 320.

  The interpolation processing unit 310 performs interpolation processing on the image captured by the image sensor 240. When the above-described trigger signal is not output from the control unit 390 (when the narrowband filter 290 is not inserted), the interpolation processing unit 310 generates an RGB image by the same method as in the first embodiment. Hereinafter, the RGB image acquired when the narrowband filter 290 is not inserted is referred to as a white light image.

  On the other hand, when the trigger signal is output from the control unit 390 (when the narrowband filter 290 is inserted), the interpolation processing unit 310 performs interpolation processing only on the G signal and the B signal. The interpolation process is, for example, a known bicubic interpolation process. In this case, the interpolation processing unit 310 generates a G image having a G signal in all pixels and a B image having a B signal in all pixels by interpolation processing. Then, the RGB image is generated by inputting the G image to the R signal of the RGB image and inputting the B image to the G signal and the B signal of the RGB image. Hereinafter, the RGB image acquired when the narrowband filter 290 is inserted is referred to as a narrowband light image (special light image).

  In the above description, the case where a narrowband light image is captured by inserting the narrowband filter 290 into the optical path has been described as an example. However, the present embodiment is not limited to this. For example, the light source unit 100 may capture a narrowband light image by emitting special light (narrowband light in a narrow sense). Alternatively, the imaging unit 200 may further include an imaging element having a color filter that transmits special light, and a narrowband light image may be captured by the imaging element. Alternatively, a narrow band light image may be generated from a white light image by image processing.

  Now, since the narrow-band light image has a narrower wavelength band of light than the white light image, the amount of light is insufficient and noise increases (S / N decreases). That is, the photographing conditions are different between the white light image and the narrow-band light image. As described above, since an afterimage is likely to appear in an NR image in an image with a reduced S / N, if the noise reduction processing of a narrowband light image is performed with the same blend ratio and threshold as the white light image, the NR image of the narrowband light image There is a problem that an afterimage appears.

  Therefore, in the present embodiment, by using the type of image as the shooting condition, a blend rate and a threshold value that match each image are set. In the case of a white light image, the blend rate and threshold set in the first embodiment are used. In the case of a narrow band light image, the amount of noise increases compared to the white light image, and the determination accuracy based on the difference average value mSAD is improved. The blend rate and threshold value are set in consideration of the decrease. Specifically, the ratio of using the spatial direction NR image is increased as compared with the white light image.

  As described above, by determining the blending rate and threshold value of the time direction NR image and the spatial direction NR image in accordance with the type of the image, and performing blending and selection, high-performance noise reduction with reduced afterimage can be realized.

  First, a case where blending is performed will be described. FIG. 19 shows a detailed configuration example of the imaging condition acquisition unit 322 in the fifth embodiment. The imaging condition acquisition unit 322 includes a motion vector detection unit 601, a motion vector alignment unit 602, a motion vector average unit 603, a brightness change amount detection unit 604, and a type information acquisition unit 605. The configurations of the noise reduction processing unit 320 and the image composition unit 325 are the same as those in the first embodiment shown in FIGS.

  The type information acquisition unit 605 acquires type information for identifying whether it is a white light image or a narrow band light image from the control unit 390. The type information is a trigger signal output from the control unit 390 when the user inserts the narrow band filter 290 by the external I / F unit 500. For example, the control unit 390 outputs “0” as a trigger signal for a white light image, and outputs “1” as a trigger signal for a narrowband light image. The type information acquisition unit 605 is connected to the image composition unit 325 and transfers the type information to the image composition unit 325.

  The image composition unit 325 calculates a blend rate α from parameters and similarities according to the type information acquired from the imaging condition acquisition unit 322, and performs a blend process based on the blend rate α.

  Specifically, the memory 250 holds a blend rate BLEND_WIL corresponding to a white light image and a blend rate BLEND_NBI corresponding to a narrowband image. BLEND_NBI and BLEND_WIL are 0 or more and 1 or less, and BLEND_NBI has a larger value than BLEND_WIL.

The blend rate determination unit 611 acquires type information from the shooting condition acquisition unit 322, and calculates a blend rate BLEND_CONDITION related to the shooting conditions by the following equation (29).

BLEND_IMAGEGETTYPE is a blend rate related to the type of image. As shown in the following formula (30), the blend rate determination unit 611 sets the blend rate BLEND_WLI corresponding to the white light image as the blend rate BLEND_IMAGETYPE when the trigger signal TS = 0. On the other hand, when the trigger signal TS = 1, the blend rate BLEND_NBI corresponding to the narrow-band light image is set as the blend rate BLEND_IMAGETYPE.

  The blend processing unit 612 calculates the blend rate α by substituting the blend rate BLEND_CONDITION set as described above into the above equation (9), and blends the time direction NR image and the spatial direction NR image by the above equation (17). .

  Next, a case where selection is performed will be described. The configurations of the imaging condition acquisition unit 322 and the noise reduction processing unit 320 are the same as those in the case of blending. The configuration of the image composition unit 325 is the same as that of the third embodiment in FIG.

  The memory 250 holds a value VAL_WLI corresponding to the white light image and a value VAL_NBI corresponding to the narrow band image. However, VAL_NBI has a larger value than VAL_WLI.

The threshold value determination unit 621 acquires type information from the imaging condition acquisition unit 322, and calculates a value CONDITION_VAL related to the imaging condition by the following equation (31).

VAL_IMAGETYPE is a value related to the type of image. As shown in the following equation (32), the threshold value determination unit 621 sets the value VAL_WLI corresponding to the white light image as the value VAL_IMAGETYPE when the trigger signal TS = 0. On the other hand, when the trigger signal TS = 1, the value VAL_NBI corresponding to the narrow band light image is set as the value VAL_IMAGEGETE.

  The threshold value determination unit 621 obtains the threshold value mSAD_TH ′ by substituting the value CONDITION_VAL set as described above into the above equation (23), and obtains either the time direction NR image or the spatial direction NR image by the above equation (25). select.

  According to the fifth embodiment described above, the shooting condition acquisition unit 322 acquires the type of the captured image as the shooting condition. Then, the image composition unit 325 changes the degree of composition between the case where the type of the captured image is the first type and the case where the type of the captured image is the second type.

  Specifically, the first type is a white light image having information in a white wavelength band, and the second type is a special light image (narrow band light image) having information in a specific wavelength band. It is. Then, when the type of the captured image is a special light image, the image composition unit 325 increases the degree of synthesis of the first image (time direction NR image) compared to the case where the type of the captured image is a white light image. While lowering, the synthesis degree of the second image (spatial direction NR image) is increased.

  In this way, by performing synthesis processing (blend processing or selection processing) with different synthesis levels depending on the type of image, the degree of synthesis of the time direction NR image even when the type of image is likely to generate afterimages. The afterimage can be suppressed by lowering.

  Specifically, in the case of a white light image, the blend ratio and the threshold set in the first embodiment are used. In the case of a narrow band light image, the amount of noise increases compared to the white light image, and the difference average value A blend rate and a threshold value are set in consideration of a decrease in determination accuracy by mSAD. In other words, afterimages can be suppressed in the final NR image by increasing the ratio of using the spatial direction NR image compared to the white light image.

  According to the fifth embodiment, the specific wavelength band is a band narrower than the white wavelength band (for example, 380 nm to 650 nm) (NBI: Narrow Band Imaging). For example, a white light image and a special light image are in-vivo images obtained by copying the inside of a living body, and a specific wavelength band included in the in-vivo image is a wavelength band of a wavelength that is absorbed by hemoglobin in blood. For example, the wavelength absorbed by the hemoglobin is 390 nm to 445 nm (first narrowband light) or 530 nm to 550 nm (second narrowband light).

  Thereby, it becomes possible to observe the structure of the blood vessel located in the surface layer part and the deep part of the living body. In addition, by inputting the obtained signals to specific channels (G2 → R, B2 → G, B), lesions that are difficult to see with normal light such as squamous cell carcinoma can be displayed in brown, etc. Oversight of parts can be suppressed. Note that 390 nm to 445 nm or 530 nm to 550 nm are numbers obtained from the characteristic of being absorbed by hemoglobin and the characteristic of reaching the surface layer or the deep part of the living body, respectively. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% due to the variation factors such as the absorption by hemoglobin and the experimental results regarding the arrival of the living body on the surface layer or the deep part. It is also conceivable that the upper limit value increases by about 0 to 10%.

7). Software In the embodiment described above, for example, each unit configuring the image processing apparatus may be configured by hardware, or the processor may perform a process of each unit for a captured image. For example, the image processing apparatus may be realized as software by a processor executing a program. Alternatively, part of the processing performed by each unit of the image processing apparatus may be configured by software. When implemented as software, the program stored in the information storage medium is read, and the processor executes the read program.

  Here, the processor may be a general-purpose processor such as a CPU, MPU, DSP, or GPU, or may be a dedicated processor such as an ASIC. That is, the processor may be a general-purpose processor that reads and executes a program describing processing from the outside, or may be a dedicated processor in which a circuit dedicated to the processing is incorporated in advance. Further, the processor may be configured by one chip or may be configured by combining a plurality of chips.

  An information storage medium (a computer-readable medium) stores programs, data, and the like. The information storage medium is provided inside and outside the computer system, including a CD-ROM and USB memory, as well as a “portable physical medium” including an MO disk, DVD disk, flexible disk (FD), magneto-optical disk, IC card, etc. Programs such as “fixed physical media” such as HDD, RAM, and ROM, public lines connected via modems, local area networks or wide area networks to which other computer systems or servers are connected It includes any recording medium that records a program that can be read by a computer system, such as a “communication medium” that stores a program in a short time during transmission.

  That is, the program is recorded on the above recording medium so as to be readable by a computer, and the computer system (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) reads the program from such a recording medium. To implement the image processing apparatus. Note that the program is not limited to be executed by a computer system, and when another computer system or server executes the program, or when they cooperate to execute the program, The present invention can be similarly applied.

  As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements from all the constituent elements described in the embodiments and modifications such as the image processing apparatus, the endoscope apparatus, and the image processing method (the operation method of the image processing apparatus, the operation method of the endoscope apparatus) May be deleted. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention. In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings.

100 light source unit, 110 white light source, 120 lens, 200 imaging unit,
210 light guide fiber, 220 illumination lens, 230 condenser lens,
240 image sensor, 250 memory, 280 filter drive unit,
290 narrowband filter, 300 controller, 310 interpolation processing unit,
315 image acquisition unit, 320 noise reduction processing unit, 321 similarity detection unit,
322 shooting condition acquisition unit, 323 first noise reduction processing unit,
324 second noise reduction processing unit, 325 image synthesis unit,
330 frame memory, 340 display image generation unit, 390 control unit,
400 display unit, 500 external I / F unit, 601 motion vector detection unit,
602 motion vector alignment unit, 603 motion vector averaging unit,
604 brightness change amount detection unit, 605 type information acquisition unit,
611 blend rate determination unit, 612 blend processing unit,
621 Threshold value determination unit, 622 selection unit

Claims (18)

An image acquisition unit for acquiring time-series captured images by the imaging unit;
A first noise reduction processing unit that performs a noise reduction process in a time direction on the captured image to generate a first image;
A second noise reduction processing unit that performs a noise reduction process in at least a spatial direction on the captured image to generate a second image;
A similarity detection unit for detecting a similarity between frames of the captured image;
An image synthesizing unit for synthesizing the first image and the second image so that the higher the similarity is, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image;
A shooting condition acquisition unit for acquiring shooting conditions of the captured image;
Including
The photographing condition acquisition unit
Obtaining movement information that changes due to relative movement between the imaging unit and the subject as the imaging condition;
The image composition unit
The image processing apparatus characterized by lowering the degree of synthesis of the first image and increasing the degree of synthesis of the second image as the relative motion represented by the motion information is larger . In claim 1 ,
The similarity detection unit
Detecting the similarity in a first region of the captured image;
The photographing condition acquisition unit
The image processing apparatus, wherein the motion information is acquired in a second area wider than the first area of the captured image. In claim 1 ,
The photographing condition acquisition unit
Obtaining a relative amount of movement between the imaging unit and the subject as the movement information;
The image composition unit
The image processing apparatus characterized by lowering the degree of synthesis of the first image and increasing the degree of synthesis of the second image as the amount of motion increases. In claim 3 ,
The photographing condition acquisition unit
An image processing apparatus, wherein a magnitude of a motion vector between frames of the subject on the captured image is acquired as the amount of motion. In claim 4 ,
The photographing condition acquisition unit
Motion vectors are detected for a plurality of regions of the captured image, the motion vectors for the plurality of regions are arranged in the order of the magnitudes of the motion vectors, and an average value of the motion vectors in a predetermined ratio from the smallest size Is acquired as the amount of motion. In claim 1 ,
The photographing condition acquisition unit
The amount of change in brightness between frames of the captured image is acquired as the motion information,
The image composition unit
The image processing apparatus characterized by lowering the degree of synthesis of the first image and increasing the degree of synthesis of the second image as the amount of change in brightness increases. In claim 1 ,
The image composition unit
The first blend ratio (α) for the first image and the second blend ratio (1-α) for the second image are determined, and the higher the similarity, the higher the first blend ratio (α) is. A rate determining unit;
A blend processing unit that performs a process of blending the first image and the second image at the first blend ratio (α) and the second blend ratio (1-α) as the synthesis process;
Have
The blend rate determination unit
The image processing apparatus according to claim 1, wherein the first blend ratio (α) is lowered as the relative movement represented by the movement information increases. In claim 1 ,
The image composition unit
A threshold value determination unit for determining a threshold value;
A selection unit that selects the first image when the similarity is greater than the threshold, and selects the second image when the similarity is less than the threshold;
Have
The threshold determination unit
The image processing apparatus, wherein the threshold value is set lower as the relative movement represented by the movement information is larger. In claim 1,
The photographing condition acquisition unit
Acquiring the type of the captured image as the imaging condition;
The image composition unit
The image processing apparatus, wherein the degree of synthesis is changed between the case where the type of the captured image is a first type and the case where the type of the captured image is a second type. In claim 9 ,
The first type is a white light image having information in a white wavelength band,
The second type is a special light image having information in a specific wavelength band,
The image composition unit
When the type of the captured image is the special light image, the degree of synthesis of the first image is lowered and the second image is compared with the case where the type of the captured image is the white light image. An image processing apparatus characterized by increasing the degree of synthesis of the image. In claim 10 ,
The image composition unit
The first blend ratio (α) for the first image and the second blend ratio (1-α) for the second image are determined, and the higher the similarity, the higher the first blend ratio (α) is. A rate determining unit;
A blend processing unit that performs a process of blending the first image and the second image at the first blend ratio (α) and the second blend ratio (1-α) as the synthesis process;
Have
The blend rate determination unit
When the type of the captured image is the special light image, the first blend ratio is lowered as compared with the case where the type of the captured image is the white light image. apparatus. In claim 10 ,
The image composition unit
A threshold value determination unit for determining a threshold value;
A selection unit that selects the first image when the similarity is greater than the threshold, and selects the second image when the similarity is less than the threshold;
Have
The threshold determination unit
The image processing apparatus, wherein when the type of the captured image is the special light image, the threshold value is set lower than when the type of the captured image is the white light image. In claim 10 ,
The specific wavelength band is
An image processing apparatus having a band narrower than the white wavelength band. In claim 1,
The similarity is a value obtained by adding an absolute value of a difference value between a pixel value of a current frame of the captured image and a pixel value of a frame before the current frame in a predetermined area,
The image processing apparatus according to claim 1, wherein the similarity is higher as the added value is smaller.   An endoscope apparatus comprising the image processing apparatus according to claim 1. An image acquisition unit for acquiring time-series captured images by the imaging unit;
A first noise reduction processing unit that performs a noise reduction process in a time direction on the captured image to generate a first image;
A second noise reduction processing unit that performs a noise reduction process in at least a spatial direction on the captured image to generate a second image;
A similarity detection unit for detecting a similarity between frames of the captured image;
An image synthesizing unit for synthesizing the first image and the second image so that the higher the similarity is, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image;
A shooting condition acquisition unit for acquiring shooting conditions of the captured image;
Including
The image composition unit
An endoscope apparatus, wherein the degree of synthesis in the synthesis process is changed based on the imaging condition. The image processing apparatus acquires a time-series captured image by the imaging unit,
The image processing apparatus performs a first noise reduction process for generating a first image by performing a noise reduction process in a time direction on the captured image;
The image processing apparatus performs a second noise reduction process for generating a second image by performing at least a spatial noise reduction process on the captured image;
The image processing device detects a similarity between frames of the captured image;
The image processing apparatus acquires movement information that changes due to relative movement between the imaging unit and a subject as a shooting condition of the captured image,
The image processing apparatus, the said first image and the second image so that the resultant degree is lower in the second image with the synthetic degree of higher similarity the first image is higher synthesizing process, the A method of operating an image processing apparatus, wherein the degree of synthesis of the first image is lowered and the degree of synthesis of the second image is increased as the relative movement represented by the motion information is larger . Acquire time-series captured images by the imaging unit,
Performing a first noise reduction process for generating a first image by performing a noise reduction process in a time direction on the captured image;
Performing a second noise reduction process for generating a second image by performing a noise reduction process in at least a spatial direction on the captured image;
Detecting the similarity between frames of the captured image;
Obtaining movement information that changes due to relative movement between the imaging unit and the subject as a shooting condition of the captured image;
The higher the similarity, the higher the degree of synthesis of the first image and the lower the degree of synthesis of the second image, so that the first image and the second image are synthesized, and the relative information represented by the motion information The greater the movement, the lower the degree of synthesis of the first image and the degree of synthesis of the second image ,
An image processing program that causes a computer to execute steps.

Images