JP2018195247A - Image processing apparatus and imaging apparatus - Google Patents

Abstract

To provide an image processing apparatus and an imaging apparatus capable of performing noise removal processing in real time by minimizing a required buffer thereby preventing delay of processing.SOLUTION: A control part 40 comprises an image processing part 44 performing various kinds of image processing. The image processing part 44 comprises a recursive variable weighted mean type noise removal filter 45 which functions as a spatial filter for performing noise removal processing.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus for performing noise removal on an image photographed by a camera, and for excitation to irradiate a subject with excitation light for exciting a fluorescent dye administered to the subject. A light source, a camera that obtains a fluorescent image by photographing the fluorescence generated from the fluorescent dye when irradiated with excitation light, and an image for processing the fluorescent image obtained by this camera and displaying it on the display unit And an imaging device including a processing unit.

  A technique called near-infrared fluorescence imaging is used for contrasting blood vessels and lymph vessels in surgery. In this near-infrared fluorescence imaging, indocyanine green (ICG), which is a fluorescent dye, is injected into the subject's body by an injector or the like and administered to the affected area. When indocyanine green is irradiated with near infrared light having a wavelength of about 600 to 850 nm (nanometer) as excitation light, indocyanine green emits near infrared fluorescence having a wavelength of about 750 to 900 nm. This fluorescence is photographed by an imaging device capable of detecting near infrared light, and the image is displayed on a display unit such as a liquid crystal display panel. According to this near-infrared fluorescence imaging, it is possible to observe blood vessels, lymph vessels, and the like existing at a depth of about 20 mm from the body surface.

  In recent years, attention has been focused on a technique in which a tumor is fluorescently labeled and used for surgical navigation. As a fluorescent labeling agent for fluorescently labeling the tumor, 5-aminolevulinic acid (5-ALA / 5-Aminolevulinic Acid) is used. When this 5-aminolevulinic acid (hereinafter abbreviated as “5-ALA”) is administered to a subject, 5-ALA is metabolized to a fluorescent dye, PpIX (protoporphyrinIX / protoporphyrinine). . This PpIX accumulates specifically in cancer cells. When PpIX, which is a metabolite of 5-ALA, is irradiated with visible light having a wavelength of about 410 nm, red visible light having a wavelength of about 630 nm is emitted as fluorescence from PpIX. By photographing and observing the fluorescence from this PpIX with an image sensor, cancer cells can be confirmed.

  In such an imaging apparatus that captures the fluorescence from the fluorescent dye that has entered the body, the weak fluorescence from the fluorescent dye is imaged by the camera, and thus it is necessary to increase the sensitivity of the camera. As described above, when the sensitivity of the camera is high, the noise in the time direction in the fluorescent image becomes large and the image quality of the fluorescent image is deteriorated.

  As a spatial filter for removing such noise, a bilateral filter that is an edge-preserving filter is generally used. In this bilateral filter, when f is an input pixel value, i and j are XY coordinates, m and n are moving amounts, and σ is a standard deviation, an output pixel value g is calculated by the following equation. ing.

  Instead of the bilateral filter, a simple smoothing filter using a convolution having a kernel size of n × m, a Gaussian filter, or a spatial filter such as a median filter may be used. These are all noise removal methods using the pixel of interest and its surroundings as parameters.

  Patent Document 1 discloses an image processing method using a bilateral filter.

JP 2014-27632 A

  There are two types of imaging devices in cameras used in such imaging apparatuses: a global shutter system that captures all pixels simultaneously, and a rolling shutter system that captures images with a time difference for each pixel. . In an image sensor that employs the global shutter system, in addition to a standby time for shooting between frames, image processing is performed thereafter, and therefore, a delay is likely to occur from shooting to pixel value output. On the other hand, in an image sensor that employs a rolling shutter system, pixel values can be acquired for each pixel, and therefore processing can be performed continuously.

  On the other hand, in the noise removal processing using the edge-preserving spatial filter described above, the pixel value of the pixel of interest and the pixel value of its neighboring pixels are required to perform the processing, so the rolling shutter method is adopted. Although continuous processing is possible in the image sensor, it is necessary to provide a buffer for pixel values for several lines. For this reason, in particular, when the resolution and the number of pixels increase, the waiting time required for processing increases, so that the delay is likely to occur. Thus, the situation requiring a buffer is a problem that may occur not only when the rolling shutter method is employed but also when the global shutter method is employed.

  The present invention has been made in order to solve the above-described problems. An image processing apparatus and an imaging device capable of preventing processing delay and performing noise removal processing in real time by minimizing a necessary buffer. An object is to provide an apparatus.

  The invention described in claim 1 is characterized in that, in an image processing apparatus for performing noise removal on an image photographed by a camera, a recursive filter is used as a spatial filter for noise removal.

  The invention according to claim 2 is the invention according to claim 1, wherein the spatial filter is a recursive variable weighted average type filter.

  According to a third aspect of the present invention, in the second aspect of the present invention, the camera includes a rolling shutter type image sensor.

  The invention according to claim 4 is the following power according to claim 3, wherein I is an input pixel value and W is a weight function, and the output pixel value F is a power exponent p. Calculate with the average formula.

The invention of claim 5 is the invention according to claim 3, the number of pixels is N, the pixel values of the image photographed by the camera and I _N, the i-th pixel value and F _i, I An image processing apparatus that calculates a pixel value F _N of an output frame by the following equation, where W is a variable weight function calculated from _N and F _i and m is the number of pixels to be used.

The invention of claim 6 is the invention according to claim 3, the number of pixels is N, the pixel values of the image photographed by the camera and I _N, the i-th pixel value and F _i, I _{When the} variable weight function calculated from _N and F _i is W and the number of pixels to be used is m, the pixel value F _N of the output frame is calculated by the following equation.

  In the invention according to claim 7, the excitation light for irradiating the subject with excitation light for exciting the fluorescent dye administered to the subject, and the excitation light irradiates the excitation light. In an imaging apparatus comprising a camera that acquires fluorescence images by photographing fluorescence generated from a fluorescent dye, and an image processing unit that performs image processing on the fluorescence image acquired by the camera and displays the image on a display unit. The image processing unit uses a recursive filter as a spatial filter for noise removal for removing noise from a fluorescent image photographed by the camera.

  According to the first to seventh aspects of the present invention, it is possible to prevent processing delay by minimizing the buffer required for noise removal processing, and to perform noise removal processing in real time. It becomes possible.

  According to the second aspect of the present invention, it is possible to satisfactorily perform edge preservation for an image by using a recursive variable weight type filter.

  According to the third aspect of the present invention, it is possible to efficiently perform noise removal processing on an image captured using a camera including a rolling shutter type imaging device.

  According to the seventh aspect of the present invention, even in the case where the camera sensitivity is increased in order to capture weak fluorescence from the fluorescent dye with the camera, noise in the fluorescence image is effectively removed, It is possible to prevent the image quality from deteriorating.

1 is a perspective view showing an imaging apparatus 1 according to the present invention together with a display apparatus 2. FIG. 2 is a schematic diagram of an illumination / photographing unit 12. FIG. 2 is a schematic diagram of a camera 21 in an illumination / photographing unit 12. FIG. 1 is a block diagram showing a main control system of an imaging apparatus 1 according to the present invention together with a display apparatus 2. FIG. It is a figure which shows the pixel value of a 3x3 original image. It is a figure which shows a 3 * 3 kernel.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a perspective view showing an imaging apparatus 1 according to the present invention together with a display apparatus 2.

  The display device 2 has a configuration in which a display unit 52 configured by a large liquid crystal display device or the like is supported by a support mechanism 51.

  The imaging apparatus 1 irradiates indocyanine green as a fluorescent dye injected into the body of the subject with excitation light, images fluorescence emitted from the indocyanine green, and captures the fluorescence image of the subject. This is for displaying on the display device 2 together with a color image which is a visible image. The imaging apparatus 1 particularly measures the fluorescence time in the region of interest of the subject by measuring the intensity of the fluorescence in the region of interest of the subject over time together with the display of the fluorescent image and the color image described above. This is for obtaining an intensity curve (Time Intensity Curve: TIC).

  The imaging apparatus 1 includes a carriage 11 having four wheels 13, an arm mechanism 30 disposed near the front of the carriage 11 in the traveling direction on the upper surface of the carriage 11, and a sub arm 41 provided on the arm mechanism 30. And an illumination / photographing unit 12 and a monitor 15. A handle 14 used when moving the carriage 11 is attached to the rear of the carriage 11 in the traveling direction. Further, on the upper surface of the carriage 11, a recess 16 for mounting an operation unit (not shown) used for remote operation of the imaging apparatus 1 is formed.

  The arm mechanism 30 described above is disposed on the front side in the traveling direction of the carriage 11. The arm mechanism 30 includes a first arm member 31 connected to a support portion 37 disposed on a support column 36 erected on the front side in the traveling direction of the carriage 11 by a hinge portion 33. The first arm member 31 can swing with respect to the carriage 11 through the support column 36 and the support portion 37 by the action of the hinge portion 33. The monitor 15 described above is attached to the support column 36.

  A second arm member 32 is connected to the upper end of the first arm member 31 by a hinge portion 34. The second arm member 32 can swing with respect to the first arm member 31 by the action of the hinge portion 34. For this reason, the first arm member 31 and the second arm member 32 are a connecting portion between the first arm member 31 and the second arm member 32 as shown in FIG. It is possible to take a photographing posture opened by a predetermined angle around a certain hinge portion 34 and a standby posture in which the first arm member 31 and the second arm member 32 are close to each other.

  A support portion 43 is connected to the lower end of the second arm member 32 by a hinge portion 35. The support portion 43 can swing with respect to the second arm member 32 by the action of the hinge portion 35. A rotating shaft 42 is supported on the support portion 43. Then, the sub arm 41 that supports the illumination / photographing unit 12 rotates around the rotation shaft 42 disposed at the tip of the second arm member 32. For this reason, the illumination / photographing unit 12 rotates the sub arm 41 so that the position of the carriage 11 in the traveling direction relative to the arm mechanism 30 for taking the photographing posture or the standby posture shown in FIG. It moves between the position of the rear side in the traveling direction of the carriage 11 with respect to the arm mechanism 30 which is the posture when moving the carriage 11.

  FIG. 2 is a schematic diagram of the illumination / photographing unit 12.

  The illumination / photographing unit 12 includes a camera 21 having a plurality of imaging elements capable of detecting near-infrared light and visible light, which will be described later, and a visible light source 22 including six LEDs disposed on the outer peripheral portion of the camera 21. And an excitation light source 23 composed of six LEDs and a confirmation light source 24 composed of one LED. The visible light source 22 emits visible light. The excitation light source 23 irradiates near infrared light having a wavelength of 760 nm, which is excitation light for exciting indocyanine green. The confirmation light source 24 emits near-infrared light having a wavelength of 810 nm, which approximates the wavelength of fluorescence generated from indocyanine green. The wavelength of the excitation light source 23 is not limited to 760 nm as long as it can excite indocyanine green. The wavelength of the light source 24 for confirmation is not limited to 810 nm, and may be longer than the wavelength emitted by indocyanine green.

  FIG. 3 is a schematic diagram of the camera 21 in the illumination / photographing unit 12.

  The camera 21 includes a movable lens 54 that reciprocates for focusing, a wavelength selection filter 53, a visible light image sensor 55, and a fluorescence image sensor 56. The visible light imaging element 55 and the fluorescence imaging element 56 are configured by a rolling shutter type imaging element such as a CMOS. The visible light imaging element 55 is capable of capturing a visible light image as a color image.

  Visible light and fluorescence incident on the camera 21 coaxially along the optical axis L pass through the movable lens 54 constituting the focusing mechanism, and then reach the wavelength selection filter 53. Of visible light and fluorescence incident coaxially, visible light is reflected by the wavelength selection filter 53 and enters the visible light imaging element 55. Of visible light and fluorescence incident coaxially, the fluorescence passes through the wavelength selection filter 53 and enters the fluorescence imaging device 56. At this time, the visible light is focused on the visible light imaging element 55 and the fluorescence is focused on the fluorescent imaging element 56 by the action of the focusing mechanism including the movable lens 54. The visible light imaging element 55 captures a visible image as a color image at a predetermined frame rate. In addition, the fluorescence imaging element 56 captures a fluorescence image that is a near-infrared image at a predetermined frame rate.

  FIG. 4 is a block diagram showing the main control system of the imaging apparatus 1 according to the present invention together with the display apparatus 2.

  The imaging apparatus 1 is composed of a CPU as a processor that executes logical operations, a ROM that stores an operation program necessary for controlling the apparatus, a RAM that temporarily stores data during control, and the like. The control part 40 to control is provided. The control unit 40 is connected to the display device 2 described above. The control unit 40 is connected to an illumination / photographing unit 12 that includes a camera 21, a visible light source 22, an excitation light source 23, and a confirmation light source 24.

  The control unit 40 includes an image processing unit 44 that executes various types of image processing. The image processing unit 44 includes a recursive variable weighted average noise removal filter 45 that functions as a spatial filter for performing a noise removal process described later.

  When performing an operation on a subject using the imaging apparatus 1 having the above-described configuration, first, the confirmation light source 24 in the illumination / imaging unit 12 is turned on, and an image of the irradiation region is displayed on the camera. 21. The confirmation light source 24 emits near-infrared light having a wavelength of 810 nm that approximates the wavelength of fluorescence generated from indocyanine green. This near infrared light cannot be confirmed by human eyes. On the other hand, when near-infrared light having a wavelength of 810 nm is emitted from the light source for confirmation 24 and an image of this irradiation region is taken by the camera 21, when the camera 21 is operating normally, near-infrared light is emitted. An image of a region irradiated with is taken by the camera 21, and the image is displayed on the display unit 52 in the display device 2. This makes it possible to easily check the operation of the camera 21.

  Thereafter, indocyanine green is injected into the subject by injection. Then, near infrared rays are emitted from the excitation light source 23 in the illumination / imaging unit 12 and visible light is emitted from the visible light source 22 toward the affected part in the organ of the subject. As the near infrared light emitted from the excitation light source 23, as described above, 760 nm near infrared light acting as excitation light for indocyanine green to emit fluorescence is employed. Thereby, indocyanine green injected into the body of the subject generates near-infrared fluorescence having a peak at about 800 nm.

  Then, the vicinity of the affected part in the organ of the subject is imaged at a predetermined frame rate by the camera 21 in the illumination / imaging unit 12. As described above, the camera 21 can detect near-infrared light and visible light. The near-infrared image and color image captured by the camera 21 at a predetermined frame rate are converted into image data that can be displayed on the display unit 52 in the display device 2 by the image processing unit 44. Are displayed on the display unit 52. Further, as necessary, the image processing unit 44 uses the near-infrared image data and the color image data to create a composite image obtained by fusing the color image and the near-infrared image.

  When imaging fluorescence from indocyanine green injected into the body of the subject in such an imaging apparatus 1 with the imaging device 56 for fluorescence in the camera 21, it is necessary to capture weak fluorescence from indocyanine green. Therefore, it is necessary to increase the sensitivity of the fluorescence imaging device 56. As described above, when the sensitivity of the fluorescence imaging element 56 is increased, the noise in the time direction in the fluorescence image becomes large, and the image quality of the fluorescence image is deteriorated. Therefore, in the imaging apparatus 1 according to the present invention, the image processing unit 44 employs a configuration in which noise removal is performed by the edge-preserving type recursive variable weighted average noise removal filter 45 that functions as a spatial filter. Yes.

  In general, a recursive edge-preserving filter such as a recursible filter is used as a temporal filter. In other words, such a recursive edge preserving filter employs a configuration that removes noise by performing operations between a plurality of consecutive frames. The present invention is characterized in that this recursive filter is used as a spatial filter. Further, as described above, the visible light imaging device 55 and the fluorescence imaging device 56 are configured by a rolling shutter type imaging device such as a CMOS that performs photographing with a time difference for each pixel.

  As the edge-preserving recursive variable weighted average noise removal filter 45 according to the present invention, a recursive variable weighted average filter that calculates pixel values by power averaging can be used. In the case of using this recursive variable weighted average filter, when I is an input pixel value and W is a weight function, the output pixel value F is expressed by the following power average equation with an exponent of power p: Calculated.

  Here, in the above formula, the following formula is established.

    The power mean is a general formula of an average using an exponent p which is also called a generalized mean.

As the edge-preserving recursive variable weighted average noise removal filter 45 according to the present invention, for example, a recursive variable weighted geometric average filter that calculates pixel values by the geometric average with the index p = 0 described above may be used. it can. When this recursive variable weighted geometric average filter is used, the pixel number is N, the pixel value of the image taken by the camera 21 is I _N , the i-th pixel value is F _i, and I _N and F _{When the} variable weight function calculated from _i is W and the number of pixels to be used is m, the pixel value F _N of the output frame is calculated by the following equation.

Further, as the edge-preserving recursive variable weighted average noise removal filter 45 according to the present invention, for example, a recursive variable weighted arithmetic average filter that calculates a pixel value by the arithmetic average with the index p = 1 is used. be able to. When this recursive variable weighted arithmetic average filter is used, the pixel number is N, the pixel value of the image captured by the camera 21 is I _N , the i-th pixel value is F _i, and I _N and F _{When the} variable weight function calculated from _i is W and the number of pixels to be used is m, the pixel value F _N of the output frame is calculated by the following equation.

  A difference between a general spatial filter and a recursive spatial filter will be described. In the following description, a case where processing is performed on a 3 × 3 area will be described as an example. FIG. 5 is a diagram illustrating pixel values of a 3 × 3 original image, and FIG. 6 is a diagram illustrating a 3 × 3 kernel applied to the original image.

  In a general spatial filter, the pixel value f ′ (i, j) after processing of the pixel of interest f (i, j) in FIG. 5 is the kernel shown in FIG. 6 with respect to the original image shown in FIG. It is calculated by scanning from the upper left to the lower right so as to move from the left end of the first row to the right end and move to the second row, multiply the values of the same position for the original image and the kernel, and then add them. This can be expressed by the following equation.

  When performing this calculation, in the recursive spatial filter according to the present invention, in the area where the calculation has already been performed, not the pixel value of the original image f (i, j) but the pixel value f ′ after the calculation. Use (i, j). That is, when f (i, j) is the target pixel in FIG. 4, f (j-1, i-1) and f (i, j-1) are obtained in the area where the calculation is performed earlier. , F (i + 1, j−1), f (i−1, j), f ′ (j−1, i−1), f ′ (i, j−1), f ′ (i + 1, j) −1), f ′ (i−1, j) are used.

  Then, by matching the scanning direction of the kernel with respect to the original image with the capturing direction of the pixel value in the fluorescence image pickup device 56 of the rolling shutter system that takes a time difference for each pixel, a buffer that has been conventionally required is reduced. It is possible to omit noise and efficiently perform noise removal processing.

  As described above, in the imaging apparatus 1 according to the present invention, a conventional general edge-preserving noise removal filter is used as a noise removal filter by using a recursive variable weighted average filter as a spatial filter. Compared to the case, the buffer required for the noise removal processing is minimized, so that the processing delay can be prevented and the noise removal processing can be executed in real time. Furthermore, since it is a variable weighted average, the occurrence of afterimages can be suppressed. And even when the sensitivity of the camera 21 is high in order to capture weak fluorescence from the fluorescent dye with the camera 21, noise in the fluorescent image is effectively removed, and the image quality of the fluorescent image is deteriorated. Can be prevented.

  In the above-described embodiment, a rolling shutter type image sensor such as a CMOS is used as the visible light image sensor 55 and the fluorescence image sensor 56, but a global shutter type image sensor such as a CCD is used. May be.

  Further, in the above-described embodiment, the present invention irradiates excitation light to indocyanine green as a fluorescent dye injected into the body of the subject, and images fluorescence emitted from the indocyanine green. In the above description, the fluorescence image is applied to the imaging apparatus 1 that displays the color image that is a visible image of the subject on the display apparatus 2. However, the present invention may be applied to other image processing apparatuses. .

DESCRIPTION OF SYMBOLS 1 Imaging apparatus 2 Display apparatus 12 Illumination / imaging | photography part 21 Camera 22 Visible light source 23 Excitation light source 24 Light source for confirmation 30 Arm mechanism 40 Control part 44 Image processing part 45 Recursive variable weighted average type noise removal filter 55 Imaging element 56 for visible light Fluorescent image sensor

Claims (7)

In an image processing apparatus for performing noise removal on an image taken by a camera,
An image processing apparatus using a recursive filter as a spatial filter for noise removal. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the spatial filter is a recursive variable weighted average filter. The image processing apparatus according to claim 2,
The camera is an image processing apparatus including a rolling shutter type imaging device. The image processing apparatus according to claim 3.
An image processing apparatus that calculates an output pixel value F according to the following power average expression, where I is an input pixel value and W is a weight function, and an exponent of power is p.

The image processing apparatus according to claim 3.
The pixel number is N, the pixel value of the image captured by the camera is I _N , the i-th pixel value is F _i, and the variable weight function calculated from I _N and F _i is W and used. image processing apparatus for calculating the number of pixels is taken as m, the pixel values F _N of the output frame by the following equation.

The image processing apparatus according to claim 3.
The pixel number is N, the pixel value of the image captured by the camera is I _N , the i-th pixel value is F _i, and the variable weight function calculated from I _N and F _i is W and used. image processing apparatus for calculating the number of pixels is taken as m, the pixel values F _N of the output frame by the following equation.

An excitation light source for irradiating the subject with excitation light for exciting the fluorescent dye administered to the subject;
A camera that obtains a fluorescence image by photographing fluorescence generated from the fluorescent dye by being irradiated with excitation light; and
An image processing unit for image processing and displaying the fluorescence image acquired by the camera on the display unit;
In an imaging apparatus comprising:
The image processing apparatus, wherein the image processing unit uses a recursive filter as a noise removing spatial filter for removing noise from a fluorescent image photographed by the camera.

Images