JP2012205619A - Image processor, control device, endoscope apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor, a control device, an endoscope apparatus, an image processing method, an image processing program, or the like which are capable of visually uniforming noise components regardless of positions in an image.SOLUTION: The image processor includes: an image acquisition section for acquiring a distortion image in which distortion is generated by optical aberration of an optical system; a correction processing section for subjecting a region to be processed in the distortion image to a transformation processing for correcting the distortion to correct the distortion image; a transformation mode information acquisition section for acquiring transformation mode information representing a transformation mode of the region to be processed by the transformation processing; and a noise reduction processing section 320 for reducing the noise components included in the distortion image based on the acquired transformation mode information.

Description

  The present invention relates to an image processing device, a control device, an endoscope device, an image processing method, an image processing program, and the like.

  Conventionally, there has been a demand for improving the accuracy of detecting lesions in a body cavity in endoscopic diagnosis, and endoscope scopes having a wide-angle optical system of 170 degrees have already been developed.

  However, in the case of an organ having a large fold such as the large intestine, it is difficult to observe the back of the fold even if an endoscope scope having such a wide-angle optical system is used. That is, it is necessary to observe with the endoscope scope tip bent greatly, or with the tip physically pressed against the fold so that the back side can be seen. In the former method, there is a problem that it takes a lot of time to examine all the folds. In the latter method, since the scope tip is pressed against the fold, there is a risk of bleeding from the living body. Bleeding from living bodies should be avoided as much as possible.

  As a technique for observing the back of the fold in a non-contact manner in a short time, for example, there is a technique using an endoscope scope having a fish-eye lens capable of obtaining an angle of view of 180 degrees or more. However, the fisheye lens has a large optical aberration (particularly distortion), and the peripheral region is greatly distorted. Therefore, there is a problem that the lesion behind the folds cannot be observed with a sufficiently large size.

  For this problem, Patent Document 1 discloses a technique for correcting distortion so as to be optimal for observation in a lumen unique to an endoscope by combining not only an optical system but also image processing. In this method, the object magnification of an equiangular object is set larger in the image peripheral region than in the image central region.

JP 2009-276371 A

  In such a technique combining a wide-angle optical system and distortion correction, the image in the central area is reduced and the image in the peripheral area is enlarged. For this reason, it is possible to display a lesion area behind the folds in a non-contact manner without bending the tip.

  However, in this method, distortion correction is performed without considering a noise component mixed in an image (for example, a noise component generated in an image sensor or the like). Therefore, there is a problem of inducing a noise component with non-uniform frequency characteristics depending on the position in the image. For example, in the distortion correction, since the peripheral area of the image is enlarged, the noise component in the peripheral area is shifted to the low frequency side. In this case, it seems that the noise component in the peripheral area has increased in human vision.

  According to some aspects of the present invention, an image processing device, a control device, an endoscope device, an image processing method, and an image processing program capable of visualizing noise components regardless of the position in the image Etc. can be provided.

  One embodiment of the present invention provides an image acquisition unit that acquires a distortion image in which distortion has occurred due to optical aberrations of an optical system, and a conversion process for correcting the distortion with respect to a processing target region in the distortion image. Performing a correction processing unit that corrects the distorted image, a conversion mode information acquisition unit that acquires conversion mode information representing a conversion mode of the processing target region by the conversion process, and the acquired conversion mode information, The present invention relates to an image processing apparatus including a noise reduction processing unit that performs processing for reducing a noise component included in the distorted image.

  According to one aspect of the present invention, a conversion process for correcting distortion is performed on a processing target area, and a process of reducing noise components included in a distortion image based on conversion mode information of the processing target area Is done. This makes it possible to visually equalize the noise component regardless of the position in the image.

  In another aspect of the present invention, an optical system that captures an image of an object, an image acquisition unit that acquires a distortion image in which distortion is caused by the optical aberration of the optical system, and a conversion process for correcting the distortion, A correction processing unit that corrects the distorted image by performing processing on the processing target region in the distorted image, and a conversion mode information acquiring unit that acquires conversion mode information representing a conversion mode of the processing target region by the conversion processing; Further, the present invention relates to an endoscope apparatus including a noise reduction processing unit that performs a process of reducing a noise component included in the distorted image based on the acquired conversion mode information.

  According to another aspect of the present invention, a distortion image in which distortion has occurred due to optical aberrations of an optical system is acquired, and a conversion process for correcting the distortion is performed on a processing target area in the distortion image. , Correcting the distorted image, acquiring conversion mode information representing a conversion mode of the processing target region by the conversion process, and reducing noise components included in the distorted image based on the acquired conversion mode information The present invention relates to an image processing method for performing processing.

  According to another aspect of the present invention, an image acquisition unit that acquires a distortion image in which distortion has occurred due to an optical aberration of an optical system, and a conversion process for correcting the distortion include a processing target region in the distortion image. A correction processing unit that corrects the distorted image, a conversion mode information acquisition unit that acquires conversion mode information representing a conversion mode of the processing target area by the conversion process, and the acquired conversion mode information. The present invention relates to an image processing program that causes a computer to function as a noise reduction processing unit that performs processing to reduce a noise component included in the distorted image.

1 is a first configuration example of an endoscope apparatus. The detailed structural example of a rotation color filter. An example of spectral characteristics of a color filter. FIG. 4A to FIG. 4C are explanatory diagrams of display forms of captured images. 3 is a detailed configuration example of an image processing unit. FIG. 6A to FIG. 6C are explanatory diagrams of spatial frequency characteristics associated with distortion correction. Explanatory drawing about the change of the noise component by distortion correction. The 1st detailed structural example of a noise reduction process part. Explanatory drawing about the division | segmentation of a frequency band. FIG. 10A shows a characteristic example of the noise estimation amount. FIG. 10B shows an example of noise reduction characteristics. Explanatory drawing about the shift by distortion correction of the divided | segmented frequency band. 3 is a detailed configuration example of a processing parameter control unit. The 2nd structural example of an endoscope apparatus. The 2nd detailed structural example of a noise reduction process part. The characteristic example of the synthetic | combination ratio of the smoothing pixel value in a frame, and the smoothing pixel value between frames. FIG. 16A and FIG. 16B are explanatory diagrams of smoothing processing for signals of the same color. Explanatory drawing about the smoothing process in a flame | frame, and the smoothing process between frames. The system block diagram which shows the structure of a computer system. The block diagram which shows the structure of the main-body part in a computer system. Example of human visual characteristics. The example in the case of setting the annular area | region centering on an optical axis as a local area | region.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline of this Embodiment First, an outline of this embodiment will be described. Consider the case of observing a lesion 10 (a region of interest in a broad sense) behind the fold of the large intestine as shown in FIG. In this embodiment, by using the fish-eye objective lens 203, it is possible to observe the lesioned part 10 on the back of the fold without bending the scope tip or contacting the fold.

  The lesioned part 10 behind the fold is reflected in the peripheral area of the image. As shown in FIG. 4B, in the fisheye lens, the peripheral area d of the image is distorted more than the central area h, so that the lesioned part 10 appears small. Therefore, in the present embodiment, as shown in FIG. 4C, image processing for enlarging the peripheral region d is performed, and the visibility of the lesioned part 10 is improved.

  However, when such distortion correction is performed, there is a problem that noise components having non-uniform characteristics are generated in the image. This point will be described. Here, it is assumed that the noise component is random white noise as indicated by A1 in FIG. Further, it is assumed that the number of pixels of the image sensor is equal to or less than the number of pixels of the display image.

  First, the central portion of the image is reduced when the captured image is converted into a display image. Therefore, when viewed in terms of frequency, the conversion processing has the characteristics of an interpolation filter (generally the frequency characteristics are flat), and fine noise is generated in the converted image as indicated by A2 in FIG. On the other hand, the peripheral portion of the image is enlarged when the conversion process is performed. Since the frequency band is shifted to the low frequency side as a whole with the enlargement process, the noise component is also shifted to the low frequency as indicated by A3. Since the human visual characteristic is more sensitive to a low frequency than a high frequency in the spatial frequency, it is recognized that the noise component has increased in the peripheral portion of the image after conversion. That is, a large amount of noise is generated around the enlarged image, and an image with little noise is displayed at the center of the image.

  As described above, there is a problem that even if the peripheral portion of the image to be noticed is enlarged and displayed, the visibility is lowered due to an increase in noise components. For example, when observing under a condition where the amount of light using narrow band light or the like is not sufficient, an image photoelectrically converted by the image sensor is amplified in order to obtain proper exposure. For this reason, an increase in noise components appears remarkably, and visual non-uniformity also becomes significant.

  In this regard, according to the present embodiment, as shown in FIG. 10B, processing for reducing noise in a higher frequency band is performed in a region where the magnification rate of distortion correction is larger. This process is performed before distortion correction. As a result, as shown in FIG. 11, among the noise components that shift to low frequencies due to enlargement, the components in the visually sensitive band (near ft) are reduced, and the noise in the image after distortion correction is visually uniform. Can be. Further, since a part of the frequency band of the image is smoothed, the frequency information about the structure of the subject can be stored.

2. First Embodiment 2.1. Endoscope Device FIG. 1 shows a configuration example of an endoscope device according to this embodiment. The endoscope device (an imaging device in a broad sense) includes a light source unit 100, an imaging unit 200, a control device 300 (processor unit), a display unit 400, and an external I / F unit 500. Note that the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of the components or adding other components are possible.

  The light source unit 100 includes a white light source 101, a rotating color filter 102 having a plurality of spectral transmittances, and a rotation driving unit 103 that drives the rotating color filter 102. The light source unit 100 also includes a condensing lens 104 that condenses light having spectral characteristics from the rotating color filter 102 on the incident end face of the light guide fiber 201.

  For example, as shown in FIG. 2, the rotation color filter 102 includes three primary color red color filters 851, a green color filter 852, a blue color filter 853, and a rotation motor 803. FIG. 3 shows the spectral characteristics of these three color filters.

  The rotation driving unit 103 rotates the rotating color filter 102 at a predetermined number of rotations in synchronization with the imaging period of the imaging element 209 based on a control signal from the control unit 302 of the control device 300. For example, if the rotating color filter 102 is rotated 20 times per second, each color filter crosses the incident white light at 1/60 second intervals. The image sensor 209 captures reflected light from the observation target for each of the three primary colors (R, G, or B) at 1/60 second intervals, and completes the transfer of the image. That is, in the present embodiment, the R image, the G image, and the B image are captured in a frame sequential manner at 1/60 second intervals, and the actual frame rate is 20 fps (fps: frame per second).

  The imaging unit 200 is formed to be elongated and bendable so as to be able to be inserted into a body cavity such as the large intestine. The imaging unit 200 includes a light guide fiber 201 for guiding the light collected by the light source unit 100, and illumination lenses 202 and 204 for diffusing the light guided to the tip by the light guide fiber 201 and irradiating the observation target. Including. The imaging unit 200 also includes an objective lens 203 that collects reflected light returning from the observation target, an image sensor 209 that detects the focused imaging light, and an analog signal photoelectrically converted by the image sensor 209 as a digital signal. And an A / D conversion unit 210 for conversion into Here, the image pickup device 209 is a monochrome single-plate image pickup device, and is configured by, for example, a CCD or a CMOS sensor.

  The objective lens 203 is composed of an optical system such as a fisheye lens having a viewing angle of 180 degrees or more, and can observe the lesioned part 10 on the back of the fold of the large intestine, for example. The illumination lenses 202 and 204 are arranged facing outward with respect to the optical axis of the objective lens 203 so that the observation target within the viewing angle of the objective lens 203 can be illuminated uniformly. In the present embodiment, the light guide fiber 201 constituted by a bundle of several thousand fibers is divided into two, but the present invention is not limited to this. For example, the light guide fiber 201 may be divided into more bundles, and an illumination lens corresponding to each bundle may be disposed at the distal end of the endoscope scope. In this way, a wide viewing angle region can be illuminated more uniformly. Further, the light source unit 100 may be configured by arranging a large number of light emitting elements such as LEDs and organic ELs at the distal end of the endoscope scope.

  The control device 300 includes an image processing unit 301 and a control unit 302. The image processing unit 301 performs distortion correction processing, noise reduction processing, and other image processing. The control unit 302 controls each unit of the endoscope apparatus.

  The display unit 400 includes a display device capable of displaying a moving image such as a CRT or a liquid crystal monitor.

  The external I / F unit 500 is an interface for performing input from the user to the endoscope apparatus. For example, the external I / F unit 500 includes a power switch for turning on / off the power, a shutter button for starting a photographing operation, a mode switching switch for switching a photographing mode and other various modes. For example, as shown in FIGS. 4B and 4C, the mode changeover switch manually switches between a plurality of projection conversion modes having different display magnifications in the central portion and the peripheral portion without changing the display area of the entire visual field. It is a switch to do in. The external I / F unit 500 outputs the input information to the control unit 302.

2.2. Distortion Correction Next, a display form of an image captured by the image capturing unit 200 will be described with reference to FIGS. The objective lens 203 of the imaging unit 200 is a fisheye lens, and has a viewing angle of 180 degrees or more.

  FIG. 4A schematically shows a tubular subject. As shown in FIG. 4A, a cylindrical lumen 390 is divided into annular regions d to h by equidistant concentric circles, and an imaging unit 200 is inserted into the lumen 390 and imaging is performed using a fisheye lens. It shall be. FIG. 4B shows a schematic diagram of an image taken with the fisheye lens. FIG. 4B corresponds to an image captured by, for example, an equisolid angle projection method or an orthographic projection method.

  As shown in FIG. 4B, since the subject located on the side surface where the distal end distance between the cylindrical lumen 390 and the imaging unit 200 is closer is distorted, the distance between the concentric circles in the peripheral portion of the image is captured in a clogged state. Is done. If it is inserted into the actual large intestine in such a state, the annular area d in the lateral field of view is pushed into a very thin display area. For this reason, even if a lesion exists in this region, the display size is reduced, and it is difficult to confirm the lesion. In particular, when the back of the fold is observed, the back of the fold can be observed when the positional relationship between the tip of the imaging unit 200 and the back of the fold is the positional relationship in which the back of the fold is reflected in the annular region d. For this reason, it is difficult to observe the target lesion on the back of the large intestine with this image as it is.

  Therefore, as illustrated in FIG. 4C, the image processing unit 301 performs projective conversion (for example, projective conversion for conversion to the polar projection method) displayed in a state where the peripheral portion of the image is enlarged. As a result, it is possible to enlarge and display a region located on the back of the side, and to greatly improve the discovery rate of fold lesions that have not been seen so far.

  Such projective transformation is performed by the image processing unit 301, and can be switched freely according to the observation state. For example, when examining the large intestine, the scope is first inserted up to the cecum, and the presence or absence of the lesion is observed in detail while removing the scope. Since it is important to secure the front visual field at the time of insertion, it is set to the projective transformation 391 that does not perform distortion correction. When observing a lesion while removing it, it is set to the projective transformation 392 because it is desired to pay attention to the peripheral visual field rather than the front visual field. Such a switching operation may be performed, for example, by a user using a mode switch or the like of the external I / F unit 500. Information on the selected projective transformation is input to the control unit 302. The control unit 302 outputs distortion correction information corresponding to the selected projective transformation to a distortion correction processing unit 330 and a noise reduction processing unit 320 described later. The distortion correction information is information indicating correspondence between coordinate information before distortion correction and coordinate information after distortion correction.

  Here, there are two pieces of distortion correction information in the present embodiment. The first distortion correction information is coordinate information indicating where the coordinates of each pixel of the image before distortion correction are located in the image after distortion correction. The second distortion correction information is coordinate information indicating where the coordinates of each pixel of the image after distortion correction are located in the image before distortion correction. The first distortion correction information is used by the noise reduction processing unit 320, and the second distortion correction information is used by the distortion correction processing unit 330.

  The reason why two pieces of distortion correction information are required is that the pixels before distortion correction and the pixels after distortion correction do not correspond one-to-one. For example, the images before and after distortion correction are raster scans. In this case, even if each pixel coordinate of the raster scan before distortion correction is projectively converted, the raster scan is not performed, and even if each pixel coordinate of the raster scan after distortion correction is reversely projected, the raster scan is not performed. Therefore, even if the first distortion correction information is reversed, the second distortion correction information is not obtained, and two pieces of distortion correction information are required.

  Specifically, since the noise reduction processing unit 320 performs noise reduction processing before distortion correction, the first distortion correction information is used to calculate the processing target pixel position before distortion correction and the positions of a plurality of surrounding pixels. The local magnification after distortion correction is calculated. And the strength of noise reduction is controlled using this local enlargement ratio. In addition, the distortion correction processing unit 330 uses the second distortion correction information to obtain the pixel value of the processing target pixel after distortion correction by interpolation. That is, a process of interpolating the pixel value of the processing target pixel from a plurality of pixels close to the pixel position (having precision after the decimal point) before distortion correction corresponding to the position of the processing target pixel (integer accuracy) (for example, known interpolation) Process). For example, in the case of bilinear interpolation, interpolation is performed using four surrounding pixels, and in the case of bicubic interpolation, interpolation is performed using 16 surrounding pixels.

  In the above description, since the noise reduction processing unit 320 is arranged in front of the distortion correction processing unit 330, two pieces of distortion correction information are necessary. However, the present embodiment is not limited to this. For example, when the noise reduction processing unit 320 is arranged after the distortion correction processing unit 330, the distortion correction information may be only the second distortion correction information.

  Two or more types of projective transformation may be prepared as the projective transformation. In this case, it is good also as a structure which a user selects optimal projection conversion according to an observation state with the above-mentioned mode change switch. For example, when examining a lesion part in a flat part, the scope is directly opposed to the lesion part and observed. In this case, a projective transformation that can enlarge and display the central portion of the image as compared with the projective transformation of FIG. 4B may be selected. This projective transformation is different from that shown in FIG. Thereby, the visibility of a lesioned part can be improved at the time of detailed examination.

2.3. Image Processing Unit Next, the image processing unit 301 in FIG. 1 will be described. FIG. 5 shows a detailed configuration example of the image processing unit 301. The image processing unit 301 includes an OB clamp processing unit 310, a noise reduction processing unit 320, a distortion correction processing unit 330, a WB processing unit 340, a three plate processing unit 350, a color correction processing unit 360, a gradation conversion processing unit 370, and an emphasis. A processing unit 380 is included.

  The image output from the A / D conversion unit 210 is input to the OB clamp processing unit 310. The control unit 302 includes an OB clamp processing unit 310, a noise reduction processing unit 320, a distortion correction processing unit 330, a WB processing unit 340, a three plate processing unit 350, a color correction processing unit 360, a gradation conversion processing unit 370, and an enhancement process. Part 380. The control unit 302 outputs processing parameters stored in advance in the control unit 302 to these units.

  For example, the processing parameters include optical black calculation area coordinates, noise reduction gain amount for each observation mode that varies depending on the observation mode, distortion correction information, white balance coefficient, and information on color light (R, G, or B) at the time of image capturing by the image sensor 209. , Color matrix coefficients, gradation conversion table, enhancement parameters, and the like. The observation mode is, for example, a normal light observation mode, a special light observation mode, or the like. The optical black calculation area coordinates are coordinates of an area for calculating an offset value representing a black level.

  The OB clamp processing unit 310 calculates the black level from the image output from the A / D conversion unit 210 based on the processing parameter output from the control unit 302 based on the optical black calculation area coordinates. The OB clamp processing unit 310 performs processing for subtracting the black level from the effective area of the image (hereinafter referred to as an effective area image). The effective area image in which the black level is corrected is output to the noise reduction processing unit 320.

  The noise reduction processing unit 320 performs noise reduction processing on the effective area image based on the noise reduction gain amount for each observation mode and the position-dependent distortion correction information from the control unit 302. The noise reduction processing unit 320 controls the amount of noise reduction based on distortion correction information based on projective transformation specified by the user based on the observation state. Specific noise reduction processing will be described later. The effective area image subjected to the noise reduction process is output to the distortion correction processing unit 330.

  The distortion correction processing unit 330 performs distortion correction processing on the input effective area image after noise reduction processing based on position-dependent distortion correction information from the control unit 302. By this distortion correction processing, an effective area image subjected to projective transformation different from the effective area image at the time of input is generated. Specifically, an effective area image converted from the projective transformation of FIG. 4B to the projective transformation of FIG. 4C is generated.

  The distortion correction processing unit 330 obtains a position in the effective area image before distortion correction corresponding to the pixel position after distortion correction based on the distortion correction information. Then, the distortion correction processing unit 330 obtains the pixel value at the obtained position by interpolation (for example, bicubic interpolation) based on the pixel values of the surrounding pixels at the obtained position, and obtains the obtained pixel value after the distortion correction. The pixel value of the position is used.

  Based on the white balance coefficient input from the control unit 302, the WB processing unit 340 adjusts the white balance for the effective area image after distortion correction. When the color light (R, G, or B) information at the time of image capturing by the image sensor 209 is R or B, the WB processing unit 340 applies the white balance coefficient corresponding to the R or B to the effective area image after distortion correction. Apply. The WB processing unit 340 outputs the processed image to the three-plate processing unit 350.

  The three-plate processing unit 350 includes (R, G, B) frame memories for three channels (not shown). Based on the color light information from the control unit 302, the triple plate processing unit 350 stores the input effective area image of each channel in the corresponding frame memory. The 3 plate processing unit 350 outputs the 3-channel effective area image (hereinafter referred to as a color effective area image) stored in the frame memory to the color correction processing unit 360.

  The color correction processing unit 360 performs color conversion on the input color effective area image based on the color matrix coefficients from the control unit 302, and generates a color effective area image corresponding to a desired color space of the display monitor. The color effective area image after color conversion is output to the gradation conversion processing unit 370.

  The gradation conversion processing unit 370 converts the input color effective area image into a color effective area image that matches the gradation characteristics of the display monitor based on the gradation conversion table from the control unit 302. The converted color effective area image is output to the enhancement processing unit 380.

  The enhancement processing unit 380 performs processing for enhancing the input color effective area image after gradation conversion based on the enhancement parameter from the control unit 302. The enhanced color effective area image is output to the display unit 400.

  Through a series of processes as described above, a captured image captured by the image sensor 209 is displayed on the display unit 400.

2.4. Noise Reduction Processing Next, the noise reduction processing performed by this embodiment will be described in detail. In the present embodiment, a wide field of view is secured, and a display area of a region of interest within the field of view is secured as large as possible. And the process which suppresses the increase in the noise component which becomes apparent by it is performed. Here, the attention area is an area where the priority of observation for the user is relatively higher than other areas.

  In FIG. 4 (A) to FIG. 4 (C), two projective transformations 391 and 392 are shown. The projective transformation 391 represents the projective transformation by the fisheye lens. Since the image distortion at the periphery of the image is strong, it is necessary to correct the distortion in order to observe the back of the fold of the large intestine. The distortion correction is performed by the distortion correction processing unit 330 so as to obtain a projective transformation 392 that reduces the image distortion in the peripheral portion of the image.

  FIG. 6A to FIG. 6C schematically show how the spatial frequency characteristics of the central portion, the peripheral portion, and the intermediate portion during imaging change with the distortion correction.

  In the central portion of the image, reduction processing is performed in the conversion from the projective transformation 391 in FIG. 4B to the projective transformation 392 in FIG. FIG. 6A shows filter characteristics for performing the reduction interpolation. This filter characteristic is a frequency characteristic accompanying distortion correction at the center of the image. On the other hand, as shown in FIG. 6C, since the peripheral portion of the image is subjected to enlargement processing, the frequency characteristic of the peripheral portion of the image after conversion is pushed into the low frequency component. As shown in FIG. 6B, the intermediate portion has an intermediate frequency characteristic between the two frequency characteristics.

  In this way, when the predetermined projective transformation is changed to a different projective transformation, a difference in frequency characteristics due to the distortion correction appears in the image. This difference in frequency characteristics not only affects the resolution of the subject, but also affects the characteristics of noise components. That is, as shown in FIG. 20, the human visual characteristics are generally more sensitive at low frequencies than at high frequencies. Therefore, the noise pushed to the low frequency side in the peripheral portion of the image shown in A3 in FIG. 7 is visually detected more clearly than the noise spread to the high frequency side in the central portion of the image shown in A4. As a result, the conversion to the projective transformation 392 was intended to improve the visibility of the peripheral portion of the image, but as a side effect, noise also increased, and the improvement effect of improving the visibility of the region of interest was sufficient. Cannot be obtained.

  Therefore, in the present embodiment, the noise reduction processing unit 320 performs control to change the noise reduction amount in accordance with the distortion correction information in the image, thereby visually equalizing the noise amount in the image after distortion correction. Let

  In FIG. 7, as shown by A1, the case where the noise component is white noise is taken as an example. As shown in A2, the band of the noise component that spreads to the high frequency side in the center of the image is actually limited to the Nyquist frequency.

  FIG. 8 shows a detailed configuration example of the noise reduction processing unit 320. The noise reduction processing unit 320 includes a low-pass processing unit 701, 706, 711, a downsampling unit 702, 707, a subtraction processing unit 703, 708, 712, a noise amount estimation unit 704, 709, 713, 715, a smoothing processing unit 705, 710, 714, 716, addition processing units 717, 719, 721, upsampling units 718, 720, and a processing parameter control unit 722.

  As shown in FIG. 9, the noise reduction processing unit 320 divides a spatial frequency band (for example, 0 Hz to Nyquist frequency) into four spatial frequency bands FB1 to FB4, and controls the noise reduction amount independently for each frequency band. To do.

  Specifically, the effective region image output from the OB clamp processing unit 310 is input to the low-pass processing unit 701 and the subtraction processing unit 703.

  The low-pass processing unit 701 performs low-pass filter processing for limiting the frequency band to half for the input effective area image. The low-pass processing unit 701 outputs the processed image as a low-frequency image L1 to the subtraction processing unit 703, the downsampling unit 702, and the noise amount estimation unit 704.

  The subtraction processing unit 703 subtracts the low frequency image L1 from the effective area image to generate the frequency band image H1 of the band FB1, and outputs the generated image H1 to the smoothing processing unit 705.

  The down-sampling unit 702 generates a low-resolution image D1 obtained by thinning the low-frequency image L1 into half the horizontal and vertical pixels, and the generated image D1 is sent to the low-pass processing unit 706, the subtraction processing unit 708, and the noise amount estimation unit 709. Output.

  The noise amount estimation unit 704 receives the low-frequency image L1 and the noise reduction amount in the band FB1 output from the processing parameter control unit 722. The noise amount estimation unit 704 calculates a smoothing process parameter based on the low frequency image L1 and the noise reduction amount. Specifically, the noise estimation amount in FB1 is estimated from the gradation value of each pixel in L1. Then, a smoothing process parameter in FB1 is calculated from the noise estimation amount and the noise reduction amount. The smoothing processing parameter is defined as an amount obtained by multiplying the noise estimation amount by the noise reduction amount output from the processing parameter control unit 722. The noise amount estimation unit 704 outputs the calculated smoothing processing parameter to the smoothing processing unit 705. Details of the noise reduction amount will be described later.

  The calculation of the noise estimation amount is uniquely determined by a predetermined noise model and the gradation value of each pixel of the low frequency image L1. Specifically, FIG. 10A shows an example of a noise model. FIG. 10A shows the relationship between the gradation value and the noise estimation amount in the four frequency bands FB1 to FB4. Since this noise model is a characteristic characteristic of the image sensor 209, it can be obtained by measuring in advance.

  The smoothing processing unit 705 receives the smoothing parameter and the frequency band image H1. The smoothing processing unit 705 smoothes the frequency band image H <b> 1 to reduce noise, and outputs the processed image H <b> 1 ′ to the addition processing unit 721.

Specifically, the smoothing processing unit 705 performs a smoothing process using a bilateral filter represented by the following formula (1). F (x, y) is a filter coefficient at coordinates (x, y). Further, σ _s is a smoothing process parameter, and is a pixel value difference that determines a weighting factor based on a difference in pixel values between the target pixel and surrounding pixels. sigma value of _s is more increased smoothing effect increases, the more smoothing effect the value of sigma _s is reduced is reduced.

Here, x is the horizontal coordinate of the frequency band image, and y is the vertical coordinate of the frequency band image. A (x, y) is a normalization coefficient of the filter. P (x ₀ , y ₀ ) is a pixel value of the target pixel (x ₀ , y ₀ ) to be filtered. P (x, y) is a pixel value of the peripheral pixel (x, y) of the target pixel.

  The low-pass processing units 706 and 711, the downsampling unit 707, the subtraction processing units 708 and 712, the noise amount estimation units 709, 713, and 715, and the smoothing processing units 710, 714, and 716 are basically the same processing as described above. To do.

  Specifically, the low pass processing unit 706 generates a low frequency image L2 by limiting the band of the low frequency image L1 to half. The subtraction processing unit 708 subtracts L2 from L1, and generates a frequency band image H2 of the band FB2. The noise amount estimation unit 709 calculates a smoothing processing parameter in FB2 based on L2 and the noise reduction amount in FB2. The smoothing processing unit 710 performs a smoothing process on H2, and generates a noise-reduced frequency band image H2 '. The downsampling unit 707 thins out the low frequency image L2 and generates a low resolution image D2.

  The low-pass processing unit 711 generates a low-frequency image L3 by limiting the band of the low-frequency image L2 to half. L3 is used as the frequency band image H4 of the band FB4. The subtraction processing unit 712 subtracts L3 from L2 to generate a frequency band image H3 of the band FB3. The noise amount estimation unit 713 calculates a smoothing process parameter in FB3 based on L3 and the noise reduction amount in FB3. The smoothing processing unit 714 performs a smoothing process on H3 to generate a frequency band image H3 'with reduced noise.

  The noise amount estimation unit 715 calculates a smoothing process parameter in FB4 based on L3 and the noise reduction amount in FB4. The smoothing processing unit 716 performs a smoothing process on H4 (L3), and generates a noise-reduced frequency band image H4 '.

  The addition processing unit 717 adds the frequency band image H3 ′ of the noise-reduced band FB3 and the frequency band image H4 ′ of the noise-reduced band FB4, and adds the frequency band image H3 ′ + H4 ′ of the added band FB3 + FB4. The data is output to the upsampling unit 718. The upsampling unit 718 performs a process of doubling the number of horizontal and vertical pixels of H3 ′ + H4 ′ and outputs the processed image U (H3 ′ + L3 ′) to the addition processing unit 719. Here, U () represents that the upsampling is performed twice.

  The addition processing unit 719 adds the noise-reduced frequency band image H2 ′ of the band FB2 and the upsampled image U (H3 ′ + L3 ′), and adds the frequency band image H2 ′ + U (H3) of the added band FB2 + FB3 + FB4. '+ L3') is output to the upsampling unit 720. The upsampling unit 720 performs a process of doubling the number of horizontal and vertical pixels of H2 ′ + U (H3 ′ + L3 ′), and adds the processed image U (H2 ′ + U (H3 ′ + L3 ′)). The data is output to the processing unit 721.

  The addition processing unit 721 adds the noise-reduced frequency band image H1 ′ of the band FB1 and the upsampled image U (H2 ′ + U (H3 ′ + L3 ′)), and the added frequency band image of the band FB1 + FB2 + FB3 + FB4 H1 ′ + U (H2 ′ + U (H3 ′ + L3 ′)) is output to the distortion correction processing unit 330. In this way, a full-frequency image of the band FB1 + FB2 + FB3 + FB4 is generated.

2.5. Processing Parameter Control Unit Next, the processing parameter control unit 722 will be described. The processing parameter control unit 722 is connected to the noise amount estimation units 704, 709, 713, and 715. The processing parameter control unit 722 determines the noise reduction amount based on the distortion correction information output from the control unit 302 and the noise reduction gain amount for each observation mode. Specifically, the processing parameter control unit 722 calculates a local enlargement ratio at each pixel position based on the distortion correction information, and calculates a noise reduction amount based on the calculated local enlargement ratio.

  More specifically, the local enlargement ratio ΔM (x, y) is calculated based on distortion correction information of a target pixel (processing target pixel) that is a pixel subjected to noise reduction processing and distortion correction information of peripheral pixels of the target pixel. calculate. The local enlargement ratio ΔM (x, y) represents how much the target pixel and its surrounding area are enlarged. Here, the distortion correction information is information in which coordinates after distortion correction are associated with coordinates of pixels before distortion correction.

The local enlargement ratio ΔM (x, y) is calculated by the following equation (2). Here, it is assumed that the distortion correction information of the target pixel is Dr (x, y) = q (x ′, y ′). The distortion correction information of the peripheral pixels is Dr (x−1, y) = q (x′−Δx _x1 ′, y′−Δy _x1 ′), Dr (x, y−1) = q (x′−Δx _y1 ′). , Y′−Δy _y1 ′). q (x ′, y ′) represents a position vector in the image after the distortion correction processing. x ′ represents the horizontal coordinate of the image after distortion correction processing, and y ′ represents the vertical coordinate of the image after distortion correction processing. Norm (q) represents the distance of the vector q.

  ΔM (x, y) in the above equation (2) represents an area (approximate area) after distortion correction with respect to a unit area before distortion correction. When ΔM (x, y) is larger than 1, it indicates that the image after distortion correction is enlarged than before distortion correction. When ΔM (x, y) is smaller than 1, it indicates that the image after distortion correction is reduced more than before distortion correction. When ΔM (x, y) is equal to 1, it indicates that the areas are equal before and after the distortion correction processing.

  As shown in FIG. 10B, the relationship between the local enlargement ratio and the amount of noise reduction is determined independently for the four frequency bands FB1 to FB4.

  As shown in FIG. 11, when the local enlargement ratio is X, the frequency band fm of the signal is compressed to the frequency band fm / X. In other words, the higher the local magnification, the higher the frequency noise shifts to the lower frequency side. Therefore, the higher the local magnification, the stronger the high frequency noise is reduced, thereby preventing the noise from being visually noticeable even after enlargement. it can.

  The difference in the noise reduction amount with respect to the local enlargement ratio in the bands FB1 to FB4 is due to the fact that the visual characteristics with respect to the spatial frequency show bandpass characteristics. That is, the frequency band for shifting to the visually most sensitive frequency differs depending on the local enlargement ratio. Specifically, as shown in FIG. 11, the bands FB1 to FB4 are shifted to the bands FB1 'to FB4' due to expansion. Assuming that the frequency sensitive to noise in the visual characteristics is ft, the band including the frequency ft among the bands FB1 'to FB4' changes according to the local enlargement ratio X. For example, when the band FB2 'includes the frequency ft, the noise component of the frequency ft that is sensitive to noise can be reduced by performing noise reduction processing stronger than the other bands on the band FB2 before distortion correction.

  FIG. 12 shows a detailed configuration example of the processing parameter control unit 722. The processing parameter control unit 722 includes a local enlargement ratio calculation unit 751, first to fourth noise reduction amount calculation units 752 to 755, and first to fourth multiplication units 756 to 759.

  The local enlargement ratio calculation unit 751 receives the target pixel output from the control unit 302 and the distortion correction information of the surrounding pixels. The local enlargement rate calculation unit 751 calculates the local enlargement rate ΔM (x, y) described in the above equation (2), and the calculated local enlargement rate ΔM (x, y) is input to the noise reduction amount calculation units 752 to 755. Output.

  Noise reduction amount calculation units 752 to 755 perform reduction amount calculation processing for reducing noise components in frequency bands FB1 to FB4, respectively. The noise reduction amount calculation unit 752 stores in advance a LUT (Look-Up Table) corresponding to the noise reduction amount of the band FB1 shown in FIG. The input of the LUT is a local enlargement ratio, and the output is a noise reduction amount. The noise reduction amount calculation unit 752 outputs the noise reduction amount of the band FB1 corresponding to the input local enlargement ratio to the multiplication unit 756.

  The multiplier 756 receives the noise reduction gain amount for each observation mode from the control unit 302. Multiplier 756 multiplies the noise reduction gain for each observation mode by the noise reduction amount of band FB1 to calculate the final noise reduction amount of band FB1, and outputs the calculated noise reduction amount to noise amount estimation unit 704. . For example, the noise reduction gain amount for each observation mode is 1.0 in the normal light observation mode, and is larger than 1.0 in the special light observation mode. The normal light is, for example, white light. Special light is light having a spectral characteristic different from that of normal light, for example, light having a narrower band than that of white light.

  In the noise reduction amount calculation unit 753, an LUT corresponding to the noise reduction amount of the band FB2 shown in FIG. 10B is stored in advance. The noise reduction amount calculation unit 753 outputs the noise reduction amount of the band FB2 corresponding to the input local enlargement ratio to the multiplication unit 757. The multiplication unit 757 receives the noise reduction gain amount for each observation mode from the control unit 302. Multiplier 757 multiplies the noise reduction gain for each observation mode by the noise reduction amount of band FB2, calculates the final noise reduction amount of band FB2, and outputs the calculated noise reduction amount to noise amount estimation unit 709. .

  In the noise reduction amount calculation unit 754, an LUT corresponding to the noise reduction amount of the band FB3 shown in FIG. 10B is stored in advance. The noise reduction amount calculation unit 754 outputs the noise reduction amount of the band FB3 corresponding to the input local enlargement ratio to the multiplication unit 758. The multiplier 758 receives the observation mode-specific noise reduction gain amount from the controller 302. Multiplier 758 multiplies the noise reduction gain amount for each observation mode by the noise reduction amount of band FB3 to calculate the final noise reduction amount of band FB3, and outputs the calculated noise reduction amount to noise amount estimation unit 713. .

  In the noise reduction amount calculation unit 755, an LUT corresponding to the noise reduction amount of the band FB4 shown in FIG. 10B is stored in advance. The noise reduction amount calculation unit 755 outputs the noise reduction amount of the band FB4 corresponding to the input local enlargement ratio to the multiplication unit 759. The multiplier 759 receives the noise reduction gain amount for each observation mode from the control unit 302. Multiplier 759 multiplies the noise reduction gain for each observation mode by the noise reduction amount of band FB4 to calculate the final noise reduction amount of band FB4 and outputs the calculated noise reduction amount to noise amount estimation unit 715. .

  The configuration of the noise reduction processing unit 320 is not limited to the above configuration, and various modifications can be made. For example, in the above description, the noise reduction processing unit 320 is arranged in front of the distortion correction processing unit 330. However, the present invention is not limited to this, and the noise reduction processing unit 320 may be arranged in the subsequent stage of the distortion correction processing unit 330. In this case, since the frequency characteristics of the noise around the enlarged image are shifted to the low frequency side, the noise reduction amount in the four frequency bands is greatly reduced by the local magnification ratio. Need to be controlled.

  Here, although the noise reduction amount is set according to the enlargement ratio in the above, the present embodiment is not limited to this, and for example, the noise reduction amount may be set according to the frequency band of the image. In this case, for example, the noise reduction processing unit 320 is placed after the distortion correction processing unit 330, and the frequency band in the local region of the image after distortion correction is calculated. As shown in FIG. 21, the local region is, for example, an annular region (an annular region having a distance R from the optical axis and a width ΔR) centered on the optical axis, and calculates a frequency characteristic band along the circumference. Then, the narrower the frequency band, the stronger, and the noise component of the predetermined band in the local region is reduced. For example, the maximum frequency of the frequency characteristic along the circumference is obtained, the local expansion rate in the ring is determined from the maximum frequency, and the degree of reduction is set according to the local expansion rate. The predetermined band is a band including, for example, ft in FIG. Since the frequency band becomes narrower and the low-frequency noise increases as the local enlargement ratio increases, the noise can be made uniform by setting the noise reduction amount according to the frequency band.

  As described above, the image processing apparatus according to the present embodiment includes the image acquisition unit, the correction processing unit, the conversion mode acquisition unit, and the noise reduction processing unit. The image acquisition unit acquires a distorted image in which distortion has occurred due to optical aberrations of the optical system. The correction processing unit performs a conversion process for correcting the distortion on the processing target area in the distortion image to correct the distortion image. The conversion mode information acquisition unit acquires conversion mode information representing the conversion mode of the processing target area by the conversion process. A noise reduction process part performs the process which reduces the noise component contained in a distortion image based on the acquired conversion aspect information.

  In this way, when the noise component changes according to the conversion mode information such as the local enlargement ratio, the noise reduction processing is performed based on the conversion mode information, so that the noise component in the visually high sensitivity band can be obtained. Uniform in the image. Thereby, since visibility improves, the oversight rate of the lesioned part on the back of the fold that has been overlooked can be reduced.

  In this embodiment, the image processing apparatus corresponds to the image processing unit 301 in FIG. The image acquisition unit corresponds to the A / D conversion unit 210 in FIG. The correction processing unit corresponds to the distortion correction processing unit 330. The conversion mode acquisition unit corresponds to the local enlargement ratio calculation unit 751 in FIG. The noise reduction processing unit corresponds to the noise reduction processing unit 320 in FIG. The optical system corresponds to the objective lens 203 in FIG.

  Distortion is a deviation from an ideal optical system image. For example, let the imaging of the point P be a point P ′, the distance from the optical axis to the point P be D, and the distance from the optical axis to the point P ′ be D ′. In an image without distortion, D ′ / D is constant regardless of D, and in an image with distortion, D ′ / D changes according to D.

  The conversion process is, for example, a process for converting pixel coordinates according to a predetermined function. That is, the correction of distortion is not limited to correcting a distorted image to an image without distortion. For example, as in the present embodiment, conversion that performs enlargement / reduction at a magnification according to the position on the image may be used.

  The processing target area may be an area composed of a plurality of pixels or an area composed of a single pixel. For example, in the present embodiment, the distortion correction processing is performed using one pixel of interest as a processing target region.

  The conversion mode information is information representing the conversion magnification of the area of the processing target area in the conversion process. That is, information indicating that the area has been expanded before and after conversion, information indicating that the area has been reduced before and after conversion, and information indicating that the area has not changed before and after conversion. For example, the conversion mode information may be the conversion magnification itself, correspondence information between the pixel coordinates before conversion and the pixel coordinates after conversion, or information on frequency characteristics of the processing target region. Good.

  In the present embodiment, the noise reduction processing unit determines a noise reduction degree, which is a degree of reducing the noise component, based on the conversion mode information, and performs noise reduction processing based on the determined noise reduction degree.

For example, in the present embodiment, the reduction processing is performed with the noise reduction amount as the noise reduction degree. As shown in FIG. 10B, the noise reduction amount is determined according to the local enlargement ratio that is the conversion mode information, and the smoothing processing parameter σ _s is set based on the noise reduction amount. The smoothing process works to reduce the estimated noise amount more as the noise reduction amount is larger.

  In this way, the noise component that changes according to the conversion mode information can be reduced to a degree according to the conversion mode information, and the noise components can be made uniform.

  In the present embodiment, as described above in Expression (2) and the like, the conversion mode information includes the size (area) of the processing target region before the conversion processing and the size (area) of the processing target region after the conversion processing. ) (Local enlargement ratio ΔM (x, y)). The noise reduction processing unit controls the degree of noise reduction according to a comparison result (local enlargement ratio ΔM (x, y)) between the size of the processing target region before the conversion processing and the size of the processing target region after the conversion processing. .

  More specifically, the noise reduction processing unit sets the noise reduction degree to the first level when the first condition that the processing target area after the conversion processing is larger than the processing target area before the conversion processing is satisfied. Make it relatively larger than when the condition other than the condition is met. On the other hand, when the noise reduction processing unit satisfies the second condition that the processing target area after the conversion processing is smaller than the processing target area before the conversion processing, the noise reduction processing unit sets the noise reduction degree to a condition other than the second condition. It is relatively smaller than the case corresponding to.

  In this way, low-frequency noise that increases or decreases according to the change in the size of the processing target area can be reduced to a degree corresponding to the change in the size of the processing target area. For example, as shown in FIG. 4C, when the enlargement process corresponds to the first condition in the peripheral part of the image and the reduction process corresponds to the second condition in the central part of the image, noise increases visually. The degree of noise reduction at the periphery of the image can be made larger than that at the center of the image.

  Here, the size of the processing target region is not limited to the area, and may be, for example, the length of a side of the processing target region, the length in a predetermined direction, or the like. The comparison result is not limited to the local enlargement ratio ΔM (x, y). For example, the ratio between the size before the conversion processing and the size after the conversion processing, or the magnitude relationship between the size before the conversion processing and the size after the conversion processing. Etc.

  In the present embodiment, the image acquisition unit acquires an image in which the degree of distortion in the processing target region differs depending on the position of the processing target region in the distorted image. The correction processing unit performs processing for converting the size of the processing target area according to the degree of distortion in the processing target area. The noise reduction processing unit controls a noise reduction degree, which is a degree of reducing the noise component, according to the conversion result of the size of the processing target region.

  More specifically, as described above with reference to FIG. 4B, the image acquisition unit acquires an image in which the degree of distortion in the central area of the distortion image is relatively smaller than the degree of distortion in the peripheral area. As described above with reference to FIG. 4C, the correction processing unit performs a process of reducing the size of the processing target area in the central area from the size before the conversion process. As described above with reference to FIG. 10B, the noise reduction processing unit makes the noise reduction degree in the central area relatively smaller than the noise reduction degree in the peripheral area.

  In addition, as described above with reference to FIG. 4C, the correction processing unit performs processing for enlarging the size of the processing target region in the peripheral region as compared with the size before the conversion processing. As described above with reference to FIG. 10B, the noise reduction processing unit makes the noise reduction degree in the peripheral area relatively larger than the noise reduction degree in the central area.

  In this way, the size of the processing target area can be converted according to the degree of distortion, and noise can be reduced to a degree according to the conversion result. Thereby, since the enlargement ratio can be increased in a region where the distortion is large, the visibility of the subject collapsed by the distortion can be improved. Further, visibility degradation due to noise in the enlarged area can be suppressed.

  Here, the degree of distortion is the degree of deviation from an ideal optical system image. For example, taking the distances D and D 'described above as an example, the deviation (for example, difference or ratio) between D' / D when there is distortion and D '/ D when there is no distortion is the degree of distortion. This degree of distortion depends on D. In an optical system that is distorted toward the periphery, the degree of distortion increases as D increases.

  The central region (central portion) is an image region including the optical axis, for example, a region within a circle centered on the optical axis. The peripheral region (peripheral portion) is a region surrounding the central region, for example, an annular region centered on the optical axis. There may be another region between the peripheral region and the central region, or the peripheral region and the central region may be in contact with each other.

  In this embodiment, the noise reduction processing unit includes a frequency band dividing unit and a smoothing processing unit. As shown in FIG. 9, the frequency band dividing unit divides the distortion image into first to nth frequency band images (FB1 to FB4) having different frequency bands. The smoothing processing unit performs different smoothing processes on the divided frequency band images.

  Specifically, as described above with reference to FIG. 10B, the smoothing processing unit performs noise included in the i-th (i is a natural number equal to or less than n) frequency band image (for example, FB2) of FB1 to FB4. The degree to which the component is reduced is controlled based on the information regarding the comparison result. As described above in Expression (1), the smoothing processing unit performs the smoothing process on the i-th frequency band image (FB2) based on the controlled noise reduction degree.

  More specifically, as described above in the above equation (2), the information related to the comparison result is the magnification ΔM (x, y) of the area of the processing target area after the conversion process with respect to the area of the processing target area before the conversion process. It is. As shown in FIG. 10B, the magnification ΔM (x, y) is divided into first to nth magnification ranges (R1 to R4). R1 to R4 are low-magnification ranges sequentially from the first to the n-th. As shown in FIG. 9, FB1 to FB4 are low-frequency band images sequentially from the first to the nth. In this case, the smoothing processing unit reduces noise of the i-th frequency band image (FB2) when the magnification ΔM (x, y) is included in the i-th magnification range (for example, R2) of R1 to R4. The degree is set to be equal to or higher than the noise reduction degree of the other frequency band images (FB1, FB3, FB4).

  In this way, as described above with reference to FIG. 11, it is possible to reduce noise components in the vicinity of the frequency ft at which noise is desired to be reduced. Further, since smoothing can be weakened in a band other than the vicinity of the frequency ft, the frequency characteristics of the structure can be left. Also, by increasing the frequency band for increasing the degree of noise reduction as the magnification ΔM (x, y) increases, the noise component in the frequency band near the frequency ft after distortion correction can be reduced before distortion correction.

  Here, in this embodiment, the frequency band dividing unit corresponds to the low-pass processing units 701, 706, and 711, the down-sampling units 702 and 707, and the subtraction processing units 703, 708, and 712 in FIG. The smoothing processing unit corresponds to the processing parameter control unit 722, the noise amount estimation units 704, 709, 712, and 715 and the smoothing processing units 705, 710, 714, and 716 in FIG.

  In the present embodiment, the optical aberration is distortion.

  Distortion is an aberration in which a straight image does not become a straight line but becomes a distorted curve. Generally, it occurs in a wide-angle objective optical system, and the distortion becomes larger as the distance from the optical axis increases.

  In this embodiment, as shown in FIG. 1, the optical system includes an objective lens 203 that images the front visual field range and the lateral visual field range.

  The front visual field range is a visual field range including the optical axis direction, for example, a range of 0 to 45 degrees with respect to the optical axis. The lateral visual field range is a visual field range including a direction orthogonal to the optical axis, and is, for example, a range of 45 degrees to 135 degrees with respect to the optical axis. The objective lens 203 of the present embodiment has a visual field range of 0 degrees to 110 degrees with respect to the optical axis, for example.

  In the present embodiment, as illustrated in FIG. 4A, the image acquisition unit acquires an image obtained by imaging the inside of the tubular subject via the objective lens 203.

  In the present embodiment, the conversion process is a projective conversion process.

  Projection represents conversion from a subject to an image by an optical system. For example, when the image of the point P is the point P ′ as described above, the distance D ′ between the optical axis and the point P ′ is a function expressed by the angle θ formed by the optical axis and the point P. Projection conversion processing is image processing that converts a projection into another projection. In the present embodiment, conversion from a subject to an image by an optical system is also referred to as projective conversion as appropriate.

  The control device 300 according to the present embodiment includes an image processing device and a control unit 302. The control unit 302 detects the observation state of the endoscope apparatus that captures the distortion image, and controls the conversion processing by the correction processing unit based on the detected observation state. Specifically, the control unit 302 controls the correction processing unit to perform a conversion process for enlarging the peripheral region when the observation state is a state in which the peripheral region of the distorted image is magnified. In addition, the control unit 302 includes a reception unit (a reception unit not illustrated) that receives input of information related to the observation state of the endoscope apparatus from the user. The control unit 302 detects the observation state of the endoscope apparatus based on the received information.

  In this way, distortion correction processing according to the observation state can be performed. For example, the correction may not be performed when inserted into the large intestine, but may be performed when the patient is examined while being pulled out. Or you may correct | amend by the normal observation observed in the direction along a lumen | bore, without correcting by the enlarged observation facing a lesioned part. Alternatively, when magnifying the lesion on the back of the fold, the scope cannot be directly opposed, so the side of the scope is brought close to the fold. In this case, correction for enlarging the peripheral portion may be performed as compared with normal distortion correction.

  In the present embodiment, the observation state is not limited to detection by a user instruction. For example, the observation state may be automatically detected by detecting that the insertion length into the large intestine has started to decrease.

  In addition, the control device 300 according to the present embodiment includes a display control unit (a display control unit (not shown)) that performs control to display the image corrected by the correction processing unit on the display unit 400.

3. Second Embodiment 3.1. Endoscope Device Next, a second embodiment will be described. In the first embodiment, frame-sequential imaging is performed, and in the second embodiment, imaging is performed by a Bayer array imaging element. In the first embodiment, noise reduction processing is performed by dividing into four frequency bands. However, in the second embodiment, noise reduction processing in the time direction and noise reduction processing in the spatial direction are performed, and these processing results are obtained. Are combined according to the local enlargement ratio.

  FIG. 13 shows a second configuration example of the endoscope apparatus. The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. Note that the description of the same components as those in the first embodiment will be omitted as appropriate, and the detailed configuration different from that in the first embodiment will be mainly described below.

  The light source unit 100 includes a white light source 101 and a condensing lens 104 that collects light on the incident end face of the light guide fiber 201. The light source unit 100 has no rotation color filter.

  The imaging unit 200 includes a light guide fiber 201, illumination lenses 202 and 204, an imaging element 209, and an A / D conversion unit 210. The image sensor 209 is different from the first embodiment, and the image sensor 209 is constituted by a primary color single plate image sensor. The primary color single-plate image sensor is a so-called Bayer array image sensor, and a CCD or CMOS can be used.

  The control device 300 includes an image processing unit 301 and a control unit 302. The image processing unit 301 has the same configuration as that of the first embodiment, and includes an OB clamp processing unit 310, a noise reduction processing unit 320, a distortion correction processing unit 330, a WB processing unit 340, a three plate processing unit 350, a color correction. A processing unit 360, a gradation conversion processing unit 370, and an enhancement processing unit 380 are included. The processing of the noise reduction processing unit 320, the distortion correction processing unit 330, the WB processing unit 340, and the 3 plate processing unit 350 is different from that of the first embodiment.

  In the first embodiment, the distortion correction processing unit 330 receives an image (R image, G image, B image) in which pixel values corresponding to three color lights are present in all the pixels of the effective area image. In the second embodiment, the Bayer array image shown in FIG. 16A is input to the distortion correction processing unit 330. Therefore, an image of the same color made up of only R pixels (or only G pixels and only B pixels) is an image with a coarse sampling interval. In this case, the kernel size of the interpolation filter used in the distortion correction process is larger than that in the first embodiment, but the basic distortion correction process is not changed, and thus description thereof is omitted.

  Since the WB processing unit 340 is basically only the multiplication of the white balance coefficient for the R pixel and the B pixel, the description thereof is omitted.

  The 3-plate processing unit 350 performs 3-plate processing by generating a color signal missing from the Bayer array by interpolation processing. For example, an interpolation process is performed by a known demosaicing process.

3.2. Noise Reduction Processing Unit Next, the noise reduction processing in the second embodiment will be described in detail. FIG. 14 shows a detailed configuration example of the noise reduction processing unit 320. The noise reduction processing unit 320 includes a processing parameter control unit 901, nonlinear filter processing units 902 and 903, a weight addition processing unit 904, and a frame memory 905.

  The control unit 302 is connected to the processing parameter control unit 901. The processing parameter control unit 901 calculates the local enlargement rate after distortion correction in the target pixel to be subjected to noise reduction processing and the surrounding pixel region based on the distortion correction information output from the control unit 302. The local enlargement ratio of the target pixel is calculated in the same manner as the above-described method in the first embodiment.

  As illustrated in FIG. 15, the processing parameter control unit 901 sets a combination ratio between the intra-frame smoothed pixel value and the inter-frame smoothed pixel value based on the local enlargement ratio, and the determined combination ratio is a weight addition processing unit. Output to 904. The relationship between the local enlargement ratio and the composition ratio shown in FIG. 15 will be described.

  As shown in FIG. 15, when the local enlargement ratio associated with distortion correction is increased, the intra-frame smoothed pixel value composite ratio α is lowered, and the inter-frame smoothed pixel value composite ratio β is increased. In particular, in order to obtain the noise reduction effect while maintaining the resolution in a region where the motion is small, it is better to increase the ratio of the noise reduction processing using the inter-frame correlation than the noise reduction processing using the intra-frame correlation.

  The reason will be described in detail. Random noise fluctuates independently in the time direction and the spatial direction, but spatial enlargement processing accompanying distortion correction shifts the spatial characteristics of the random noise to a lower frequency side (becomes low frequency noise). On the other hand, the temporal frequency change of random noise does not occur.

  The noise reduction process is basically a low-pass filter process, but in order to keep the subject's spatial blurring amount constant before and after the enlargement process by distortion correction, it is preferable to fix the frequency characteristic of the low-pass filter. However, if the low-pass filter characteristics in the spatial direction are fixed, random noise will be spatially shifted to the low frequency side after enlargement processing by distortion correction. It manifests itself as high low frequency noise. Therefore, it is perceived that the noise reduction effect is relatively lowered. This reduction is enhanced by noise reduction processing using inter-frame correlation that suppresses noise fluctuations in the time direction.

The noise reduction process using the correlation between frames is a low-pass filter process in the time direction, and when there is no motion in the subject, random noise can be cut while suppressing the occurrence of spatial blur. (In this case, σ _s2 in the above equation (4) needs to be set smaller than σ _{s1 in the} above equation (3)).

  On the other hand, since a high frequency component is generated in the time direction in a region where the motion is large, if the low-pass filter processing in the time direction is strengthened, spatial blur due to motion blur occurs. Further, since the low-pass filter processing is an IIR (infinite impulse response) type, a lot of afterimages are generated.

  That is, it is possible to apply the noise reduction process using the inter-frame correlation more positively to an area where the motion is small. Further, since the peripheral area of the image before the enlargement process by distortion correction is a reduced area compared to the central area of the image, the apparent movement is a small area. Therefore, even if the noise reduction process using the inter-frame correlation is more strongly applied to the peripheral area of the image to compensate for the reduction in the noise reduction effect in the spatial direction, it is possible to suppress the blur and obtain the noise reduction effect. . In particular, the noise reduction processing using inter-frame correlation is performed using the peripheral area of the past frame corresponding to the target pixel of the current frame shown in FIG. Can be used.

  The effective area image output from the OB clamp processing unit 310 is input to the nonlinear filter processing units 902 and 903. The non-linear filter processing units 902 and 903 perform a smoothing process using a non-linear filter such as a bilateral filter, for example.

  The non-linear filter processing unit 902 performs an intra-frame smoothing process for smoothing in the spatial direction. As shown in FIG. 14, the non-linear filter processing unit 902 has the same color included in the pixel of interest in the current (processing target) frame image and its peripheral region (for example, 9 × 9 pixels on the Bayer array). Only the signal is extracted, and smoothing processing is performed on the extracted signal of the same color. The same color is an R image composed of R pixels, a G image composed of G pixels, and a B image composed of B pixels. As shown in FIG. 16B, the G pixel is further divided into a Gr image composed of Gr pixels located in the same row as the R pixel and a Gb image composed of Gb pixels located in the same row as the B pixel. Also good. In this case, the relative sampling positions used in the nonlinear filter processing described later can be made common to all the R, Gr, Gb, and B images, and the circuit scale can be reduced.

  The non-linear filter processing unit 903 performs inter-frame smoothing processing for smoothing in the time direction. As illustrated in FIG. 17, the non-linear filter processing unit 903 performs processing on the target pixel of the current frame image and the pixels included in the peripheral region of the past frame image. The past frame image is a noise-reduced image stored in the frame memory 905, for example, an image one frame before the current frame image. The peripheral area is a pixel around the pixel corresponding to the target pixel of the current frame image.

The nonlinear filter processing unit 902 performs a smoothing process using, for example, a nonlinear filter represented by the following expression (3). The following equation (3) shows the filter coefficient F _intra (x, y, t) of the bilateral filter.

Here, x is the horizontal coordinate of the image at time t, and y is the vertical coordinate of the image at time t. B (x, y, t) is a normalization coefficient of the filter. P (x ₀ , y ₀ , t) is a pixel value of the target pixel (x ₀ , y ₀ , t) to be filtered. P (x, y, t) is a pixel value of the peripheral pixel (x, y, t) of the target pixel.

The nonlinear filter processing unit 903 performs a smoothing process using, for example, a nonlinear filter represented by the following expression (4). The following expression (4) shows the filter coefficient F _inter (x, y, t) of the bilateral filter.

Here, x is the horizontal coordinate of the image at time t or time t-1, and y is the vertical coordinate of the image at time t or time t-1. C (x, y, t) is a normalization coefficient of the filter. Po (x, y, t−1) is a pixel (x corresponding to the pixel of interest (x ₀ , y ₀ , t) at time t in the noise-reduced image at time t−1 stored in the frame memory 905. _This is the pixel value of the peripheral pixel (x, y, t−1) of ₀ , y ₀ , t−1).

  The weight addition processing unit 904 receives the intra-frame smoothed pixel value of the target pixel from the nonlinear filter processing unit 902 and the inter-frame smoothed pixel value of the target pixel from the nonlinear filter processing unit 903. In addition, the weight addition processing unit 904 receives from the processing parameter control unit 901 the composition ratio of the output intra-frame smoothed pixel value and the inter-frame smoothed pixel value. The weight addition processing unit 904 weights and averages the intra-frame smoothed pixel value and the inter-frame smoothed pixel value based on the combination ratio, and the weighted average weighted average value Po is sent to the distortion correction processing unit 330 and the frame memory 905. Output.

Specifically, the weight addition processing unit 904 calculates the weighted average value Po (x0, y0, t) at the target pixel (x0, y0, t) by the following equation (5).

Here, α is a synthesis ratio of the intra-frame smoothed pixel values output from the processing parameter control unit 901. β is a synthesis ratio of the inter-frame smoothed pixel values output from the processing parameter control unit 901. For example, α + β = 1. Σ _xy represents the total sum for pixels of the same color in the peripheral region of the pixel of interest (x ₀ , y ₀ , t).

  By performing the noise reduction process using the inter-frame correlation in this way, it is possible to prevent in advance that the noise in the peripheral portion of the image magnified by the subsequent distortion correction processing unit 330 is visually noticeable.

In the above description, the noise reduction processing unit 320 is placed before the distortion correction processing unit 330. However, the noise reduction processing unit 320 may be placed after the distortion correction processing unit 330. In this case, in the noise reduction processing using the correlation between frames, the peripheral area of the past frame corresponding to the target pixel of the current frame shown in FIG. 17 is resized according to the local enlargement ratio (if the enlargement ratio is high, the size is increased. And σ _d2 in the above equation (4) may be set large (it becomes easier to find a pixel having a high inter-frame correlation). Alternatively, a motion compensation process based on a motion vector calculated by block matching between the peripheral area of the current frame and the peripheral area of the past frame is added without changing the size of the peripheral area, and the motion vector is searched according to the enlargement ratio. The range may be variable. This makes it possible to effectively use noise reduction processing using inter-frame correlation. If the maximum local enlargement ratio is determined by design, the size of the surrounding area or the search range for the motion vector may be set based on the local enlargement ratio. In this case, it goes without saying that the size of the peripheral region or the search range of motion vectors need not be variable.

  Here, in the first embodiment and the second embodiment, the fisheye lens is used as the objective lens, but the present invention is not limited to this, and for example, another optical system that can obtain a wide field of view may be used. For example, a reflective objective optical system may be used.

  According to the above, as shown in FIG. 14, the noise reduction processing unit 320 includes a time smoothing processing unit (nonlinear filter processing unit 902), a spatial smoothing processing unit (nonlinear filter processing unit 903), and a synthesis processing unit. (Weight addition processing unit 904). As described above in equation (4), the time smoothing processing unit performs a process of smoothing the noise component in the time direction, and outputs the processed pixel value as the time smoothed pixel value. As described above in Equation (3), the spatial smoothing processing unit performs processing to smooth the noise component in the spatial direction, and outputs the processed pixel value as the spatial smoothed pixel value. As described above in equation (5), the synthesis processing unit synthesizes the temporally smoothed pixel value and the spatially smoothed pixel value at predetermined ratios α and β.

  More specifically, as illustrated in FIG. 15, the synthesis processing unit includes information (local enlargement ratio ΔM (x) regarding the comparison result between the size of the processing target area before the conversion process and the size of the processing target area after the conversion process. , Y)) to control the predetermined ratios α and β.

  In this way, when the peripheral portion of the image is enlarged and displayed, it is possible to perform noise reduction that effectively uses the inter-frame correlation with respect to the peripheral portion of the image. As a result, it is possible to suppress visually increased noise due to the enlarged display, and the visibility can be improved. By improving visibility, it is possible to reduce the oversight rate of lesions behind the folds that have been missed.

4). Software In the above-described embodiment, each unit configuring the image processing unit 301 is configured by hardware. However, the present invention is not limited to this. For example, the CPU may be configured to perform processing of each unit on an image acquired in advance using an imaging device such as a capsule endoscope, and may be realized as software by the CPU executing a program. Alternatively, a part of processing performed by each unit may be configured by software.

  When the imaging unit is a separate unit and the processing performed by each unit of the image processing unit 301 is realized as software, a known computer system such as a workstation or a personal computer can be used as the image processing apparatus. Then, a program (image processing program) for realizing the processing performed by each unit of the image processing unit 301 is prepared in advance, and this image processing program is executed by the CPU of the computer system.

  FIG. 18 is a system configuration diagram showing the configuration of the computer system 600 in this modification, and FIG. 19 is a block diagram showing the configuration of the main body 610 in this computer system 600. As shown in FIG. 18, the computer system 600 includes a main body 610, a display 620 for displaying information such as an image on the display screen 621 according to an instruction from the main body 610, and various information on the computer system 600. A keyboard 630 for input and a mouse 640 for designating an arbitrary position on the display screen 621 of the display 620 are included.

  Further, as shown in FIG. 19, a main body 610 in the computer system 600 includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, a CD-ROM drive 615 that accepts a CD-ROM 660, and a USB memory. USB port 616 to which 670 is detachably connected, I / O interface 617 to which display 620, keyboard 630 and mouse 640 are connected, and a LAN interface for connection to a local area network or wide area network (LAN / WAN) N1 618.

  Further, the computer system 600 is connected to a modem 650 for connecting to a public line N3 such as the Internet, and is another computer system via a LAN interface 618 and a local area network or a wide area network N1. A personal computer (PC) 681, a server 682, a printer 683, and the like are connected.

  The computer system 600 implements an image processing apparatus by referring to an image processing program recorded on a predetermined recording medium and reading and executing an image processing program for realizing a processing procedure to be described later. Here, the predetermined recording medium is a “portable physical medium” including an MO disk, a DVD disk, a flexible disk (FD), a magneto-optical disk, an IC card, etc. in addition to the CD-ROM 660 and the USB memory 670, a computer A “fixed physical medium” such as HDD 614, RAM 612, and ROM 613 provided inside and outside the system 600, a public line N3 connected via the modem 650, and another computer system (PC) 681 or server 682 are connected. It includes any recording medium that records an image processing program readable by the computer system 600, such as a “communication medium” that stores the program in a short time when transmitting the program, such as a local area network or a wide area network N1.

  That is, the image processing program is recorded on a recording medium such as “portable physical medium”, “fixed physical medium”, and “communication medium” in a computer-readable manner. An image processing apparatus is realized by reading and executing an image processing program from a medium. Note that the image processing program is not limited to be executed by the computer system 600, and when the other computer system (PC) 681 or the server 682 executes the image processing program or in cooperation therewith. The present invention can be similarly applied to a case where an image processing program is executed.

  The present embodiment can also be applied to a computer program product in which a program code for realizing each unit (noise reduction processing unit, distortion correction processing unit, etc.) of the present embodiment is recorded.

  Here, the computer program product includes, for example, an information storage medium (an optical disk medium such as a DVD, a hard disk medium, a memory medium, etc.) on which the program code is recorded, a computer on which the program code is recorded, and an Internet system on which the program code is recorded. (For example, a system including a server and a client terminal) such as an information storage medium, apparatus, device, or system in which a program code is incorporated. In this case, each component and each processing process of this embodiment are mounted by each module, and the program code constituted by these mounted modules is recorded in the computer program product.

  As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in each embodiment or modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

  In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings.

100 light source unit, 101 white light source, 102 rotating color filter,
103 rotational drive unit, 104 condenser lens, 200 imaging unit,
201 light guide fiber, 202 illumination lens, 203 objective lens,
209 imaging device, 210 A / D converter, 300 controller,
301 image processing unit, 302 control unit, 310 OB clamp processing unit,
320 noise reduction processing unit, 330 distortion correction processing unit, 340 WB processing unit,
350 3 plate processing unit, 360 color correction processing unit, 370 gradation conversion processing unit,
380 enhancement processing unit, 390 lumen, 391 projective transformation, 392 projective transformation,
400 display unit, 500 external I / F unit,
600 computer system, 610 main body,
611 CPU, 612 RAM, 613 ROM, 614 HDD,
615 CD-ROM drive, 616 USB port,
617 I / O interface, 618 LAN interface,
620 display, 621 display screen, 630 keyboard, 640 mouse,
650 modem, 660 CD-ROM, 670 USB memory, 681 PC,
682 server, 683 printer,
701 Low-pass processing unit, 702 downsampling unit, 703 subtraction processing unit,
704 noise amount estimation unit, 705 smoothing processing unit, 706 low-pass processing unit,
707 downsampling unit, 708 subtraction processing unit, 709 noise amount estimation unit,
710 smoothing processing unit, 711 low-pass processing unit, 712 subtraction processing unit,
713 noise amount estimation unit, 714 smoothing processing unit, 715 noise amount estimation unit,
716 smoothing processing unit, 717 addition processing unit, 718 upsampling unit,
719 addition processing unit, 720 upsampling unit, 721 addition processing unit,
722 processing parameter control unit, 751 local enlargement rate calculation unit,
752 to 755 first to fourth noise reduction amount calculation units,
756-759 1st-4th multiplication part, 803 rotary motor,
851 Red color filter, 852 Green color filter, 853 Blue color filter,
901 processing parameter control unit, 902 nonlinear filter processing unit,
903 nonlinear filter processing unit, 904 weight addition processing unit, 905 frame memory,
FB1 to FB4 First to fourth frequency bands, F _inter filter coefficients,
F _intra filter coefficient, N1 wide area network, N3 public line,
Po weighted average value, d peripheral area, h central area, t time,
ΔM (x, y) Local magnification, α composition ratio, β composition ratio,
_s smoothing parameter

Claims (25)

An image acquisition unit that acquires a distortion image in which distortion has occurred due to optical aberrations of the optical system;
A correction processing unit that performs a conversion process for correcting the distortion on a processing target area in the distortion image, and corrects the distortion image;
A conversion mode information acquisition unit that acquires conversion mode information representing a conversion mode of the processing target area by the conversion process;
A noise reduction processing unit that performs a process of reducing a noise component included in the distorted image based on the acquired conversion mode information;
An image processing apparatus comprising: In claim 1,
The conversion mode information is
An image processing apparatus, comprising: information related to a comparison result between the size of the processing target area before the conversion process and the size of the processing target area after the conversion process. In claim 2,
The noise reduction processing unit
A noise reduction degree, which is a degree of reducing the noise component, is controlled according to the comparison result between the size of the processing target area before the conversion process and the size of the processing target area after the conversion process. An image processing apparatus. In claim 3,
The noise reduction processing unit
When the first condition that the area to be processed after the conversion process is larger than the area to be processed before the conversion process corresponds to the condition other than the first condition, the noise reduction degree An image processing apparatus characterized in that the image processing apparatus is relatively larger than the case where it is performed. In claim 3,
The noise reduction processing unit
If the processing target area after the conversion process satisfies the second condition that the processing target area is smaller than the processing target area before the conversion process, the noise reduction degree corresponds to a condition other than the second condition. An image processing apparatus characterized in that the image processing apparatus is made relatively smaller than the case where it is performed. In claim 1,
The noise reduction processing unit
An image processing apparatus, wherein a noise reduction degree, which is a degree of reducing the noise component, is determined based on the conversion mode information, and the noise reduction processing is performed based on the noise reduction degree. In claim 1,
The image acquisition unit
The distortion degree of the distortion in the processing target area acquires an image that differs depending on the position of the processing target area in the distortion image,
The correction processing unit
According to the degree of distortion in the processing target area, to perform a process of converting the size of the processing target area,
The noise reduction processing unit
An image processing apparatus that controls a noise reduction degree, which is a degree of reducing the noise component, according to a conversion result of a size of the processing target area. In claim 7,
The image acquisition unit
Obtaining an image in which the degree of distortion in the central area of the distortion image is relatively smaller than the degree of distortion in the peripheral area;
The correction processing unit
A process of reducing the size of the processing target area in the central area than the size before the conversion process,
The noise reduction processing unit
The image processing apparatus according to claim 1, wherein the noise reduction degree in the central area is relatively smaller than the noise reduction degree in the peripheral area. In claim 7,
The image acquisition unit
Obtaining an image in which the degree of distortion in the central area of the distortion image is relatively smaller than the degree of distortion in the peripheral area;
The correction processing unit
Performing a process of enlarging the size of the processing target area in the peripheral area than the size before the conversion process;
The noise reduction processing unit
The image processing apparatus according to claim 1, wherein the noise reduction degree in the peripheral area is relatively larger than the noise reduction degree in the central area. In claim 1,
The noise reduction processing unit
A frequency band dividing unit that divides the distortion image into first to n-th frequency band images (n is a natural number of 2 or more) having different frequency bands;
A smoothing processing unit that performs different smoothing processing on each of the divided frequency band images;
An image processing apparatus comprising: In claim 10,
The conversion mode information is
Information regarding a comparison result between the size of the processing target area before the conversion process and the size of the processing target area after the conversion process;
The smoothing processing unit
Controlling the degree of reducing the noise component included in the i-th frequency band image (i is a natural number equal to or less than n) of the first to n-th frequency band images based on the information on the comparison result;
The smoothing processing unit
An image processing apparatus that performs the smoothing process on the i-th frequency band image based on the controlled noise reduction degree. In claim 11,
Information about the comparison result is
A magnification of the area of the processing target region after the conversion processing with respect to the area of the processing target region before the conversion processing;
The first to nth frequency band images before the conversion process are band images of low frequency sequentially from 1st to nth, and the magnification is divided into first to nth magnification ranges, When the 1st to nth magnification range is a magnification range of low magnification sequentially from 1st to nth,
The smoothing processing unit
When the magnification is included in the i-th magnification range of the first to n-th magnification ranges, the noise reduction degree of the i-th frequency band image is greater than or equal to the noise reduction degree of other frequency band images. An image processing apparatus. In claim 1,
The noise reduction processing unit performs processing for smoothing a noise component in a time direction, and outputs a pixel value after processing as a time smoothed pixel value;
A spatial smoothing processing unit that performs a process of smoothing noise components in the spatial direction and outputs the processed pixel value as a spatially smoothed pixel value;
A synthesis processing unit that synthesizes the temporally smoothed pixel value and the spatially smoothed pixel value at a predetermined ratio;
An image processing apparatus comprising: In claim 13,
The conversion mode information is
Information regarding a comparison result between the size of the processing target area before the conversion process and the size of the processing target area after the conversion process;
The synthesis processing unit
An image processing apparatus that controls the predetermined ratio based on information relating to the comparison result. In claim 1,
The image processing apparatus, wherein the optical aberration is distortion. In claim 1,
The optical system is
An image processing apparatus having an objective lens that images a front visual field range and a lateral visual field range. In claim 16,
The image acquisition unit
An image processing apparatus that acquires an image obtained by imaging an inside of a tubular subject through the objective lens. In claim 1,
The image processing apparatus, wherein the conversion process is a projective conversion process. An image processing apparatus according to any one of claims 1 to 18,
A control unit that detects an observation state of the endoscope apparatus that captures the distortion image, and controls the conversion processing by the correction processing unit based on the detected observation state;
The control apparatus characterized by including. In claim 19,
The controller is
When the observation state is a state of magnifying and observing the peripheral area of the distorted image, the control apparatus controls the correction processing unit to perform a conversion process for magnifying the peripheral area. In claim 19,
The controller is
A receiving unit that receives input of information related to the observation state of the endoscope apparatus from a user;
The controller is
A control apparatus that detects the observation state of the endoscope apparatus based on the received information. An image processing apparatus according to any one of claims 1 to 18,
A display control unit that performs control to display the image corrected by the correction processing unit on a display unit;
The control apparatus characterized by including. An optical system for imaging a subject;
An image acquisition unit for acquiring a distortion image in which distortion is caused by the optical aberration of the optical system;
A correction processing unit that performs a conversion process for correcting the distortion on a processing target area in the distortion image, and corrects the distortion image;
A conversion mode information acquisition unit that acquires conversion mode information representing a conversion mode of the processing target area by the conversion process;
A noise reduction processing unit that performs a process of reducing a noise component included in the distorted image based on the acquired conversion mode information;
An endoscopic device comprising: Acquire a distortion image that is distorted by the optical aberration of the optical system,
A conversion process for correcting the distortion is performed on a processing target area in the distortion image, the distortion image is corrected,
Obtaining conversion mode information representing a conversion mode of the processing target area by the conversion processing;
An image processing method comprising: performing a process of reducing a noise component included in the distorted image based on the acquired conversion mode information. An image acquisition unit that acquires a distortion image in which distortion has occurred due to optical aberrations of the optical system;
A correction processing unit that performs a conversion process for correcting the distortion on a processing target area in the distortion image, and corrects the distortion image;
A conversion mode information acquisition unit that acquires conversion mode information representing a conversion mode of the processing target area by the conversion process;
Based on the acquired conversion mode information, as a noise reduction processing unit that performs processing to reduce noise components included in the distortion image,
An image processing program for causing a computer to function.