JP2010193042A - Fixed-pattern noise removal unit, imaging unit, and electronic endoscope system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a fixed-pattern noise removal unit for accurately removing fixed-pattern noise with a simple structure without arranging a mechanical shutter nor an optical black while simplifying a manufacturing process. <P>SOLUTION: A timing controller causes an imaging element to generate an image signal in an exposure time of 200 ns. An A/D converter 25 converts the image signal to image data. An image processing part 30 includes a region averaging circuit 31, a determination circuit 32, an FPN creation circuit 33, and an FPN correction circuit 34. The region averaging circuit 31 stores pseudo black pixel data constituting the image data in a DRAM 27. The region averaging circuit 31 averages the entire pseudo black pixel data stored in the DRAM 27. The determination circuit 32 determines whether the pseudo black pixel data are used for noise data based on a reference luminance range and the averaged pseudo black pixel data. The FPN creation circuit 33 generates noise data in the row direction and the column direction by the pseudo black pixel data determined to be used for the noise data. The FPN correction circuit 34 removes FPN from imaging pixel data using the noise data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a fixed pattern noise removing unit that effectively removes fixed pattern noise that differs from pixel to pixel with a simple configuration, such as a CMOS image sensor, an imaging unit, and an electronic that creates an image from which fixed pattern noise has been removed. The present invention relates to an endoscope system.

  A CMOS image sensor that can reduce power consumption and manufacturing cost is known. Since the CMOS image sensor is provided with an amplifier for each pixel, individual fixed pattern noise is generated for each pixel.

  In order to remove fixed pattern noise, it has been proposed to generate a fixed pattern noise component by performing imaging while the mechanical shutter is closed, and remove the fixed pattern noise component from image data at the time of imaging (patent). Reference 1).

  In addition, in order to remove fixed pattern noise, an optical black is formed with some pixels shielded from light, and the pixel signal generated by the optical black is used as a fixed pattern noise component. It has been proposed (see Patent Document 2).

  However, since removal of such fixed pattern noise requires the provision of a mechanical shutter or optical black, the imaging device is complicated and the manufacturing cost is increased. Further, it has been difficult to apply to an imaging apparatus that is difficult to provide a mechanical shutter, such as an electronic endoscope system.

  In addition, it is also known to store a fixed pattern noise component generated by shielding an image sensor during manufacturing of an imaging device, and to remove the fixed pattern noise component from image data during normal use.

  However, in order to generate and store a fixed pattern noise component in advance, it is necessary to provide a large-capacity nonvolatile memory such as an EEPROM or a flash memory. . In addition, a fixed pattern noise component generation process and a storage process are necessary at the time of manufacturing, which increases the man-hours and requires an apparatus for performing such a process in the imaging apparatus.

JP-A-8-51571 JP 2004-15712 A

  Accordingly, an object of the present invention is to provide a fixed pattern noise removing unit that removes fixed pattern noise with high accuracy with a simple configuration without simplifying the manufacturing process and without providing a mechanical shutter and optical black.

  The fixed pattern noise removal unit of the present invention includes a receiving unit that receives a pixel signal generated by a plurality of pixels arranged in an image sensor according to the amount of received light, and an exposure time of pixels in the image sensor as a first exposure time or By switching to a second exposure time shorter than the first exposure time, an imaging pixel signal that is a pixel signal corresponding to the amount of received light in the first exposure time or a pixel signal corresponding to the amount of received light in the second exposure time. A fixed pattern in a pixel corresponding to a pseudo black pixel signal based on an imaging element control unit that generates a pseudo black pixel signal, a predetermined parameter calculated by a pseudo black pixel signal generated in a different pixel, and a first threshold value A discrimination unit that discriminates whether or not it can be regarded as noise, and a noise signal based on a pseudo black pixel signal that is regarded as fixed pattern noise by the discrimination unit It is characterized in that it comprises a subtraction unit for subtracting from the image pixel signal.

  Further, the subtraction unit averages the pseudo black pixel signal regarded as fixed pattern noise by the determination unit for each row of pixels, and the average pseudo black pixel signal for each row is used as a noise signal in the corresponding row. It is preferable to subtract from the imaging pixel signal.

  The subtracting unit preferably averages a predetermined range including a median or a predetermined number of pseudo black pixel signals in the distribution range of the signal level of the pseudo black pixel signal for each row.

  The subtracting unit averages a plurality of pseudo black pixel signals regarded as fixed pattern noise by the discrimination unit for each column of pixels, and corresponds the averaged pseudo black pixel signal for each column as a noise signal. Preferably, it is subtracted from the imaging pixel signal in the column.

  The image sensor control unit causes the image sensor to perform imaging for the second exposure time a plurality of times, thereby generating a plurality of pseudo black pixel signals corresponding to each pixel, and the subtracting unit includes a plurality of pseudo black pixel signals. Preferably, the noise signal based on the pseudo black pixel signal regarded as the fixed pattern noise by the determination unit is subtracted from the imaging pixel signal.

  Further, the subtracting unit averages a plurality of pseudo black pixel signals generated by imaging at a plurality of times of the second exposure time and regarded as fixed pattern noise by the determination unit for each pixel. The averaged pseudo black pixel signal is preferably subtracted from the image pickup pixel signal for each pixel as a noise signal.

  In addition, the image sensor control unit causes the image sensor to have a second exposure time until m (m is an integer of 2 or more) pseudo black pixel signals that are regarded as fixed pattern noise by the determination unit in the same pixel. It is preferable to repeat the imaging, and the subtraction unit averages a plurality of pseudo black pixel signals in the same pixel regarded as fixed pattern noise.

  In addition, an averaging unit is provided that generates an averaged signal by averaging pseudo black pixel signals corresponding to all or some of the pixels arranged in the image sensor, and the determination unit averages based on the black pixel signal It is preferable to determine whether the signal can be regarded as fixed pattern noise.

  In addition, the imaging element control unit causes the imaging element to perform imaging for the second exposure time a plurality of times to generate a plurality of pseudo black pixel signals corresponding to each pixel, and the subtraction unit regards it as fixed pattern noise by the determination unit. Preferably, the noise signal based on the pseudo black pixel signal used to generate the averaged signal that has been deceived is subtracted from the imaging pixel signal.

  Further, the subtracting unit is a plurality of pseudo black pixel signals generated by imaging a plurality of times of the second exposure time, and is used to generate a plurality of average signals regarded as fixed pattern noise by the determining unit. It is preferable that the pseudo black pixel signal is averaged for each pixel, and the averaged pseudo black pixel signal for each pixel is subtracted from the imaging pixel signal as a noise signal.

  In addition, the image sensor control unit causes the image sensor to repeat imaging at the second exposure time until the averaged signal regarded as fixed pattern noise by the determination unit reaches m (m is an integer of 2 or more), The subtracting unit preferably averages the pseudo black pixel signal used to generate m averaged signals regarded as fixed pattern noise.

  An averaging unit that generates an averaged signal by averaging the pseudo black pixel signals corresponding to a plurality of pixels in the first and second small areas that are a part of the entire area where the pixels are arranged. Preferably, the determination unit determines, for each of the first and second small regions, whether the averaged signal can be regarded as fixed pattern noise based on the black pixel signal.

  In addition, the imaging element control unit causes the imaging element to perform imaging for the second exposure time a plurality of times to generate a plurality of pseudo black pixel signals corresponding to each pixel, and the subtraction unit regards it as fixed pattern noise by the determination unit. Preferably, the noise signal based on the pseudo black pixel signal used to generate the averaged signal that has been deceived is subtracted from the imaging pixel signal.

  Further, the subtracting unit is a plurality of pseudo black pixel signals generated by imaging a plurality of times of the second exposure time, and is used to generate a plurality of average signals regarded as fixed pattern noise by the determining unit. It is preferable that the pseudo black pixel signal is averaged for each pixel, and the averaged pseudo black pixel signal for each pixel is subtracted from the imaging pixel signal for each pixel as a noise signal.

  In addition, the image sensor control unit sets the second average image signal to the image sensor until the average signal regarded as fixed pattern noise by the determination unit becomes m (m is an integer of 2 or more) for each of the first and second small regions. It is preferable that the imaging with the exposure time is repeated, and the subtraction unit averages the pseudo black pixel signals used to generate a plurality of averaged signals regarded as fixed pattern noise.

  In addition, it is preferable that the imaging element control unit stops imaging at the second exposure time when the imaging element is imaged n times (n is an integer greater than m) for the second exposure time.

  In addition, it is preferable that the imaging element control unit generates a pseudo black pixel signal by switching the pixel exposure time to the second exposure time when the imaging apparatus provided with the fixed pattern noise removing unit is started.

  In addition, an input unit for inputting a command for generating a pseudo black pixel signal is provided, and the imaging element control unit sets the pixel exposure time to the second exposure time when a command for generating the pseudo black pixel signal is input to the input unit. Preferably, the pseudo black pixel signal is generated by switching to the above.

  The image sensor control unit sets the pixel exposure time to the second exposure time when the luminance component of the image pickup pixel signal generated by setting the pixel exposure time to the first exposure time is less than the second threshold. When the discrimination unit considers the pseudo black pixel signal as fixed pattern noise, the image sensor control unit returns the pixel exposure time to the first exposure time, Preferably, it is generated.

  Further, it is preferable to include an output unit that continues to output the most recently generated imaging pixel signal until the determination unit considers the pseudo black pixel signal as fixed pattern noise.

  The first threshold value is preferably set to a predetermined parameter when the pseudo black pixel signal can be regarded as fixed pattern noise.

  The predetermined parameter is preferably at least one of an average value, maximum value, median value, MSE, and PSNR of the signal levels of the plurality of pseudo black pixel signals.

  It is also preferable to use the standard deviation of the signal levels of the plurality of pseudo black pixel signals as the predetermined parameter.

  Further, a lens is disposed on the light receiving surface of the image sensor, and the determination unit determines whether or not the pseudo black pixel signal of a pixel disposed in a region away from the optical axis of the lens can be regarded as fixed pattern noise, It is preferable that the subtracting unit subtracts a noise signal based on the pseudo black pixel signal that the determining unit considers as fixed pattern noise from the imaging pixel signal.

  Further, it is preferable that the second exposure time is set to be less than a time period in which the integrated exposure amount to the effective pixel can be regarded as an integrated exposure amount corresponding to black.

  The second exposure time is preferably the shortest exposure time that can be set in the image sensor.

  An imaging unit of the present invention includes a plurality of pixels that output a pixel signal that is generated according to the amount of light received by the pixels, and the exposure time of the pixels in the imaging device is the first exposure time or the first exposure time. By switching to a shorter second exposure time, an imaging pixel signal that is a pixel signal corresponding to the amount of received light in the first exposure time or a pseudo black pixel signal that is a pixel signal corresponding to the amount of received light in the second exposure time The pseudo black pixel signal can be regarded as fixed pattern noise in the corresponding pixel based on the imaging element control unit to be generated and a predetermined parameter calculated by the pseudo black pixel signal generated in a different pixel and the first threshold value. And a subtractor for subtracting a noise signal based on a pseudo black pixel signal regarded as fixed pattern noise by the discriminator from the imaging pixel signal. It is characterized in that it comprises a part.

  The electronic endoscope system of the present invention includes a plurality of pixels that output a pixel signal that is generated according to the amount of light received by the pixels, and the exposure time of the pixels in the imaging device is the first exposure time or the first exposure time. By switching to a second exposure time shorter than the exposure time, an imaging pixel signal that is a pixel signal corresponding to the amount of light received in the first exposure time or a pseudo black pixel that is a pixel signal corresponding to the amount of light received in the second exposure time The pseudo black pixel signal is regarded as a fixed pattern noise in a corresponding pixel based on a predetermined parameter and a first threshold value calculated by a pseudo black pixel signal generated in a pixel different from the image sensor control unit that generates the signal. A subtracting unit that determines whether or not to generate a noise signal based on a pseudo black pixel signal that is regarded as fixed pattern noise by the determining unit. It is characterized by having an imaging unit and a part.

  According to the present invention, it is possible to remove fixed pattern noise with high accuracy with a simple configuration without providing a mechanical shutter, optical black, and nonvolatile memory while simplifying the manufacturing process.

1 is a block diagram schematically showing an internal configuration of an electronic endoscope system having a fixed pattern noise removal unit to which a first embodiment of the present invention is applied. FIG. It is a block diagram which shows roughly the internal structure of the image process part of 1st Embodiment. It is a flowchart for demonstrating the process of the removal of the FPN component in 1st Embodiment. It is a flowchart for demonstrating the subroutine of noise data generation in 1st Embodiment. It is a block diagram which shows roughly the internal structure of the image process part of 2nd Embodiment. It is a flowchart for demonstrating the process of the removal of FPN component in 2nd Embodiment. It is a 1st flowchart for demonstrating the subroutine of noise data generation in 2nd Embodiment. It is a 2nd flowchart for demonstrating the subroutine of the noise data generation in 2nd Embodiment. It is a figure which shows the positional relationship of an image pick-up element and an objective lens, in order to show the area | region where the pixel corresponding to the pseudo | simulation black pixel data used for generation | occurrence | production of noise data in 3rd Embodiment is arrange | positioned.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram schematically showing an internal configuration of an electronic endoscope system having a fixed pattern noise removing unit to which the first embodiment of the present invention is applied.

  The electronic endoscope system 10 includes an electronic endoscope 20, an endoscope processor 40, and a monitor 11. The endoscope processor 40 is connected to the electronic endoscope 20 and the monitor 11.

  The endoscope processor 40 is provided with a light source unit (not shown). Illumination light emitted from the light source unit is transmitted to the distal end of the insertion tube 22 by a light guide (not shown) extending from the connector 21 of the electronic endoscope 20 to the distal end of the insertion tube 22.

  Illumination light transmitted by the light guide is irradiated near the tip of the insertion tube 22. The subject irradiated with the illumination light is imaged by the electronic endoscope 20. An image signal generated by imaging of the electronic endoscope 20 is sent to the endoscope processor 40.

  The endoscope processor 40 includes an image processing circuit 41, and predetermined signal processing is performed on an image signal obtained from the electronic endoscope 20 in the image processing circuit 41. The image signal subjected to the predetermined signal processing is transmitted to the monitor 11, and an image corresponding to the transmitted image signal is displayed on the monitor 11.

  The operation of the light source unit and image processing circuit 41 is controlled by a system controller 42 provided in the endoscope processor 40. The system controller 42 is also connected to the electronic endoscope 20 and controls the operation of each part of the electronic endoscope 20.

  Note that the endoscope processor 40 is provided with an input unit 43. When a command from the user is input, an order signal corresponding to the input command is transmitted from the input unit 43 to the system controller 42. The system controller 42 controls the operation of each part according to the received order signal.

  The electronic endoscope 20 includes an objective lens 23, an image sensor 24, an A / D converter 25 (receiver), an image processor 30, a D / A converter 26, a DRAM 27, a timing controller 28 (image sensor controller), and the like. Provided.

  As described above, the optical image of the subject irradiated with the illumination light reaches the light receiving surface of the image sensor 24 through the objective lens 23. Based on the driving of the timing controller 28, the image sensor 24 generates an image signal corresponding to the optical image that has reached the light receiving surface.

  The image sensor 24 is a CMOS image sensor, and a plurality of pixels are arranged in a matrix on the light receiving surface. In each pixel, a pixel signal corresponding to the amount of received light is generated.

  The pixel signal generated in each pixel has a fixed pattern noise (FPN) component and a received light amount component. The FPN component is a noise component unique to each pixel, and has a constant magnitude regardless of the exposure time. The received light amount component has a magnitude corresponding to the integrated exposure amount to the pixel, that is, the product of the amount of incident light and the exposure time.

  In general, the signal level of the FPN component unique to a pixel tends to change depending on the row and column in which the pixel is arranged. Therefore, the FPN component of each pixel has a row direction component and a column direction component. The row direction component is common to pixels arranged in the same row, and the column direction component is common to pixels arranged in the same column.

  The output timing of the pixel signal generated by each pixel is controlled by the timing controller 24, and the pixel signal is sequentially output from the image sensor 24. An image signal of one frame is constituted by pixel signals generated by all the pixels arranged on the light receiving surface.

  As will be described later, the timing controller 28 can also control the exposure time of the pixels. Further, the timing controller 28 also controls the operation timing in the A / D converter 25 and the image processing unit 30.

  The generated image signal is transmitted from the image sensor 24 to the A / D converter 25. A / D conversion is performed by the A / D converter 25, and the analog image signal is converted into digital image data. The image data is constituted by pixel data obtained by A / D converting the pixel signal.

  The image data is transmitted to the image processing unit 30. In the image processing unit 30, predetermined image processing including FPN removal is performed on the image data. Note that the image processing unit 30 is connected to the DRAM 27. The DRAM 27 stores pixel data for FPN removal, as will be described later.

  Image data that has undergone predetermined image processing is transmitted to the D / A converter 26. D / A conversion is performed by the D / A converter 26, and the digital image data is converted into an analog image signal. The converted image signal is transmitted to the image processing circuit 41 of the endoscope processor 40. As described above, the image processing circuit 41 performs predetermined image processing on the image signal. After the image processing, the image signal is transmitted to the monitor 11 and an image corresponding to the image signal is displayed on the monitor 11.

  Next, the configuration of the image processing unit 30 and the FPN removal executed in the image processing unit 30 will be described. As shown in FIG. 2, the image processing unit 30 includes a region averaging circuit 31 (averaging unit), a discrimination circuit 32 (discrimination unit), an FPN creation circuit 33 (subtraction unit), and an FPN correction circuit 34 (subtraction unit, output). Part), a general image processing circuit 35, and the like.

  In order to remove the FPN component from the pixel signal, a noise signal that is an FPN component unique to each effective pixel is required. As described above, the FPN component has a row direction component and a column direction component, and the row direction component and the column direction component of the noise signal are necessary for each row and for each column in order to remove the FPN. Before the FPN removal, the image processing unit 30 generates row direction noise data and column direction noise data.

  In order to generate noise data, the timing controller 28 causes the image sensor 24 to generate an image signal with the shortest exposure time that can be set, for example, an exposure time of 200 ns, when the electronic endoscope 20 is activated. When the exposure time is 200 ns, the generated pixel signal is transmitted to the area averaging circuit 31 as pseudo black pixel data.

  The area averaging circuit 31 stores the received pseudo black pixel data in the DRAM 27. In the DRAM 27, a storage area of each pixel is determined, and pseudo black pixel data received in order is stored in the determined storage area.

  When the pseudo black pixel data constituting one frame of image data is stored in the DRAM 27, all the pseudo black pixel data constituting one frame of image data is read out by the area averaging circuit 31, and the luminance data component is averaged. Is done. In addition, the standard deviation of the luminance data component is calculated by the area averaging circuit 31.

  Average pseudo black data and standard deviation data, which are averaged pseudo black pixel data, are transmitted to the determination circuit 32. The determination circuit 32 has a ROM (not shown), and stores reference luminance range data and reference standard deviation data for comparing the average value and standard deviation of luminance data components. In the discrimination circuit 32, the average pseudo black data and the reference luminance range data are compared, and the standard deviation data and the reference standard deviation data are compared.

  The determination circuit 31 determines whether or not the data level of the average pseudo black data is within the data level range of the reference luminance range data. Further, the determination circuit 31 determines whether or not the data level of the standard deviation data is lower than the data level of the reference standard deviation data.

  The pseudo black pixel data, that is, the pixel data includes the FPN component and the received light amount component as described above. As described above, the received light amount component has a magnitude corresponding to the product of the amount of incident light and the exposure time. Therefore, if the incident light amount and the exposure time are sufficiently short, a pseudo black pixel signal is generated in which the signal level of the received light amount component can be regarded as zero. However, even if the exposure time is sufficiently shortened, if the amount of incident light is large, a pseudo black pixel signal including a received light amount component having a signal level greater than zero is generated.

  Therefore, the discrimination circuit 32 compares the average pseudo black data with the reference luminance range data and the standard deviation data with the reference standard deviation data, so that the amount of light incident on the image sensor 24 is sufficiently small and the pseudo black pixel signal is substantially reduced. In particular, it is possible to determine whether or not it has only the FPN component.

  If the data level of the average pseudo black data is outside the range of the data level of the reference luminance range data, or the data level of the standard deviation data exceeds the data level of the standard standard deviation data, the average pseudo black data is generated. The used frame image data is erased from the DRAM 27.

  On the other hand, if the data level of the average pseudo black data is within the range of the data level of the reference luminance range data and the data level of the standard deviation data is less than the data level of the standard standard deviation data, the generation of the average pseudo black data is generated. The image data of the frame used for is stored in the DRAM 27 as it is.

  When the storage of one frame of image data in the DRAM 27, the averaging in the area averaging circuit 31 and the calculation of the standard deviation, and the determination in the determination circuit 32 are completed, the timing controller 28 again images the image sensor 24 with an exposure time of 200 ns. Generate a signal.

  As described above, the newly generated image data of one frame is stored in the DRAM 27, averaged in the area averaging circuit 31 and calculated as the standard deviation, and determined by the determination circuit 32. Until the image data not erased by the discrimination circuit 32 reaches 20 frames, the data is stored in the DRAM 27, averaged by the area averaging circuit 31, and discriminated by the discrimination circuit 32. When the image data that is not erased by the determination circuit 32 reaches 20 frames, imaging with an exposure time of 200 ns is stopped.

  The 20 frames of image data that have not been erased by the discrimination circuit 32 are read out to the FPN creation circuit 33. The FPN generation circuit 33 generates row direction noise data and column direction noise data for each row and each column.

  In order to generate row direction noise data and column direction noise data, 20 pseudo black pixel data corresponding to the same pixel and having different frames are averaged. Next, using the averaged pseudo black pixel data, row direction noise data and column direction noise data are generated in the following procedure.

  The averaged pseudo black pixel data is extracted for each row and each column. Next, the extracted pseudo black pixel data averaged for each row and each column is rearranged in ascending order of data levels. Next, a predetermined number of averaged pseudo black pixel data including the median value of the rearranged averaged pseudo black pixel data is extracted. For example, when 600 × 600 pixels are arranged, pseudo-black pixel data of 200 pixels including the median is extracted for each row and each column.

  Next, the average value of the extracted 200-pixel averaged pseudo black pixel data is calculated. The calculated average value is generated as row direction noise data and column direction noise data corresponding to each row and each column. The generated row direction noise data and column direction noise data are stored in the DRAM 27.

  When the noise data is stored in the DRAM 27, the timing controller 28 causes the image sensor 24 to generate an image signal with an exposure time for taking a normal image, for example, an exposure time of 1 / 60s. As described above, each pixel signal constituting an image signal of one frame is sequentially transmitted to the image processing unit 30 via the A / D converter 25 as an imaging pixel signal.

  When the exposure time is 1/60 s, the imaged pixel data generated and A / D converted is transmitted to the FPN correction circuit 34. The FPN correction circuit 34 reads out row direction noise data and column direction noise data corresponding to the row and column of the received imaging pixel data from the DRAM 27. The FPN component is removed by subtracting the row direction noise data and the column direction noise data from the imaged pixel data by the FPN correction circuit 34.

  The corrected imaging pixel data from which the FPN component has been removed is transmitted to the general image processing circuit 35, and subjected to predetermined image processing such as color interpolation processing and gamma correction processing. The corrected imaging pixel data subjected to the predetermined image processing is transmitted to the D / A converter 26 as described above.

  Next, noise data generation and FPN component removal processing from the imaged pixel data performed by the timing controller 28 and the image processing unit 30 will be described with reference to the flowcharts of FIGS. 3 and 4.

  FIG. 3 is a flowchart for explaining processing for removing the FPN component. FIG. 4 is a flowchart for explaining a subroutine for generating row direction noise data and column direction noise data. The process of removing the FPN component is started when the electronic endoscope system 10 is turned on.

  When the electronic endoscope 20 is activated in step S100, the process proceeds to step S200, and row direction noise data and column direction noise data are generated for each row and each column. When the generation of noise data ends, the process proceeds to step S101.

  In step S101, the image pickup device 24 picks up an image of a subject and generates a one-frame image signal. As will be described later, the exposure time of the image sensor 24 is set to 1/60 s. When one frame image signal is generated, the process proceeds to step S102.

  In step S102, the FPN component is removed from the image signal generated in step S101. That is, by subtracting the row direction noise data and the column direction noise data stored in the DRAM 27 in step S200 from each imaging pixel data constituting image data obtained by A / D converting the image signal, an FPN component unique to each imaging pixel data is obtained. Remove. When the FPN component of all the imaging pixel data is removed, the process proceeds to step S103.

  In step S103, predetermined image processing such as color interpolation processing and gamma processing is performed on the image data from which the FPN component has been removed. When predetermined image processing is performed, the process proceeds to step S104, and it is determined whether or not a command for ending the moving image display is input.

  When the command for ending the moving image display has not been input, the process returns to step S101, and steps S101 to S104 are repeated until the command for ending the moving image display is input. When the command for ending the moving image display is input, the FPN component removal processing is ended.

  Next, a noise data generation subroutine will be described. As described above, after activation of the electronic endoscope 20 (step S100), a noise data generation subroutine is started. In step S201, the exposure time of the image sensor 24 is set to 200 ns. After setting the exposure time of the image sensor 24, the process proceeds to step S202.

  In step S <b> 202, the image pickup device 24 picks up an image of the subject to generate one frame of image signal. In the next step S203, pseudo black pixel data constituting image data obtained by A / D converting the image signal is stored in the DRAM 27. When all the pseudo black pixel data constituting one frame of image data is stored in the DRAM 27, the process proceeds to step S204.

  In step S204, average pseudo black data is generated by reading all pseudo black pixel data from the DRAM 27 and averaging the luminance data components. Further, the standard deviation data is generated by calculating the standard deviation of the luminance data. After calculating the average pseudo black data and the standard deviation data, the process proceeds to step S205.

  In step S205, it is determined whether or not the data level of the average pseudo black data generated in step S204 is within the data level range of the reference luminance range data. If it is within the range, the process proceeds to step S206. On the other hand, if it is out of range, the process proceeds to step S207.

  In step S206, it is determined whether or not the data level of the standard deviation data generated in step S204 is less than the data level of the reference standard deviation data. If the data level of the standard deviation data exceeds the data level of the reference standard deviation data, the process proceeds to step S207. If the data level of the standard deviation data is less than the data level of the standard standard deviation data, the process proceeds to step S208.

  In step S207, the pseudo black pixel data stored in the DRAM 27 in the most recent step S203 is erased. After erasing data, the process returns to step S202. Thereafter, in step S205 and step S206, until the data level of the average pseudo black data falls within the range of the data level of the reference luminance range data and the data level of the standard deviation data becomes less than the data level of the reference standard deviation data, The processing from step S202 to step S207 is repeated.

  In step S208, it is determined whether or not 20 frames of image data are stored in the DRAM 27. If 20 frames of image data are not stored, the process returns to step S202. Thereafter, the processing from step S202 to step S208 is repeated until it is determined in step S208 that 20 frames of image data are stored. If 20 frames of image data are stored, the process proceeds to step S209.

  In step S209, the pseudo black pixel data constituting the 20-frame image signal stored in the DRAM 27 is averaged for each pixel. When the pseudo black pixel data is averaged over all the pixels, the process proceeds to step S210.

  In step S210, row direction noise data and column direction noise data are generated for each row and each column. As described above, the average pseudo black pixel data of each row is rearranged in ascending order of the data level, and the average pseudo black pixel data of 200 pixels including the median is averaged to generate row direction noise data. Further, the averaged pseudo black pixel data of each column is rearranged in ascending order of the data level, and the averaged pseudo black pixel data of 200 pixels including the median is averaged to generate column direction noise data.

  The generated noise data is stored in the DRAM 27. When the row direction noise data and the column direction noise data of all rows and all columns are generated and stored, the process proceeds to step S211.

  In step S211, the exposure time of the image sensor 24 is set to 1/60 s. After setting the exposure time of the image sensor 24, the process proceeds to step S101 as described above.

  According to the fixed pattern noise removal unit to which the first embodiment as described above is applied, it is possible to remove the FPN without providing a mechanical shutter, an optical black, and a large-capacity nonvolatile memory.

  Further, as described above, when the amount of incident light on the image sensor 24 is large when generating the pseudo black pixel signal, the FPN can be removed with high accuracy even if the pseudo black pixel signal generated by shortening the exposure time is used. It was difficult to do. However, in the fixed pattern noise removal unit of the present embodiment, it is determined in advance whether or not it is appropriate to use the pseudo black pixel signal for removal of the fixed pattern noise based on the luminance range assumed as the FPN component. Therefore, it is possible to remove fixed pattern noise with high accuracy.

  Further, even if the pseudo-black pixel signal includes a non-negligible received light amount component in some pixels, the average value of the luminance data components of all pixels may be included in the range of the reference luminance range data. . Although the image signal of such a frame is not suitable for noise removal, the fixed pattern noise removal unit of the present embodiment compares the standard standard deviation with the standard deviation to remove the pseudo black pixel signal from the fixed pattern noise. Since it is determined whether or not it is appropriate to use, it is possible to remove fixed pattern noise with higher accuracy.

  Next, a fixed pattern noise removing unit to which the second embodiment of the present invention is applied will be described. The second embodiment differs from the first embodiment in the generation time and generation method of noise data. Hereinafter, the second embodiment will be described with a focus on portions different from the first embodiment. In addition, the same code | symbol is attached | subjected to the site | part which has the same function.

  Configurations and functions of parts other than the timing controller, the image processing unit, and the DRAM of the electronic endoscope of the second embodiment are the same as those of the electronic endoscope of the first embodiment. In the second embodiment, the endoscope processor and the monitor are the same as the endoscope processor and the monitor in the first embodiment.

  A description will be given of FPN removal executed in the image processing unit, along with the configuration and functions of the image processing unit of the second embodiment. As shown in FIG. 5, the image processing unit 300 includes a region averaging circuit 310, a determination circuit 320, an FPN creation circuit 33, an FPN correction circuit 34, a general image processing circuit 35, and the like.

  Similar to the first embodiment, the row direction noise data and the column direction noise data are generated by the image processing unit 300 before the FPN removal. Unlike the first embodiment, noise data is not generated when the electronic endoscope 20 is activated, and a time suitable for noise data generation is determined during observation by the electronic endoscope 20, and noise data is generated at that time. Is done.

  As will be described below, the luminance of the image signal is calculated, and a time suitable for noise data generation is determined based on the calculated luminance.

  After the electronic endoscope 20 is activated, the timing controller 28 causes the image sensor 24 to generate an image signal with an exposure time for capturing a normal image, for example, an exposure time of 1/60 s. Each pixel signal constituting one frame of image signal is transmitted to the image processing unit 300 via the A / D converter 25 as an imaging pixel signal.

  When the exposure time is 1/60 s and no noise data is stored in the DRAM 27, the generated and A / D converted image data is transmitted to the area averaging circuit 310 and the FPN correction circuit 34.

  Note that since no noise data is stored in the DRAM 27 until noise data is generated after the electronic endoscope 20 is started up, imaging pixels until noise data generation is started after the electronic endoscope 20 is started up. The data is transmitted to the area averaging circuit 310 and the FPN correction circuit 34.

  In the FPN correction circuit 34, the row direction noise data and the column direction noise data stored in the DRAM 27 are subtracted from the image pickup pixel data as in the first embodiment. However, since noise data is not stored in the DRAM 27 until noise data is generated after the electronic endoscope 20 is activated, an FPN correction circuit is generated until generation of noise data is started after the electronic endoscope 20 is activated. In 34, the noise data is output without being subtracted from the imaged pixel data.

  The area averaging circuit 310 stores the received imaging pixel data in the DRAM 27. When imaging pixel data constituting one frame of image data is stored in the DRAM 27, the area averaging circuit 310 reads all the imaging pixel data constituting one frame of image data and averages the luminance data components. .

  The averaged luminance data component is transmitted to the discrimination circuit 320. The determination circuit 320 has a ROM and stores brightness threshold data for comparison with the brightness data. The discrimination circuit 320 compares the averaged luminance data with the luminance threshold data.

  It is determined whether the data level of the luminance data averaged by the determination circuit 320 is less than the data level of the luminance threshold data. When noise data is generated by shortening the exposure tube, it is preferable that light with low intensity is incident on the image sensor 24. Therefore, when the data level of the averaged luminance data is lower than the data level of the luminance threshold data, it can be determined that the time is suitable for noise data generation.

  When the data level of the averaged luminance data is lower than the data level of the luminance threshold data, the timing controller 28 uses the shortest exposure time that can be set for noise data generation, for example, an exposure time of 200 ns, and the image sensor 24. To generate an image signal.

  The pseudo black pixel data constituting the image data obtained by A / D converting the image signal is not transmitted to the FPN correction circuit 34 but is transmitted only to the area averaging circuit 310. As in the first embodiment, the area averaging circuit 310 stores the received pseudo black pixel data in the DRAM 27.

  Unlike the first embodiment, when the pseudo black pixel data constituting one frame of image data is stored in the DRAM 27, a part of the pseudo black pixel data is read out by the area averaging circuit 310, and the luminance data component is averaged. It becomes.

  A part of the pseudo black pixel data read by the area averaging circuit 310 will be described. First to 256 small regions are defined in which the light receiving surfaces of the image sensor 24 are classified into 256 vertical and horizontal 16 × 16. The pseudo black pixel data of the pixels arranged in the first small area is read out to the area averaging circuit 310, and the luminance data component is averaged. In addition, the area average circuit 310 calculates the standard deviation of the luminance data components of the pixels arranged in the first small area.

  The first standard corresponding to the standard deviation of the luminance component of the average pseudo black pixel data of the first small region and the luminance component of the first black pseudo pixel data of the first small region The deviation data is transmitted to the determination circuit 320.

  As in the first embodiment, the determination circuit 320 has a ROM (not shown) and stores reference luminance range data and reference standard deviation data. As in the first embodiment, the determination circuit 320 compares the first average pseudo black data with the reference luminance range data, the standard deviation data, and the reference standard deviation data.

  Similar to the first embodiment, the data level of the first average pseudo black data is outside the range of the data level of the reference luminance range data, or the data level of the standard deviation data is the data level of the reference standard deviation data. When exceeding, the pseudo black pixel data of the pixels arranged in the first small area is erased from the DRAM 27.

  Similarly to the first embodiment, the data level of the first average pseudo black data is within the data level range of the reference luminance range data, and the data level of the first standard deviation data is the reference standard deviation data. If it is less than the data level, the image data of the frame used to generate the first average pseudo black data is stored in the DRAM 27 as it is.

  Similar to the first average pseudo black data corresponding to the first small area, second to 256 average pseudo black data corresponding to the second to 256 small areas are generated, and the pseudo average black data is generated for each small area. It is determined whether or not the black pixel data is erased.

  As in the first embodiment, when the discrimination by the discrimination circuit 320 is completed for the average pseudo black data corresponding to all small regions, the timing controller 28 causes the image sensor 24 to generate an image signal again with an exposure time of 200 ns. .

  Thereafter, until the pseudo black pixel data of each small area that is not erased by the discrimination circuit 320 reaches 20 frames, the data is stored in the DRAM 27, the average and the standard deviation are calculated by the area averaging circuit 310, and the discrimination by the discrimination circuit 320 is performed. It is. When the pseudo black pixel data of each small area that is not erased by the determination circuit 320 reaches 20 frames, imaging with an exposure time of 200 ns is stopped.

  However, even if the pseudo black pixel data that is not erased by the determination circuit 320 does not become 20 frames, imaging with an exposure time of 200 ns is stopped even when an image signal of a predetermined number of frames, for example, 40 frames is created.

  Similar to the first embodiment, the FPN creation circuit 33 generates row direction noise data and column direction noise data for each row and each column. Note that, unlike the first embodiment, 20 frames of pseudo black pixel data may not be stored in some pixels. Pixels in which 20 frames of pseudo black pixel data are not stored are excluded from the generation of noise data. As in the first embodiment, the generated noise data is stored in the DRAM 27.

  Unlike the first embodiment, the image signal generated by imaging with the exposure time of 1 / 60s immediately before switching to the exposure time of 200 ns from the start of the generation of noise data until the generation of noise data is completed is FPN. Output from the correction circuit 34. As described above, by repeatedly outputting the image signal with the exposure time of the latest 1/60 s, it becomes possible to continuously display the image on the monitor 11.

  As in the first embodiment, when the row direction noise data and the column direction noise data are stored in the DRAM 27, the timing controller 28 causes the image sensor 24 to generate an image signal with an exposure time of 1/60 s. In addition, each imaging pixel signal constituting one frame of image signal is sequentially transmitted to the image processing unit 300 via the A / D converter 25.

  As in the first embodiment, when the exposure time is 1/60 s, the generated and A / D converted imaging pixel data is transmitted to the FPN correction circuit 34. As in the first embodiment, the FPN component is removed by subtracting the row direction noise data and the column direction noise data stored in the DRAM 27 from the imaging pixel data.

  Next, noise data creation processing and FPN component removal processing from the imaging pixel data performed by the timing controller 28 and the image processing unit 300 will be described with reference to the flowcharts of FIGS.

  FIG. 6 is a flowchart for explaining processing for removing the FPN component. FIG. 7 is a first flowchart for explaining a subroutine of noise data generation. FIG. 8 is a second flowchart for explaining a subroutine of noise data generation. The process of removing the FPN component is started when the electronic endoscope system 10 is turned on.

  When the electronic endoscope 20 is activated in step S300, the process proceeds to step S301, and the exposure time of the image sensor 24 is set to 1/60 s. After setting the exposure time of the image sensor 24, the process proceeds to step S302.

  In step S <b> 302, the image pickup device 24 picks up an image of the subject and generates a one-frame image signal. When the image signal is generated, the process proceeds to step S303, and it is determined whether or not row direction noise data and column direction noise data are stored in the DRAM 27. If no noise data is stored, the process proceeds to step S304. If noise data is stored, the process proceeds to step S305.

  In step S304, it is determined whether or not the average luminance of the entire image signal generated in step S302 is less than the luminance threshold. When the average luminance is less than the luminance threshold, the process proceeds to step S400, and row direction noise data and column direction noise data are generated for each row and each column. When the average luminance exceeds the luminance threshold value or after generating the noise data in step S400, step S305 is skipped and the process proceeds to step S306.

  In step S305, the FPN component is removed from the image signal generated in step S302. That is, the FPN component is removed by subtracting the row direction noise data and the column direction noise data stored in the DRAM 27 in step S400 from each image pickup pixel data constituting the image data obtained by A / D converting the image signal. When the FPN component of all the imaging pixel data is removed, the process proceeds to step S306.

  Thereafter, the same processing as Step S103 and Step S104 in the first embodiment is executed in Step S306 and Step S307.

  Next, a noise data generation subroutine will be described. As described above, when the average luminance is less than the luminance threshold value in step S304, the noise data generation subroutine is started. In steps S401 to S403, the same processing as in steps S101 to S103 in the first embodiment is executed.

  In step S404, the number n of the small area for generating the average pseudo black data is set to 1. After setting the small area number n, the process proceeds to step S405. In step S405, the pseudo-black pixel data of all pixels arranged in the n-th small area is read from the DRAM 27, and the luminance data component is averaged to generate the n-th average pseudo-black data. Further, the standard deviation of the pseudo black pixel data of all the pixels arranged in the nth small region is calculated. After the generation of the n-th average pseudo black data and standard deviation data, the process proceeds to step S406.

  In step S406, it is determined whether or not the data level of the average pseudo black data generated in step S405 is within the range of the data level of the reference luminance range data. If it is within the range, the process proceeds to step S407. On the other hand, if it is out of range, the process proceeds to step S408.

  In step S407, it is determined whether or not the data level of the standard deviation data generated in step S405 is less than the data level of the reference standard deviation data. If the data level of the standard deviation data exceeds the data level of the reference standard deviation data, the process proceeds to step S408. If the data level of the standard deviation data is less than the data level of the standard standard deviation data, the process proceeds to step S409.

  In step S408, the pseudo black pixel data corresponding to the nth small region is erased from the pseudo black pixel data stored in the DRAM 27 in the most recent step S403. After erasing data, the process proceeds to step S409.

  In step S409, it is determined whether the small area number n is 256 or not. If n is not 256, the process proceeds to step S410, 1 is added to n, and the process returns to step S405. Thereafter, the processing from step S405 to step S410 is repeated until n becomes 256 in step S409. If n is 256 in step S409, the process proceeds to step S411.

  In step S411, it is determined whether or not the imaging operation in step S402 has reached 40 times. If it has not reached 40, the process proceeds to step S412. If the number has reached 40, step S412 is skipped and the process proceeds to step S413.

  In step S 412, it is determined whether or not 20 frames of pseudo black pixel data are stored in the DRAM 27 in all small areas. If there is a small region in which 20 frames of pseudo black pixel data are not stored, the process returns to step S402. Thereafter, the processing from step S402 to step S412 is repeated until 20 frames of pseudo black pixel data are stored in all the small areas. If pseudo black pixel data of 20 frames is stored in all small areas, the process proceeds to step S413.

  In subsequent steps S413 to S415, the same processing as in steps S209 to S211 in the first embodiment is executed to generate noise data, store it in the DRAM 27, and set the exposure time to 1/60 s. After setting the exposure time, the process proceeds to step S416.

  In step S416, the image signal with the exposure time of 1 / 60s most recently generated in step S302 is repeatedly output. After outputting the image signal whose exposure time is 1/60 s, the process proceeds to step S306 as described above.

  The same effect as that of the first embodiment can also be obtained by the fixed pattern noise removing unit to which the second embodiment as described above is applied.

  In addition, according to the present embodiment, the time suitable for generating the noise data during the observation by the electronic endoscope 20 is determined and the generation of the noise data is started, which is sufficient to generate the noise data. The pseudo black pixel data of the frame can be acquired at an early stage.

  Further, according to the present embodiment, it is possible to efficiently generate noise data even when the amount of incident light is partially large in the image sensor 24. For example, when the amount of incident light is partially large in the image sensor 24, the pseudo black pixel data in other regions may not be used for generating noise data even if the received light amount component is not included. However, according to the present embodiment, since it is determined whether or not to use noise data for each small region, the speed of noise data generation can be increased.

  In addition, according to the present embodiment, since the imaging with an exposure time of 200 ns is stopped after imaging 40 times, generation of noise data is started even if pseudo black pixel data of a necessary frame is not stored. . Therefore, the speed of noise data generation can be increased.

  Next, a fixed pattern noise removing unit to which the third embodiment of the present invention is applied will be described. The third embodiment is different from the first embodiment in that the position of a pixel that generates a pseudo black pixel signal used for generating noise data is limited. Hereinafter, the third embodiment will be described with a focus on portions different from the first embodiment. In addition, the same code | symbol is attached | subjected to the site | part which has the same function.

  In the third embodiment, functions other than the area averaging circuit are the same as those in the first embodiment. The area averaging circuit according to the third embodiment stores, in the DRAM 27, only pseudo black pixel data corresponding to pixels arranged in a specific area, as will be described below, among the received pseudo black pixel data.

  As shown in FIG. 9, the positions of both are adjusted so that the optical axis of the objective lens 23 and the center C of the image sensor 24 overlap. Only the pseudo black pixel data in the pixels arranged in the area around the edge of the image sensor 24 (see the shaded area in FIG. 9) is used to generate noise data.

  As described above, when generating ideal pseudo black pixel data, it is desirable that the amount of light incident on the pixel is low. The amount of light transmitted through the objective lens decreases as the angle between the emission direction and the optical axis increases, that is, as the distance from the center of the image sensor 24 increases.

  Therefore, by using the pseudo black pixel data generated by the pixels around the edge of the image sensor 24, noise data that does not substantially include the received light amount component is generated even if the amount of incident light to the image sensor 24 is large. It becomes possible.

  In the first to third embodiments, the row direction noise data and the column direction noise data are generated and subtracted from the imaging pixel data. However, only one of the noise data is generated and subtracted. Also good. Even if only noise data in a direction in which streak noise is strongly generated is subtracted from the imaging pixel data, the FPN component can be sufficiently removed.

  In the first to third embodiments, the noise data is generated using a predetermined number of pseudo black pixel data including the median value of all the pseudo black pixel data in each row and each column. You may use the pseudo black pixel data which is the data level of the predetermined range centering on a value.

  In the first to third embodiments, the row direction noise data and the column direction noise data are generated using pseudo black pixel data of a part of each row and each column. Single pseudo black pixel data may be used, or may be generated using all the pseudo black pixel data for each row and each column. However, the accuracy of noise data is improved by using a plurality of pseudo black data instead of a single one. Further, even when some pseudo black pixel data has a light reception amount component, by using some pseudo black pixel data, the pseudo black pixel data having the light reception amount component is excluded from the generation of noise data. It is possible.

  In the first to third embodiments, the row direction noise data and the column direction noise data are generated for each row and for each column and subtracted from the imaging pixel data. However, the noise data is generated for each pixel. Thus, it may be configured to subtract from the imaged pixel data.

  Further, in the first and third embodiments, the pseudo black pixel data of all the pixels is averaged, and in the second embodiment, the light receiving surface is divided into a plurality of small areas, and the pseudo black pixel data for each small area. Is averaged to determine whether or not the received light amount component is sufficiently small, but may be performed for each pixel.

  In the first to third embodiments, noise data is generated by averaging 20 frames of pseudo black pixel data. However, the number of frames to be averaged is not limited to 20. A plurality of pseudo black pixel data may be averaged, or a single pseudo black pixel data may be used as noise data. However, the accuracy of the noise data improves as the pseudo black pixel data for generating the noise data increases.

  In the first to third embodiments, noise data is generated by averaging 20 frames of pseudo black pixel data. However, the method of generating noise data is not limited to averaging. Noise data may be generated by other methods using a plurality of pseudo black pixel data.

  Further, in the second embodiment, the imaging is performed 40 times with the exposure time of 200 ns, that is, the imaging with the exposure time of 200 ns is stopped after the generation of the image signal of 40 frames, but how many times the imaging is performed. It is not necessary to stop the imaging until 20 frames of pseudo black pixel data are stored without being set. However, in order to speed up the generation of noise data as described above, it is preferable to stop imaging with a short exposure time after a predetermined number of imaging. Moreover, the structure which stops the imaging of such short-time exposure at 40 times is applicable also to the 1st, 3rd embodiment.

  In the first and third embodiments, immediately after the activation of the electronic endoscope 20, in the second embodiment, noise data is automatically generated and stored in the DRAM 27 while the subject is being observed by the electronic endoscope. In any of the embodiments, generation of noise data may be automatically started immediately after the electronic endoscope 20 is activated and during observation of the subject. Furthermore, generation of noise data may be started based on a noise data generation command input to the input unit 43.

  In the second embodiment, it is configured to determine the appropriate time for generating the noise data and start generating the noise data. However, the generation of the noise data is started without determining the appropriate time. Also good.

  In the second embodiment, after the start of generation of noise data, until the storage in the DRAM 27 is completed, image data having imaging pixel data of the latest frame, that is, 1/60 s is generated as the exposure time. The image data is repeatedly output from the FPN correction circuit 34, but may not be repeatedly output. However, since no image is displayed on the monitor 11, it is preferable that the latest image data is continuously output.

  In the first to third embodiments, the average value of the data levels of the plurality of pseudo black pixel data is calculated, and it is determined whether the average value is within the reference luminance range. Alternatively, other parameters may be calculated and compared with a threshold value to determine whether the pixel data is pseudo black pixel data suitable for noise data generation. For example, the maximum value, median value, PSNR, MSE, etc. may be calculated. When PSNR or MSE is used, for example, PSNR or MSE is calculated on the basis of the average value of the data levels of a plurality of pseudo black pixel data.

  In the first to third embodiments, the standard deviation of the data level of the plurality of pseudo black pixel data is calculated and compared with the reference standard deviation. However, the standard deviation and the reference standard deviation are not compared. May be. However, as described above, the accuracy of the noise data can be improved by comparing the standard deviation with the reference standard deviation.

  In the first to third embodiments, the image sensor 24 is driven so as to capture an image with the shortest exposure time that can be set in the image sensor 24 for noise data generation. It may be set to an exposure time that can be regarded as a value when. Since the integrated exposure amount is also affected by the luminance of the subject itself, it can be used for generation of noise data if the image is taken with an exposure time that is at least shorter than the exposure time during which normal photographing is possible.

  In the first to third embodiments, the fixed pattern noise removal unit is configured to be applied to an electronic endoscope system, but may be applied to other imaging devices such as a digital still camera and a digital video camera. Good.

  In the first to third embodiments, the image pickup device 24 is a CMOS image pickup device. However, the image pickup device 24 is also applied to other types of image pickup devices in which an amplifier is provided in each pixel and pixel-specific fixed pattern noise occurs. Is possible.

DESCRIPTION OF SYMBOLS 10 Electronic endoscope system 20 Electronic endoscope 23 Objective lens 24 Image pick-up element 25 A / D converter 27 DRAM
28 Timing controller 30, 300 Image processing unit 31, 310 Area averaging circuit 32, 320 Discrimination circuit 33 FPN creation circuit 34 FPN correction circuit 40 Endoscope processor 42 System controller 43 Input unit

Claims (28)

A receiving unit that receives a pixel signal generated by a plurality of pixels arranged in the imaging device according to the amount of received light;
By switching the exposure time of the pixel in the image sensor to the first exposure time or the second exposure time shorter than the first exposure time, the pixel signal corresponding to the amount of received light in the first exposure time. An image sensor control unit for generating a pseudo black pixel signal which is the pixel signal according to a certain imaging pixel signal or the amount of received light in the second exposure time;
Whether the pseudo black pixel signal can be regarded as fixed pattern noise in the corresponding pixel based on a predetermined parameter calculated by the pseudo black pixel signal generated in the different pixel and a first threshold value A determination unit for determining
A fixed pattern noise removing unit, comprising: a subtracting unit that subtracts a noise signal based on the pseudo black pixel signal regarded as the fixed pattern noise by the determining unit from the imaging pixel signal.   The subtracting unit averages the pseudo black pixel signal regarded as the fixed pattern noise by the determination unit for each row in which the pixels are arranged, and the averaged pseudo black pixel signal for each row is the noise signal. The fixed pattern noise removal unit according to claim 1, wherein the fixed pattern noise removal unit is subtracted from the image pickup pixel signal in a corresponding row.   The subtracting unit averages a predetermined range including a median or a predetermined number of the pseudo black pixel signals in a distribution range of signal levels of the pseudo black pixel signals for each row. The fixed pattern noise elimination unit according to 2.   The subtracting unit averages the plurality of pseudo black pixel signals regarded as the fixed pattern noise by the determining unit for each column in which the pixels are arranged, and calculates the average pseudo black pixel signal for each column. The fixed pattern noise removing unit according to claim 2 or 3, wherein the fixed pattern noise removing unit is subtracted from the imaging pixel signal in a corresponding column as a noise signal. The imaging element control unit causes the imaging element to perform imaging for the second exposure time a plurality of times, thereby generating a plurality of pseudo black pixel signals corresponding to the pixels,
The subtracting unit subtracts, from the imaging pixel signal, the noise signal based on the pseudo black pixel signal regarded as the fixed pattern noise by the determination unit among the plurality of pseudo black pixel signals. The fixed pattern noise removal unit according to any one of claims 1 to 4. The subtracting unit is the plurality of pseudo black pixel signals generated by imaging the second exposure time a plurality of times, and the plurality of pseudo black pixel signals regarded as the fixed pattern noise by the determination unit The fixed pattern noise removal unit according to claim 5, wherein the averaged pseudo black pixel signal is subtracted from the imaging pixel signal for each pixel as the noise signal. The image sensor control unit controls the image sensor until the number of the pseudo black pixel signals regarded as the fixed pattern noise by the determination unit in the same pixel becomes m (m is an integer of 2 or more). Repeat the imaging with the exposure time of 2,
The fixed pattern noise removal unit according to claim 6, wherein the subtraction unit averages a plurality of the pseudo black pixel signals in the same pixel regarded as the fixed pattern noise. An averaging unit that generates an averaged signal by averaging the pseudo black pixel signals corresponding to all or some of the pixels arranged in the image sensor;
The said discrimination | determination part discriminate | determines whether the said average signal can be considered as the said fixed pattern noise based on the said black pixel signal. The any one of Claims 1-4 characterized by the above-mentioned. Fixed pattern noise elimination unit as described. The imaging element control unit causes the imaging element to perform imaging for the second exposure time a plurality of times, thereby generating a plurality of pseudo black pixel signals corresponding to the pixels,
The subtracting unit subtracts the noise signal based on the pseudo black pixel signal used to generate the averaged signal regarded as the fixed pattern noise by the determining unit from the imaging pixel signal. The fixed pattern noise removing unit according to claim 8.   The subtracting unit is configured to generate the averaged signal which is the plurality of pseudo black pixel signals generated by imaging the second exposure time a plurality of times and is regarded as the fixed pattern noise by the determination unit. 10. The plurality of pseudo black pixel signals used are averaged for each pixel, and the averaged pseudo black pixel signal for each pixel is subtracted from the imaging pixel signal as the noise signal. Fixed pattern noise elimination unit as described in 1. The image sensor control unit applies the second exposure time to the image sensor until the averaged signal regarded as the fixed pattern noise by the determination unit reaches m (m is an integer of 2 or more). Repeat the imaging,
The fixed pattern noise according to claim 10, wherein the subtraction unit averages the pseudo black pixel signal used to generate the m averaged signals regarded as the fixed pattern noise. Removal unit. Averaging that generates an averaged signal by averaging the pseudo black pixel signals corresponding to the plurality of pixels in the first and second subregions that are part of the entire region where the pixels are arranged. Part
The determination unit determines, for each of the first and second small regions, whether or not the averaged signal can be regarded as the fixed pattern noise based on the black pixel signal. The fixed pattern noise removal unit according to claim 1. The imaging element control unit causes the imaging element to perform imaging for the second exposure time a plurality of times, thereby generating a plurality of pseudo black pixel signals corresponding to the pixels,
The subtracting unit subtracts the noise signal based on the pseudo black pixel signal used to generate the averaged signal regarded as the fixed pattern noise by the determining unit from the imaging pixel signal. The fixed pattern noise elimination unit according to claim 12.   The subtracting unit is configured to generate the averaged signal which is the plurality of pseudo black pixel signals generated by imaging the second exposure time a plurality of times and is regarded as the fixed pattern noise by the determination unit. The plurality of pseudo black pixel signals used are averaged for each pixel, and the averaged pseudo black pixel signal for each pixel is subtracted from the imaging pixel signal for each pixel as the noise signal. The fixed pattern noise removing unit according to claim 13. The image sensor control unit is configured such that the average signal, which is regarded as the fixed pattern noise by the determination unit, is m (m is an integer of 2 or more) for each of the first and second small regions. Causing the imaging device to repeat imaging at the second exposure time;
The fixed pattern noise removal according to claim 14, wherein the subtracting unit averages the pseudo black pixel signals used to generate the plurality of averaged signals regarded as the fixed pattern noise. unit.   The imaging element control unit stops imaging at the second exposure time when the imaging element is executed n times (n is an integer larger than m) at the second exposure time. The fixed pattern noise removing unit according to any one of claims 7, 11, and 15.   The image sensor control unit is configured to generate the pseudo black pixel signal by switching an exposure time of the pixel to the second exposure time when an image pickup apparatus provided with the fixed pattern noise removal unit is started. The fixed pattern noise removal unit according to any one of claims 1 to 16. An input unit for inputting a command for generating the pseudo black pixel signal;
When the command for generating the pseudo black pixel signal is input to the input unit, the imaging element control unit switches the exposure time of the pixel to the second exposure time to generate the pseudo black pixel signal. The fixed pattern noise removing unit according to any one of claims 1 to 16, wherein The imaging element control unit sets the exposure time of the pixel when the luminance component of the imaging pixel signal generated by setting the exposure time of the pixel to the first exposure time is less than a second threshold value. Switching to the second exposure time to generate the pseudo black pixel signal;
When the determination unit regards the pseudo black pixel signal as the fixed pattern noise, the imaging element control unit returns the exposure time of the pixel to the first exposure time to generate the imaging pixel signal. The fixed pattern noise removing unit according to any one of claims 1 to 16, wherein   20. The output unit according to claim 18, further comprising: an output unit that continuously outputs the imaging pixel signal generated most recently until the determination unit considers the pseudo black pixel signal as the fixed pattern noise. Fixed pattern noise removal unit.   The said 1st threshold value is defined in the said predetermined parameter in case the said pseudo black pixel signal can be considered as the said fixed pattern noise, The any one of Claims 1-20 characterized by the above-mentioned. Fixed pattern noise removal unit.   The predetermined parameter is at least one of an average value, maximum value, median value, MSE, and PSNR of signal levels of the plurality of pseudo black pixel signals. The fixed pattern noise removing unit according to claim 1.   The fixed pattern noise removal unit according to claim 22, wherein a standard deviation of signal levels of the plurality of pseudo black pixel signals is also used as the predetermined parameter. A lens is disposed on the light receiving surface of the image sensor,
The determination unit determines whether the pseudo black pixel signal of the pixel arranged in a region away from the optical axis of the lens can be regarded as the fixed pattern noise,
The subtracting unit subtracts a noise signal based on the pseudo black pixel signal, which the determining unit considers as the fixed pattern noise, from the imaging pixel signal. Fixed pattern noise elimination unit as described in 1.   25. The second exposure time according to any one of claims 1 to 24, wherein the second exposure time is set to be less than a time when the integrated exposure amount to the effective pixel can be regarded as an integrated exposure amount corresponding to black. Fixed pattern noise elimination unit as described in 1.   The fixed pattern noise elimination unit according to any one of claims 1 to 24, wherein the second exposure time is a shortest exposure time that can be set in the imaging device. An image sensor having a plurality of pixels and outputting a pixel signal generated according to the amount of light received by the pixels;
By switching the exposure time of the pixel in the image sensor to the first exposure time or the second exposure time shorter than the first exposure time, the pixel signal corresponding to the amount of received light in the first exposure time. An image sensor control unit for generating a pseudo black pixel signal which is the pixel signal according to a certain imaging pixel signal or the amount of received light in the second exposure time;
Whether the pseudo black pixel signal can be regarded as fixed pattern noise in the corresponding pixel based on a predetermined parameter calculated by the pseudo black pixel signal generated in the different pixel and a first threshold value A determination unit for determining
An imaging unit comprising: a subtracting unit that subtracts a noise signal based on the pseudo black pixel signal regarded as the fixed pattern noise by the determination unit from the imaging pixel signal.   An electronic endoscope system comprising the imaging unit according to claim 27.

Images