JP5588729B2 - Image signal processing device - Google Patents

Description

  The present invention relates to an image signal processing apparatus. For example, the present invention relates to an image signal processing apparatus that receives a captured image having fixed pattern noise as input and corrects the fixed pattern noise.

  As a background field of the present technology, for example, Japanese Patent Publication No. 6-103925 (Patent Document 1) is available. This publication aims to provide a sensitivity correction method for an image sensor capable of accurately correcting variations in sensitivity characteristics of each pixel of the image sensor over the entire amount of incident light. The output signal of each pixel of the image sensor corresponding to the density, black reference density, and at least one gray reference density is obtained as white reference density data, black reference density data, and gray reference density data, and the document is read by the image sensor. It is determined which of the at least three reference density data corresponding to each pixel the image signal level of each pixel when read is between the reference density data, and sandwiches the image signal level of each pixel 2 The image signal of each pixel is corrected based on the relationship between one reference density data and a known reference density corresponding to each reference density data. It discloses a technique that.

No. 6-103925

  In the image signal obtained by imaging with an image sensor, the amount of signal obtained even when the same amount of light is incident on a pixel-by-pixel basis due to variations in characteristics such as dark current generated in the photodiode and sensitivity of the amplifier. Therefore, there is a problem that it appears as noise on the image. Noise that appears at a fixed position for each pixel is called fixed pattern noise. In order to generate a high-quality image, it is necessary to correct the fixed pattern noise by performing signal processing on the image signal.

  In Patent Document 1, a calibration object having a predefined white, black, or arbitrary gray density is photographed, and an output value of each pixel of the obtained image signal, that is, a pixel value, and a predefined white , Black, and an arbitrary gray density are associated with each other. When shooting a target object, the pixel value of each pixel of the obtained image signal is linearly interpolated based on this association and converted to a predetermined pixel value, thereby generating the sensitivity generated for each pixel. A method of correcting variation is disclosed. In Patent Document 1, by providing a plurality of arbitrary gray densities between white and black as a reference, correction can be performed with high accuracy even when the sensitivity variation is nonlinear with respect to the incident light quantity. is there. However, the method disclosed in Patent Document 1 has the following problems.

  In recent years, for the purpose of improving the dynamic range of an image sensor, a CMOS image sensor has been developed in which a pixel value to be output with respect to an incident light quantity has logarithmic characteristics. Such an image sensor is characterized in that the sensitivity itself is non-linear with respect to the amount of incident light, not the variation in sensitivity. Therefore, when the correction is performed by the method disclosed in Patent Document 1, the non-linear sensitivity characteristic becomes linear. There is a problem that the dynamic range is lost due to correction.

  Further, for the purpose of improving the dynamic range of an image sensor, an image sensor capable of performing exposure with a wide dynamic range by providing a plurality of charge storage locations in one pixel has been developed. In such an image sensor, the pixel value to be output does not continuously change with respect to the amount of incident light. Therefore, when the correction is performed by the method disclosed in Patent Document 1, the relationship between the subject density and the pixel value of the image signal changes. There is a problem.

  Therefore, in the present invention, each pixel of the image signal output from the image sensor is corrected with a different correction function depending on the pixel value in consideration of noise characteristics of fixed pattern noise including variations in sensitivity of the image sensor. For the purpose. Thereby, even in an image sensor characterized by having non-linearity or discontinuity with respect to the amount of incident light, it is possible to correct the fixed pattern noise with high accuracy while maintaining the imaging characteristics of the image sensor.

  The above object can be achieved by the invention described in the claims.

  According to the present invention, for each image signal, an image sensor having any characteristic can be obtained by performing correction with a different correction function depending on the pixel value in consideration of the noise characteristic of the fixed pattern noise. Fixed pattern noise can be corrected with high accuracy.

1 is a first schematic diagram showing an image signal processing apparatus according to a first embodiment of the present invention. FIG. 1 is a first diagram showing an example of a fixed pattern noise correction function selection method and a fixed pattern noise correction method according to the first embodiment of the present invention; FIG. 2 is a second diagram showing an example of a fixed pattern noise correction function selection method and a fixed pattern noise correction method according to the first embodiment of the present invention. First schematic diagram showing an image signal processing apparatus according to a second embodiment of the present invention. The figure which shows an example of the storage system of the correction function which concerns on 2nd Example of this invention. The figure which shows an example of the fixed pattern noise correction function selection process and fixed pattern noise correction process sequence which concern on 2nd Example of this invention. Second schematic diagram showing an image signal processing apparatus according to a second embodiment of the present invention. The figure which shows an example of the fixed pattern noise correction method at the time of the temperature change which concerns on 2nd Example of this invention. The figure which shows an example of the fixed pattern noise correction method at the time of the gain change which concerns on 2nd Example of this invention. FIG. 1 is a first diagram showing an example of a fixed pattern noise correction method according to a third embodiment of the present invention; 2nd figure which shows an example of the fixed pattern noise correction method which concerns on 3rd Example of this invention. Schematic diagram showing an image signal processing apparatus according to a fourth embodiment of the present invention. The figure which shows an example of the noise emphasis method based on 4th Example of this invention.

Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a first schematic diagram showing an image signal processing apparatus according to a first embodiment of the present invention. In FIG. 1, 0101 is an image signal input unit, 0102 is a fixed pattern noise correction function selection unit, and 0103 is a fixed pattern noise correction unit.

  In the image signal processing apparatus shown in FIG. 1, an image signal input unit 0101 is connected to an imaging device or a video device via Ethernet (registered trademark), USB, a capture board, etc., and inputs an image signal having fixed pattern noise. To do. Alternatively, the image signal input unit 0101 may include a lens, an image sensor, and an A / D converter, and may generate an image signal by performing an imaging process. Alternatively, the image signal stored in an image recording unit (not shown) may be read. Here, the fixed pattern noise is a characteristic of each characteristic such as a color filter, a photoelectric conversion unit, a readout drive circuit, a signal amplifier, an A / D conversion unit, and a microlens array in the device used for imaging. Due to the variation, even when the same amount of light is incident on the image sensor, the signal amount of the output image signal is different for each pixel, and this indicates noise inherent to the pixel. The fixed pattern noise correction function selection unit 0102 acquires the image signal input from the image signal input unit 0101, evaluates the pixel value of each pixel of the image signal, and corrects the fixed pattern noise component included in the pixel value. A correction function is selected from a plurality of correction functions prepared in advance, and is output to the fixed pattern noise correction unit 0103. A method for selecting a plurality of correction functions to be prepared and a necessary correction function from among the plurality of correction functions will be described later with reference to FIG. The correction function and the conditions for selecting the correction function may be recorded in advance in a memory unit (not shown) and read and used when an image signal is input from the image signal input unit 0101. Or it is good also as a form which a user inputs from a user interface not shown. In addition, when the image signal input by the image signal input unit 0101 can be captured by a plurality of types of imaging devices, a condition for selecting a correction function and a correction function is prepared for each imaging device, and is selected and used. It is also good. The fixed pattern noise correction unit 0103 acquires the image signal input from the image signal input unit 0101, the correction function of each pixel of the image signal from the fixed pattern noise correction function selection unit 0102, and the pixel of each pixel of the image signal The value is corrected using a corresponding correction function and output as a corrected image signal. As a result, it is possible to generate an image signal in which the fixed pattern noise is corrected. Note that the fixed pattern noise correction function selection process and the fixed pattern noise correction process are performed by, for example, an application on a CPU in a personal computer, a microcomputer, a DSP, a dedicated LSI, or the like in an embedded device.

FIG. 2 is a first diagram illustrating an example of a fixed pattern noise correction function selection method and a fixed pattern noise correction method according to the first embodiment of the present invention. In the present invention, the fixed pattern noise correction function selection process is performed by the fixed pattern noise correction function selection unit 0102, and the fixed pattern noise correction process is performed by the fixed pattern noise correction unit 0103. FIG. 2A is a diagram illustrating signal characteristics of an image signal having fixed pattern noise input by the image signal input unit 0101. The incident light amount when the image signal input by the image signal input unit 0101 is imaged. The relationship of the pixel value of each pixel of the image signal with respect to is shown. As shown in FIG. 2A, in the case of an image signal having fixed pattern noise, various variations depending on the type of the image sensor, such as variations in dark current and sensitivity of photodiodes included in the image sensor, and variations in sensitivity of the amplifier, etc. For this reason, the signal characteristics of each pixel vary with respect to the ideal signal characteristics. This variation, that is, the size of the fixed pattern noise is not only different for each pixel, but also varies with the amount of incident light. FIG. 2B is a diagram illustrating noise characteristics of fixed pattern noise of an image signal having fixed pattern noise input by the image signal input unit 0101. Ideal pixels of the image signal input by the image signal input unit 0101 are illustrated in FIG. The relationship of the noise level to the value is shown. The noise level can be obtained, for example, by calculating the standard deviation of the fixed pattern noise of each pixel of the image signal when the subject having the same luminance is imaged. In this way, if the noise level shows a characteristic that can be approximated by a function within a local range with respect to the pixel value, it is possible to correct the fixed pattern noise using the approximate function. For example, in the case of FIG. 2B, the noise level can be approximated by a linear function within two ranges. Although this example shows the noise characteristics with a simple model, of course, in actual application, the noise level may be approximated within three or more ranges, or may be approximated with a multidimensional function. An optimum model may be selected in consideration of approximation accuracy and calculation cost. In this example, the fixed pattern noise correction function selection method 0102 corrects the fixed pattern noise in the first range in which the noise level can be approximated by the function, the second range in which the function approximation can be performed, and the respective ranges. The correction function is estimated and prepared. FIG. 2 (3) is a diagram showing a method for selecting a correction function for a fixed pattern noise of an image signal having a fixed pattern noise input from the image signal input unit 0101, and correcting the pixel value of each pixel of the input image signal. The relationship between subsequent pixel values is shown. The fixed pattern noise correction function selection unit 0102 refers to the pixel value of each pixel of the image signal having the fixed pattern noise input from the image signal input unit 0101, and has a first range in which a noise level prepared in advance can be approximated by a function. It is determined whether the pixel value is within one of the second ranges where the noise level can be approximated by a function. If the pixel value is within the first range, the first correction function is within the second range. In this case, the second correction function is selected. In addition, when the magnitude of the fixed pattern noise is different for each pixel, it is preferable to perform correction using the first correction function and the second correction function for each pixel in order to perform highly accurate correction. That is, when the pixel value of the first pixel a is in the first range in which the noise level can be approximated by a function, the noise level can be approximated by the function of the first correction function f _a1 (x) corresponding to the pixel a. If the second correction function f _a2 (x) corresponding to the pixel a is in the second range, the pixel value of the second pixel b is in the first range in which the noise level can be approximated by a function. _{Includes a} first correction function f _b1 (x) corresponding to the pixel b, and a second correction function f _b2 (x) corresponding to the pixel a when the noise level is in a second range in which the function can be approximated. Are prepared in advance, and the fixed pattern noise correction function selection unit 0102 may select an optimal correction function from a combination of a pixel to be referred to and a pixel value thereof. FIG. 2 (4) is a diagram showing a method for correcting fixed pattern noise of an image signal having fixed pattern noise input from the image signal input unit 0101, and shows the relationship of the pixel value of each pixel of the image signal with respect to the amount of incident light. ing. The fixed pattern noise correction function selection unit 0103 uses the correction function selected by the fixed pattern noise correction function selection unit 0102 for the pixel value of each pixel of the image signal having the fixed pattern noise input from the image signal input unit 0101. Correct fixed pattern noise. That is, when the pixel value of the target pixel is within the first range in which the noise level can be approximated by a function, the pixel value of the target pixel is corrected using the first correction function, and the noise level is in the second range where the function can be approximated. If there is a pixel value of the target pixel, it is corrected using the second correction function. As a result, the pixel values of all the corrected pixels can be matched with the pixel values in ideal signal characteristics. As a result, the fixed pattern noise can be corrected without impairing the signal characteristics of the image signal with respect to the original incident light quantity, and even with an image sensor having nonlinear or discontinuous signal characteristics with respect to the incident light quantity Pattern noise can be corrected.

  FIG. 3 is a second diagram illustrating an example of a fixed pattern noise correction function selection method and a fixed pattern noise correction method according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating noise characteristics for each color of fixed pattern noise of an image signal having fixed pattern noise input by the image signal input unit 0101, and ideal pixels of the image signal input by the image signal input unit 0101 The relationship of the noise level for each color to the value is shown. In FIG. 3, when an image signal having a different color component for each pixel is input without being subjected to demosaicing processing and captured through an RGB color filter, a fixed pattern is set in a different range for each RGB component. An example in which the noise level of noise is approximated by a function is shown. By preparing a range that approximates the noise level for each identical color component with different sensitivities and a correction function within that range, it is possible to correct fixed pattern noise with higher accuracy for color images. Become.

  As described above, according to this embodiment, it is possible to correct only the fixed pattern noise with high accuracy for the image signal having the fixed pattern noise without impairing the original signal characteristics.

  FIG. 4 is a first schematic diagram showing an image signal processing apparatus according to the second embodiment of the present invention. In this embodiment, an example of an embodiment in which the present invention is applied to an imaging device such as a digital still camera or a digital video camera is shown. In FIG. 4, 0410 is an imaging unit, 0411 is a lens group, 0412 is an imaging element, 0413 is an A / D conversion unit, 0402 is a fixed pattern noise correction function selection unit, 0403 is a fixed pattern noise correction unit, and 0404 is range information storage. , 0405 is a fixed pattern noise correction function storage unit, and 0406 is a signal processing unit.

  In the image signal processing apparatus shown in FIG. 4, the imaging unit 0410 corresponds to the image signal input unit 0101 in FIG. 1, and includes a lens group 0411, an imaging element 0412, and an A / D conversion unit 0413. Light incident on the image sensor 0412 via 0411 is photoelectrically converted and output as a digital image signal by the A / D converter 0413. An analog front end unit (not shown) may be provided as necessary. For example, when the image sensor 0412 is a CCD, a CDS, an AGC, or the like exists between the image sensor 0412 and the A / D converter 0413. The fixed pattern noise correction function selection unit 0402 acquires the image signal output from the imaging unit 0410, and uses the range information stored in the range information storage unit 0404, so that each pixel value of the image signal has two or more pixel values. Evaluating whether it is within an arbitrary range, and if it is within any range, the fixed pattern noise included in the pixel value from among a plurality of correction functions stored in the fixed pattern noise correction function storage unit 0405 A correction function for correcting the component is selected and output to the fixed pattern noise correction unit 0403. The fixed pattern noise correction unit 0403 acquires the correction function of each pixel of the image signal from the fixed pattern noise correction function selection unit 0402 from the image signal output from the imaging unit 0410, and sets the pixel value of each pixel of the image signal. The signal is corrected using a corresponding correction function and output to the signal processing unit 0406. The range information storage unit 0404 stores, for example, a range upper limit threshold and a lower limit threshold as range information necessary for the determination for the fixed pattern noise correction function selection unit 0402 to select an optimal correction function from a plurality of correction functions. . The fixed pattern noise correction function selection unit 0402 stores a plurality of correction functions necessary for the fixed pattern noise correction unit 0403 to correct the fixed pattern noise of each pixel of the image signal. The range information to be stored by the range information storage unit 0404 and the plurality of correction functions to be stored by the fixed pattern noise correction function storage unit 0405 evaluate the noise characteristics of the fixed pattern noise of the image sensor 0412 as shown in FIG. By doing so, an optimal one may be generated and stored in advance. The signal processing unit 0406 performs various signal processing for improving image quality such as demosaicing processing, low-pass processing, enhancer processing, and gamma processing on the image signal corrected by the fixed pattern noise output from the fixed pattern noise correction unit 0403. And output to a video signal generation unit and an image recording unit (not shown). Thereby, it is possible to capture a high-quality image in which the fixed pattern noise is corrected in real time. The fixed pattern noise correction function selection process, the fixed pattern noise correction process, and the signal processing are performed by a microcomputer, a DSP, a dedicated LSI, or the like. The range information storage unit 0404 and the fixed pattern noise correction function storage unit 0405 correspond to a nonvolatile memory such as an EEPROM or FRAM.

  FIG. 5 is a diagram illustrating an example of a correction function storing method according to the second embodiment of the present invention. In the present invention, the correction function is stored in the fixed pattern noise correction function storage unit 0405. As shown in FIG. 2, different correction functions are prepared for each pixel of the image signal output from the imaging unit 0410 in a range in which the noise level of the fixed pattern noise of the image signal can be approximated by a function. preferable. Therefore, in this example, when the correction function is approximated by a linear function in each range where the noise level can be approximated by a function, the slope and intercept of the correction function are stored in the form of a lookup table. Yes. For example, for pixel 1 as the first pixel, the slope 11 and intercept 11 of the correction function in the first range in which the noise level can be approximated by function is within the Mth range in which the noise level can be approximated by function. The correction function slopes 1M and intercept 1M correspond to the correction function, and this correspondence is stored for each of the N pixels. As a result, the amount of information to be stored as a correction function is reduced, so that even an embedded device with a small amount of memory can be readily realized.

  FIG. 6 is a diagram showing an example of a fixed pattern noise correction function selection process and a fixed pattern noise correction process sequence according to the second embodiment of the present invention. In the fixed pattern noise correction function selection process and the fixed pattern noise correction process sequence of FIG. 6, in ST0601, the fixed pattern noise correction function selection unit 0402 and the fixed pattern noise correction unit 0403 obtain an image signal from the imaging unit 0410, and the line memory The pixel value of the first pixel stored in etc. is acquired. In ST0602, the fixed pattern noise correction function selection unit 0402 acquires range information from the range information storage unit 0404, determines whether the pixel value is within the first range stored as range information, and the pixel value is the first value. If it is within the range of 1, go to ST0403, otherwise go to ST0406. In ST0403, when the pixel value is within the first range, the lookup table stored in the fixed pattern noise correction function storage unit 0405 is referred to, and the slope and intercept of the corresponding correction function are selected and fixed. Output to the pattern noise correction unit 0403. In ST0404, the fixed pattern noise correction unit 0403 adds the intercept acquired in ST0403 to the pixel value acquired in ST0401. In ST0405, the fixed pattern noise correction unit 0403 performs an integration process on the slope acquired in ST0403 for the pixel value acquired in ST0401, and proceeds to ST0407. In ST0406, it is determined whether the pixel value is in any range stored as the range information for all the ranges required in advance. The process proceeds to ST0407 without correcting the value. If not done for all ranges, repeat from ST0402 for the next range. That is, when the pixel value is not within the first range, it is determined for all the ranges whether it is within the second range. In ST0407, it is determined whether correction processing has been performed for all the pixels of the image signal output by the imaging unit 0410. If all pixels have been corrected, the processing for one screen is terminated. If not done for all pixels, the process repeats again from ST0401 for the next pixel. As a result, the correction process for one pixel only needs to be added and integrated once, so that high-precision fixed pattern noise can be corrected with a small circuit configuration.

  FIG. 7 is a second schematic diagram showing an image signal processing apparatus according to the second embodiment of the present invention. In FIG. 7, 0710 is an imaging unit, 0711 is a lens group, 0712 is an imaging device, 0713 is an A / D conversion unit, 0702 is a fixed pattern noise correction function selection unit, 0703 is a fixed pattern noise correction unit, and 0704 is a range information storage. 0705 is a fixed pattern noise correction function storage unit, 0706 is a signal processing unit, 0707 is a gain control unit, and 0708 is a temperature measurement unit. The temperature measurement unit 0707 and the gain control unit 0708 are added to the schematic diagram shown in FIG. It has an added configuration.

  In the image signal processing apparatus shown in FIG. 7, the temperature measurement unit 0707 measures the temperature of the imaging unit 0710 when the imaging unit 0710 performs imaging, and outputs the temperature to the fixed pattern noise correction function selection unit 0702. The magnitude of the fixed pattern noise in the image signal may vary depending on the temperature at the time of imaging. Therefore, the range information storage unit 0704 and the fixed pattern noise correction function storage unit 0705 store range information and a plurality of correction functions in consideration of temperature fluctuations, respectively, and the fixed pattern noise correction function selection unit 0702 captures images from the temperature measurement unit 0707. The temperature of the hour is acquired, and an appropriate correction function is selected based on the range information corresponding to the temperature at the time of imaging and a plurality of correction functions. This makes it possible to correct fixed pattern noise with high accuracy even when the operating environment changes. The gain control unit 0708 is an analog image output by the image sensor 0712 by photoelectric conversion based on automatic exposure control based on the signal amount of the image signal output from the image capturing unit 0710 or user input from a user interface (not shown). Control the gain of the signal. For example, when the image sensor 0712 is a CCD, the gain is controlled by controlling the gain amount of AGC at an analog front end (not shown). For example, when the image sensor 0712 is a CMOS, the amplification of the amplifier is performed. What is necessary is just to control quantity. At this time, the magnitude of the fixed pattern noise in the image signal is also amplified according to the gain of the analog image signal. Therefore, range information storage unit 0704 and fixed pattern noise correction function storage unit 0705 store range information and a plurality of correction functions in consideration of gain fluctuations, and fixed pattern noise correction function selection unit 0702 captures images from gain control unit 0708. The time gain is acquired, and an appropriate correction function is selected based on the range information corresponding to the gain at the time of imaging and a plurality of correction functions. Thereby, even when the gain of the image signal obtained by imaging changes according to the scene, it is possible to correct the fixed pattern noise with high accuracy.

  FIG. 8 is a diagram showing an example of a fixed pattern noise correction method at the time of temperature change according to the second embodiment of the present invention. It is a figure which shows the noise characteristic for every temperature of the fixed pattern noise of the image signal which has the fixed pattern noise imaged with the imaging part 0710, and the noise level for every temperature with respect to the ideal pixel value of the image signal imaged with the imaging part 0710 Showing the relationship. In this way, if the temperature level can be approximated by a function with a certain temperature α, or if the noise level fluctuates, the range information corresponding to the temperature change and multiple correction functions are prepared in advance. What is necessary is just to store in the range information storage part 0704 and the fixed pattern noise correction function storage part 0705. At this time, for example, within the range of the guaranteed operating temperature of the image signal processing apparatus, the noise level when imaging is performed at a temperature at regular intervals, and range information and a plurality of correction functions at each temperature are prepared. Are stored in the range information storage unit 0704 and the fixed pattern noise correction function storage unit 0705, and the fixed pattern noise correction function selection unit 0702 stores the range information at the time closest to the temperature at the time of imaging, information on a plurality of correction functions, or By selecting the appropriate correction function using the range information at the vicinity of the temperature and information obtained by interpolation from multiple correction functions, it is possible to correct fixed pattern noise with high accuracy even when the temperature changes. It is. As another method, if the range information and the plurality of correction functions change regularly due to a temperature change, for example, the reference temperature range information and the plurality of correction functions and the temperature change range Information and difference information of a plurality of correction functions are stored in the range information storage unit 0704 and the fixed pattern noise correction function storage unit 0705, and the fixed pattern noise correction function selection unit 0702 determines the difference between the temperature at the time of imaging and the reference temperature. It is possible to reduce the amount of data to be stored by calculating optimal range information and a plurality of correction functions for the temperature at the time of imaging and selecting an appropriate correction function.

  FIG. 9 is a diagram showing an example of a fixed pattern noise correction method at the time of gain change according to the second embodiment of the present invention. It is a figure which shows the noise characteristic for every gain of the fixed pattern noise of the image signal which has the fixed pattern noise imaged with the imaging part 0710, and the noise level for every gain with respect to the ideal pixel value of the image signal imaged with the imaging part 0710 Showing the relationship. As described above, the range in which the noise level can be approximated by the function according to the gain and the noise level also fluctuate. Therefore, range information corresponding to the gain change and a plurality of correction functions are prepared in advance, and each of the range information storage unit 0704 and the fixed level It may be stored in the pattern noise correction function storage unit 0705. At this time, for example, within the controllable gain range of the gain control unit 0708, the noise level when imaging is performed while controlling the gain at regular intervals, and range information and a plurality of corrections at each gain are evaluated. A function is prepared and stored in the range information storage unit 0704 and the fixed pattern noise correction function storage unit 0705. The fixed pattern noise correction function selection unit 0702 stores the range information at the time of gain closest to the gain at the time of imaging and a plurality of correction functions. By selecting an appropriate correction function using information, or range information at a nearby temperature, or information obtained by interpolation from multiple correction functions, even if the gain changes, high-precision fixed pattern noise Correction is possible. As another method, since the range information and the plurality of correction functions regularly change with the same gain due to the gain change, for example, the reference temperature range information and the plurality of correction functions and the gain change Range information and difference information of a plurality of correction functions are stored in the range information storage unit 0704 and the fixed pattern noise correction function storage unit 0705, and the fixed pattern noise correction function selection unit 0702 is a ratio between a gain at the time of imaging and a reference gain. Thus, it is possible to reduce the amount of data to be stored by calculating optimal range information and a plurality of correction functions for the gain at the time of imaging and selecting an appropriate correction function.

  As described above, according to this embodiment, only fixed pattern noise is accurately obtained in real time in an imaging device such as a digital video camera or a digital still camera, without impairing the original signal characteristics with respect to an image signal having fixed pattern noise. It becomes possible to correct to.

  In this embodiment, in the image signal processing apparatus according to the first embodiment of the present invention shown in FIG. 1, the image signal input unit 0101 inputs an image signal having a plurality of pixels and the fixed pattern noise having the same characteristics. In the image signal processing apparatus according to the example of the embodiment or the second example of the present invention illustrated in FIG. 4, the imaging unit 0410 captures an image signal having a plurality of pixels and the same fixed pattern noise characteristic. This is an example of an embodiment, and in a device used for imaging, a plurality of pixels such as a color filter, a photoelectric conversion unit, a readout drive circuit, a signal amplifier, an A / D conversion unit, and a microlens array are provided. It is the structure shared by.

  In this embodiment, it is possible to share a correction function or a part of a correction function for correcting a fixed pattern noise component among a plurality of pixels.

  FIG. 10 is a first diagram illustrating an example of a fixed pattern noise correction method according to the third embodiment of the present invention. FIG. 10 (1) is an example of signal readout when a CCD is used as the image sensor, and FIG. 10 (2) shows the inclination of the correction function when the fixed pattern noise in the image signal can be approximated by a plurality of linear functions. FIG. 10 (3) is an example of a look-up table in which the intercept of the correction function is recorded. In the example shown in FIG. 10 (1), each pixel of the image sensor has a photodiode, and the signal obtained by photoelectric conversion is transferred by the vertical CCD for each column, and further transferred by the horizontal CCD for each row. To output as an image signal. At this time, for example, variations in fixed pattern noise caused by transfer deterioration in the vertical CCD are generated in the same amount in all pixels in the same column. If transfer deterioration in the vertical CCD is the main cause of variation in the offset component of the fixed pattern noise, the variation in the offset component of the fixed pattern noise due to other factors is sufficiently small that correction is unnecessary. If there is, as shown in FIGS. 10 (2) and 10 (3), the inclination of the correction function is prepared for all pixels, and the intercept of the correction function is prepared so as to share the same value for each column. Then, it is only necessary to create a lookup table for each. This makes it possible to reduce the amount of memory with respect to the lookup table shown in FIG. Further, if the variation of the gain component of the fixed pattern noise is sufficiently small, it is not necessary to use a look-up table relating to the inclination of the correction function, and a further reduction in the memory amount can be realized.

  FIG. 11 is a second diagram illustrating an example of the fixed pattern noise correction method according to the third embodiment of the present invention. FIG. 11 (1) is an example of signal readout when a CMOS having an amplifier for each column is used as the image sensor, and FIG. 11 (2) can approximate fixed pattern noise in an image signal by a plurality of linear functions. FIG. 11 (3) shows an example of a look-up table in which the intercept of the correction function is recorded. In the example shown in FIG. 11 (1), each pixel of the image sensor has a photodiode, and a signal obtained by photoelectric conversion is transferred by a signal line for each column and amplified by an amplifier for each column. The image signal is output by transferring the signal line for each row. At this time, for example, variations in the fixed pattern noise caused by the gain of the amplifier occur with the same slope characteristic in all the pixels in the same column. If the gain variation of the amplifier is the main factor of the variation of the gain component of the fixed pattern noise, and the variation of the gain component of the fixed pattern noise due to other factors is sufficiently small that correction is unnecessary As shown in FIGS. 11 (2) and 11 (3), the intercept of the correction function is prepared for all pixels, and the inclination of the correction function is prepared so as to share the same value for each column. Create a lookup table for each. This makes it possible to reduce the amount of memory with respect to the lookup table shown in FIG. Further, if the variation of the offset component of the fixed pattern noise is sufficiently small, it is not necessary to use a lookup table relating to the intercept of the correction function, and a further reduction in the memory amount can be realized.

  As described above, according to the present embodiment, the correction function or a part of the correction function is shared by a plurality of pixels in accordance with the characteristics of the fixed pattern noise of the image signal. Fixed pattern noise can be corrected.

  FIG. 12 is a schematic diagram showing an image signal processing apparatus according to the fourth embodiment of the present invention. In FIG. 12, reference numeral 1201 denotes an image signal input unit, 1202 denotes a fixed pattern noise correction function selection unit, and 1203 denotes a fixed pattern noise enhancement unit. The image signal processing apparatus according to the first embodiment of the present invention shown in FIG. The fixed pattern noise correction unit 0103 in the first schematic diagram shown is replaced with a fixed pattern noise enhancement unit 1203.

  In the image signal processing apparatus shown in FIG. 12, the fixed pattern noise enhancement unit 1203 receives the image signal input from the image signal input unit 1201 and the correction function of each pixel of the image signal from the fixed pattern noise correction function selection unit 1202. The noise amount is acquired based on a correction function corresponding to the pixel value of each pixel of the image signal, and added to the original image signal even if the noise amount is enhanced. As a result, it is possible to generate an image signal in which noise is enhanced as a special effect on the image. Note that the fixed pattern noise enhancement processing is performed by an application on a CPU for a personal computer, a microcomputer, a DSP, a dedicated LSI, or the like for an embedded device, for example.

  FIG. 13 shows an example of a noise enhancement method according to the fourth embodiment of the present invention. In the present invention, the noise enhancement method is performed by the fixed pattern noise enhancement unit 1203. FIG. 13 shows the pixel value of each pixel of the image signal when a subject with uniform brightness is photographed, the horizontal axis is the spatial position of the image signal, and the vertical axis is the pixel value at each spatial position. The fixed pattern noise enhancement unit 1203 has no noise amount, that is, no fixed pattern noise, for each pixel of the image signal input from the image signal input unit 1201 based on the correction function acquired from the fixed pattern noise correction function selection unit 1202. The difference between the pixel value in the ideal signal characteristic and the actual pixel value is calculated. This can be calculated, for example, by taking the difference between the pixel value after correction by the correction function and the pixel value before correction. By adding this amount of noise to the original pixel value, it is possible to obtain a noise-enhanced image signal. At this time, a weight input from a user input unit (not shown), a set value recording unit, or the like may be added to or accumulated in the noise amount, and the obtained value may be added to the original pixel value. Further, the noise amount may be amplified by inverting the sign of the coefficient of the correction function according to the order or integrating a constant value to the coefficient without estimating the noise amount. As a result, it is possible to generate an image signal in which noise is intentionally enhanced.

  Thus, according to the present embodiment, it is possible not only to correct fixed pattern noise with the same configuration, but also to generate an image that intentionally gives a sense of noise by enhancing the fixed pattern noise.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.

  The present invention can be used for an imaging apparatus, a PC application, an image processing apparatus, etc., for example, in consumer use, monitoring, in-vehicle use, and business use.

0101 Image signal input unit 0102 Fixed pattern noise correction function selection unit 0103 Fixed pattern noise correction unit 0402 Fixed pattern noise correction function selection unit 0403 Fixed pattern noise correction unit 0404 Range information storage unit 0405 Fixed pattern noise correction function storage unit 0406 Signal processing unit 0410 Imaging unit 0411 Lens group 0412 Imaging element 0413 A / D conversion unit 0707 Gain control unit 0708 Temperature measurement unit 0702 Fixed pattern noise correction function selection unit 0703 Fixed pattern noise correction unit 0704 Range information storage unit 0705 Fixed pattern noise correction function storage unit 0706 Signal processing unit 0710 Imaging unit 0711 Lens group 0712 Image sensor 0713 A / D conversion unit 1201 Image signal input unit 1202 Fixed pattern noise correction function selection unit 1203 Fixed pattern noise enhancement unit

Claims (15)

Image signal input means for inputting an image signal having fixed pattern noise;
From a plurality of correction functions prepared in advance for correcting the fixed pattern noise of each pixel of the image signal, the imaging condition of the image input from the image signal input unit and each pixel of the image signal input from the image signal input unit Fixed pattern noise correction function selection means for selecting and outputting an optimal correction function based on the pixel value of
Fixed pattern noise correction means for correcting the pixel value of each pixel of the image signal input from the image signal input means based on a correction function output from the fixed pattern noise correction function selection means, and outputting the corrected image signal as a corrected image signal;
With
The fixed pattern noise correction function selection means predetermines two or more arbitrary ranges for the pixel value of each pixel of the image signal for each imaging condition, and sets each pixel of the image signal input from the image signal input means. An image signal processing apparatus, wherein a different correction function is selected depending on which range the pixel value is. The image signal processing apparatus according to claim 1,
The imaging condition is the type of color filter in the image sensor,
When the image signal input means inputs an image signal having at least two or more different types of color components for each pixel, two or more arbitrary ranges predetermined for the pixel value of each pixel of the image signal are: , Different for each color component in the entire image or an arbitrary region of the image signal,
The fixed pattern noise correction function selection means selects two or more predetermined ranges based on which color component is the pixel value of each pixel of the input image signal, and the pixel value is selected An image signal processing apparatus that selects a different correction function depending on which of two or more arbitrary ranges is present. The image signal processing apparatus according to claim 1,
Furthermore, range information storage means for storing range information defining two or more predetermined ranges for the pixel value of each pixel of the image signal;
With
The fixed pattern noise correction function selection means has two or more arbitrary predetermined pixel values of pixel values of the image signal input from the image signal input means based on the range information stored in the range information storage means. An image signal processing apparatus for determining whether the range is within one of the ranges. The image signal processing apparatus according to claim 3 .
Furthermore, fixed pattern noise correction function storage means for storing a plurality of correction functions for correcting the fixed pattern noise of each pixel of the image signal,
An image signal processing apparatus comprising: The image signal processing apparatus according to claim 4 ,
The imaging condition is a temperature when the image signal is captured,
An imaging temperature input means for inputting a temperature when the image signal input to the image signal input means is imaged;
With
The range information storage means stores range information at a plurality of temperatures,
The fixed pattern noise correction function storage means stores correction functions at a plurality of temperatures,
The fixed pattern noise correction function selection means selects or interpolates information corresponding to the temperature at the time of imaging for the range information and the correction function based on the temperature information input from the temperature input means at the time of imaging. An image signal processing apparatus. The image signal processing apparatus according to claim 4 ,
The imaging condition is a temperature when the image signal is captured,
An imaging temperature input means for inputting a temperature when the image signal input to the image signal input means is imaged;
With
The range information storage means stores range information at a predetermined temperature and variation information of the range information at a temperature change,
The fixed pattern noise correction function storage means stores correction information at a predetermined temperature and fluctuation information of the correction function at a temperature change,
The fixed pattern noise correction function selection means corrects and uses the range information and the correction function to information corresponding to the temperature at the time of imaging based on the temperature information input from the imaging temperature input means. An image signal processing apparatus. The image signal processing apparatus according to claim 4 ,
The imaging condition is a gain when imaging the image signal,
The image signal input means includes a lens, an image sensor, and a gain control unit, and inputs a signal obtained by controlling the gain of a signal obtained by photoelectric conversion of light incident on the image sensor via the lens as an image signal. ,
The range information storage means stores range information for a plurality of gains, respectively.
The fixed pattern noise correction function storage means stores correction functions for a plurality of gains, respectively.
The fixed pattern noise correction function selection means, and characterized by the use of pre-SL retrieves the gain information from the image signal input unit, select or interpolate to the information corresponding to the gain at the time of imaging for the range information and the correction function An image signal processing device. The image signal processing apparatus according to claim 4 ,
The imaging condition is a gain when imaging the image signal,
The image signal input means includes a lens, an image sensor, and a gain control unit, and inputs a signal obtained by controlling the gain of a signal obtained by photoelectric conversion of light incident on the image sensor via the lens as an image signal. ,
The range information storage unit stores the variation information ranging information gain during variation and Mukuo range information during a predetermined gain,
The fixed pattern noise correction function storage means stores a correction function at a predetermined gain and variation information of the correction function at a gain change,
The fixed pattern noise correction function selecting means obtains the gain information from the previous SL image signal input unit, an image, which comprises using corrected to the range information and the correction function information corresponding to the gain at the time of imaging for Signal processing device . The image signal processing apparatus according to claim 1,
Each of the plurality of correction functions prepared in advance is a linear function based on a noise characteristic of fixed pattern noise of the image signal. The image signal processing apparatus according to claim 1,
An image signal processing apparatus, wherein the plurality of correction functions prepared in advance differ for each pixel of the image signal. The image signal processing apparatus according to claim 1,
An image signal processing apparatus characterized in that two or more arbitrary ranges predetermined for pixel values of each pixel of the image signal are the same in the entire image of the image signal or in an arbitrary region. The image signal processing apparatus according to claim 1,
The image signal input unit includes an image sensor, and inputs a signal obtained by photoelectrically converting light incident on the image sensor as an image signal. The image signal processing apparatus according to claim 4,
The fixed pattern noise correction function storage means associates each correction function with information specifying the position of each pixel of the image signal and information regarding a slope or intercept of a linear function for correcting the pixel. An image signal processing apparatus, which is stored as a look-up table. The image signal processing device according to claim 1,
An image signal processing apparatus, wherein the plurality of correction functions prepared in advance or a part of the plurality of correction functions are the same in a plurality of pixels of the image signal. The image signal processing device according to any one of claims 1 to 14,
The fixed pattern noise correction unit emphasizes the fixed pattern noise of each pixel of the image signal input from the image signal input unit based on a correction function output from the fixed pattern noise correction function selection unit. Image signal processing device.

Images