JP2018022951A - Image processing apparatus and black-and-white image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress deterioration of picture quality incident to high sensitivity or expansion of dynamic range.SOLUTION: An image processing apparatus includes a linear/nonlinear conversion circuit, a nonlinear digital gain circuit, and a nonlinear/linear conversion circuit. The linear/nonlinear conversion circuit applies linear/nonlinear conversion of input-output 1:1 for a linear RAW signal, and generates a first nonlinear RAW signal having bit number larger than that of the linear RAW signal. The nonlinear digital gain circuit multiplies a second nonlinear RAW signal, based on the first nonlinear RAW signal, by a digital gain and obtains a third nonlinear RAW signal. The nonlinear/linear conversion circuit applies nonlinear/linear conversion, corresponding to inverse transformation of linear/nonlinear conversion, for nonlinear brightness based on the third nonlinear RAW signal or the RGB signal, and generates a linear brightness signal or the RGB signal.SELECTED DRAWING: Figure 3

Description

  Embodiments relate to image processing.

  Conventionally, for example, a solid-state imaging device such as a complementary metal oxide semiconductor (CMOS) sensor can perform imaging with sensitivity suitable for the brightness (illuminance) of a subject without adjusting an optical aperture. Such imaging sensitivity control can be realized by adjusting the exposure time (that is, the shutter speed) of the solid-state imaging device and the gain applied to the pixel signal. Here, the gain includes a digital gain applied in an ISP (Image Signal Processor) in addition to an analog gain applied in the solid-state imaging device.

  However, the adjustment of the digital gain means that the digital pixel signal is multiplied by a digital value (gain) larger than 1, and the quantization noise is amplified to deteriorate the image quality. Since the digital gain needs to be increased in proportion to the target imaging sensitivity, the image quality such as SNR (Signal to Noise Ratio) is greatly deteriorated as the sensitivity is increased.

  Control of imaging sensitivity is also utilized in so-called HDR (High Dynamic Range) imaging technology. For example, HDR composition (also referred to as multiple exposure HDR imaging) is to perform imaging using a plurality of different exposure times, and expand the dynamic range within one screen by combining after combining the sensitivity of the captured image. Technology. Also known is a system (also called single exposure system HDR imaging) that expands the dynamic range by increasing sensitivity (that is, increasing the input saturation level setting). According to these HDR imaging technologies, it is possible to appropriately represent an image with a large luminance difference. For example, it is possible to reduce overexposure of a subject imaged in a dark environment, or to reduce blackout of a subject imaged in a bright environment.

  In the multiple exposure method, the digital gain is adjusted in order to match the sensitivity of the synthesized image (in other words, to increase the sensitivity of the short-time exposure channel). Even in the single exposure method, the digital gain is adjusted for high sensitivity. As described above, these digital gain adjustments amplify quantization noise and degrade image quality. As a result, the image quality such as SNR is greatly deteriorated as the dynamic range is expanded.

JP 2012-029029 A

  An object of the embodiment is to suppress deterioration in image quality accompanying an increase in sensitivity or an expansion of a dynamic range.

  According to the embodiment, the image processing apparatus includes a first linear / non-linear conversion circuit, a non-linear digital gain circuit, a non-linear / linear conversion circuit, and a second linear / non-linear conversion circuit. The first linear / nonlinear conversion circuit applies a first linear / nonlinear conversion of input / output 1: 1 to the second linear RAW signal based on the first linear RAW signal, and the second linear RAW A first non-linear RAW signal having more bits than the signal is generated. The non-linear digital gain circuit multiplies the second non-linear RAW signal based on the first non-linear RAW signal by a digital gain larger than x0 times to obtain a third non-linear RAW signal. The non-linear / linear conversion circuit applies non-linear / linear conversion corresponding to inverse conversion of the first linear / non-linear conversion to the first non-linear luminance signal or non-linear RGB signal based on the third non-linear RAW signal. 1 linear luminance signal or linear RGB signal is generated. The second linear / non-linear conversion circuit applies the second linear / non-linear conversion to the second linear luminance signal or the linear RGB signal based on the first linear luminance signal or the linear RGB signal, and the second non-linear conversion circuit is applied. A luminance signal or a non-linear RGB signal is generated.

  According to another embodiment, a black and white image processing apparatus includes a linear / nonlinear conversion circuit, a non-linear digital gain circuit, and a non-linear / nonlinear conversion circuit. The linear / non-linear conversion circuit applies linear / non-linear conversion of input / output 1: 1 to the second linear RAW signal based on the first linear RAW signal, and has a bit number larger than that of the second linear RAW signal. A large number of first nonlinear RAW signals are generated. The non-linear digital gain circuit multiplies the second non-linear RAW signal based on the first non-linear RAW signal by a digital gain larger than x0 times to obtain a third non-linear RAW signal. The non-linear / non-linear conversion circuit applies non-linear / non-linear conversion to the first non-linear luminance signal based on the third non-linear RAW signal to generate a second non-linear luminance signal.

FIG. 5 is a block diagram illustrating an imaging / image processing apparatus according to first to third embodiments. FIG. 4 is a block diagram illustrating a camera module included in an imaging / image processing apparatus according to the first embodiment and the second embodiment. The block diagram which illustrates the ISP pre-processing part contained in the imaging / image processing apparatus which concerns on 1st Embodiment and 3rd Embodiment. FIG. 3 is a block diagram illustrating an ISP post-processing unit included in an imaging / image processing apparatus according to the first embodiment and the second embodiment. 4 is a graph illustrating γ correction performed by the γ correction circuit of FIG. 3. The table which illustrates the setting value of a non-linear RAW signal in case a linear RAW (pixel) signal is 0 [%] (black) reference value +/- 7LSB. The graph which plotted the weighted sum in case the nonlinear RAW signal corresponding to the estimation normal pixel group which consists of four pixels is distributed in the range of 0 [%] reference value +/- 3LSB. The graph which plotted the weighted sum in case the nonlinear RAW signal corresponding to the estimation normal pixel group which consists of four pixels distributes in the range of 0 [%] reference value +/- 5LSB. The graph which plotted the weighted sum in case the nonlinear RAW signal corresponding to the estimation normal pixel group which consists of four pixels distributes in the range of 0 [%] reference value +/- 7LSB. The table which illustrates an input saturation level setting and the linear digital gain and nonlinear digital gain corresponding to the said input saturation level setting. 5 is a graph illustrating reverse γ correction performed by the reverse γ correction circuit of FIG. 4. FIG. 9 is a block diagram illustrating an ISP preprocessing unit included in an imaging / image processing apparatus according to a second embodiment. FIG. 10 is a block diagram illustrating a camera module included in an imaging / image processing apparatus according to a third embodiment. FIG. 10 is a block diagram illustrating an ISP post-processing unit included in an imaging / image processing apparatus according to a third embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. Hereinafter, the same or similar elements as those already described are denoted by the same or similar reference numerals, and redundant description is basically omitted.

  As illustrated in FIG. 1, the imaging / image processing apparatus according to the first to third embodiments includes a camera module 10 and an ISP core 20. The ISP core 20 is further divided into an ISP pre-processing unit 21 and an ISP post-processing unit 22. The ISP core 20 can also be called an image processing apparatus alone.

  Note that the functional division in FIG. 1 is merely an example. For example, the ISP preprocessing unit 21 may be incorporated in the camera module 10. Further, the ISP post-processing unit 22 may be incorporated in a chip different from the ISP pre-processing unit 21 such as an image processing chip or an image recognition chip.

  The camera module 10 captures an image, generates a linear RAW signal, and outputs the linear RAW signal to the ISP preprocessing unit 21. The camera module 10 may be a CMOS sensor. The ISP preprocessing unit 21 applies γ correction, which will be described later, to the linear RAW signal, expands the dynamic range by multiplying the digital gain (enhances sensitivity), and then performs HDR compression or clipping (hereinafter also simply referred to as compression). To be applied). The ISP preprocessing unit 21 outputs the compressed non-linear RAW signal to the ISP postprocessing unit 22.

  The ISP post-processing unit 22 performs various image processing such as contour extraction / correction and brightness / contrast adjustment on the nonlinear RAW signal from the ISP pre-processing unit 21. Further, the ISP post-processing unit 22 applies inverse γ correction corresponding to the inverse conversion of the γ correction, performs image processing such as color correction as necessary, and then applies the (re) γ correction. The ISP post-processing unit 22 outputs the processed signal to a subsequent device (for example, a display device, a communication device, a recording device, etc.) not shown.

(First embodiment)
The color imaging / image processing apparatus according to the first embodiment supports single-exposure type HDR imaging. The camera module 100, the ISP pre-processing unit 110, and the ISP post-processing unit 120 of this imaging / image processing apparatus are illustrated in FIGS. 2, 3, and 4, respectively.

  The camera module 100 in FIG. 2 includes an optical lens 101, a color filter 102, a light receiving pixel 103, an ADC (Analog to Digital Converter) 104, and a logic signal processing unit 105.

  The logic signal processing unit 105 may be incorporated in the same chip as the optical lens 101, the color filter 102, the light receiving pixel 103, and the ADC 104, or may be incorporated in a different chip. The logic signal processing unit 105 may be incorporated in the ISP preprocessing unit 110.

  The optical lens 101 condenses incident light from the outside on the light receiving pixel 103 via the color filter 102. The light receiving pixel 103 converts (photoelectric) the incident light that has passed through the optical lens 101 into an electric signal having a signal level corresponding to the amount of light. The light receiving pixel 103 has an electronic shutter function, and accumulates electric charge over a period (exposure time) determined by a set shutter speed. The light receiving pixel 103 outputs the generated electric signal to the ADC 104.

  The ADC 104 receives an electrical signal from the light receiving pixel 103 and performs analog / digital conversion on the electrical signal to generate a digital linear RAW signal. The ADC 104 outputs a linear RAW signal to the logic signal processing unit 105.

  The value of the linear RAW signal depends on the amount of incident light in the corresponding light receiving pixel 103. The linear RAW signal is, for example, an RG / GB line sequential signal in a primary color Bayer array (R-Gr-Gb-B). Note that the pixel array of the linear RAW signal is determined depending on the color array of the color filter 102. The color filter 102 includes colors other than red (R), green (G), and blue (B) such as white (Wh or Y), yellow (Ye), cyan (Cy), and magenta (Mg). It is also possible to have a color array (for example, a complementary color system array). The pixel array of the RAW signal is unchanged during a series of image processing (ISP preprocessing) performed by the ISP preprocessing unit 110 described later.

  The logic signal processing unit 105 receives the linear RAW signal from the ADC 104 and performs various signal processing such as temperature compensation at high temperature, optical shading correction, and static flaw correction. The static type defect correction is a technique for correcting coordinates by specifying (holding) known defective pixels at the time of shipment of the camera module 100, and is also referred to as manual defect correction. Note that the logic signal processing unit 105 generally performs signal processing on the linear RAW signal in the linear region. The logic signal processing unit 105 outputs the processed linear RAW signal to the ISP preprocessing unit 110. The linear RAW signal is a 10-bit to 16-bit signal, for example, but is assumed to be a 12-bit signal in the following description.

  3 includes a black level adjustment circuit 111, a linear digital gain circuit 112, a pre-γ correction circuit 113, a flaw correction / noise reduction processing circuit 114, a non-linear digital gain circuit 115, and HDR compression. A circuit 116, an input test image generation circuit 117, and an automatic exposure control circuit 118 are included.

  The black level adjustment circuit 111 receives a linear RAW signal from the camera module 100 and performs black level adjustment (OB (Optical Black) clamp). The black level adjustment circuit 111 outputs the linear RAW signal whose black level has been adjusted to the linear digital gain circuit 112.

  The linear digital gain circuit 112 receives a linear RAW signal (first linear RAW signal) from the black level adjustment circuit 111. The linear digital gain circuit 112 multiplies the linear RAW signal by the linear digital gain. The linear digital gain circuit 112 outputs the gain-corrected linear RAW signal to the pre-γ correction circuit 113. Note that the name of the first linear RAW signal can also be used to indicate a linear RAW signal output from the camera module 100 or the ADC 104, for example.

  This linear digital gain includes a first gain set for each color for correcting the color tone (typically adjusting the white balance) in addition to the imaging sensitivity setting of the entire imaging apparatus in FIG. It may be determined depending on at least one of the second gains set according to the position in the screen for shading correction.

  The pre-γ correction circuit 113 receives the linear RAW signal (second linear RAW signal) from the linear digital gain circuit 112. The pre-γ correction circuit 113 applies γ correction (first linear / nonlinear conversion) to the linear RAW signal, thereby generating a non-linear RAW signal (first non-linear RAW signal) having a larger number of bits than the linear RAW signal. Generate. The pre-γ correction circuit 113 outputs the non-linear RAW signal to the defect correction / noise reduction processing circuit 114.

  Specifically, the pre-γ correction circuit 113 linearly performs multi-bit γ correction of input / output 1: 1 (that is, there is no overlap between outputs corresponding to different inputs) shown in the following formula (1), for example. It can be applied to RAW signals.

  Equation (1) shows ideal characteristics in γ correction, and in this case, ideal linear / nonlinear conversion obtained by calculation (rounded off after the decimal point).

  In Equation (1), OUT (x) represents the value of the nonlinear RAW signal when the input signal level of the ADC 104 is x [%], and IN (x) is the linear RAW when the input signal level is x [%]. Represents the value of the signal. IN (0) and IN (100) represent the set values of the linear RAW signal when the input signal level is 0 [%] (black) and 100 [%] (white), respectively.

  γ is a γ correction coefficient determined according to input / output characteristics of a display device (not shown) provided at the subsequent stage of the ISP post-processing unit 120, for example. However, unlike the γ correction circuit 127, the pre-γ correction circuit 113 and the inverse γ correction circuit 125 may be fixed at ideal coefficient = 0.45.

  For example, in the case of γ = 0.45, if the input / output 1: 1 is maintained, the number of output bits becomes 2 bits or more larger than the number of input bits. γ may be a value different from 0.45, and may be simplified as 0.5, for example.

  OUT (0) and OUT (100) represent the setting values of the nonlinear RAW signal when the input signal levels are 0 [%] (black) and 100 [%] (white), respectively. If OUT (100) −OUT (0) is increased, the number of bits of the nonlinear RAW signal can be arbitrarily increased. However, since the γ correction is a linear / nonlinear conversion of input / output 1: 1, the number of gradations of the nonlinear RAW signal itself is the linear RAW signal no matter how much the number of bits of the nonlinear RAW signal is increased compared to the linear RAW signal. It will never fall below or exceed the number of tones.

  When IN (0) = 256, IN (100) = 2816, γ = 0.45, OUT (0) = 15360, OUT (100) = 56320, the pre-γ correction circuit 113 is γ illustrated in FIG. The correction is applied to the 12-bit linear RAW signal to generate a 16-bit non-linear RAW signal.

  Here, a total of 2560 (= 2816-256) gradations exist in the range from IN (0) to IN (100). 2560 corresponds to 10/16 of 4096, which is a gradation that can be expressed by a 12-bit linear RAW signal. Since IN (0) = 256 = 4096 × 1/16 and IN (100) = 2816 = 4096 × 11/16, 4095 which is the maximum value of the nonlinear RAW signal is IN (150) = 4096. Is almost equal.

  Similarly, a total of 40960 (= 56320-15360) gradations exist in the range from OUT (0) to OUT (100). 40960 corresponds to 10/16 of 65536, which is a gray scale that can represent a 16-bit non-linear RAW signal. Since OUT (0) = 15360 = 65536 × 3.75 / 16 and OUT (100) = 65536 × 13.75 / 16, 65535 which is the maximum value of the non-linear RAW signal is OUT (122.122). 5) = approximately equal to 65536. However, the value of the nonlinear RAW signal when IN (x) is the minimum value (0) is 827 (> 0), and the value of the nonlinear RAW signal when IN (x) is the maximum value (4095) Is 64513 (<65535). Therefore, whatever value IN (x) is, OUT (x) is not saturated.

  According to the γ correction of FIG. 5, an individual gain is given to the linear RAW signal for each signal level. The nonlinear RAW signal changes nonlinearly on the low luminance (low density) side, but changes substantially linearly on the high luminance (high density) side. That is, the non-linear RAW signal has a large gradation step size and is likely to change on the low luminance side (the gradation step size monotonously decreases as the luminance increases). On the other hand, the non-linear RAW signal has a small gradation step size on the high luminance side, and the gradation step width hardly changes but does not become zero.

  Note that the γ correction may be defined as a function, or may be defined using a table such as a lookup table. As described above, the input / output characteristics of γ correction may be fixed. In this case, the pre-γ correction circuit 113 may perform γ correction by referring to a fixed look-up table stored in a ROM (Read Only Memory), for example, or use HWL (Hard Wired Logic). Γ correction may be performed. By fixing the input / output characteristics of γ correction in this way, the circuit scale of the pre-γ correction circuit 113 can be suppressed. The input / output characteristics of γ correction can be easily calculated using, for example, spreadsheet software. Further, the γ correction can be replaced with other input / output 1: 1 multi-bit linear / non-linear conversion.

  The defect correction / noise reduction processing circuit 114 receives the non-linear RAW signal from the pre-γ correction circuit 113. The flaw correction / noise reduction processing circuit 114 applies at least one of flaw correction processing and noise reduction processing to the non-linear RAW signal. The defect correction / noise reduction processing circuit 114 outputs the processed non-linear RAW signal to the non-linear digital gain circuit 115.

  Specifically, the defect correction / noise reduction processing circuit 114 simultaneously extracts five lines of non-linear RAW signals and extracts a 5 × 5 pixel region (= same color 3 × 3 pixel region). The 5-line synchronization process can be implemented using, for example, a 4H line memory. A pixel occupying the center of the extracted 5 × 5 pixel region is a processing target (target pixel) by the scratch correction / noise reduction processing circuit 114. The target pixel is sequentially selected according to the scanning order of the effective pixel region of the light receiving pixel 103. That is, any pixel can be selected as the target pixel. The defect correction / noise reduction processing circuit 114 calculates a weighted sum of values of a plurality of pixels (in the nonlinear RAW signal) belonging to the surrounding pixel group including eight pixels having the same color as the target pixel. This weighted sum can be used for both the defect correction process and the noise reduction process. The flaw correction / noise reduction processing circuit 114 sets the value of the target pixel (in the non-linear RAW signal) to the sum of weights (Rc) for convenience of flaw correction processing and noise reduction processing.

  Note that the flaw correction / noise reduction processing circuit 114 does not have to calculate the weighted sum using all the pixels belonging to the surrounding pixel group. On the contrary, when a pixel that is estimated to correspond to a scratch (white scratch or black scratch) is included in the surrounding pixel group, it is preferable to exclude the pixel. Therefore, as described below, the defect correction / noise reduction processing circuit 114 calculates the weighted sum of the values of the pixels belonging to the estimated normal pixel group excluding the pixels from the surrounding pixel group.

  In this embodiment, the pixel is assumed to be either a flaw (white flaw or black flaw) or normal. In reality, there are pixels corresponding to fine flaws that are buried in various noises, but such flaws are treated as normal pixels because they are difficult to detect and correct appropriately.

  In the following description, eight identical color pixel values belonging to the surrounding pixel group are represented by Rn, Rm, Rk, Rj, Rh, Rg, Rf, and Re in descending order.

  When the maximum allowable number of white scratches is one, only the pixel having the maximum value (Rn) among the pixels belonging to the surrounding pixel group is excluded from the estimated normal pixel group because it may correspond to the white scratch. The When the maximum allowable number of white scratches is two, pixels having the second largest pixel value (Rm) in addition to the pixels having the maximum value (Rn) among the pixels belonging to the surrounding pixel group are white scratches. Are excluded from the estimated normal pixel group.

  If the number of white flaws is three, flaw correction does not work because white flaws arranged in a straight line cannot be distinguished from the stripe image. Therefore, the maximum allowable number of white scratches is at most 2. The maximum allowable number of black scratches is at most 2 for the same reason.

  Similarly, when the maximum allowable number of black scratches is 1, only the pixel having the minimum value (Re) among the pixels belonging to the surrounding pixel group may correspond to the black scratch, and therefore the estimated normal pixel group Excluded from. When the maximum allowable number of black scratches is two, pixels having the second smallest pixel value (Rf) in addition to the pixels having the minimum value (Re) among the pixels belonging to the surrounding pixel group are black scratches. Are excluded from the estimated normal pixel group.

  In other words, when the maximum allowable number of white and black flaws is one, the flaw correction / noise reduction processing circuit 114 calculates six pixels having values of Rm, Rk, Rj, Rh, Rg, and Rf. Specify as an estimated normal pixel group.

  When the maximum allowable number of white and black scratches is 1 and 2, respectively, the scratch correction / noise reduction processing circuit 114 estimates five pixels having values of Rm, Rk, Rj, Rh, and Rg. It is specified as a normal pixel group.

  When the maximum allowable number of white and black scratches is 2 and 1, respectively, the scratch correction / noise reduction processing circuit 114 estimates five pixels having values of Rk, Rj, Rh, Rg, and Rf. It is specified as a normal pixel group.

  When the maximum allowable number of white and black flaws is two, the flaw correction / noise reduction processing circuit 114 identifies four pixels having values of Rk, Rj, Rh, and Rg as an estimated normal pixel group. To do.

  Note that four pixels having values of Rk, Rj, Rh, and Rg are always included in the estimated normal pixel group regardless of whether the maximum allowable number of white scratches and black scratches is 1 or 2, respectively. Therefore, the scratch correction / noise reduction processing circuit 114 may calculate the weighted sum (Ra) of Rk, Rj, Rh, and Rg according to, for example, the following equation (2) regardless of the maximum allowable number of scratches and black scratches. Good.

Here, m and n represent weights. By using the weighted sum (Ra) calculated by the mathematical formula (2) as a theoretical reference value of the value (Rc) of the target pixel in the scratch correction process or the noise reduction process, the quantum of the value (Rc) of the target pixel The conversion accuracy is improved by at least x (2m + 2n) times (that is, + log ₂ (2m + 2n) bits). For example, if (m, n) = (0, 1), the quantization accuracy is improved by at least x2 (+1 bit). If (m, n) = (1, 1), the quantization accuracy is improved by at least x4 (+2 bits). If (m, n) = (1, 3), the quantization accuracy is improved by at least x8 (+3 bits). If (m, n) = (1, 7), the quantization accuracy is improved by at least x16 times (+4 bits). In the following description, it is assumed that (m, n) = (1, 3) and the number of bits of the target pixel value (Rc) is 19 (= 16 + 3) bits.

  The effect of improving the quantization accuracy by using the weighted sum (Ra) for the defect correction process or the noise reduction process varies greatly depending on whether the value (Rc) of the target pixel is near black or white, The vicinity is not much different from the above numerical example, but it is considerably higher than the above numerical example near black. As described above, since the step width between gradations of a non-linear RAW signal is likely to change in the vicinity of black, the weighted sum (Ra) of values in the vicinity of black can be dispersed over a wide range to express a finer gradation.

FIG. 6 shows values of the nonlinear RAW signal up to ± 7LSB with IN (0) (= 256) as the center. When the linear RAW signal increases / decreases by 1 LSB from IN (0), the increase / decrease of the non-linear RAW signal is 1199 (> 2 ¹⁰ ). That is, the substantial quantization accuracy near black of the 16-bit non-linear RAW signal can be estimated to be less than 6 bits (<16-10).

Assuming that all values of the normal pixel group are within the range of IN (0) ± 3LSB (that is, 7 values of 13395 to 17325), 113288 (= 8) of the weighted sums for all combinations of these values The graph of FIG. 7 is obtained by plotting what is in the range of (× (IN (0) −1LSB)) to 132472 (= 8 × (IN (0) + 1LSB)). In the example of FIG. 7, the number of gradations of the weighted sum within the range is 128 (= 2 × 2 ⁶ ) or more. Therefore, if scratch correction processing or noise reduction processing is performed using this weighted sum (Ra), the quantization accuracy of the value (Rc) of the target pixel near black is restored to about 12 bits (= 6 + 6 = 19-7). there's a possibility that.

Similarly, it is assumed that all the values of the normal pixel group are within the range of IN (0) ± 5LSB (that is, 11 values of 12887 to 17833), and 11288 of the weighted sums for all combinations of these values. The graph of FIG. 8 is obtained by plotting what is in the range from (= 8 × (IN (0) −1LSB)) to 132472 (= 8 × (IN (0) + 1LSB)). In the example of FIG. 8, the number of gradations of the weighted sum within the range is 512 (= 2 × 2 ⁸ ) or more. Therefore, if scratch correction processing or noise reduction processing is performed using this weighted sum (Ra), the quantization accuracy of the value (Rc) of the target pixel near black is restored to about 14 bits (= 6 + 8 = 19-5). there's a possibility that. In FIG. 8, an S-shaped curve with a moderate weighted sum is drawn, which is considered to be because the vicinity of the reference black level is limited to a finite value.

Similarly, it is assumed that all the values of the normal pixel group are within the range of IN (0) ± 7LSB (that is, 15 values of 12483 to 18237), and 11288 of the weighted sums for all combinations of these values. The graph of FIG. 9 is obtained by plotting what is in the range from (= 8 × (IN (0) −1LSB)) to 132472 (= 8 × (IN (0) + 1LSB)). In the example of FIG. 9, the number of weighted sum gradations within the range is 1024 (= 2 × 2 ⁹ ) or more, and the step size between adjacent gradations is approximately 48 or less. Therefore, if the defect correction process or the noise reduction process is performed using the weighted sum (Ra), the quantization accuracy of the value (Rc) of the target pixel near black is restored to about 15 bits (= 6 + 9 = 19−4). there is a possibility.

  Note that how much the value of the normal pixel group is dispersed depends on local changes in luminance, noise components, and the like in the linear RAW signal. However, any of the ranges assumed in FIGS. 7 to 9 includes at most 15 values, which is less than 16 values that are 1/16 of IN (0) = 256. Therefore, except for the situation where the luminance change in the normal pixel group is extremely small, if the flaw correction processing or noise reduction processing is performed using the weighted sum (Ra) as follows, the value of the target pixel near black ( The effect of greatly improving the quantization accuracy of Rc) can be expected.

  The flaw correction / noise reduction processing circuit 114 performs flaw detection on the target pixel based on, for example, the value (Rc) of the target pixel and the values of a plurality of pixels (nonlinear RAW signal) in the surrounding pixel group. . If the defect correction / noise reduction processing circuit 114 detects the target pixel as a defect, the defect correction / noise reduction processing circuit 114 corrects the value (Rc) of the target pixel based on at least the weighted sum (Ra).

  Alternatively, the flaw correction / noise reduction processing circuit 114 calculates, for example, a difference between the weighted sum (Ra) and the value (Rc) of the target pixel as a noise basic component. Then, the flaw correction / noise reduction processing circuit 114 multiplies the basic noise component by a coefficient that determines the noise reduction intensity, and subtracts these products from the value (Rc) of the target pixel (or reduces the product to the value of the target pixel ( Add noise to Rc) to reduce noise. Note that the flaw correction / noise reduction processing circuit 114 may reduce noise using a technique different from the technique described here.

  Note that the flaw correction / noise reduction processing generally targets a linear RAW signal, but the flaw correction / noise reduction processing circuit 114 targets a non-linear RAW signal. For example, a non-linear RAW signal obtained by γ correction of γ = 0.45 is suitable for human visual characteristics that are sensitive to low luminance compared to high luminance. For this reason, the flaw correction / noise reduction processing circuit 114 can realize flaw detection / correction and noise reduction with high accuracy by calculation using the non-linear RAW signal. Therefore, improvement in the yield of the camera module 100 and improvement in SNR can be expected.

  The non-linear digital gain circuit 115 receives the non-linear RAW signal (second non-linear RAW signal) from the defect correction / noise reduction processing circuit 114. The non-linear digital gain circuit 115 multiplies the non-linear RAW signal by a non-linear digital gain larger than x0. The non-linear digital gain circuit 115 outputs the gain-corrected non-linear RAW signal (third non-linear RAW signal) to the HDR compression circuit 116.

  This nonlinear digital gain affects the imaging sensitivity of the entire imaging apparatus (and the dynamic range of the nonlinear RAW signal). As will be described later, for example, if other elements such as a linear digital gain are fixed, the sensitivity is doubled by converting the non-linear digital gain to about 1.366 times (linear digital gain conversion).

  The non-linear digital gain (NLDG) that achieves a given sensitivity can be calculated by the following equation (3) based on the linear digital gain (LDG) that achieves the same sensitivity.

  In Equation (3), the power index of LDG is set to 0.45, but this power index is equal to the value of the γ correction coefficient (γ) used by the pre-γ correction circuit 113.

FIG. 10 shows an input saturation level (ie, sensitivity) setting (IDRS: Input Dynamic Range Setting), and a linear digital gain and a non-linear digital gain corresponding to the sensitivity setting. When both the linear digital gain and the non-linear digital gain are set to x1 or more, there is an effect of increasing the sensitivity by increasing the gradation interval. For example, if the linear digital gain (LDG) is doubled, the sensitivity is also doubled. On the other hand, the non-linear digital gain required to double the sensitivity is only about 1.366 times. This is clear from the above formula (3), which is 2 ^0.45 = 1.366.

  As illustrated in Equation (3) and FIG. 10, the non-linear digital gain can improve the sensitivity equally at a lower magnification than the linear digital gain. That is, the linear digital gain can be replaced by a smaller non-linear digital gain. Therefore, if the non-linear digital gain is increased in preference to the linear digital gain, the sensitivity can be improved while suppressing the deterioration of SNR.

  Assuming that the digital multiplier as the non-linear digital gain circuit 115 supports x0 to x63 times, the non-linear digital gain circuit 115 is x581 times the non-linear digital corresponding to x8192 times the linear digital gain. Gain can be used. As a result, the theoretical dynamic range of the gain-corrected nonlinear RAW signal is 150 [dB], and the number of bits is 25 (= 19 + 6). Even if a 12-bit linear RAW signal is simply multiplied by × 8192, the number of bits is the same 25 (= 12 + 13), but the substantial quantization accuracy included in the RAW signal is greatly different.

  The non-linear digital gain may be determined depending on the third gain set in common for all colors based on the HDR ratio (dynamic range expansion ratio) in addition to the imaging sensitivity setting of the entire imaging apparatus in FIG. Good. If the HDR ratio is larger than x1 times, the third gain is smaller than the HDR ratio as is apparent from the above equation (3).

  The nonlinear digital gain may be determined in consideration of the first gain or the second gain described above in addition to the imaging sensitivity setting (and the third gain) of the entire imaging apparatus in FIG. For example, the linear digital gain circuit 112 may be removed, and the non-linear digital gain circuit 115 may further perform color tone correction or shading correction.

  However, the linear digital gain circuit 112 multiplies the linear RAW signal by a digital gain, whereas the non-linear digital gain circuit 115 multiplies the non-linear RAW signal by a digital gain. Therefore, the gains required for color tone correction and shading correction differ between the linear digital gain circuit 112 and the non-linear digital gain circuit 115.

  The HDR compression circuit 116 receives the non-linear RAW signal from the non-linear digital gain circuit 115. The HDR compression circuit 116 multiplies the nonlinear RAW signal by a compression coefficient to compress the dynamic range. The HDR compression circuit 116 outputs the dynamic range-compressed non-linear RAW signal (fourth non-linear RAW signal) to the ISP post-processing unit 120. Note that the dynamic range of the output signal of the HDR compression circuit 116 is determined so as to match the dynamic range that the ISP post-processing unit 120 can support.

  Specifically, the HDR compression circuit 116 compresses the dynamic range of the nonlinear RAW signal while maintaining local contrast as much as possible using, for example, tone mapping.

  The number of bits of the nonlinear RAW signal output from the HDR compression circuit 116 is smaller than the number of bits of the input nonlinear RAW signal (for example, about 16 to 20 bits). Therefore, as illustrated in FIG. 3, the HDR compression circuit 116 is arranged as the last block in the ISP preprocessing unit 110, so that the transmission signal between the ISP preprocessing unit 110 and the ISP postprocessing unit 120 is transmitted. The number of bits can be minimized.

  Note that the HDR compression circuit 116 can be replaced with a clipping circuit that clips an excessive portion of the nonlinear RAW signal. In this case, since the information of the high luminance part is lost, the ISP preprocessing unit 110 does not support HDR imaging.

  The input test image generation circuit 117 generates a test input image. The automatic exposure control circuit 118 controls the imaging sensitivity by controlling part or all of the shutter speed, analog gain, linear digital gain, and non-linear digital gain based on, for example, luminance.

  The ISP post-processing unit 120 in FIG. 4 includes an m-line synchronization circuit 121, a contour extraction circuit 122, a contour correction / edge extraction circuit 123, a demosaic (RGB synchronization) circuit 124, an inverse γ correction circuit 125, Linear RGB processing circuit 126, γ correction circuit 127, YUV matrix circuit 128, YUV processing circuit 129, auto white balance circuit 130, YUV band limiting circuit 131, output test image generation circuit 132, And a matrix (YUV / RGB reverse conversion) circuit 133.

  The m line synchronization circuit 121 receives a non-linear RAW signal from the ISP preprocessing unit 110. m is, for example, 7, but may be another odd number. The m line synchronization circuit 121 synchronizes the non-linear RAW signal with m lines. The m line synchronization processing can be implemented using, for example, (m−1) H line memory. The m line synchronization circuit 121 outputs the m-line synchronized non-linear RAW signal to the contour extraction circuit 122 and the demosaic circuit 124.

  The contour extracting circuit 122 receives the m-line synchronized non-linear RAW signal (fifth non-linear RAW signal) from the m-line synchronizing circuit 121. The contour extraction circuit 122 extracts a contour in each of the horizontal direction and the vertical direction, for example. For the contour extraction, for example, LPF (Low Pass Filter) and BPF (Band Pass Filter) can be used in combination. The contour extraction circuit 122 outputs the extracted contour signal to the contour correction / edge extraction circuit 123.

  Note that the contour extraction generally targets a linear RAW signal, but the contour extraction circuit 122 targets a non-linear RAW signal. As described above, for example, a nonlinear RAW signal obtained by γ correction of γ = 0.45 is suitable for human visual characteristics. For this reason, the contour extraction circuit 122 can extract the contour with high accuracy by calculation using the non-linear RAW signal. By using the contour extracted in this way, further improvement in resolution can be expected.

  The contour correction / edge extraction circuit 123 receives the contour signal from the contour extraction circuit 122. The contour correction / edge extraction circuit 123 corrects the contour signal and further extracts an edge. The correction of the contour signal can include, for example, a coring process and a clip process. The contour correction / edge extraction circuit 123 outputs the corrected contour signal and edge signal to the YUV processing circuit 129.

  The demosaic circuit 124 receives the non-linear RAW signal synchronized with m lines from the m line synchronization circuit 121. The demosaic circuit 124 performs demosaic (also referred to as RGB synchronization or color separation) on the nonlinear RAW signal to generate an RGB 3-channel nonlinear RAW signal. The demosaic circuit 124 outputs the RGB 3-channel non-linear RAW signal to the inverse γ correction circuit 125.

  Note that the demosaic generally targets a linear RAW signal, but the demosaic circuit 124 targets a non-linear RAW signal. As described above, for example, a nonlinear RAW signal obtained by γ correction of γ = 0.45 is suitable for human visual characteristics. For this reason, the demosaic circuit 124 can perform the interpolation operation (demosaic) with high accuracy using the non-linear RAW signal and improve the color reproducibility.

  When the color space of the input non-linear RAW signal is different from RGB, the demosaic circuit 124 first synchronizes the three to four color pixels of the non-linear RAW signal, and then converts the color space to RGB to convert the RGB three-channel non-linear. An RGB (primary color) signal is generated. Specifically, the demosaic circuit 124 performs an interpolation calculation according to the pixel position, and then performs color conversion of the nonlinear RAW signal using a basic expression exemplified by the following Expression (4).

  In order to use the above formula (4), the pre-γ correction circuit 113 needs to perform an ideal γ correction of γ = 0.45.

  If the demosaic circuit 124 has such a color conversion function, the ISP pre-processing unit 110 and the ISP post-processing unit 120 can be used regardless of the color arrangement of the color filter 102 included in the camera module 100. (Excluding the demosaic circuit 124) can perform common processing. That is, the ISP preprocessing unit 110 and the ISP postprocessing unit 120 can support a wide variety of color filters 102. Note that the demosaic circuit 124 is not limited to Equation (4), and if the relationship between the color space of the input nonlinear RAW signal and RGB is known, color conversion may be performed using the relationship.

  The inverse γ correction circuit 125 receives the RGB three-channel non-linear RGB signal (first non-linear RGB signal) from the demosaic circuit 124. The inverse γ correction circuit 125 generates an RGB 3-channel linear RGB signal (first linear RGB signal) by applying inverse γ correction (nonlinear / linear conversion) to each nonlinear RGB signal. The inverse γ correction circuit 125 outputs the linear RGB signal to the linear RGB processing circuit 126.

  Specifically, the inverse γ correction circuit 125 uses a γ correction coefficient (for example, 2.2 = 1 / 0.45) equal to the reciprocal of the γ correction coefficient (γ) used by the pre-γ correction circuit 113, and is non-linear. Apply (reverse) gamma correction to RGB signals.

  For example, the inverse γ correction circuit 125 converts a 16-bit non-linear RGB signal into a 20-bit linear RGB signal. In the non-linear RGB signal, values of 0 [%] (black) and 100 [%] (white) are set to 15360 and 56320, respectively. In the linear RGB signal, the values in the case of 0 [%] (black) and 100 [%] (white) are set to 65536 and 720896, respectively. FIG. 11 illustrates inverse γ correction based on such numerical examples.

  However, the inverse γ correction in FIG. 11 is not input / output 1: 1. In order for the inverse γ correction circuit 125 to realize the inverse γ correction of the input / output 1: 1, the number of bits of the output linear RGB signal must be set very large, which is not realistic. Therefore, the inverse γ correction circuit 125 does not have to comply with the input / output 1: 1. According to the example of FIG. 11, the maximum discrete amount of the linear RGB signal with respect to the minimum input effective value = 827 vicinity is about 10, and the maximum discrete amount of the linear RGB signal with respect to the maximum input effective value = 64513 is about 45. Good color reproducibility can be achieved by performing post-stage linear RGB color correction processing using the 20-bit linear RGB signal.

  The inverse γ correction may be defined as a function, or may be defined using a table such as a lookup table. The input / output characteristics of inverse γ correction may be fixed. In this case, for example, the inverse γ correction circuit 125 may perform the inverse γ correction by referring to a fixed lookup table stored in the ROM, or may perform the inverse γ correction using the HWL. . By fixing the input / output characteristics of inverse γ correction in this way, the circuit scale of the inverse γ correction circuit 125 can be suppressed. The input / output characteristics of inverse γ correction can be easily calculated using, for example, spreadsheet software. Further, the inverse γ correction can be replaced with other non-linear / linear conversion.

  The linear RGB processing circuit 126 receives RGB 3-channel linear RGB signals from the inverse γ correction circuit 125. The linear RGB processing circuit 126 performs linear RGB processing such as white balance adjustment (for example, R / B gain adjustment) and linear matrix color correction (that is, color mixture correction), for example. When performing both white balance adjustment and linear matrix color correction, it is preferable to perform white balance adjustment first. The linear RGB processing circuit 126 outputs processed RGB 3-channel linear RGB signals to the γ correction circuit 127.

  Since color mixing between RGB and level imbalance occur around the color filter 102, the light receiving pixel 103, and the ADC 104, they are all byproducts of signal processing in the linear region. Therefore, the linear RGB signal is more suitable for white balance adjustment and linear matrix color correction than the non-linear RGB signal. In the ISP pre-processing unit 110 in FIG. 3 and the ISP post-processing unit 120 in FIG. 4, as much signal processing as possible is performed on a non-linear RGB signal suitable for human visual characteristics as compared to a linear RGB signal. An inverse γ correction circuit 125 is provided in the preceding stage of the linear RGB processing circuit 126. Therefore, the linear RGB processing circuit 126 can achieve high color reproducibility by performing highly accurate linear RGB processing on the linear RGB signal.

  As described above, the number of bits of the HDR-compressed non-linear RAW signal is assumed to be 16 to 20 bits. However, depending on the configuration of the demosaic circuit 124 and the inverse γ correction circuit 125, it is possible to further increase the number of bits. Therefore, the quantization accuracy of the RGB 3-channel linear RGB signal given to the linear RGB processing circuit 126 can be sufficiently secured.

  Note that the subject that adjusts the white balance is not limited to the linear RGB processing circuit 126. For example, the linear digital gain circuit 112 may perform white balance primary adjustment, and the linear RGB processing circuit 126 may perform white balance secondary adjustment. According to such a configuration, it is possible to prevent false coloring at high luminance and improve color reproducibility.

  The γ correction circuit 127 receives an RGB 3-channel linear RGB signal (second linear RGB signal) from the linear RGB processing circuit 126. The γ correction circuit 127 applies γ correction (second linear / nonlinear conversion) to the linear RGB signal, thereby generating an RGB 3-channel non-linear RGB signal (second non-linear RGB signal). The γ correction circuit 127 outputs the generated non-linear RGB signal to the YUV matrix circuit 128.

  The γ correction input / output characteristics of the γ correction circuit 127 may be the same as or different from the γ correction input / output characteristics of the pre-γ correction circuit 113. Further, the input / output characteristics of γ correction of the γ correction circuit 127 may be rewritable (programmable).

  For example, the user may arbitrarily set the γ correction input / output characteristics of the γ correction circuit 127 in advance, or the γ correction circuit 127 may be input every V blanking period or every discontinuous V blanking period. The output characteristics may be dynamically rewritten according to user settings (also referred to as a dynamic variable system). Since the influence of the γ correction of the pre-γ correction circuit 113 is canceled by the reverse γ correction performed by the reverse γ correction circuit 125, the γ correction circuit 127 is not affected by the γ correction and is input by the user. Γ correction faithful to the output characteristics (linear / nonlinear characteristics) can be performed.

  Therefore, the imaging / image processing apparatus including the ISP pre-processing unit 110 and the ISP post-processing unit 120 can be used for various purposes (for example, output images adapted to viewing environments such as ambient brightness and input / output characteristics of a display device not shown). Can be adapted). As described above, the γ correction input / output characteristics of the pre-γ correction circuit 113 and the inverse γ correction input / output characteristics of the inverse γ correction circuit 125 may be fixed. In this case, the user can use the γ correction circuit 127. It is sufficient to control only the input / output characteristics of γ correction.

  The γ correction circuit 127 realizes ideal ISP processing by performing γ correction on the ideal linear RGB signal obtained by the linear RGB processing circuit 126.

  The YUV matrix circuit 128 receives RGB 3-channel non-linear RGB signals from the γ correction circuit 127. The YUV matrix circuit 128 multiplies the nonlinear RGB signal by a matrix for RGB-YUV conversion to generate a YUV 3-channel nonlinear YUV (Y color difference) signal. The YUV matrix circuit 128 outputs the generated non-linear YUV signal to the YUV processing circuit 129. The relationship between RGB and YUV or YCbCr can be expressed by a basic formula exemplified by the following formula (5).

  Here, α represents a fixed offset value for obtaining a positive value (straight binary).

  The YUV processing circuit 129 receives a contour signal and an edge signal from the contour correction / edge extraction circuit 123 and receives a YUV three-channel non-linear YUV signal from the YUV matrix circuit 128. The YUV processing circuit 129 uses the contour signal and the edge signal to convert the non-linear YUV signal into, for example, brightness / contrast adjustment, color suppression in the high / low luminance region, color suppression at the edge, contour suppression at the high / low luminance, Various YUV processes such as contour addition are performed. The YUV processing circuit 129 outputs the processed YUV 3-channel non-linear YUV signal to a device (for example, a display device, a recording device, or a communication device) not shown.

  The auto white balance circuit 130 automatically controls the gain of each color (RGB) used in the white balance adjustment performed by the linear RGB processing circuit 126, for example. The YUV band limiting circuit 131 is a space LPF (Low Pass Filter), for example, and limits the band of the YUV 3-channel nonlinear YUV signal. The output test image generation circuit 132 generates a test output image.

  The RGB rematrix circuit 133 multiplies the YUV3 channel non-linear YUV signal by a matrix for YUV-RGB reconversion to generate an RGB3 channel non-linear RGB signal. That is, the ISP post-processing unit 120 can support both standard YUV (non-linear) output and standard RGB (non-linear) output.

  As described above, the imaging / image processing apparatus according to the first embodiment applies the input / output 1: 1 multi-bit / linear conversion to the linear RAW signal, and multiplies the nonlinear RAW signal by the nonlinear digital gain. To do. Therefore, the imaging / image processing apparatus can expand the dynamic range equally by using a small digital gain as compared with the case where the linear RAW signal is multiplied by the digital gain. Therefore, according to this imaging / image processing apparatus, it is possible to suppress the degradation of SNR accompanying the increase in the dynamic range. Furthermore, the point that the imaging / image processing apparatus performs defect correction / noise reduction processing using a non-linear RAW signal also contributes to the improvement of SNR.

  In addition, since the imaging / image processing apparatus performs the white balance adjustment or the linear matrix color correction after the non-linear RGB signal is once converted back into the linear RGB signal, good color reproducibility is achieved. The image processing apparatus again performs linear / nonlinear conversion of the linear RGB signal, but the input / output characteristics of this conversion can be rewritten. Therefore, the imaging / image processing apparatus can be applied to various applications.

  Furthermore, the point that this imaging / image processing apparatus performs highly accurate demosaicing using a non-linear RAW signal also contributes to the improvement of the color reproducibility described above. Note that when the color space of the nonlinear RAW signal is other than RGB, the imaging / image processing apparatus performs demosaicing and then converts the color space to RGB. Therefore, the imaging / image processing apparatus can support a wide variety of color filters. In addition, since the imaging / image processing apparatus performs highly accurate contour extraction using the non-linear RAW signal, there is also an effect of improving the resolution.

(Second Embodiment)
The imaging / image processing apparatus according to the second embodiment is different from the imaging / image processing apparatus according to the first embodiment in that it supports multiple exposure HDR imaging instead of a single exposure method. The camera module 100, the ISP pre-processing unit 200, and the ISP post-processing unit 120 of the imaging / image processing apparatus according to the present embodiment are illustrated in FIGS. 2, 12, and 4, respectively.

  In this embodiment, the camera module 100 outputs a linear RAW signal of a plurality of channels (three channels in the following description) having different exposure times, but the other operations are the same as those in the first embodiment. is there. The operation of the ISP post-processing unit 120 in this embodiment is the same as that in the first embodiment.

  12 includes a linear HDR synthesizing circuit 201, a signal dividing circuit 202, a linear digital gain circuit 211, a pre-γ correction circuit 212, a scratch correction / noise reduction processing circuit 213, and a non-linear digital gain. Circuit 214, linear digital gain circuit 221, pre-γ correction circuit 222, scratch correction / noise reduction processing circuit 223, non-linear digital gain circuit 224, non-linear HDR synthesis circuit 231, HDR compression circuit 232, input test An image generation circuit 233 and an automatic exposure control circuit 234 are included.

  The linear HDR synthesizing circuit 201 receives three-channel linear RAW signals (for example, 12 bits each) having different exposure times from the camera module 100, and HDR-synthesizes them. The linear HDR synthesis circuit 201 outputs a synthesized linear RAW signal (for example, 32 bits) to the signal division circuit 202.

  The signal dividing circuit 202 receives the linear RAW signal from the linear HDR synthesizing circuit 201, and divides the linear RAW signal into an upper bit signal and a lower bit signal (subdivision). However, there may be overlap between the upper bit signal and the lower bit signal. For example, the lower n bits of the upper bit signal and the upper n bits of the lower bit signal may overlap. Further, the signal dividing circuit 202 does not need to divide the linear RAW signal into two equal parts. For example, the signal dividing circuit 202 outputs the upper 16 bits of the input linear RAW signal to the linear digital gain circuit 211 as the linear RAW signal of the short exposure (low sensitivity) channel, and the lower 16 bits of the input linear RAW signal is long. A linear RAW signal of a time exposure (high sensitivity) channel is output to the linear digital gain circuit 221.

  When the ISP preprocessing unit 200 inputs a 2-channel linear RAW signal, the linear HDR synthesis circuit 201 and the signal division circuit 202 may be omitted.

  The linear digital gain circuit 211 receives the linear RAW signal of the short exposure channel from the signal dividing circuit 202. The linear digital gain circuit 211 multiplies the linear RAW signal by the linear digital gain. The linear digital gain circuit 211 outputs the linear RAW signal after gain correction to the pre-γ correction circuit 212.

  The pre-γ correction circuit 212 receives the linear RAW signal from the linear digital gain circuit 211. The pre-γ correction circuit 212 generates a nonlinear RAW signal (for example, 20 bits) having a larger number of bits than the linear RAW signal by applying γ correction to the linear RAW signal. The pre-γ correction circuit 212 outputs the non-linear RAW signal to the defect correction / noise reduction processing circuit 213.

  The defect correction / noise reduction processing circuit 213 receives the non-linear RAW signal from the pre-γ correction circuit 212. The flaw correction / noise reduction processing circuit 213 applies at least one of flaw correction processing and noise reduction processing to the non-linear RAW signal. The defect correction / noise reduction processing circuit 213 outputs a processed non-linear RAW signal (for example, 23 bits) to the non-linear digital gain circuit 214.

  Note that the flaw correction / noise reduction processing generally targets a linear RAW signal, but the flaw correction / noise reduction processing circuit 213 targets a non-linear RAW signal. For example, a non-linear RAW signal obtained by γ correction of γ = 0.45 is suitable for human visual characteristics that are sensitive to low luminance compared to high luminance. For this reason, the flaw correction / noise reduction processing circuit 213 can realize flaw detection / correction and noise reduction with high accuracy by calculation using the non-linear RAW signal. Therefore, improvement in the yield of the camera module 100 and improvement in SNR can be expected.

  The non-linear digital gain circuit 214 receives the non-linear RAW signal from the defect correction / noise reduction processing circuit 213. The non-linear digital gain circuit 214 multiplies the non-linear RAW signal by a non-linear digital gain larger than x1. The non-linear digital gain circuit 214 outputs a gain-corrected non-linear RAW signal (for example, 31 bits) to the non-linear HDR synthesizing circuit 231.

  The operations of the linear digital gain circuit 221, the pre-γ correction circuit 222, the scratch correction / noise reduction processing circuit 223, and the non-linear digital gain circuit 224, except for the non-linear digital gain, are the linear digital gain circuit 211, the pre-γ correction circuit 212, and the scratch. Since it may be the same as the correction / noise reduction processing circuit 213 and the non-linear digital gain circuit 214, detailed description thereof will be omitted. However, if the non-linear digital gain used by the non-linear digital gain circuit 224 can be fixed to x1, the non-linear digital gain circuit 224 is not necessary.

  The linear digital gain circuit 211 and the linear digital gain circuit 221 can be collectively referred to as a linear digital gain circuit group. The pre-γ correction circuit 212 and the pre-γ correction circuit 222 can be collectively referred to as a γ correction circuit group. The flaw correction / noise reduction processing circuit 213 and the flaw correction / noise reduction processing circuit 223 can be collectively referred to as a flaw correction / noise reduction processing circuit group. The non-linear digital gain circuit 214 and the non-linear digital gain circuit 224 can be collectively referred to as a non-linear digital gain circuit group.

  The non-linear HDR synthesizing circuit 231 receives the non-linear RAW signal of the short-time exposure channel from the non-linear digital gain circuit 214, receives the non-linear RAW signal of the long-time exposure channel from the non-linear digital gain circuit 224, and HDR synthesizes them. The non-linear HDR synthesis circuit 231 outputs the synthesized linear RAW signal (for example, 31 bits) to the HDR compression circuit 232.

  For example, the non-linear HDR synthesizing circuit 231 receives a non-linear RAW signal (for example, 31 (= 23 + 8) bits) multiplied by × 20.854 by the non-linear digital gain circuit 214, and is non-linear multiplied by × 1 by the non-linear digital gain circuit 224. A RAW signal (eg, 23 bits) is received. The non-linear HDR synthesizing circuit 231 can synthesize these and generate a non-linear RAW signal having a theoretical dynamic range of 186 [dB].

  The HDR compression circuit 232 receives the nonlinear RAW signal from the nonlinear HDR synthesis circuit 231. The HDR compression circuit 232 multiplies the nonlinear RAW signal by a compression coefficient to compress the dynamic range. The HDR compression circuit 232 outputs the non-linear RAW signal subjected to dynamic range compression to the ISP post-processing unit 120. Note that the dynamic range of the output signal of the HDR compression circuit 232 is determined so as to match the dynamic range that the ISP post-processing unit 120 can support.

  The number of bits of the nonlinear RAW signal output from the HDR compression circuit 232 is smaller than the number of bits of the input nonlinear RAW signal (for example, about 16 to 24 bits). Therefore, as illustrated in FIG. 12, the HDR compression circuit 232 is arranged as the last block inside the ISP preprocessing unit 200, so that it is transmitted between the ISP preprocessing unit 200 and the ISP postprocessing unit 120. It is also possible to minimize the number of bits of the signal.

  The input test image generation circuit 233 generates a test input image. The automatic exposure control circuit 234 controls imaging sensitivity by controlling part or all of the shutter speed, analog gain, linear digital gain, and non-linear digital gain based on, for example, luminance.

  As described above, the imaging / image processing apparatus according to the second embodiment is the same as the imaging / image processing apparatus according to the first embodiment except that it supports multiple-exposure type HDR imaging. The effect is the same.

  That is, this imaging / image processing apparatus applies multi-bit / linear conversion with 1: 1 input / output to each of a plurality of channels of linear RAW signals having different exposure times, and multiplies each nonlinear RAW signal by a nonlinear digital gain. To do. Therefore, the imaging / image processing apparatus can expand the dynamic range equally by using a small digital gain as compared with the case where the linear RAW signal is multiplied by the digital gain. Therefore, according to this imaging / image processing apparatus, it is possible to suppress the degradation of SNR accompanying the increase in the dynamic range. Furthermore, the point that the imaging / image processing apparatus performs defect correction / noise reduction processing using a non-linear RAW signal also contributes to the improvement of SNR.

  In addition, since the imaging / image processing apparatus performs the white balance adjustment or the linear matrix color correction after the non-linear RGB signal is once converted back into the linear RGB signal, good color reproducibility is achieved. The image processing apparatus again performs linear / nonlinear conversion of the linear RGB signal, but the input / output characteristics of this conversion can be rewritten. Therefore, the imaging / image processing apparatus can be applied to various applications.

  Furthermore, the point that this imaging / image processing apparatus performs highly accurate demosaicing using a non-linear RAW signal also contributes to the improvement of the color reproducibility described above. Note that when the color space of the non-linear RAW signal is other than RGB, the imaging / image processing apparatus first synchronizes the color space (demosaic) and then converts the color space to RGB. Therefore, the imaging / image processing apparatus can support a wide variety of color filters. In addition, since the imaging / image processing apparatus performs highly accurate contour extraction using the non-linear RAW signal, there is also an effect of improving the resolution.

(Third embodiment)
The imaging / image processing apparatus according to the third embodiment is different from the imaging / image processing apparatus according to the first embodiment or the second embodiment in that it supports monochrome HDR imaging instead of color HDR imaging. The camera module 300, ISP pre-processing unit 110 (or ISP pre-processing unit 200), and ISP post-processing unit 310 of the imaging / image processing apparatus according to the present embodiment are shown in FIGS. 13, 3 (or 13) and 14, respectively. Illustrated. In the present embodiment, the ISP preprocessing unit 110 (or ISP preprocessing unit 200) handles the luminance (Y) signal of one channel, but the basic operation is the first embodiment (or the second embodiment). ).

  A camera module 300 in FIG. 13 includes an optical lens 301, a light receiving pixel 302, an ADC 303, and a logic signal processing unit 304. The logic signal processing unit 304 may be incorporated in the same chip as the optical lens 301, the light receiving pixel 302, and the ADC 303, or may be incorporated in a different chip.

  The optical lens 301 collects incident light from the outside on the light receiving pixel 302. The light receiving pixel 302 converts (photoelectrically) the incident light that has passed through the optical lens 301 into an electric signal having a signal level corresponding to the amount of light. The light receiving pixel 302 has an electronic shutter function, and accumulates charges over a period determined by the set shutter speed. The light receiving pixel 302 outputs the generated electric signal to the ADC 303.

  The ADC 303 receives an electrical signal from the light receiving pixel 302 and performs analog / digital conversion on the electrical signal to generate a digital linear RAW signal (luminance Y signal). The ADC 303 outputs a linear luminance signal to the logic signal processing unit 304. The value of the linear luminance signal depends on the amount of incident light in the corresponding light receiving pixel 302.

  The logic signal processing unit 304 receives the linear luminance signal from the ADC 303 and performs various signal processing such as temperature compensation at high temperature, optical shading correction, and static flaw correction. Note that the logic signal processing unit 304 generally performs signal processing on the linear luminance signal in the linear region. The logic signal processing unit 304 outputs the processed linear luminance signal to the ISP preprocessing unit 110.

  14 includes an m-line synchronization circuit 311, a contour extraction circuit 312, a contour correction circuit 313, an inverse γ correction circuit 314, a γ correction circuit 315, and a brightness / contrast adjustment circuit 316. , A contour suppression / addition circuit 317, a luminance band limiting circuit 318, and an output test image generation circuit 319.

  The m line synchronization circuit 311 receives a non-linear luminance signal from the ISP preprocessing unit 110. m is, for example, 5, but may be another natural number. The m line synchronization circuit 311 synchronizes the non-linear luminance signal for m lines. The m line synchronization processing can be implemented using, for example, (m−1) H line memory. The m-line synchronization circuit 311 outputs the non-linear luminance signal synchronized with m lines to the contour extraction circuit 312, and outputs one line of non-linear luminance signal to the inverse γ correction circuit 314.

  The contour extraction circuit 312 receives the non-linear luminance signal synchronized with m lines from the m line synchronization circuit 311. The contour extraction circuit 312 extracts a contour in each of the horizontal direction and the vertical direction, for example. For example, BPF can be used for the contour extraction. The contour extraction circuit 312 outputs the extracted contour signal to the contour correction circuit 313.

  Note that contour extraction generally targets a linear luminance signal, but the contour extraction circuit 312 targets a non-linear luminance signal. As described above, for example, the non-linear luminance signal obtained by γ correction of γ = 0.45 is suitable for human visual characteristics. For this reason, the contour extraction circuit 312 can realize the contour extraction with high accuracy by the calculation using the non-linear luminance signal. By using the contour extracted in this way, further improvement in resolution can be expected.

  The contour correction circuit 313 receives the contour signal from the contour extraction circuit 312. The contour correction circuit 313 corrects the contour signal. The correction of the contour signal can include, for example, a coring process and a clip process. The contour correction circuit 313 outputs the corrected contour signal to the contour suppression / addition circuit 317.

  The inverse γ correction circuit 314 receives the non-linear luminance signal (first non-linear luminance signal) from the m line synchronization circuit 311. The inverse γ correction circuit 314 generates a linear luminance signal (first linear luminance signal) by applying inverse γ correction to the non-linear luminance signal. The inverse γ correction circuit 314 outputs the linear luminance signal to the γ correction circuit 315.

  The γ correction circuit 315 receives the linear luminance signal (second linear luminance signal) from the inverse γ correction circuit 314. The γ correction circuit 315 generates a non-linear luminance signal (second non-linear luminance signal) by applying γ correction to the linear luminance signal. The γ correction circuit 315 outputs the generated non-linear luminance signal to the brightness / contrast adjustment circuit 316.

  The γ correction input / output characteristics of the γ correction circuit 315 may be the same as or different from the γ correction input / output characteristics of the pre-γ correction circuit 113 (or the pre-γ correction circuit 212 and the pre-γ correction circuit 222). May be. Further, the γ correction input / output characteristics of the γ correction circuit 315 may be rewritable (programmable).

  In the ISP post-processing unit 310, no processing circuit exists between the inverse γ correction circuit 314 and the γ correction circuit 315. Therefore, they can be integrated as a non-linear / non-linear conversion circuit that performs (programmable) non-linear / non-linear conversion. The input / output characteristics of the nonlinear / nonlinear conversion can be derived by combining the input / output characteristics of the inverse γ correction of the inverse γ correction circuit 314 and the input / output characteristics of the γ correction of the γ correction circuit 315.

  The brightness / contrast adjustment circuit 316 receives the non-linear luminance signal from the γ correction circuit 315. The brightness / contrast adjustment circuit 316 performs brightness / contrast adjustment on the non-linear luminance signal. The brightness / contrast adjustment circuit 316 outputs the processed non-linear luminance signal to the contour suppression / addition circuit 317.

  The contour suppression / addition circuit 317 receives a contour signal from the contour correction circuit 313 and receives a non-linear luminance signal from the brightness / contrast adjustment circuit 316. The contour suppression / addition circuit 317 performs contour suppression / addition at high / low luminance on the non-linear luminance signal using the contour signal. The contour suppression / addition circuit 317 outputs the processed non-linear luminance signal to a subsequent device (for example, a display device, a recording device, a communication device, etc.) not shown.

  The luminance band limiting circuit 318 is a space LPF, for example, and limits the band of the non-linear luminance signal. The output test image generation circuit 319 generates a test output image.

  As described above, the imaging / image processing apparatus according to the third embodiment applies the multi-bit / linear conversion of input / output 1: 1 to the linear luminance signal and multiplies the nonlinear luminance signal by the nonlinear digital gain. To do. Therefore, the imaging / image processing apparatus can expand the dynamic range equally by using a small digital gain as compared to the case where the linear gain signal is multiplied by the digital gain. Therefore, according to this imaging / image processing apparatus, it is possible to suppress the degradation of SNR accompanying the increase in the dynamic range. Furthermore, the point that the imaging / image processing apparatus performs defect correction / noise reduction processing using the non-linear luminance signal also contributes to the improvement of SNR. Furthermore, since the imaging / image processing apparatus performs highly accurate contour extraction using the non-linear luminance signal, there is also an effect of improving the sense of resolution.

  In addition, this imaging / image processing apparatus once converts a non-linear luminance signal back to a linear luminance signal and then performs linear / non-linear conversion again. The input / output characteristics of this linear / non-linear conversion (or non-linear / non-linear conversion) are rewritten. Is possible. Therefore, the imaging / image processing apparatus can be applied to various applications.

  The various functional units described in the above embodiments may be realized by using a circuit. The circuit may be a dedicated circuit that realizes a specific function, or may be a general-purpose circuit such as a processor.

  At least a part of the processing of each of the above embodiments can also be realized by using a general-purpose computer as basic hardware. A program for realizing the above processing may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable file or an executable file. Examples of the recording medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The recording medium may be any recording medium as long as it can store the program and can be read by the computer. The program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

10, 100, 300 ... Camera module 20 ... ISP core 21, 110, 200 ... ISP pre-processing unit 22, 120, 310 ... ISP post-processing unit 101, 301 ... Optical lens 102 ..Color filters 103, 302... Light receiving pixels 104, 303... ADC
105, 304 ... logic signal processing unit 111 ... black level adjustment circuit 112, 211, 221 ... linear digital gain circuit 113, 212, 222 ... pre-γ correction circuit 127, 315 ... main γ Correction circuit 114, 213, 223 ... Scratch correction / noise reduction processing circuit 115, 214, 224 ... Non-linear digital gain circuit 116, 232 ... HDR compression circuit 117, 233 ... Input test image generation circuit 118 , 234... Automatic exposure control circuit 121, 311... M-line synchronization circuit 122, 312... Contour extraction circuit 123 ... contour correction / edge extraction circuit 124 .. demosaic (RGB synchronization) circuit 125, 314 ... Inverse γ correction circuit 126 ... Linear RGB processing (WB multiplication, color correction) circuit 128 YUV matrix circuit 129 ... YUV processing circuit 130 ... auto white balance circuit 131 ... YUV band limiting circuit 132,319 ... output test image generation circuit 133 ... RGB re-matrix (YUV / RGB inverse) Conversion) circuit 201... Linear HDR synthesizing circuit 231... Non-linear HDR synthesizing circuit 202... Signal dividing circuit 313... Contour correction circuit 316... Brightness / contrast adjustment circuit 317. Circuit 318 ... Luminance band limiting circuit

Claims (11)

A first linear / nonlinear conversion with an input / output ratio of 1: 1 is applied to a second linear RAW signal based on the first linear RAW signal, and the first number having a larger number of bits than that of the second linear RAW signal. A first linear / nonlinear conversion circuit for generating a non-linear RAW signal;
A non-linear digital gain circuit that multiplies a second non-linear RAW signal based on the first non-linear RAW signal by a digital gain greater than x0 to obtain a third non-linear RAW signal;
Applying a non-linear / linear conversion equivalent to the inverse of the first linear / non-linear conversion to the first non-linear luminance signal or the non-linear RGB signal based on the third non-linear RAW signal, thereby providing a first linear luminance signal Or applying a second linear / nonlinear conversion to the non-linear / linear conversion circuit for generating a linear RGB signal and the second linear luminance signal or the linear RGB signal based on the first linear luminance signal or the linear RGB signal, An image processing apparatus comprising: a second linear / nonlinear conversion circuit that generates a second non-linear luminance signal or a non-linear RGB signal. A demosaicing circuit that generates a first non-linear RGB signal by demosaicing a fifth non-linear RAW signal based on the third non-linear RAW signal;
A first processing circuit that performs at least one of white balance adjustment and linear matrix color correction on the first linear RGB signal to generate the second linear RGB signal;
The image processing apparatus according to claim 1. The input / output characteristics of the second linear / nonlinear conversion can be rewritten.
The image processing apparatus according to claim 1. The input / output characteristics of the first linear / nonlinear conversion are fixed,
The input / output characteristics of the non-linear / linear conversion are fixed,
The image processing apparatus according to claim 1.   2. The apparatus according to claim 1, further comprising a second processing circuit that applies at least one of a scratch correction process and a noise reduction process to the first non-linear RAW signal to generate the second non-linear RAW signal. Image processing device. A compression circuit for compressing a dynamic range of the third nonlinear RAW signal and generating a fourth nonlinear RAW signal;
The demosaic circuit is incorporated in a chip different from the compression circuit.
The image processing apparatus according to claim 2.   When the color space of the fifth non-linear RAW signal is different from RGB, the demosaic circuit synchronizes each color arrangement included in the fifth non-linear RAW signal and then converts the color space to RGB. The image processing apparatus according to claim 2, wherein the first nonlinear RGB signal is generated. A linear digital gain circuit that obtains the second linear RAW signal by multiplying the first linear RAW signal by a digital gain for preset white balance adjustment;
The first processing circuit performs at least main white balance adjustment on the first linear RGB signal to generate the second linear RGB signal.
The image processing apparatus according to claim 2.   The image processing apparatus according to claim 1, further comprising a contour extraction circuit that extracts a contour in each of a horizontal direction and a vertical direction from a fifth nonlinear RAW signal based on the third nonlinear RAW signal. A first linear RAW signal having a number of bits larger than that of the second linear RAW signal by applying linear / non-linear conversion of input / output 1: 1 to the second linear RAW signal based on the first linear RAW signal. A linear / non-linear conversion circuit that generates
A non-linear digital gain circuit that multiplies a second non-linear RAW signal based on the first non-linear RAW signal by a digital gain greater than x0 to obtain a third non-linear RAW signal;
A monochrome image processing apparatus, comprising: a non-linear / non-linear conversion circuit that applies a non-linear / non-linear conversion to the first non-linear luminance signal based on the third non-linear RAW signal to generate a second non-linear luminance signal.   The monochrome image processing apparatus according to claim 10, wherein input / output characteristics of the non-linear / non-linear conversion are rewritable.