JP2017163260A - Imaging control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve high-sensitivity while suppressing deterioration of a SNR.SOLUTION: According to an embodiment, an imaging control device includes a liner/non-liner conversion circuit, a non-liner digital gain circuit, a luminance calculation circuit, and a sensitivity control part. The liner/non-liner conversion circuit forms a first non-liner pixel signal with more bit numbers than a second liner pixel signal by applying a liner/non-liner conversion of a 1:1 input output to the second liner pixel signal on the basis of a first liner pixel signal formed by an analog pixel part. The non-liner digital gain circuit multiples a non-liner digital gain to the first non-liner pixel signal, and thereby acquires a second non-liner pixel signal. The luminance calculation circuit calculates a luminance on the basis of the second non-liner pixel signal. The sensitivity control part controls an imaging sensitivity by controlling at least non-liner digital gain on the basis of the luminance.SELECTED DRAWING: Figure 1

Description

  Embodiments relate to control of imaging sensitivity, that is, automatic exposure (AE) technology.

  Conventionally, solid-state imaging devices such as CMOS (Complementary Metal Oxide Semiconductor) sensors do not have to adjust the optical aperture by adjusting the shutter speed (that is, the exposure time) and the gain applied to the pixel signal. Imaging can be performed with a sensitivity suitable for the brightness (illuminance) of the subject.

  However, since the SNR (Signal to Noise Ratio) deteriorates as the gain increases, it is not practical to set the gain to an unlimited large value. Similarly, the subject blur increases as the shutter speed decreases, and the upper limit of the shutter speed is limited by the frame rate when a moving image is captured. In particular, when performing fluorescent lamp flickerless imaging, the settable shutter speed is further limited. Further, for example, in an in-vehicle camera, the sensitivity is lowered due to the miniaturization of the pixel cell size accompanying the increase in the number of pixels.

  That is, there is a limit to the illuminance of an object that can be appropriately captured by a solid-state image sensor, and in particular, visible 1 million magnification light imaging equivalent to human vision (for example, only by adjusting the shutter speed of the CMOS sensor and subsequent gain adjustment). ) It is difficult to realize. For this reason, for example, a situation may occur in which an in-vehicle camera cannot capture a subject that can be seen by a human when traveling at night.

Japanese Patent Laying-Open No. 2015-192222

  An object of the embodiment is to achieve high sensitivity while suppressing deterioration of SNR.

  According to the embodiment, the imaging control apparatus includes a linear / nonlinear conversion circuit, a non-linear digital gain circuit, a luminance calculation circuit, and a sensitivity control unit. The linear / non-linear conversion circuit applies the input / output 1: 1 linear / non-linear conversion to the second linear pixel signal based on the first linear pixel signal generated by the analog pixel unit, so that the second The first non-linear pixel signal having a larger number of bits than the linear pixel signal is generated. The non-linear digital gain circuit multiplies the first non-linear pixel signal by the non-linear digital gain to obtain a second non-linear pixel signal. The luminance calculation circuit calculates the luminance based on the second non-linear pixel signal. The sensitivity control unit controls the imaging sensitivity by controlling at least the nonlinear digital gain based on the luminance.

1 is a block diagram illustrating an imaging apparatus according to a first embodiment. The figure which illustrates the pixel array of a RAW signal. 3 is a graph illustrating gamma correction performed by the γ correction circuit of FIG. 1. The table which illustrates an input saturation level setting and the linear digital gain and nonlinear digital gain corresponding to the said input saturation level setting. The figure which illustrates the color arrangement | sequence of a 3x3 pixel area | region extractable from the pixel arrangement | sequence of FIG. The figure which illustrates the color arrangement | sequence of a 3x3 pixel area | region extractable from the pixel arrangement | sequence of FIG. The figure which illustrates the color arrangement | sequence of a 3x3 pixel area | region extractable from the pixel arrangement | sequence of FIG. The figure which illustrates the color arrangement | sequence of a 3x3 pixel area | region extractable from the pixel arrangement | sequence of FIG. The figure which illustrates the weight for luminance calculation allocated in common to the pixel area of Drawing 5A, Drawing 5B, Drawing 5C, and Drawing 5D. The block diagram which illustrates the sensitivity control part of FIG. The figure which shows an example of the frequency distribution which the frequency calculation circuit of FIG. 7 calculates. The graph which illustrates the reverse gamma correction which the reverse gamma correction circuit of FIG. 7 performs. FIG. 8 is a diagram illustrating an integration target region in which the luminance integration circuit of FIG. 7 integrates linear luminance. The figure which illustrates the weight allocated to the integration | accumulation object area | region of FIG. FIG. 8 is a diagram illustrating counter addition / subtraction values generated by the counter control circuit of FIG. 7. The figure which shows the numerical example of FIG. The graph which illustrates transition of the parameter in imaging sensitivity control concerning a comparative example. 2 is a graph illustrating the transition of parameters in imaging sensitivity control in the imaging apparatus of FIG. 1.

  Hereinafter, embodiments will be described with reference to the drawings. Hereinafter, the same or similar elements as those already described are denoted by the same or similar reference numerals, and redundant description is basically omitted.

(First embodiment)
As illustrated in FIG. 1, the imaging apparatus according to the first embodiment includes an optical lens 101, a color filter 102, a light receiving pixel 103, an ADC (Analog-to-Digital Converter) 104, and a signal processing circuit. 110 and a sensitivity control unit 130. In the following description, the optical lens 101, the color filter 102, the light receiving pixel 103, and the ADC 104 may be collectively referred to as an analog pixel unit. Also, elements related to control of imaging sensitivity (particularly, part of the signal processing circuit 110 and the sensitivity control unit 130) in the imaging apparatus of FIG. 1 can be collectively referred to as an imaging control apparatus.

  The optical lens 101 condenses incident light from the outside on the light receiving pixel 103 via the color filter 102. The light receiving pixel 103 converts (photoelectric) the incident light that has passed through the optical lens 101 into an electric signal having a signal level corresponding to the amount of light. The light receiving pixel 103 has an electronic shutter function, and accumulates charges over a period (exposure time) determined by a set shutter speed (ES). The light receiving pixel 103 outputs the generated electric signal to the ADC 104.

  The shutter speed of the light receiving pixel 103 is controlled by the sensitivity control unit 130. The shutter speed affects the gain (sensitivity) of the entire imaging apparatus in FIG. If other elements (specifically, analog gain (AG), which is the conversion gain of ADC 104, and linear digital gain (LDG) and non-linear digital gain (NLDG) applied in signal processing circuit 110) are fixed, The sensitivity is also doubled by halving the shutter speed (that is, the exposure time is doubled).

  The ADC 104 receives an electrical signal from the light receiving pixel 103 and performs analog / digital conversion on the electrical signal to generate a digital linear RAW signal (first linear pixel signal). The ADC 104 outputs a linear RAW signal to the signal processing circuit 110.

  The analog gain that is the conversion gain of the ADC 104 is controlled by the sensitivity control unit 130. The analog gain affects the gain of the entire imaging apparatus in FIG. If other elements (specifically, the shutter speed, the linear digital gain, and the non-linear digital gain described above) are fixed, doubling the analog gain doubles the sensitivity.

  The value of the RAW signal depends on the amount of incident light in the corresponding light receiving pixel 103. The RAW signal is, for example, an RGB signal having a primary color Bayer array as shown in FIG. In the pixel array of FIG. 2, a block of four pixels including a red pixel (R), two green pixels (Gr, Gb), and a blue pixel (B) is regularly arranged. Note that the pixel array of the RAW signal is determined depending on the array of the color filters 102. The pixel array of the RAW signal is unchanged during a series of image processing (ISP preprocessing) performed by the signal processing circuit 110 described later.

  The signal processing circuit 110 receives a linear RAW signal from the ADC 104. The signal processing circuit 110 performs various image processing including application of linear digital gain and nonlinear digital gain to the linear RAW signal. The linear digital gain and the non-linear digital gain are controlled by the sensitivity control unit 130. The signal processing circuit 110 outputs the processed signal to a subsequent device (for example, a display device, a communication device, a recording device, etc.) not shown. Furthermore, the signal processing circuit 110 feeds back a luminance signal representing the (non-linear) luminance of the pixel to the sensitivity control unit 130 for sensitivity control.

  Specifically, the signal processing circuit 110 includes a first digital gain circuit 111, a γ correction circuit 112, a 5-line synchronization circuit 113, a flaw detection / correction circuit 114, a noise reduction circuit 115, a second Digital gain circuit 116, three-line synchronization circuit 117, HDR compression circuit 118, luminance calculation circuit 119, and HDR compression coefficient generation circuit 120.

  The first digital gain circuit 111 (linear digital gain circuit) receives a linear RAW signal from the ADC 104. The first digital gain circuit 111 multiplies the linear RAW signal by the linear digital gain to obtain a first gain-corrected signal (second linear pixel signal). The first digital gain circuit 111 outputs the first gain corrected signal to the γ correction circuit 112.

  This linear digital gain is controlled by the sensitivity control unit 130. The linear digital gain affects the gain of the entire imaging apparatus in FIG. If other elements (specifically, the shutter speed, analog gain, and non-linear digital gain) are fixed, the sensitivity is doubled by doubling the linear digital gain.

  This linear digital gain includes a first gain set for each color for correcting the color tone (typically adjusting the white balance) in addition to the imaging sensitivity setting of the entire imaging apparatus in FIG. It may be determined depending on at least one of the second gains set according to the position in the screen for shading correction.

  The γ correction circuit 112 receives the first gain corrected signal from the first digital gain circuit 111. The γ correction circuit 112 applies gamma correction (corresponding to linear / non-linear conversion) to the first gain-corrected signal, so that the gamma-corrected signal (the first number) having a larger number of bits than the first gain-corrected signal. 1 non-linear pixel signal). The γ correction circuit 112 outputs the gamma corrected signal to the 5-line synchronization circuit 113.

  Specifically, the γ correction circuit 112 performs the first multi-bit gamma correction of the input / output 1: 1 shown in the following formula (1) (that is, there is no overlap between outputs corresponding to different inputs). It can be applied to gain corrected signals.

  Equation (1) shows ideal characteristics in gamma correction, and in this case, ideal linear / nonlinear conversion obtained by calculation (rounded off after the decimal point).

  In Equation (1), OUT (x) represents the value of the gamma corrected signal when the input signal level of the ADC 104 is x [%], and IN (x) is the first value when the input signal level is x [%]. 1 represents the value of the gain-corrected signal. IN (0) and IN (100) represent the setting values of the first gain-corrected signal when the input signal level is 0 [%] (black) and 100 [%] (white), respectively.

  γ is a gamma correction coefficient determined according to input / output characteristics of a display device (not shown) provided at the subsequent stage of the signal processing circuit 110, for example. For example, in the case of γ = 0.45, the number of output bits becomes 2 bits or more than the number of input bits if the input / output 1: 1 is maintained.

  OUT (0) and OUT (100) represent the set values of the gamma corrected signal when the input signal level is 0 [%] (black) and 100 [%] (white), respectively. If OUT (100) -OUT (0) is increased, the number of bits of the gamma corrected signal can be arbitrarily increased. Since the gamma correction is an input / output 1: 1 linear / nonlinear conversion, no matter how much the number of bits of the gamma-corrected signal is increased compared to the first gain-corrected signal, the gradation of the gamma-corrected signal itself The number does not fall below or exceed the number of gradations of the first gain-corrected signal. However, if the number of bits of the gamma-corrected signal is set large, there is an effect of improving the quantization accuracy in the process of various bit operations (specifically, flaw correction and noise reduction) described later.

  In the case of IN (0) = 256, IN (100) = 2816, γ = 0.45, OUT (0) = 15360, OUT (100) = 56320, the γ correction circuit 112 performs the gamma correction illustrated in FIG. Is applied to the 12-bit first gain-corrected signal to generate a 16-bit gamma-corrected signal.

  According to the gamma correction of FIG. 3, the first gain corrected signal is given an individual gain for each signal level. The gamma-corrected signal has a resolution (resolution) on the low luminance side (side where the value of the first gain corrected signal is small) compared to the high luminance side (side where the value of the first gain corrected signal is large). Is rough. On the low luminance side, the gradation step size is large and is likely to change (the gradation step size monotonously decreases as the luminance increases). On the other hand, on the high luminance side, the gradation step width is small, and the gradation step width hardly changes. As described above, since the input / output is 1: 1, the gradation step size does not become 0 even on the high luminance side.

  In the 12-bit input and 16-bit output γ correction of FIG. 3, the maximum step size near black = 1199 and the step size near white = 7-8. By using flaw correction and noise reduction, the step size near black = 16 and the step size near white = 1 can be suppressed, and the quantization accuracy can be improved. This is an improvement effect due to a so-called non-linear characteristic that the step size of the gradation is large and easily changes on the low luminance side.

  The gamma correction may be defined as a function, or may be defined using a table such as a lookup table. Further, the γ correction can be replaced with other input / output 1: 1 multi-bit linear / non-linear conversion.

  The 5-line synchronization circuit 113 can be mounted using, for example, a 4H line memory of SRAM (Static Random Access Memory). The 5-line synchronization circuit 113 receives the γ corrected signal from the γ correction circuit 112. The 5-line synchronization circuit 113 extracts a 5 × 5 pixel region by synchronizing 5 lines of the γ-corrected signal.

  A pixel occupying the center of the extracted 5 × 5 pixel region is a target for defect detection in the defect detection / correction circuit 114. In addition to the target pixel, scratch detection is performed using the values of eight pixels belonging to the surrounding pixel group having the same color as the target pixel. Scratch detection target pixels are sequentially selected according to the scanning order of the effective pixel region of the light receiving pixel 103. The 5-line synchronization circuit 113 outputs a pixel signal representing the pixel values of the target pixel and the surrounding pixel group to the defect detection / correction circuit 114.

  The defect detection / correction circuit 114 receives pixel signals representing pixel values of the target pixel and the surrounding pixel group from the 5-line synchronization circuit 113. The flaw detection / correction circuit 114 detects flaws in the target pixel. If the target pixel is detected as a flaw, the value of the target pixel (of the gamma corrected signal) is corrected using the value of the surrounding pixel group. To do. The scratch detection / correction circuit 114 outputs the scratch corrected signal to the noise reduction circuit 115.

  The noise reduction circuit 115 generates a noise reduction signal by applying noise reduction processing to the scratch corrected signal. The noise reduction circuit 115 outputs a noise reduction signal to the second digital gain circuit 116.

  Specifically, the noise reduction circuit 115 applies a median filter to the noise reduction target pixel and the values of the surrounding pixel group, and extracts the median value of these pixels. The noise reduction circuit 115 calculates a difference between the extracted median value and the value of the target pixel as a noise basic component. The noise reduction circuit 115 multiplies this basic noise component by a coefficient that determines the intensity of noise reduction, and subtracts these products from the value of the target pixel to generate a noise reduction signal. Note that the noise reduction circuit 115 may reduce noise using a technique different from the technique described here.

  The second digital gain circuit 116 (nonlinear digital gain circuit) receives the noise reduction signal from the noise reduction circuit 115. The second digital gain circuit 116 multiplies the noise reduction signal by a non-linear digital gain to generate a second gain-corrected signal (second non-linear pixel signal). The second digital gain circuit 116 outputs the second gain corrected signal to the three-line synchronization circuit 117.

  This nonlinear digital gain is controlled by the sensitivity control unit 130. The non-linear digital gain affects the gain of the entire imaging apparatus in FIG. As will be described later, if other factors (specifically, the shutter speed, analog gain, and linear digital gain described above) are fixed, the sensitivity is doubled by increasing the non-linear digital gain by about 1.366 times. It becomes.

  That is, this non-linear digital gain can improve the sensitivity equally at a lower magnification than the shutter speed, analog gain, and linear digital gain described above. Therefore, if the non-linear digital gain is increased in preference to the linear digital gain, the sensitivity can be improved while suppressing the deterioration of SNR.

  This non-linear digital gain is determined depending on the third gain set in common for all colors based on the HDR ratio (dynamic range expansion ratio) in addition to the imaging sensitivity setting of the entire imaging apparatus of FIG. Also good. The third gain is smaller than the HDR ratio if the HDR ratio is greater than x1.

  The nonlinear digital gain may be determined in consideration of the first gain or the second gain described above in addition to the imaging sensitivity setting (and the third gain) of the entire imaging apparatus in FIG. For example, the first digital gain circuit 111 may be removed, and the second digital gain circuit 116 may further perform color tone correction or shading correction. However, while the first digital gain circuit 111 multiplies the linear RAW signal by the digital gain, the second digital gain circuit 116 multiplies the non-linear noise reduction signal by the digital gain. Therefore, the gains required for color tone correction and shading correction differ between the first digital gain circuit 111 and the second digital gain circuit 116.

FIG. 4 shows an input saturation level (ie, sensitivity) setting (IDRS: Input Dynamic Range Setting), and a linear digital gain and a non-linear digital gain corresponding to the sensitivity setting. When both the linear digital gain and the non-linear digital gain are set to x1 or more, there is an effect of increasing the sensitivity by increasing the gradation interval. For example, if the linear digital gain (LDG) is doubled, the sensitivity is also doubled. On the other hand, the non-linear digital gain required to double the sensitivity is only about 1.366 times. This is clear from the formula (2) described later, and 2 ^0.45 = 1.366.

Also, sensitivity control by linear digital gain sacrifices quantization accuracy greatly. Multiplication of the linear digital gain corresponds to an n-bit left shift operation. Thus, if a linear digital gain to the sensitivity to 2 ⁿ times the 2 ⁿ times, the upper n bits of information multiplication previous pixel signal is lost, substantial quantization accuracy is deteriorated n bits . In other words, in order to maintain the quantization accuracy of the pixel signal after multiplication as 12 bits, it is theoretically necessary to multiply the pixel signal having the quantization accuracy of (12 + n) bits by the linear digital gain. . If the sensitivity is increased 256 times, the substantial quantization accuracy is deteriorated by 8 bits. If the quantization accuracy of a 12-bit pixel signal is lowered by 8 bits, the substantial quantization accuracy is 4 bits, which is not a level that can be used as a pixel signal.

  On the other hand, sensitivity control by nonlinear digital gain does not sacrifice quantization accuracy much. For example, since the non-linear digital gain necessary for increasing the sensitivity to 256 times is less than 16 times, the substantial deterioration of the quantization accuracy can be suppressed to less than 4 bits.

  Further, although the gamma correction itself increases the number of bits of the signal, it does not improve the quantization accuracy. However, the improvement effect of the quantization accuracy by flaw correction and noise reduction is at least +2 bits (in practice, from +3 bits to +6 bits) It can be assumed that an improvement effect is expected). Therefore, the quantization accuracy of a 16-bit noise reduction signal is estimated to be at least 12 + 2 = 14 bits. That is, even if the non-linear digital gain is increased by about 12.13 times in order to increase the sensitivity to 256 times, the second gain-corrected signal can secure a quantization accuracy of at least 10 bits and is sufficient as a pixel signal. It is a level that can be used.

  In order to ensure the SNR, it is preferable to suppress the deterioration of the quantization accuracy due to the multiplication of the non-linear digital gain to the extent that it is covered by the improvement effect of the quantization accuracy by the flaw correction and noise reduction. As described above, the improvement effect of quantization accuracy by flaw correction and noise reduction is about 3 to 6 bits. However, even if the upper limit of the non-linear digital gain is set to x8.877 times with a margin, x128 is converted into linear gain. A sensitivity improvement effect equivalent to double is obtained.

  The non-linear digital gain (NLDG) that achieves a given sensitivity can be calculated by the following formula (2) based on the linear digital gain (LDG) that achieves the same sensitivity.

  In Equation (2), the power index of LDG is set to 0.45, but this power index is equal to the value of the γ correction coefficient (γ) used by the γ correction circuit 112.

  The 3-line synchronization circuit 117 can be mounted using, for example, an SRAM 2H line memory. The three-line synchronization circuit 117 receives the second gain-corrected signal from the second digital gain circuit 116. The 3-line synchronization circuit 117 extracts a 3 × 3 pixel region by synchronizing the second gain-corrected signal with 3 lines.

  The extracted 3 × 3 pixel area is subjected to luminance calculation in the luminance calculation circuit 119. The 3-line synchronization circuit 117 outputs the 3 × 3 pixel area to the luminance calculation circuit 119, and the HDR compression circuit 118 uses the value of the central pixel of the 3 × 3 pixel area (ie, the second gain-corrected signal). To output.

  The HDR compression circuit 118 receives the second gain-corrected signal from the second digital gain circuit 116 and receives the compression coefficient from the HDR compression coefficient generation circuit 120. The HDR compression circuit 118 generates an HDR compressed signal by compressing the dynamic range of the second gain-corrected signal using a compression coefficient. The HDR compression circuit 118 outputs the HDR compression signal to a subsequent device (for example, a display device) (not shown). The dynamic range of the HDR compressed signal is determined so as to be compatible with the dynamic range that can be supported by the latter apparatus.

  Specifically, the HDR compression circuit 118 compresses the dynamic range of the second gain-corrected signal while maintaining local contrast as much as possible using, for example, tone mapping.

  The luminance calculation circuit 119 receives a 3 × 3 pixel region from the three-line synchronization circuit 117. The luminance calculation circuit 119 obtains a luminance signal representing the (non-linear) luminance value of the region by calculating a weighted average of the pixel values of the 3 × 3 pixel region. The luminance calculation circuit 119 outputs a luminance signal for each region to the HDR compression coefficient generation circuit 120 and the sensitivity control unit 130.

  For example, the color arrangement of a 3 × 3 pixel region that can be extracted from the pixel arrangement of FIG. 2 can be roughly divided into four patterns illustrated in FIGS. 5A, 5B, 5C, and 5D. That is, a pattern in which the color of the center pixel is red (R) (FIG. 5A), a color in which the color of the center pixel is green (Gr), and a color of adjacent pixels on the same line as the center pixel is red (FIG. 5B), There are four patterns: a pattern in which the color of the center pixel is green (Gb) and the color of an adjacent pixel in the same line as the center pixel is blue (B) (FIG. 5C), and a pattern in which the color of the center pixel is blue (FIG. 5D) . In general, the following formula (3) is known as a conversion formula from RGB to luminance (Y in YUV format).

  However, the luminance calculation circuit 119 may calculate the luminance with an accuracy that can be used for controlling the imaging sensitivity and the HDR compression coefficient. Therefore, the luminance calculation circuit 119 calculates the weighted average of the following formula (4) as pseudo luminance.

  Using Equation (4), the luminance calculation circuit 119 simply calculates the pseudo luminance using the weights shown in FIG. 6 regardless of the color pattern of the 3 × 3 pixel region in any of the above four patterns. it can. In addition, when distinguishing Gr and Gb, it can replace with Numerical formula (4) and can use the following Numerical formula (5).

  The HDR compression coefficient generation circuit 120 receives a luminance signal from the luminance calculation circuit 119 and receives a frequency distribution described later from the sensitivity control unit 130. The HDR compression coefficient generation circuit 120 generates a compression coefficient for each luminance class based on the frequency distribution. Then, the HDR compression coefficient generation circuit 120 applies the compression coefficient applied (multiplied) to each of the second gain corrected signals by the HDR compression circuit 118 based on the luminance signal corresponding to the second gain corrected signal. decide. The HDR compression coefficient generation circuit 120 outputs the compression coefficient to the HDR compression circuit 118.

  The sensitivity control unit 130 receives a luminance signal from the luminance calculation circuit 119. The sensitivity control unit 130 controls the imaging sensitivity by controlling the shutter speed, analog gain, linear digital gain, and nonlinear digital gain described above based on the luminance signal. Further, as will be described later, the sensitivity control unit 130 calculates the frequency distribution of the luminance signal for each 1V. The sensitivity control unit 130 outputs the frequency distribution to the HDR compression coefficient generation circuit 120 as well as using the frequency distribution internally for controlling the imaging sensitivity.

  Specifically, as illustrated in FIG. 7, the sensitivity control unit 130 includes a frequency distribution calculation circuit 131, an inverse γ correction circuit 132, a luminance integration circuit 133, an index calculation circuit 134, and a parameter control unit 140. Including.

  The frequency distribution calculation circuit 131 receives a luminance signal from the luminance calculation circuit 119. The frequency distribution calculation circuit 131 calculates the frequency distribution of the luminance signal as illustrated in FIG. 8 for each 1V. In FIG. 8, the vertical axis represents the frequency, and the horizontal axis represents the luminance class. The frequency distribution calculation circuit 131 outputs the frequency distribution to the HDR compression circuit 118 and the parameter control unit 140.

  The frequency distribution calculation circuit 131 can calculate a frequency distribution by classifying and counting luminance signals for 1 V into a plurality of luminance classes. The width of the luminance class is determined depending on the characteristics of γ correction applied by the γ correction circuit 112 and the non-linear digital gain.

For example, the number of bits of the input / output signal for γ correction is “12” and “16”, respectively, and IN (0) = 256, IN (100) = 2816, γ = 0.45, OUT (0) = 15360, OUT (100) = 56320 and the non-linear digital gain is x16.56424 times. In this case, substituting IN (x) = 0 and 4095 (= 2 ¹² −1) into Equation (1), respectively, derives that the minimum and maximum values of the gamma-corrected signal are 827 and 64513, respectively. it can.

  Since the luminance represented by the luminance signal is a weighted average of the second gain-corrected signal, its maximum value and minimum value are equal to the maximum value and minimum value of the second gain-corrected signal, respectively. Therefore, the maximum luminance value can be calculated by the following formula (6).

  On the other hand, the minimum luminance value represented by the luminance signal can be calculated by the following mathematical formula (7).

  If 15360 corresponding to OUT (0) to 56320 corresponding to OUT (100) are divided into 10 classes (that is, in increments of 10%), the width of one luminance class is 4096 (= (56320-15360). / 10) value.

  Therefore, the region with positive luminance can be divided into 203 luminance classes as shown in the following formula (8).

  Similarly, the area where the luminance is negative can be divided into 56 luminance classes as shown in the following formula (9).

  Therefore, according to the above numerical example, the frequency distribution has a total of 259 luminance classes of 203 in the positive region and 56 in the negative region.

  In the example of FIG. 7, the frequency distribution is calculated using a (non-linear) luminance signal after gamma correction. Human visual characteristics are known to be sensitive to low brightness while insensitive to high brightness. Since gamma correction is a correction that matches such visual characteristics of humans, humans can perceive the luminance signal after gamma correction almost uniformly from the dark part to the bright part. Therefore, by using a non-linear luminance signal, it is possible to calculate a non-biased frequency distribution without changing the width of the luminance class between the low luminance side and the high luminance side.

  The inverse γ correction circuit 132 receives the luminance signal from the luminance calculation circuit 119. The inverse γ correction circuit 132 generates a linear luminance signal by applying inverse gamma correction (corresponding to non-linear / linear conversion) to the luminance signal. The inverse γ correction circuit 132 outputs a linear luminance signal to the luminance integrating circuit 133. Specifically, the inverse γ correction circuit 132 uses a gamma correction coefficient (for example, 2.2 = 1 / 0.45) equal to the inverse of the gamma correction coefficient (γ) used by the γ correction circuit 112 to generate a luminance signal. Apply (reverse) gamma correction to.

  For example, the input (luminance signal) is converted to 12 bits, and the values in the case of 0 [%] (black) and 100 [%] (white) are 960 (= 15360/16) and 3520 (= 56320/16), respectively. Set to The number of bits of the output (linear luminance signal) is 16, and the values in the case of 0 [%] (black) and 100 [%] (white) are 4096 (= 256 × 16) and 45056 (= 2816 ×), respectively. 16). Inverse gamma correction based on such numerical examples is illustrated in FIG.

  The luminance integration circuit 133 receives the linear luminance signal from the inverse γ correction circuit 132. The luminance integration circuit 133 integrates the values of the linear luminance signal in each of a plurality of integration target regions every 1V. The plurality of integration target areas may be determined as shown in FIG. 10, for example. The luminance integration circuit 133 outputs the luminance integration value of each integration target area to the index calculation circuit 134.

  In the example of FIG. 10, a part of the effective pixel area is defined as 9 × 7 integration target areas (A1 to A63), and the remaining part is defined as a non-accumulation area (A0). Each integration target area is composed of p × q pixels (≧ 2). Note that it is not necessary to integrate all of the p × q pixels, and some of them can be thinned out. The luminance integration circuit 133 integrates the linear luminance signal value in each integration target area.

  The reason for integrating the linear luminance signal using inverse γ correction is that the obtained integrated value is proportional to the camera sensitivity, and exposure control using the shutter speed, analog gain, linear digital gain, and digital gain (hereinafter referred to as “gain control”) , ALC (also called Automatic Luminance-level Control) is easy to perform. Here, ALC is a type of AE, and refers to a technology that realizes sensitivity control by controlling the shutter speed and various gains without depending on the illumination light source, the optical aperture in the imaging lens, or the like.

  In the example of FIG. 10, the shape, area (size), and integrated pixel number of each integration target region are uniform. By imposing such conditions, it is possible to simplify the rearrangement of the luminance integrated values described later. Note that the shape, area, and number of integrated pixels of each integration target region are not limited to the example of FIG. 10 and can be arbitrarily set.

  A weight as illustrated in FIG. 11 is assigned to each integration target area. The weight assignment shown in FIG. 11 is suitable for center-weighted photometry, and the weight decreases as the distance from the center increases. In the example of FIG. 11, since the weights assigned to the four corners of the integration target area are 0, the integration of these integration target areas can be omitted. The weight assignment shown in FIG. 11 is merely an example, and any assignment can be adopted.

  In the example of FIG. 11, since the weight is an integer from 0 to 7, this information can be expressed by 3 bits. On the other hand, since the area number is an integer from 1 to 63, 6 bits are required to express this information. Furthermore, the index calculation circuit 134 in the subsequent stage only needs to be able to calculate the weighted average of the luminance integrated values of each region, and does not need to grasp the region number corresponding to each luminance integrated value. Therefore, the luminance integration circuit 133 may add a weight (3-bit information) to the luminance integration value instead of the area number (6-bit information) and output the luminance integration value to the index calculation circuit 134 in order to reduce the circuit scale.

  The index calculation circuit 134 receives the luminance integration value of each region from the luminance integration circuit 133. Here, there is a possibility that values of areas with excessively high luminance or areas with excessively low luminance may be mixed in the luminance integrated value. Especially when such a value is multiplied by a large weight, the weighted average of the luminance integrated value is calculated. There is a risk that the current screen average brightness level may deviate greatly.

  Therefore, the index calculation circuit 134 may sort the luminance integrated values in ascending or descending order, and calculate the weighted average after excluding the upper r (for example, 31) value and the lower s (for example, 16) value. In this way, the effect of backlight correction can be obtained by excluding the luminance integrated value in a region where the luminance is excessively high or low.

  The index calculation circuit 134 outputs a weighted average of the luminance integrated values to the parameter control unit 140 as an index for sensitivity control. It should be noted that the process of excluding the luminance integrated value described above can be omitted.

  The parameter control unit 140 receives the frequency distribution from the frequency distribution calculation circuit 131 and receives the index from the index calculation circuit 134. The parameter control unit 140 adjusts the sensitivity setting value based on the index, and controls the shutter speed, analog gain, linear digital gain, and nonlinear digital gain according to the adjusted sensitivity.

  The parameter control unit 140 need not always perform sensitivity adjustment. Specifically, the parameter control unit 140 may switch whether to continue sensitivity adjustment or to omit (maintain current sensitivity setting) based on the frequency distribution. For example, when the high-luminance pixels temporarily increase during backlighting or when traveling in a short tunnel, the frequency distribution may have a new mountain on the high-luminance side in addition to the mountain on the low-luminance side. When sensitivity adjustment is executed in such a situation, the parameter control unit 140 may lower the sensitivity due to the influence of high-luminance pixels. However, it is not always preferable to adjust the sensitivity by being swayed by such temporary brightness fluctuations. Therefore, for example, when a new mountain occurs on the high luminance side, the sensitivity may be stabilized by temporarily omitting the sensitivity adjustment.

  In addition, since parameter control should just be performed once for every 1V, the parameter control part 140 is not restricted to hardware mounting (refer FIG. 7), It can also mount software. When the parameter control unit 140 is implemented by software, for example, a microcontroller can be used. An advantage of implementing the parameter control unit 140 by software is that the degree of freedom of control is high.

  7 includes a counter control circuit 141, an ALC counter 142, an address generation circuit 143, an SRAM core 144, and a parameter control circuit 145.

  The counter control circuit 141 receives the frequency distribution from the frequency distribution calculation circuit 131 and receives the index from the index calculation circuit 134. The counter control circuit 141 generates an addition / subtraction value for the ALC counter 142 that manages the sensitivity setting value of the entire imaging apparatus of FIG. 1 based on the frequency distribution and the index. The counter control circuit 141 outputs the addition / subtraction value to the ALC counter 142.

  Specifically, the counter control circuit 141 compares the index with a plurality of comparison criteria, and generates an addition / subtraction value corresponding to the comparison result. In general, the counter control circuit 141 generates 0 if the index is equal to the target value, generates a negative value (subtraction value) if the index is larger than the target value, and is positive if the index is smaller than the target value. Value (added value) is generated. The target value indicates the ideal brightness of the screen and defines a sensitivity convergence target.

The plurality of comparison criteria can be determined, for example, every t times and every 1 / t times based on a certain target value. The case of t = 2 ^1/2 (= √2) is illustrated in FIG. According to FIG. 12, if the index is greater than or equal to √2 times and less than twice the target value, the counter control circuit 141 generates “−128 × m + n”. On the other hand, if the index is greater than ½ times the target value and less than 1 / √2 times, the counter control circuit 141 generates “128 × mn”. In FIG. 12, m, n, j, and k are arbitrary coefficients that determine the convergence speed. FIG. 13 illustrates a comparison reference and addition / subtraction values when target value = 1024, m = 1, n = 1, k = 63, and j = 7.

In the example of FIG. 12, each time the count value increases by “128 × m”, the sensitivity setting increases by ^2½ times. Since the index corresponds to a weighted average of luminance integrated values in each area of the linear luminance signal, it is substantially proportional to the sensitivity. Therefore, for example, if the index is twice the target value, the sensitivity is considered to be about twice as large as the convergence target. Therefore, the counter control circuit 141 generates a value of “−256 × m + n” to increase the sensitivity. What is necessary is just to reduce to 1/2 time.

The ALC counter 142 receives the addition / subtraction value from the counter control circuit 141. The ALC counter 142 applies the addition / subtraction value to the count value, and outputs the count value after application to the address generation circuit 143. The number of bits of the count value counted by the ALC counter 142 is not particularly limited, but may be 13, for example. If the count value is Sadamere such increase to twice the sensitivity for each increase 256 (= 2 ^8), can be expressed sensitivity to about × 2 ³² × 1 × × by 13-bit count value (2 ^13/256 = 8192/256 = 32). In this example, the sensitivity (S) and the count value (C) have the relationship shown in the following mathematical formula (10).

  The address generation circuit 143 receives the count value from the ALC counter 142. The address generation circuit 143 generates an address value for reading from the SRAM core 144 according to the count value.

  The SRAM core 144 stores a combination of shutter speed, analog gain, linear digital gain, and non-linear digital gain for achieving sensitivity corresponding to the count value in the area specified by the address value. Note that the SRAM 144 may store the values of these parameters in a rewritable (programmable) manner. According to such a configuration, the user can freely design the transition of each parameter during sensitivity control.

  The parameter control circuit 145 reads the shutter speed, analog gain, linear digital gain, and non-linear digital gain values from the SRAM core 144 using the address value generated by the address generation circuit 143. The parameter control circuit 145 sets the read shutter speed, analog gain, linear digital gain, and non-linear digital gain values in the light receiving pixel 103, the ADC 104, the first digital gain circuit 111, and the second digital gain circuit 116, respectively.

  The parameter control circuit 145 changes the shutter speed, the analog gain, the linear digital gain, and the non-linear digital gain as shown in FIG. it can.

  An example of control of each parameter is shown in FIG. In FIG. 15, the shutter speed (exposure time) is discretely controlled from 1H / 4 to 1 / 2H, 1H, 2H, 4H,... 511H, 512H, and the analog gain is from x1 to x16 times. Continuously controlled, linear digital gain is continuously controlled from x1 to x4, and non-linear digital gain is discrete from x1 to x8.877 (equivalent to x128 times the linear digital gain). Controlled.

  The lowest sensitivity that defines the highest subject illuminance (which is defined as x1) can be achieved by setting the shutter speed, analog gain, linear digital gain, and non-linear digital gain all to the minimum value. That is, the parameter control circuit 145 sets the shutter speed to 1H / 4 and the analog gain, linear digital gain, and non-linear digital gain to x1.

  In order to achieve higher sensitivity, either parameter needs to be increased. Which parameter is selected is arbitrary, but it is preferable to increase the shutter speed first from the viewpoint of preventing the deterioration of SNR. However, if the shutter speed is changed from 1H / 4 to the second smallest 1 / 2H, the sensitivity is doubled. As described above, since the shutter speed and the non-linear digital gain greatly change the sensitivity, interpolation using an analog gain is performed in the example of FIG. Note that interpolation can be performed using linear digital gain instead of analog gain.

That is, when the count value is 1 to 255, the parameter control circuit 145 continuously increases the analog gain from x1 to about x2 without increasing the shutter speed. When the count value reaches 256, the parameter control circuit 145 increases the shutter speed to 1H / 2 and returns the analog gain to x1. By performing interpolation for discrete control in this way, fine sensitivity control with × 2 ^{1/256 times accuracy} becomes possible. While repeating such control, the parameter control circuit 145 increases the shutter speed by 1H, 2H, 4H,.

Further, when the count value is 2560 (= 256 × 10) or later, the parameter control circuit 145 can increase the shutter speed by 1H for each bit of the count value, so that the analog gain is fixed to × 1 and the shutter speed is increased. the 512H upper from 256H (sensitivity × ^{2 11} times) continuously increases up. By increasing the shutter speed instead of the analog gain, SNR degradation can be suppressed. When the count value reaches 2816 (= 256 × 11), the parameter control circuit 145 fixes the shutter speed to 512H. Then, the parameter control circuit 145 continuously increases the analog gain again from x1 to about x2.

  When the count value reaches 3072 (= 256 × 12), the parameter control circuit 145 increases the non-linear digital gain by × 1.366 (corresponding to × 2 times the linear digital gain) and returns the analog gain to × 1. . Then, the parameter control circuit 145 continuously increases the analog gain again from x1 to about x2. While repeating such control, the parameter control circuit 145 increases the non-linear digital gain by a factor of 1.86 times, x2.549 times,..., X8.877 times (linear gain conversion). .

When the count value is 4608 (= 256 × 18) or later, the parameter control circuit 145 fixes the non-linear digital gain to × 8.877 times (× 128 times in terms of linear gain conversion) and the analog gain from × 1 to the upper limit × Increase continuously up to 16 times. When the count value reaches 5632 (= 256 × 22), the parameter control circuit 145 fixes the analog gain to x16. The sensitivity at this point is × ²² times, that is, over × 4 million times. Incidentally, at the time of setting the analog gain to × 4 times, the sensitivity is over a million-fold × a × 2 ^20.

  Therefore, the parameter control circuit 145 can control the sensitivity from x1 to over x1,000,000 times even if the linear digital gain is excluded from the control target. However, the parameter control circuit 145 can further improve the upper limit of sensitivity by including the linear digital gain in the control target.

Specifically, after the count value is 5632 (= 256 × 22), the parameter control circuit 145 increases the linear digital gain from x1 times to about x4 times) (the maximum sensitivity is about x2 ^24). That is, it can be increased up to x16 million times. Therefore, the minimum subject illuminance is about 1/16 million of the maximum subject illuminance.

  Note that the sensitivity described here is not an absolute size, but a relative size when the minimum sensitivity is x1. Therefore, if the ratio of the maximum value to the minimum value of sensitivity exceeds 1 million, it does not necessarily guarantee the realization of visible 1 million times optical imaging. However, in the example of FIG. 14, the ratio reaches more than 16 million, and further improvement is possible depending on the setting of the upper limit of the parameter.

On the other hand, if the non-linear digital gain is excluded from the control target, the upper limit of sensitivity is greatly reduced. Such a comparative example is shown in FIG. In Figure 14, the shutter speed, analog gain and linear digital gain respectively 512H (upper), even increased to 16 times × (upper limit) and × 4 times, sensitivity × 2 ¹⁷ times, ie × 13 thousand times approximately Only. If the linear digital gain is increased up to x32 times, it is possible to increase the sensitivity to over x1 million times, but such linear digital gain is not practical because it causes a deterioration of quantization accuracy of 5 bits. .

  The contribution of non-linear digital gain is also significant in the imaging sensitivity control when the fluorescent lamp flickerless imaging function is enabled. In order to make the flickerless imaging function effective, it is necessary to fix the shutter speed to a value based on a given commercial power supply frequency (specifically, an integral multiple of a half cycle). If the frame rate is fixed at 60 fps and the variable range of the shutter speed is from 1H / 4 to 512H, the shutter speed is fixed to one of the following.

In Equation (11), ES ₅₀ represents a shutter speed that can be set when the commercial power supply frequency is 50 Hz. ES _60, and _{ES 60, 2} represents the settable shutter speed when the commercial power supply frequency is 60 Hz.

  For example, when the commercial power supply frequency is 50 Hz, by controlling the analog gain, the linear digital gain, and the non-linear digital gain, the sensitivity region where flickerless imaging can be performed becomes x1228 times to over x10 million times. It should be noted that if the non-linear digital gain is not included in the control target, the upper limit of the sensitivity region where flickerless imaging can be performed is reduced to about x80,000 times.

  Similarly, when the commercial power supply frequency is 60 Hz, by controlling the analog gain, linear digital gain, and non-linear digital gain, the sensitivity region where flickerless imaging can be performed becomes x1024 times to more than x16 million times. It should be noted that if the non-linear digital gain is not included in the control target, the upper limit of the sensitivity area where flickerless imaging can be performed is reduced to about x130,000 times.

  As described above, the imaging apparatus according to the first embodiment applies the input / output 1: 1 linear / nonlinear conversion to the pixel signal and multiplies the variable non-linear digital gain. This imaging apparatus can automatically control imaging sensitivity over a wide range by controlling a plurality of parameters including the nonlinear digital gain based on luminance. In particular, by including the shutter speed and the analog gain in addition to the non-linear digital gain in the control target, it is possible to achieve a high sensitivity of x1 million times or more while suppressing the SNR deterioration. In addition, according to this imaging device, it is also possible to greatly expand the sensitivity area where flickerless imaging is possible.

  Further, in this imaging apparatus, the three-line synchronization circuit, the luminance calculation circuit, and the frequency distribution calculation circuit are shared by both processing of imaging sensitivity control and HDR compression coefficient generation. Therefore, this imaging apparatus can realize imaging sensitivity control and single exposure HDR with a small circuit scale.

  In addition, the imaging apparatus can include a storage circuit that rewritably stores changes in shutter speed, analog gain, linear digital gain, and nonlinear digital gain that are set when imaging sensitivity is controlled. Therefore, the user can freely design a control range of the imaging sensitivity, a combination of parameters for achieving a given imaging sensitivity, and the like.

  The various functional units described in the above embodiments may be realized by using a circuit. The circuit may be a dedicated circuit that realizes a specific function, or may be a general-purpose circuit such as a processor.

  At least a part of the processing of each of the above embodiments can also be realized by using a general-purpose computer as basic hardware. A program for realizing the above processing may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable file or an executable file. Examples of the recording medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The recording medium may be any recording medium as long as it can store the program and can be read by the computer. The program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 101 ... Optical lens 102 ... Color filter 103 ... Light receiving pixel 104 ... ADC
DESCRIPTION OF SYMBOLS 110 ... Signal processing circuit 111 ... 1st digital gain circuit 112 ... gamma correction circuit 113 ... 5-line synchronization circuit 114 ... Scratch detection / correction circuit 115 ... Noise reduction circuit 116 ... Second digital gain circuit 117 ... 3-line synchronization circuit 118 ... HDR compression circuit 119 ... Luminance calculation circuit 120 ... HDR compression coefficient generation circuit 130 ... Sensitivity control unit 131 ... Frequency distribution calculation circuit 132... Inverse gamma correction circuit 133 .. luminance integration circuit 134... Index calculation circuit 140... Parameter control unit 141 ... counter control circuit 142 ... ALC counter 143 .... Address generation circuit 144 ... SRAM core 145 ... parameter control circuit

Claims (9)

By applying linear / non-linear conversion of input / output 1: 1 to the second linear pixel signal based on the first linear pixel signal generated by the analog pixel unit, a bit more than the second linear pixel signal. A linear / nonlinear conversion circuit for generating a large number of first nonlinear pixel signals;
A non-linear digital gain circuit for multiplying the first non-linear pixel signal by a non-linear digital gain to obtain a second non-linear pixel signal;
A luminance calculation circuit for calculating luminance based on the second non-linear pixel signal;
An imaging control apparatus comprising: a sensitivity control unit that controls imaging sensitivity by controlling at least the nonlinear digital gain based on the luminance.   The imaging control apparatus according to claim 1, wherein the sensitivity control unit further controls at least one of a shutter speed and an analog gain set in the analog pixel unit based on the luminance. A linear digital gain circuit for multiplying the first linear pixel signal by a linear digital gain to obtain the second linear pixel signal;
The sensitivity control unit further controls the linear digital gain based on the luminance.
The imaging control apparatus according to claim 1 or 2.   The imaging control apparatus according to claim 2, wherein the sensitivity control unit discretely controls the nonlinear digital gain, continuously controls the analog gain, and interpolates the discrete control of the nonlinear digital gain.   The imaging control apparatus according to claim 3, wherein the sensitivity control unit discretely controls the nonlinear digital gain, and continuously controls the linear digital gain to interpolate the discrete control of the nonlinear digital gain.   The imaging control apparatus according to claim 2, wherein the sensitivity control unit includes a storage circuit that rewritably stores transitions of the shutter speed, the analog gain, and the nonlinear digital gain that are set when the imaging sensitivity is controlled.   The imaging control device according to claim 2, wherein a ratio of a maximum value to a minimum value of the imaging sensitivity exceeds 1 million.   The imaging control apparatus according to claim 2, wherein the shutter speed is fixed to a value based on a given commercial power supply frequency. A frequency distribution calculation circuit for calculating the frequency distribution of the luminance;
A generating circuit for generating a compression coefficient for each luminance class based on the frequency distribution;
The imaging control apparatus according to claim 1, further comprising: a compression circuit that compresses a dynamic range of the second non-linear pixel signal using the compression coefficient.