JP2012205030A - Image signal processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which an image signal processing apparatus performing a wide dynamic range (WDR) processing, capable of obtaining an image with few blown-out highlights and blocked-up shadows by combining two images of a high-speed exposure image and low-speed exposure image, may cause many blown-out highlights and blocked-up shadows depending on a subject, particularly under a condition where the WDR is required, so that a feedback control, using the detection data, needs a time to optimally control detection data with an extreme blown-out highlight, obtained from the subject.SOLUTION: According to a detection result obtained by signal level detecting means for an image sensor with nonlinear photoelectric conversion characteristics, a long-time exposure and a short-time exposure by an image sensor with linear photoelectric conversion characteristics are controlled.

Description

  The present invention relates to an image signal processing apparatus.

  As background art in this technical field, for example, there is JP-A-2002-279256 (Patent Document 1). In this publication, as a problem, “in the imaging apparatus capable of capturing an image having a wide dynamic range such as the above-mentioned wide dynamic range camera, the dynamic range of the output image that can be reproduced as a video signal corresponds to the luminance signal level being high luminance. The ratio of the high luminance portion and the luminance signal level assigned to the low / medium luminance portion corresponding to the low luminance / medium luminance was always fixed regardless of the subject. ”[Means for solving the problem] , “Image picking up the subject; detecting the image signal of the long exposure image in which the exposure time of the subject imaged is relatively long and the short exposure image in which the exposure time is relatively short; switching determined from the image signal Generating a composite image from the long-exposure image and the short-exposure image based on the luminance signal level; compressing the composite image according to the luminance region occupied in the composite image; The dynamic range of the output image to be output as an image signal is dynamically allocated; the dynamic range of the composite image is compressed based on the dynamic allocation of the dynamic range of the output image ” Yes.

  Moreover, there exists Unexamined-Japanese-Patent No. 2002-77733 (patent document 2), for example. In this publication, as a [problem], “the present invention is capable of automatically switching between a logarithmic conversion operation and a linear conversion operation according to the amount of incident light incident on the photoelectric conversion unit without switching the bias voltage. The object is to provide a device. ”As a solution,“ By giving a pulse signal having a voltage VL lower than the voltage VH applied to the source of the MOS transistor T1 to the signal φVPS during imaging, Since the gate voltage of the MOS transistor T1 is lower than the source voltage, the MOS transistor T1 is cut off until the subject exceeds a predetermined luminance value at the time of imaging. When the output of the object exceeds a predetermined luminance value, the MOS transistor T1 is set in the subthreshold region. The logarithm-converted electrical signal is output because the operation is performed. "

Japanese Patent Application No. 2002-279256 JP 2002-77733 A

Image signal processing that performs wide dynamic range processing (hereinafter referred to as WDR = Wide Dynamic Range processing) that can produce images with little overexposure and underexposure by combining two images, a high-speed exposure image and a low-speed exposure image In order to obtain an appropriate WDR-processed image in the apparatus, in Patent Document 1, “every time a composite image is generated, it is determined whether or not the distribution ratio is an appropriate dynamic range. The detection data obtained from the image is acquired and analyzed by `` correcting the distribution ratio '', and these are feedback control using the detection data, so in particular in the environment where WDR is required, There is a problem in that detection data obtained from a subject that is often blacked out and is extremely out of focus requires time for optimal control.

  Further, in Patent Document 2, there is switching between a logarithmic (hereinafter referred to as non-linear) conversion operation and a linear conversion operation. However, in the non-linear operation, an amount of noise equivalent to the image in the linear operation and an image quality related to color reproduction cannot be obtained. There was a problem that images with a wide dynamic range could not be obtained during linear operation.

  An object of the present invention is to provide an image signal processing apparatus capable of controlling exposure control at high speed when using linear and nonlinear image processing techniques.

  In order to solve the above object, the configuration described in the claims is adopted.

  According to the present invention, it is possible to provide an image signal processing apparatus capable of controlling exposure control at high speed when using linear and non-linear image processing techniques.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

2 is a diagram illustrating an example of a basic configuration in Embodiment 1. FIG. It is a figure explaining the detail of an image sensor. It is a figure which shows the example of control of the histogram of the image in which whiteout has generate | occur | produced. It is a figure which shows the example of the histogram of the image at the time of using a linear sensor and a nonlinear sensor together. It is a figure which shows an example of the arrangement | sequence of a color filter. It is a figure which shows an example of the arrangement | sequence of a color filter. It is a figure which shows an example of the arrangement | sequence of a color filter.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

A system configuration of the first embodiment will be described with reference to FIG.
FIG. 1 is a diagram illustrating an example of a basic configuration according to the first embodiment of the present invention.
Details of the imaging unit 101 will be described with reference to FIG.

  The imaging unit 101 includes a diaphragm (not shown) that adjusts the amount of incident light from the subject, a lens (not shown) that collects light that passes through the diaphragm, and photoelectrically converts the light collected by the lens to generate an image. An image sensor that outputs as a signal and has an input / output characteristic that is linear and an image sensor that is non-linear, each image sensor includes an electronic shutter controller (not shown) and a gain controller (not shown). Is equipped.

  The image input unit 102 is an input interface process for performing image signal processing in a block that receives an image signal from the imaging unit 101 and performs subsequent signal processing. For example, the image input unit 102 performs AD (Analog to Digital) conversion processing. Also, for example, it is a synchronization adjustment process for adjusting the processing timing of the image sensor and the signal processing unit. The image input unit 102 may include predetermined image signal processing such as gain control and filter processing for signal level adjustment.

  The linear sensor exposure control unit 103 controls the aperture of the imaging unit 101, the electronic shutter control of the imaging device, the gain control, the gain of the image input unit 102, and the like. Further, in order to obtain a WDR image, the long time exposure and the short time exposure control are performed on the linear sensor in a time division manner to obtain a long time exposure image and a short time exposure image.

  The non-linear sensor exposure control unit 104 changes the amount of incident light with respect to the non-linear sensor by controlling the aperture of the imaging unit 101 so that the linear sensor exposure control unit 103 controls long exposure and short exposure in a time-sharing manner. When the linear sensor exposure control unit 103 controls the gain of the image input unit 102 or the like, the electronic shutter of the image sensor is maintained so as to keep the amount of the nonlinear sensor image signal output from the image input unit 102 at a predetermined level. And / or gain control.

  The signal level detection unit 105 detects the signal level of the image signal from the linear sensor and the nonlinear sensor from the image input unit 102. The numerical value detected by the signal level detection unit 105 is, for example, the absolute value of the input pixel signal. Also, for example, numerical values serving as information indicating the luminance distribution state of the signal in the image area, such as a histogram distribution, maximum value minimum value, and average value of the input pixel signal. If the above information is detected, the amount of incident light when the pixel signal is obtained can be estimated.

  The image quality control unit 106 applies image signals such as noise removal, gamma correction, contour enhancement, filter processing, zoom processing, camera shake correction, and image recognition to the long exposure image signal and the short exposure image signal from the signal level detection unit 105. Process.

The image synthesis unit 107 synthesizes the WDR image from the long exposure image signal and the short exposure image signal from the image quality control unit, and the image output unit 108 performs predetermined processing on the image signal from the image synthesis unit 107 and outputs it.

  The predetermined process is an output interface process for converting to a signal format of an output device such as a TV or a storage, for example, converting to an NTSC or PAL video output, for example, converting to an HDMI signal. For example, the signal is converted into a predetermined signal for network transmission. The image output unit 110 may be configured to include predetermined image signal processing such as gain control for signal level adjustment, filter processing, and compression processing by encoding.

Next, the details of the image sensor which is a part of the imaging unit 101 will be described with reference to FIG.
The color filter array shown in FIG. 2 uses a linear image sensor as an image sensor in which R, G, and B filters are arranged with respect to each RGB color filter called a general Bayer array. A portion corresponding to half of the filter is a non-linear image sensor.

  Further, the linear sensor exposure control unit 103 performs electronic shutter and gain control for the linear pixel in which the R, G, and B filters of FIG. 2 are arranged. The non-linear sensor exposure control unit 104 performs electronic shutter and gain control on the non-linear pixel (L pixel) in FIG. 2 to individually control the amount of linear pixel and non-linear pixel signals output from the image input unit 102. Control becomes possible.

  Next, WDR control solved by this embodiment will be described. First of all, WDR control is a control that can obtain an image with little overexposure and underexposure by combining two images, a high-speed exposure image and a low-speed exposure image. In order to obtain an appropriate WDR-processed image, the detection data obtained from the image is acquired and analyzed, and the composite image is compressed according to the luminance region occupied in the composite image, and the dynamic range of the output image output as a video signal Dynamic allocation is performed. In order to acquire the high-speed exposure image and low-speed exposure image necessary for this dynamic allocation, shutter control, aperture control, and gain control are performed on the high-speed exposure image to suppress overexposure. The same control is performed so as to suppress blackout on the exposed image.

  However, in the conventional linear sensor, since the dynamic range is narrow, each of these images has whiteout or blackout, especially when the data acquired by the sensor is saturated with regard to whiteout. Since it is not possible to obtain the control amount that will be the optimal image in the control, it is necessary to repeat the detection data acquisition, analysis, and control, so it takes time to optimize the control. I was sorry.

  Therefore, in this embodiment, the linear sensor and the non-linear sensor are used instead of repeating the control until the optimum long exposure control and short exposure control are performed from the image that has been blown out or blacked out as in the conventional WDR control. Combined with this, by analyzing the histogram and brightness data obtained from detection data without whiteout or blackout that can always be acquired by a non-linear sensor, the shutter can acquire image data from the linear image sensor without whiteout or blackout. The control value, aperture control value, and gain control value can be obtained.

The difference in control will be described with reference to FIGS.
FIG. 3 shows a control example of a histogram of an image in which whiteout occurs due to conventional WDR. FIG. 3A shows a state in which a signal having a saturation level a or higher of the sensor exists. FIG. 3A shows a state in which there is no signal above the saturation level a by the WDR control, but it is not below the optimum signal level b. FIG. 3B shows a state where the optimum signal level is reached.

  In the conventional WDR control, it cannot be determined whether or not the whiteout signal is at such a luminance level because the signal of the sensor is saturated, so the state shown in FIG. The control amount until the state (b) is reached cannot be obtained. For this reason, as shown in the flowchart of FIG. 3D, the control amount when the signal is saturated can only be controlled by repeatedly changing an appropriate constant control amount and gradually changing the control value. If the control amount is increased, there is a possibility that the control will be excessively performed beyond the optimum exposure state, and if the control amount is decreased, the control will take time.

  FIG. 4 shows an example of an image histogram when a linear sensor and a nonlinear sensor are used in combination.

  FIG. 4A shows a state where a signal equal to or higher than the saturation level a of the sensor exists on the assumption that the same subject as in FIG. 3 is being photographed. FIG. 4B is an example of a histogram photographed by a non-linear sensor at the same timing as FIG. FIG. 4A shows that a signal that has become a signal equal to or higher than the saturation level can be acquired as a histogram of the optimum signal level in the nonlinear sensor as shown in FIG. 4B. . Further, since the correlation between the signal level of the linear sensor and the nonlinear sensor is determined by the input / output characteristics of the sensor, the gain amount of each element, etc., it is assigned to the brightness c in FIG. 4B of the histogram acquired by the nonlinear sensor. A control amount for controlling the signal level to the optimum signal level b in FIG. 4C of the linear sensor can be obtained.

  Therefore, it is not necessary to gradually change the control amount so as to obtain an appropriate control value as in the conventional linear sensor, and the linear sensor can be controlled to the optimum exposure state by a short time control.

  However, because long-time exposure and short-time exposure control are performed for an image pickup unit with linear input characteristics, the input / output characteristics are nonlinear by changing the mechanical shutter and diaphragm used in common for both sensors. The input signal to the unit also varies.

However, even when the input signal of the linear image sensor fluctuates due to the mechanical shutter or aperture control, the control amount of the linear image sensor can be obtained from the signal imaged by the nonlinear image sensor by considering the control amount of the mechanical shutter and the aperture amount. It is possible to control the amount of control of the shutter and the amount of control of the aperture by using electronic shutter control and gain control of the nonlinear imaging device, so that the signal amount of the nonlinear imaging device can be controlled by the shutter and aperture gain control. Can be equivalent to a fixed state.

  That is, by controlling the electronic shutter and gain of the nonlinear imaging unit so as to cancel the exposure control by the linear imaging unit, it is possible to make the input signal to the nonlinear imaging unit have a certain signal amount, By controlling in this way, it can be assumed that the nonlinear imaging element is always operating with a constant control amount, and the control amount of the linear element can be easily obtained from the detection result such as a histogram acquired by the nonlinear element. It becomes.

  By controlling in this way, it becomes possible to use detection data from an image having no overexposure or underexposure obtained from the characteristic that the dynamic range of the nonlinear imaging device is wide for controlling the linear imaging device.

  The same applies to the short-time exposure control, and it is possible to obtain an optimal long-time exposure control amount without requiring a control time for a subject in which blackout occurs in the linear imaging device.

  In addition to the color filter arrangement shown in FIG. 2, the arrangement of the color filters may be such that the position of the G filter as the non-linear image sensor is the pattern shown in FIG. Alternatively, FIG. 7 may be used as a G filter color reproduction-oriented pattern.

DESCRIPTION OF SYMBOLS 101 ... Imaging part, 102 ... Image input part, 103 ... Linear sensor exposure control part, 104 ... Nonlinear sensor exposure control part, 105 ... Signal level detection part, 106 ... Image quality control part, 107 ... Image composition part 108: Image output unit 201: R: Linear pixel and R filter 202: G: Linear pixel and G filter 203 ... B: Linear pixel and B filter, 204 ... L: Non-linear pixel

Claims (2)

A first image sensor having a non-linear photoelectric conversion characteristic and a second image sensor having a linear photoelectric conversion characteristic;
Image input means for inputting a signal from the image sensor;
Signal level detection means for detecting the signal level from the image input means;
Image signal combining means for combining a high-speed exposure signal and a low-speed exposure signal; first exposure control means for an image pickup device whose photoelectric conversion characteristics are nonlinear;
A second exposure control means of the image pickup device having a linear photoelectric conversion characteristic;
Comprising
The second exposure control means includes a long-time exposure control and a short time by an imaging element having a linear photoelectric conversion characteristic according to a detection result obtained by the signal level detection means of the imaging element in which the photoelectric conversion is nonlinear. An image signal processing apparatus for controlling exposure control. 2. The first imaging device according to claim 1, wherein the detection result for the imaging device having a non-linear photoelectric conversion characteristic cancels the exposure control amount for the imaging device having a linear photoelectric conversion property. An image signal processing apparatus controlled by exposure control means.

Images