CN100539648C - Imaging apparatus and camera head and method - Google Patents
Imaging apparatus and camera head and method Download PDFInfo
- Publication number
- CN100539648C CN100539648C CNB2007101074229A CN200710107422A CN100539648C CN 100539648 C CN100539648 C CN 100539648C CN B2007101074229 A CNB2007101074229 A CN B2007101074229A CN 200710107422 A CN200710107422 A CN 200710107422A CN 100539648 C CN100539648 C CN 100539648C
- Authority
- CN
- China
- Prior art keywords
- mentioned
- exposure
- pixel data
- standard
- standard exposure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention provides a kind of imaging apparatus and camera head, it can be according to the pixel data of each pixel of being exposed in the time for exposure (non-standard time for exposure) shorter than the standard exposure time, whether prediction this each pixel when the exposure of standard exposure time saturated, also can according to above-mentioned predict the outcome the pixel data of standard exposure time and the pixel data of non-standard time for exposure synthesized generate the HDR image.Camera head (1) comprises: shooting treatment system (10), its between 1 standard exposure period in, come photographic images by the exposure for standard exposure time of the exposure area of sensor cell array (56), the exposure by the non-standard time for exposure simultaneously comes photographic images; And image processing system (12), its view data according to the non-standard time for exposure predicts whether each pixel saturated under the exposure of standard exposure time, and according to the photographed images data that predict the outcome synthetic standards time for exposure and non-standard time for exposure to generate the HDR view data.
Description
Technical field
The present invention relates to use destroy and read mode and nondestructive read is got the imaging apparatus and the camera head of mode reading electric charges from photo-electric conversion element.
Background technology
As using with digital camera, can enumerate the narrow and small problem of dynamic range to one of problem points of the camera head of the solid-state imager of representative.Therefore, in taking the resulting image of the very large scene of contrast, can produce in white sudden strain of a muscle (white と び) in the high-brightness region (pixel saturated) and the low brightness area blackspot (?つ ぶ れ).
Current, multiple gimmick has been proposed in order to address the above problem.As one, have the gimmick of 2 different (or many) images of synthetic exposure, as the known example of using this gimmick, for example can enumerate patent documentation 1 described camera head etc.
This camera head can obtain the video standard signal of suitable light exposure and the camera head of the non-standard picture signal that light exposure is less than video standard signal by the light exposure changing cell that the change light exposure is photographed for utilizing; It is take two following unit as feature, that is: the value of making the video standard signal that expression exports whether surpass threshold value (judging the saturated of pixel) saturated judgement signal (being in the present invention shade (mask) picture signal) the unit and switch the unit (replacing satisfied pixel in the video standard signal with the value of non-standard picture signal) of 2 picture signals according to this saturated judgement signal. Obtain the image of wide dynamic range thus.
[patent documentation 1]: TOHKEMY 2004-32583 communique
But, in the prior art of above-mentioned patent documentation 1, when making saturated judgement signal (establish saturated pixel and be 1, non-saturated pixel be 0) according to the video standard signal that has applied noise, if carry out the synthetic of image according to this judgements signal then may cause the image quality reduction of composograph.Particularly, the dark current noise of imaging apparatus passes through in time and increases, and is subjected to The noise.And the difference of the sensitivity of each pixel, the difference of saturation capacity etc. also all is the wrong reason of judging.This be because: for example, as shown in figure 15, former pixel that should be all becomes white pixel, but because the difference of above-mentioned pixel has produced a little black pixel.If use this saturated judgement signal composograph, be mixed with video standard signal and non-standard picture signal (originally full criteria of right and wrong picture signal) then undesiredly.Can address this problem though change threshold value, need come setting threshold very numerous and diverse processing like this according to each pixel by property difference according to each pixel.
On the other hand, the method (Figure 16) of the image that obtains wide dynamic range by video standard signal and non-standard picture signal are added up mutually in addition.This simple phase computation system is called associating (Knee) mode herein.Because not carrying out the level of pixel value, associated form do not judge, so do not produce the such problem of the prior art of above-mentioned patent documentation 1.And, associated form is owing to carry out the level compression to high-brightness region, so in the device that can show wide dynamic range (for example can carry out the contrast device shown of 10000:1 etc.), generation is diminished the problem of the gray scale of high-brightness region, and because linear disappearance the, so problem that can't faithful reappearance color (situation of the transducer of chromatic filter has been installed).
And, as other problem, have with dividing other regularly to come to take standard exposure image and non-standard exposure image 2 times, after being synthesized, they become the such a problem of ambiguity diagram.
Summary of the invention
So, the present invention is conceived to the unsolved problem that this prior art has and finishes, its purpose be to provide a kind of can be according to the pixel data of each pixel of exposing by the time for exposure shorter than the standard exposure time, whether saturated prediction when the exposure of standard exposure time this each pixel imaging apparatus and camera head, can synthesize the camera head that generates the wide dynamic range image according to the above-mentioned pixel data that predicts the outcome the pixel data of standard exposure time and time for exposure shorter than the standard exposure time but also provide a kind of.
[mode 1]
In order to reach above-mentioned purpose, the imaging apparatus of mode 1 has the electronic shutter function of photo-electric conversion element being arranged to the time for exposure of how rectangular photoelectric conversion part and the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that, this imaging apparatus has: the 1st reading unit, and it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes; The 2nd reading unit, its between the exposure period identical with above-mentioned the 1st reading unit in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time; And saturated predicting unit, it is according to the non-standard exposure pixel data that above-mentioned the 2nd electric charge that uses above-mentioned the 2nd reading unit to be read is derived, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated.
According to this structure, can use the 1st reading unit from each pixel of the above-mentioned photo-electric conversion element formation of above-mentioned photoelectric conversion part, read mode with destruction and read in the electric charge that the standard exposure time exposed, can use the 2nd reading unit between the exposure period identical with above-mentioned the 1st reading unit in, from each pixel that above-mentioned photo-electric conversion element constitutes, reading the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting is the electric charge that short exposure time is exposed, can also be by saturated predicting unit according to the non-standard exposure pixel data that the electric charge of the above-mentioned short exposure time that is read by above-mentioned the 2nd reading unit constitutes, predict each pixel when exposing in the above-mentioned standard exposure time whether accumulate the quantity of electric charge saturated.
Therefore, for example for by paying close attention to the shooting object that subject and background thereof constituted, because can be to the standard exposure time (standard exposure time) of the setting of the concern subject in this object of making a video recording as correct exposure, use the 1st reading unit to read in the electric charge of each pixel of the photoelectric conversion part that this standard exposure time exposed, therefore can obtain the shooting object pixel data (standard exposure pixel data) of being exposed in the standard exposure time.On the other hand, in between the exposure period of above-mentioned standard exposure time, owing to use the 2nd reading unit to read in the electric charge of above-mentioned each pixel that the time for exposure (short exposure time) shorter than the above-mentioned standard exposure time exposed, so can obtain the pixel data (non-standard exposure pixel data) of the exposure (less signal amplitude) that the exposure (signal amplitude) when exposing in the standard exposure time lacks in the nondestructive read mode of getting.That is, in 1 time of standard exposure time exposure, can obtain the pixel data (standard exposure pixel data) when expose and the pixel data (non-standard exposure pixel data) when exposing in the non-standard time for exposure in the standard exposure time.And then, since when predicting that according to the above-mentioned non-standard exposure pixel data that reads out the pixel corresponding with this non-standard exposure pixel data exposed in the standard exposure time whether accumulate the quantity of electric charge saturated, so can judge thus whether (the electric charge accumulation amount of short exposure time) be saturated under, the repressed state of noise effect less at the dark current noise, thereby each pixel when exposing in the criterion time for exposure accumulate the quantity of electric charge when whether saturated, can obtain suppressing the effect of the difference of saturated/unsaturated judged result that noise causes.
The following describes the principle of the difference that suppresses saturated/unsaturated judged result.Instinctively, the pixel data of short exposure time nearly all becomes the blackspot zone, only manifests luminance signal at specific high-brightness region (the bigger zone of irradiation light quantity).It is that the signal level of dark portion and highlights is 1:500 than also under the situation that the ratio of dark portion and highlights is, the time for exposure is identical more than the 1:500 that wide dynamic range is for example taken in light amount ratio, and secretly portion becomes blackspot.And, as mentioned above, suppressed in time through and the The noise of the dark current noise that increases etc. can suppress mistake and judge.And then also can suppress mistake for the difference of the saturation level of each pixel, sensitivity and judge.That is, owing to judge,, be difficult to be subjected to the influence of the difference of saturation level so can make decision level become for example 1/10 of saturation level with the short exposure image.About sensitivity difference, owing to determine signal level, judge under and the state that signal level is lower less, so be difficult to be subjected to the influence of sensitivity difference at light income by the product of charge/voltage conversion gain (sensitivity) and light income.As mentioned above, by the difference of the characteristic of pixel and The noise all less during in only detect (judgements) and have the zone that this signal is exported, thereby the difference that can suppress to judge.Below, also identical in about the mode of camera head, camera system, image capture method.
Herein, above-mentioned " photoelectric conversion part " for example uses the CMOS technology to constitute, as the imaging apparatus that the carried out nondestructive read that uses the CMOS technology is got, has threshold value modulation type imaging apparatus (VMIS (Threshold Voltage Modulation Image Sensor: the voltage modulated imageing sensor)) etc. for example.Below, also identical in about the mode of camera head, camera system, image capture method.
And above-mentioned " destruction is read mode " is when reading electric charges from photo-electric conversion element (picture element signal), and following and making the electric charge that is accumulated in this photo-electric conversion element is the mode of the reset processing of sky.Below, also identical in about the mode of camera head, camera system, image capture method.
In addition, above-mentioned " nondestructive read is got mode " is when reading electric charges from photo-electric conversion element (picture element signal), do not make the electric charge that is accumulated in this photo-electric conversion element keep the mode that accumulated state reads for sky.That is, owing to when electric charge reads, do not carry out reset processing, so the accumulating in the way of electric charge, till the time for exposure that sets, the electric charge that can carry out arbitrarily time for the different time for exposure reads.Below, also identical in about the mode of camera head, camera system, image capture method.
And, above-mentioned " saturated " is meant the state of the maximum amount of charge that the light that is input in the photo-electric conversion element surpasses this photo-electric conversion element and can accumulate (below be called maximum accumulate the quantity of electric charge), and the gray scale that the image that comprises the pixel data that becomes this saturated brightness value (below be called saturation value) produces this part disappears and is called as the phenomenon of white sudden strain of a muscle.Be that background is taken under the situation in house with the snow scenes look for example, if according to exposing to taking as the house of paying close attention to subject, then the major part of the pixel data of snow scenes color part becomes saturation value and produces white sudden strain of a muscle.Below, also identical in about the mode of camera head, camera system, image capture method.
[mode 2]
On the other hand, in order to reach above-mentioned purpose, the camera head of mode 2 has the electronic shutter function of photo-electric conversion element being arranged to the time for exposure of how rectangular photoelectric conversion part and the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that, this camera head has: the 1st reading unit, and it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes; The 2nd reading unit, its between the exposure period identical with above-mentioned the 1st reading unit in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time; Saturated predicting unit, it is according to the non-standard exposure pixel data that above-mentioned the 2nd electric charge that uses above-mentioned the 2nd reading unit to be read is derived, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated; And high dynamic range images data generating unit, it predicts the outcome according to above-mentioned saturated predicting unit, synthetic standard exposure pixel data and the above-mentioned non-standard exposure pixel data that uses above-mentioned the 1st electric charge and derive generates high dynamic range (High Dynamic Range) view data.
According to this structure, can use the 1st reading unit from each pixel of the above-mentioned photo-electric conversion element formation of above-mentioned photoelectric conversion part, read mode with destruction and read in the electric charge that the standard exposure time exposed, can use the 2nd reading unit between the exposure period identical with above-mentioned the 1st reading unit in, from each pixel that above-mentioned photo-electric conversion element constitutes, reading the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting is the electric charge that short exposure time is exposed, can also be by saturated predicting unit according to the non-standard exposure pixel data that the electric charge of the above-mentioned short exposure time that is read by above-mentioned the 2nd reading unit constitutes, predict each pixel when exposing in the above-mentioned standard exposure time whether accumulate the quantity of electric charge saturated.
And then, by the high dynamic range images data generating unit, according to predicting the outcome of above-mentioned saturated predicting unit, can synthesize by above-mentioned the 1st reading unit read expose in the above-mentioned standard exposure time time electric charge the standard exposure pixel data and the above-mentioned non-standard exposure pixel data that constitute, to generate the high dynamic range images data.
Therefore, for example for by paying close attention to the shooting object that subject and background thereof constituted, because can be to the standard exposure time (standard exposure time) of the setting of the concern subject in this object of making a video recording as correct exposure, use the 1st reading unit to read in the electric charge of each pixel of the photoelectric conversion part that this standard exposure time exposed, therefore can obtain the shooting object pixel data (standard exposure pixel data) of being exposed in the standard exposure time.On the other hand, in between the exposure period of above-mentioned standard exposure time, owing to use the 2nd reading unit to read in the electric charge of above-mentioned each pixel that the time for exposure (short exposure time) shorter than the above-mentioned standard exposure time exposed, so can obtain the pixel data (non-standard exposure pixel data) of the exposure (less signal amplitude) that the exposure (signal amplitude) when exposing in the standard exposure time lacks in the nondestructive read mode of getting.That is, in based on 1 time of standard exposure time exposure, can obtain the pixel data (standard exposure pixel data) when expose and the pixel data (non-standard exposure pixel data) when exposing in the non-standard time for exposure in the standard exposure time.Therefore, when Synthetic 2 image, can generate and suppressed fuzzy image.And then, for example when predicting that (judgement) is saturated, select non-standard exposure pixel data, choice criteria exposure pixel data when predicting that (judgement) is unsaturated, synthetic these selected non-standard exposure pixel data and standard exposure pixel datas, can generate the view data (HDR view data) of wide dynamic range, so can obtain to obtain the image quality ratio effect of stable wide dynamic range image in the past.
[mode 3]
Then, the camera head of mode 3 is characterised in that, in the camera head of mode 2, above-mentioned the 2nd reading unit between the exposure period identical with above-mentioned the 1st reading unit in, from each pixel that above-mentioned photo-electric conversion element constitutes, read in the 3rd electric charge that exposure was produced of the multiple short exposure time shorter successively than the above-mentioned standard exposure time in the nondestructive read mode of getting
The non-standard exposure element data that above-mentioned saturated predicting unit derives according to the 3rd electric charge that uses above-mentioned the 2nd reading unit to be read, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated.
According to this structure, for example for by paying close attention to the shooting object that subject and background thereof constituted, concern subject in this shooting object has been set under the situation of the standard exposure time (standard exposure time) as correct exposure, in between based on the exposure period of this standard exposure time, owing to can use the 2nd reading unit to read in the electric charge of above-mentioned each pixel that the multiple short exposure time (for example 2 kind) shorter than the above-mentioned standard exposure time exposed, so can obtain the pixel data (non-standard exposure pixel data) of the multiple exposure that the exposure when exposing in the standard exposure time lacks in the nondestructive read mode of getting.And then, according to the non-standard exposure pixel data of this multiple exposure predict when standard exposure time of the pixel corresponding exposure with this non-standard exposure pixel data whether accumulate the quantity of electric charge saturated, so can be predicted under the less state (the electric charge accumulation amount of short exposure time) of noise effect that the difference in each pixel causes whether saturated effect thus more accurately.And, for example unsaturated under short exposure time t2, at short exposure time t1 (when t2＜t1) is saturated down, owing to can use the non-standard exposure pixel data of short exposure time t2 to generate the view data (HDR view data) of wide dynamic range, can obtain to obtain to carry out the effect of the wide dynamic range image of gray scale performance more accurately.
[mode 4]
In addition, the camera head of mode 4 is characterised in that in the camera head of mode 2, this camera head has operation processing unit, this operation processing unit is to the 2nd electric charge that uses above-mentioned the 2nd reading unit and read and the calculation process that the non-standard exposure pixel data of deriving is stipulated
The non-standard exposure pixel data of above-mentioned saturated predicting unit after according to above-mentioned calculation process predicts whether the brightness value that above-mentioned standard exposure pixel data is represented is saturation value,
Above-mentioned high dynamic range images data generating unit predicts the outcome according to above-mentioned saturated predicting unit, and non-standard exposure pixel data and above-mentioned standard exposure pixel data after the synthetic above-mentioned calculation process generate above-mentioned high dynamic range images data.
According to this structure, can use arithmetic element to come the calculation process that the above-mentioned non-standard exposure pixel data that the electric charge that is read by above-mentioned the 2nd reading unit constitutes is stipulated, the non-standard exposure pixel data of above-mentioned saturated predicting unit after according to above-mentioned calculation process, can predict whether the brightness value that above-mentioned standard exposure pixel data is represented is saturation value, above-mentioned high dynamic range images data generating unit predicting the outcome according to above-mentioned saturated predicting unit, can synthesize non-standard exposure pixel data and above-mentioned standard exposure pixel data after the above-mentioned calculation process, generate the high dynamic range images data.
Therefore, can carry out the calculation process that is suitable for synthetic pixel data that for example treatment for correcting, the noise of intensity level are removed processing etc. to non-standard exposure pixel data, can also use the non-standard exposure pixel data that has carried out this calculation process to generate the view data (HDR view data) of wide dynamic range, so the effect of the image quality of the image of the wide dynamic range that can be improved.
[mode 5]
In addition, the camera head of mode 5 is characterised in that, in the camera head of mode 4, this camera head has the 3rd reading unit, the 3rd reading unit reads the electric charge after just resetting from each pixel that above-mentioned photo-electric conversion element constitutes, above-mentioned operation processing unit is carried out following calculation process: deduct corresponding with this non-standard exposure pixel data the above-mentioned non-standard exposure pixel data of the deriving from using above-mentioned the 2nd electric charge that above-mentioned the 2nd reading unit reads, the benchmark pixel data of using the electric charge after just the resetting that above-mentioned the 3rd reading unit reads and deriving.
According to this structure, can use the 3rd reading unit to read the electric charge afterwards that just resets from each pixel that above-mentioned photo-electric conversion element constitutes, above-mentioned operation processing unit can be carried out following calculation process: deduct the benchmark pixel data that the electric charge afterwards that just resets corresponding with this non-standard exposure pixel data, that read by above-mentioned the 3rd reading unit constitutes from the above-mentioned non-standard exposure pixel data that the electric charge that is read by above-mentioned the 2nd reading unit constitutes.
Therefore, owing to deduct just the calculation process of the brightness value that (making the electric charge of accumulating of pixel the be empty action) pixel data afterwards that resets represents the brightness value that the non-standard exposure pixel data that can carry out reading from the 2nd reading unit is represented, so can from non-standard exposure pixel data, remove the fixed pattern noise that the property difference separately because of each photo-electric conversion element (pixel) of formation photoelectric conversion part produces.Promptly, get in the mode at nondestructive read, owing in the non-standard exposure pixel data that electric charge constituted that reads out, sneaked into a lot of noises, so handle and to remove the noise component of sneaking in the non-standard exposure pixel data by carrying out above-mentioned calculus of differences, thereby can obtain the effect that the non-standard exposure pixel data according to more accurate intensity level carries out saturated/unsaturated prediction.And then, owing to can use the non-standard exposure pixel data of more accurate intensity level to generate the view data (HDR view data) of wide dynamic range, so the effect of the image quality of the image of the wide dynamic range that can be improved.
In " fixed pattern noise ", for example have the various noises that produce because when time exposure, becoming transducer sensitivity different of the dark current blackspot of problem and each pixel etc. herein.
[mode 6]
In addition, the camera head of mode 6 is characterised in that, in the camera head of mode 2, above-mentioned high dynamic range images data generating unit is selected the 1st non-standard exposure pixel data corresponding be predicted as saturated pixel by above-mentioned saturated predicting unit from the non-standard exposure pixel data that above-mentioned the 2nd reading unit is read, from the standard exposure pixel data that above-mentioned the 1st reading unit is read, select and be predicted as the 1st corresponding standard exposure pixel data of undersaturated pixel by above-mentioned saturated predicting unit, above-mentioned the 1st standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data are synthesized, generate above-mentioned high dynamic range images data.
According to this structure, do not become the standard exposure of maximum brightness value pixel data and use non-standard exposure pixel data owing to can not using when saturated in prediction (judgement), when predicting (judgement), use the standard exposure pixel data to generate the view data (HDR view data) of wide dynamic range, so can obtain to obtain the image quality ratio effect of the image of stable wide dynamic range in the past for unsaturation.
[mode 7]
In addition, the camera head of mode 7 is characterised in that, in the camera head of mode 6, when the 2nd non-standard exposure pixel data that do not have the time for exposure shorter than the short exposure time corresponding with the above-mentioned the 1st non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit generates and the corresponding high dynamic range pixel data of the above-mentioned the 1st non-standard exposure pixel data according to the 2nd standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data corresponding with the above-mentioned the 1st non-standard exposure pixel data; When having the above-mentioned the 2nd non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit is according to the above-mentioned the 2nd non-standard exposure pixel data and above-mentioned the 2nd standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data, generate and the corresponding high dynamic range pixel data of the above-mentioned the 1st non-standard exposure pixel data, and synthesize above-mentioned high dynamic range pixel data and above-mentioned the 1st standard exposure pixel data generates above-mentioned high dynamic range images data.
For example, merely increasing the intensity level of proofreading and correct non-standard exposure pixel data, when using non-standard exposure pixel data after this increase is proofreaied and correct to generate the photographed images data, because by increasing the also increase thereupon of noise that correction makes that the difference of pixel causes, so might become the state of in the non-standard exposure pixel data that increases after proofreading and correct, sneaking into the noise of amplification.Certainly, this noise can become the reason that image quality reduces.
According to said structure, because when not having the non-standard exposure pixel data of the time for exposure shorter than the short exposure time corresponding with above-mentioned selected non-standard exposure pixel data, in the correction of intensity level, can use this selected non-standard exposure pixel data and corresponding with it standard exposure pixel data, so, can reduce the noise that increases timing for example by making the processing of non-standard exposure pixel data and the addition of standard exposure pixel data.And, when having the non-standard exposure pixel data of the time for exposure shorter than the short exposure time corresponding with above-mentioned selected non-standard exposure pixel data, in the correction of intensity level, can use this selected non-standard exposure pixel data and corresponding with it above-mentioned non-standard exposure pixel data and the standard exposure pixel data of short time for exposure, so for example, can further reduce the noise that increases timing by making different 2 kinds of non-standard exposure pixel datas and the addition of standard exposure pixel data of time for exposure.
And, owing to can use the non-standard exposure pixel data that has reduced noise to generate the high dynamic range images data as mentioned above, can obtain obtaining the image quality ratio effect of the image of stable wide dynamic range in the past.
[mode 8]
In addition, the camera head of mode 8 is characterised in that, in the camera head of mode 6, the brightness value of accumulating the quantity of electric charge of each pixel of above-mentioned saturated predicting unit comparison during and the saturated prediction brightness value of setting according to the time for exposure according to the exposing of above-mentioned non-standard exposure pixel data prediction in the standard exposure time, according to this comparative result predict each pixel when exposing in the above-mentioned standard exposure time whether accumulate the quantity of electric charge saturated
When not existing than the time with the 2nd non-standard exposure pixel data of shorter time for exposure of the above-mentioned the 1st non-standard exposure pixel data short exposure time that promptly the 1st non-standard exposure pixel data is corresponding, above-mentioned high dynamic range images data generating unit deducts the above-mentioned saturated prediction of using with the 1st non-standard exposure pixel data from the represented brightness value of the above-mentioned the 1st non-standard exposure pixel data be the result that the 1st saturated prediction obtains divided by the 1st normalisation coefft with brightness value with brightness value, this result who subtracts each other and above-mentioned the 1st normalisation coefft are multiplied each other, make this multiplied result add that the 2nd standard exposure pixel data corresponding with the 1st non-standard exposure pixel data generates above-mentioned high dynamic range pixel data then.
According to this structure, when do not exist than with the non-standard exposure pixel data of shorter time for exposure of the above-mentioned selected non-standard exposure pixel data short exposure time that promptly the 1st non-standard exposure pixel data is corresponding promptly during the 2nd non-standard exposure pixel data, in the correction of the intensity level of the above-mentioned the 1st non-standard exposure pixel data, can make and from the 1st non-standard exposure pixel data, deduct the on duty of the 1st saturated prediction usefulness brightness value with the 1st normalisation coefft, make the brightness value of this result and standard exposure pixel data carry out the addition processing then, handle by this addition and can reduce the noise that increases timing.That is, multiply by the 1st normalisation coefft, can reduce the noisiness that amplifies when multiplying each other compared with the brightness value that only makes the 1st non-standard exposure pixel data, in addition, owing to also carry out handling with the addition of standard exposure pixel data, thereby can further reduce noise.
And, owing to can use this 1st non-standard exposure pixel data that has reduced noise to generate the high dynamic range pixel data, can obtain obtaining the image quality ratio effect of the image of stable wide dynamic range in the past.
And the high dynamic range images in the manner generates different with above-mentioned associated form, is that the linearity of keeping picture signal is carried out the processing that dynamic range enlarges, and can realize loyal color reprodubility thus.Also identical in the camera head of mode 10 below.
[mode 9]
In addition, the camera head of mode 9 is characterised in that, in the camera head of mode 8, when the above-mentioned standard exposure time be made as " a ", when will the short exposure time corresponding being made as " b " with the non-standard exposure pixel data that is used to generate above-mentioned imaging apparatus, will above-mentioned unsaturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " a/b " as above-mentioned saturated prediction brightness value
The normalisation coefft that uses above-mentioned " a/b " to use as the non-standard exposure pixel data that is used to generate above-mentioned imaging apparatus.
According to this structure, in the correction of the intensity level of the 1st non-standard exposure pixel data, can make the brightness value of the 1st non-standard exposure pixel data multiply by 1st normalisation coefft corresponding, proofread and correct so can more suitably carry out intensity level with the ratio of short exposure time and standard exposure time.
[mode 10]
In addition, the camera head of mode 10 is characterised in that, in the camera head of mode 8, when having than the above-mentioned the 2nd non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit deducts the above-mentioned saturated prediction of using with the 2nd non-standard exposure pixel data from the represented brightness value of the above-mentioned the 2nd non-standard exposure pixel data be the result that the 2nd saturated prediction obtains divided by the 2nd normalisation coefft with brightness value with brightness value, make this subtract each other the result again and above-mentioned the 2nd normalisation coefft multiplies each other, the added luminance that this multiplied result and the above-mentioned the 1st non-standard exposure pixel data are represented, from the result of this addition, deduct the above-mentioned the 1st saturated prediction brightness value, make this subtract each other the result and multiply by above-mentioned the 1st normalisation coefft, make this multiplied result and the represented added luminance of above-mentioned the 2nd standard exposure pixel data generate above-mentioned high dynamic range pixel data.
According to this structure, have than with the non-standard exposure pixel data of shorter time for exposure of the above-mentioned selected non-standard exposure pixel data short exposure time that promptly the 1st non-standard exposure pixel data is corresponding promptly during the 2nd non-standard exposure pixel data, in the correction of the intensity level of the above-mentioned the 1st non-standard exposure pixel data, can make and from the 2nd non-standard exposure pixel data, deduct the 2nd saturated prediction and multiply each other with resulting value of brightness value and the 2nd normalisation coefft, and the added luminance that this multiplied result and the 1st non-standard exposure pixel data are represented, the processing that makes this addition result multiply by the added luminance of the multiplication result of the 1st normalisation coefft and standard exposure pixel data can be handled the noise that further reduce the increase timing by this addition.Promptly, multiply by the 1st normalisation coefft compared with the brightness value that only makes the 1st non-standard exposure pixel data, can reduce the noisiness that amplifies when multiplying each other, in addition, owing to also carry out handling, thereby can further reduce noise with the addition of the 2nd non-standard exposure pixel data and standard exposure pixel data.
And, owing to can use this 1st non-standard exposure pixel data that has reduced noise to generate the high dynamic range pixel data, can obtain obtaining the image quality ratio effect of the image of stable wide dynamic range in the past.
[mode 11]
In addition, the camera head of mode 11 is characterised in that, in the camera head of mode 10, when the above-mentioned standard exposure time be made as " a ", when will the short exposure time corresponding being made as " b " with the above-mentioned the 1st non-standard exposure pixel data, will above-mentioned unsaturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " a/b " as the above-mentioned the 1st saturated prediction brightness value; When the above-mentioned standard exposure time be made as " a ", when will the short exposure time corresponding being made as " c " with the above-mentioned the 2nd non-standard exposure pixel data, will do not reach above-mentioned saturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " (ab)/(ac) " as the above-mentioned the 1st saturated prediction brightness value
Use above-mentioned " a/b " as above-mentioned the 1st normalisation coefft, use above-mentioned " (ab)/(ac) " as above-mentioned the 2nd normalisation coefft.
According to this structure, in the correction of the intensity level of the 1st non-standard exposure pixel data, can make the brightness value of the 1st non-standard exposure pixel data multiply by 1st normalisation coefft corresponding with the ratio of this short exposure time and standard exposure time, simultaneously can make the brightness value of the 2nd non-standard exposure pixel data multiply by 2nd normalisation coefft corresponding, thereby can access the effect of more suitably carrying out the intensity level correction with the ratio of this short exposure time and standard exposure time.
[mode 12]
On the other hand, in order to reach above-mentioned purpose, the shooting assembling system of mode 12 has the electronic shutter function of photo-electric conversion element being arranged to the time for exposure of how rectangular photoelectric conversion part and the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that this camera system has:
The 1st reading unit, it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes;
The 2nd reading unit, its between the exposure period identical with above-mentioned the 1st reading unit in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time;
Saturated predicting unit, it is according to the non-standard exposure pixel data that above-mentioned the 2nd electric charge that uses above-mentioned the 2nd reading unit to be read is derived, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated; And
The high dynamic range images data generating unit, it predicts the outcome according to above-mentioned saturated predicting unit, the standard exposure pixel data and the above-mentioned non-standard exposure pixel data that synthesize the electric charge when using exposing in the above-mentioned standard exposure time that above-mentioned the 1st reading unit read and derive generate high dynamic range (High Dynamic Range) view data.
Can obtain effect and the effect equal thus with the camera head of mode 2.
Herein, native system can be implemented as single device, terminal and other equipment (this moment is equal with the camera head of mode 2), can also realize the network system that can be connected with multiple arrangement, terminal and other equipment communicatedly.In the latter case, each inscape be as long as can connect communicatedly respectively, just can belong in a plurality of equipment etc. any one.
[mode 13]
In addition, in order to reach above-mentioned purpose, the image capture method of mode 13 is used to have the camera head of photo-electric conversion element being arranged to how rectangular photoelectric conversion part and the electronic shutter function of the time for exposure of the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that this image capture method has:
The 1st read step, it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes;
The 2nd read step, its with above-mentioned the 1st read step in exposure period between between identical exposure period in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time;
Saturated prediction steps, it is according to the non-standard exposure pixel data that uses above-mentioned the 2nd electric charge read in above-mentioned the 2nd read step to derive, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated; And
The high dynamic range images data generate step, it predicts the outcome according to above-mentioned saturated prediction steps, the standard exposure pixel data and the above-mentioned non-standard exposure pixel data that synthesize the electric charge when using exposing in the above-mentioned standard exposure time of reading in above-mentioned the 1st read step and derive generate high dynamic range (High Dynamic Range) view data.
Can obtain effect and the effect equal thus with the camera head of mode 2.
Description of drawings
Fig. 1 is the block diagram of the summary structure of expression camera head 1 of the present invention.
Fig. 2 is the block diagram of the internal structure of the internal structure of expression shooting treatment system 10 and host computer system 2.
Fig. 3 is expression 1AFE (Analog Front End: the AFE (analog front end)) figure of 102 internal structure.
Fig. 4 is the block diagram that the non-destruction of expression is scanned the internal structure of corresponding type imaging apparatus 100.
Fig. 5 is the figure of the internal structure of expression scan line scanner 54.
Fig. 6 is the figure of the detailed structure of expression sensor cell array 56.
Fig. 7 is the figure of an example that reads action of the exposure of every line of each pixel in the sensor cell array 56 of expression imaging apparatus 100 and picture element signal.
Fig. 8 is the block diagram of the internal structure of expression image processing system 12.
Fig. 9 is the figure of the internal structure of expression non-standard view data generating unit/saturated prediction section 12c.
Figure 10 (a) is the figure of the internal structure of expression the 1st non-standard image production part 70, (b) is the figure of the internal structure of expression the 1st saturated prediction section 72, (c) is the figure of the internal structure of expression the 1st composograph generating unit 73.
Figure 11 is the figure that expression destroys the variation of accumulating the quantity of electric charge of reading the pixel under the mode.
Figure 12 is the figure of each pixel of expression sensor cell array 56 with respect to an example of the variation of time for exposure of different irradiation light quantities and signal level (intensity level).
Figure 13 is the figure of internal structure of the HDR image production part 12d of expression the 1st execution mode.
Figure 14 is the figure of internal structure of the HDR image production part 12d of expression the 2nd execution mode.
Figure 15 is the figure of an example of saturated/unsaturated false judgment of representing that the difference of pixel value in the past causes.
Figure 16 is the figure that represents the relation of the illuminance of synthesis mode subject in the past and output valve.
Embodiment
[the 1st execution mode]
Below, the 1st execution mode of imaging apparatus of the present invention, camera head, camera system and image capture method is described with reference to the accompanying drawings.Fig. 1 to Figure 13 is the figure of the 1st execution mode of expression imaging apparatus of the present invention, camera head, camera system and image capture method.
Below, the summary structure of camera head 1 of the present invention is described according to Fig. 1.Herein, Fig. 1 is the block diagram of the summary structure of expression camera system 3 of the present invention.
As shown in Figure 1, camera system 3 constitutes and comprises camera head 1 and host computer system 2.
And then, as shown in Figure 1, camera head 1 constitutes and comprises: image processing system (DSP) 12, its corresponding pixel data of the exposure with short exposure time (below be called the short exposure pixel data) according to the output of shooting treatment system 10 and with the corresponding pixel data of the exposure of ultrashort time for exposure (below be called ultrashort exposure pixel data), predict whether the pixel corresponding with short exposure pixel data and ultrashort exposure pixel data be saturated when the exposure of standard exposure time, according to this synthetic pixel data corresponding that predict the outcome (below be called the standard exposure pixel data) with the exposure of standard exposure time, above-mentioned short exposure pixel data and ultrashort exposure pixel data generate wide dynamic range view data (below be called the HDR view data); And frame memory 14, the various view data of its Memory Reference view data (aftermentioned), the ultrashort exposure image data that constitute by above-mentioned ultrashort exposure pixel data, the short exposure image data that constitutes by above-mentioned short exposure pixel data, HDR view data etc.
And then, according to Fig. 2 to Fig. 6 illustrate the shooting treatment system 10 internal structure.Herein, Fig. 2 is the block diagram of the internal structure of the internal structure of expression shooting treatment system 10 and host computer system 2.Fig. 3 is the figure of the internal structure of expression 1AFE (Analog Front End) 102.Fig. 4 is the block diagram that the non-destruction of expression is scanned the internal structure of corresponding type imaging apparatus 100.Fig. 5 is the figure of the internal structure of expression scan line scanner 54.Fig. 6 is the figure of the detailed structure of expression sensor cell array 56.
As shown in Figure 2, shooting treatment system 10 constitutes and comprises non-destruction and scan corresponding type imaging apparatus 100,1AFE 102,2AFE 104,3AFE 106 and 4AFE 108.
Non-destruction is scanned corresponding type imaging apparatus 100 (below be called imaging apparatus 100) and is gone up the light of concentrating from subject with pick-up lens (not shown) at sensor cell array 56 (aftermentioned), and electric charge accumulation that will be corresponding with this light harvesting amount is on each pixel of sensor cell array 56.And, the electric charge group that the drive signal (pixel clock, horizontal-drive signal and vertical synchronizing signal 0) that imaging apparatus 100 is exported according to the timing controller 12b (aftermentioned) of image processing system 12 will be accumulated in the exposure of standard exposure time on each pixel column of sensor cell array 56 is converted to the voltage group successively, and the electric charge group of each pixel column after will just resetting simultaneously is converted to the voltage group successively.And, the electric charge group that the vertical synchronizing signal 1 that imaging apparatus 100 is generated according to scan line scanner 54 described later will be accumulated in the exposure of ultrashort time for exposure on each pixel column of sensor cell array 56 is converted to the voltage group successively, and the electric charge group that the vertical synchronizing signal 2 that is generated according to scan line scanner 54 described later will be accumulated in the exposure of short exposure time on each pixel column of sensor cell array 56 is converted to the voltage group successively.And, imaging apparatus 100 is by giving drive control signal from host computer system 2 (aftermentioned), can control the processing of reading of picture element signal, except the combination of above-mentioned time for exposure, can also carry out the processing of reading of picture element signal with standard exposure time and short exposure time or a certain side's in the ultrashort time for exposure combination.That is, except the combination of above-mentioned standard exposure time, ultrashort time for exposure and short exposure time, the combination of all right choice criteria time for exposure and ultrashort time for exposure, standard exposure time and short exposure time.
That the electric charge group that imaging apparatus 100 will expose in the time to the standard exposure that is about to reset is carried out is that converted voltage group (picture element signal) has via horizontal transport portion 58 (aftermentioned), constitute the 1st delivery channel 58a that comprises the 1st linear memory (below be called CH1) exports to 1AFE102 successively; Will to the electric charge group after just resetting carry out that converted voltage group (picture element signal) has via horizontal transport portion 58 (aftermentioned), constitute the 2nd delivery channel 58b that comprises the 2nd linear memory (below be called CH2) and export to 2AFE102 successively.In addition, imaging apparatus 100 will to the electric charge group of exposing in the short exposure time carry out that converted voltage group has via horizontal transport portion 58 (aftermentioned), constitute the 3rd delivery channel 58c that comprises the 3rd linear memory (below be called CH3) and export to 3AFE106 successively; Will to the electric charge group of exposing in the ultrashort time for exposure carry out that converted voltage group has via horizontal transport portion 58 (aftermentioned), constitute the 4th delivery channel 58d that comprises the 4th linear memory (below be called CH4) and export to 4AFE 108 successively.
In the present embodiment, imaging apparatus 100 is for the exposure area of sensor cell array 56, read mode with destruction and independently carry out reading respectively from the electric charge before and after the resetting of each pixel via CH1 and CH2, reading of the electric charge that independently carries out short exposure time respectively via CH3 and CH4 in the nondestructive read mode of getting and reading of the electric charge of ultrashort time for exposure, thereby using between 1 time exposure period of electronic shutter function in (standard exposure time) the electric charge group when independently reading standard exposure time interior exposure respectively, electric charge group after just having resetted, electric charge group during exposure in short exposure time and ultrashort time for exposure.
Explanation herein destroys reads the different of the action of getting with nondestructive read.It is to carry out reset processing (making the electric charge that is accumulated in the sensor unit be empty processing) after reading at once that destruction is read, and reads action again.The signal (analogue data) that reads before resetting is kept in the 1st linear memory, and the signal that reads after resetting is kept in the 2nd linear memory.In differential amplifier 60 (aftermentioned), carry out the subtraction process of corresponding picture element signal and carry out the detection of signal level and the removal of noise then.On the other hand, to get be not carry out reset processing after reading to nondestructive read.Signal after reading (analogue data) is kept at respectively in the 3rd linear memory and the 4th linear memory.The picture element signal and the pixel clock that are kept at respectively in the 1st～the 4th linear memory synchronously are output to the 1st～the 4AFE 102～108 respectively.
The 1st～the 4AFE 102～108 will be converted to numerical data (below be called pixel data) via corresponding picture element signal (analogue data) of the different separately time for exposure of the CH1 of horizontal transport portion 58～CH4 output.Then, the 1st～the 4AFE 102～108 pixel data that will generate is exported to the non-standard view data generating unit/saturated prediction section 12c (aftermentioned) and the HDR image production part 12d (aftermentioned) of image processing system 12 respectively.
The internal structure of 1AFE 102 is described according to Fig. 3 then.
As shown in Figure 3,1AFE 102 constitutes and comprises clamp circuit 102a, amplifying circuit 102b and A/D change-over circuit 102c.
Amplifying circuit 102b makes the picture element signal after the clamper amplify in the mode that the input range with A/D change-over circuit 102c is complementary, and the picture element signal after this amplification is exported to A/D change-over circuit 102c.
A/D change-over circuit 102c will export to image processing system 12 after will being converted to pixel data (numerical data) from the picture element signal (analogue data) of amplifying circuit 102b.
And because the 1st～the 4AFE 102～108 has identical internal structure, so omit explanation to the internal structure of the 2nd～the 4AFE 104～108.
And then, the internal structure of imaging apparatus 100 is described according to Fig. 4.
As shown in Figure 4, imaging apparatus 100 constitutes benchmark timing generator 50, driving pulse generator 52, scan line scanner 54, sensor cell array 56 and the horizontal transport portion 58 of comprising.
Driving pulse generator 52 bases select signal and read line to select signal to produce driving pulse from the reference timing signal of benchmark timing generator 50 with from the reset line of scan line scanner 54, and offer sensor cell array 56.
The internal structure of scan line scanner 54 is described according to Fig. 5 below.
As shown in Figure 5, scan line scanner 54 constitutes and comprises common scan counter 54a, common scan address decoder 54b, the 1st non-destruction scan counter 54c, the 1st non-destruction scan address decoder 54d, the 2nd non-destruction scan counter 54e, the 2nd non-destruction scan address decoder 54f and OR logic device 54g.
Usually scan counter 54a is according to vertical synchronizing signal 0 and the action of horizontal-drive signal repeated incremental counting from benchmark timing generator 50.Herein, the value of counter is corresponding with the wire size sign indicating number of the pixel of exposure area, and this wire size sign indicating number is exported to common scan address decoder 54b.
Usually scan address decoder 54b makes line from the wire size sign indicating number of common scan counter 54a as " read line " and effectively, line in addition is invalid.And then the read line control signal that will be expressed as effective line position (address) is exported to OR logic device 54g, and selects signal to export to driving pulse generator 52 as reset line this read line control signal.
The 1st non-destruction scan counter 54c is according to the information and common scan counter 54a asynchronous ground repeated incremental counting action of expression from the initial number of communicator/DSP operation control part 12a.Herein, the value of counter is corresponding with the wire size sign indicating number of the pixel of exposure area, and this wire size sign indicating number is exported to the 1st non-destruction scan address decoder 54d.In addition, the 1st non-destruction scan counter 54c generates that to be used for carrying out the vertical synchronizing signal that nondestructive read gets by the exposure of short exposure time be vertical synchronizing signal 1, and the vertical synchronizing signal 1 that will generate is exported to the timing controller 12b of image processing system 12.
The 1st non-destruction scan address decoder 54d makes from the line of the wire size sign indicating number of the 1st non-destruction scan counter 54c as " read line " and effectively, line in addition is invalid.And then the read line control signal that will be expressed as effective line position (address) is exported to OR logic device 54g.
The 2nd non-destruction scan counter 54e and the 1st non-destruction scan counter 54c have same structure, with the action of asynchronous ground of common scan counter 54a repeated incremental counting, the wire size sign indicating number that count value is represented are exported to the 2nd non-destruction scan address decoder 54d.In addition, the 2nd non-destruction scan counter 54e generates that to be used for carrying out the vertical synchronizing signal that nondestructive read gets by the exposure of ultrashort time for exposure be vertical synchronizing signal 2, the vertical synchronizing signal 2 of this generation is exported to the timing controller 12b of image processing system 12.
The 2nd non-destruction scan address decoder 54f and the 1st non-destruction scan address decoder 54d have same structure, make line from the wire size sign indicating number of the 2nd non-destruction scan counter 54e as " read line " and effectively, line in addition is invalid.And then the read line control signal that will be expressed as effective line position (address) is exported to OR logic device 54g.
OR logic device 54g basis is from the read line control signal of common scan address decoder 54b, from the 1st and the 2nd non-destruction scan address decoder 54d and 54f read line control signal, every line is carried out the OR computing, generate final read line selection signal with respect to each time for exposure of exposure area.Select signal to export to driving pulse generator 52 read line of these generations.
The detailed structure of sensor cell array 56 is described according to Fig. 6 then.
As shown in Figure 6, sensor cell array 56 will use CMOS and a plurality of sensor units (pixel) 56a of constituting is configured to rectangular, according to each pixel column, address wire, reset line and read line jointly are connected on the sensor unit 56a that constitutes each pixel column, various drive signals are sent to the sensor unit 56a that constitutes each pixel column via these 3 control lines.Then, constitute when address wire and read line are effective, will accumulate charge transfer via holding wire shown in Figure 6 and give on some among the 1st～the 4th delivery channel 58a～58d.By this structure, making the pixel column (selection) that carries out homing action or read action by means of address wire is effectively, each sensor unit 56a for the pixel column of selecting signal to select by this, the signal that under the situation of carrying out homing action, moves via reset line input indication reset, carrying out under the situation about reading of picture element signal, accumulating the signal of the transmission of electric charge via read line input indication.
And then, describe to the control method of the time for exposure of imaging apparatus 100 with from the read method of the picture element signal of sensor cell array 56 according to Fig. 7.Herein, Fig. 7 is the figure of an example that reads action of the exposure of every line of each pixel in the sensor cell array 56 of expression imaging apparatus 100 and picture element signal.
Herein, the control of time for exposure of the present invention is the exposure area of setting for sensor cell array 56, carry out each pixel line accumulate resetting and common scan line (read line) L1 that reads of the picture element signal of standard exposure time of electric charge, and set exposure area, carry out high-velocity scanning line (read line) L3 that high-velocity scanning line (read line) L2 that the nondestructive read of the picture element signal of short exposure time gets and the nondestructive read that carries out the picture element signal of ultrashort time for exposure are got for sensor cell array 56.Then, between 1 time exposure period in (standard exposure time), independent respectively carry out picture element signal when standard exposure expose in the time read and the nondestructive read of the picture element signal when resetting and exposing in short exposure time with in the ultrashort time for exposure is got.Promptly, usually scan line L1 and high-velocity scanning line L2, L3 are as shown in Figure 7, when the electric charge of standard exposure time quantum was accumulated successively to the line of the pixel of exposure area, scan line L1 was set to the picture element signal of the line that reads each pixel successively usually, and made this accumulate electric charge successively to reset.On the other hand, on the line of each pixel after the resetting of exposure area, in during the electric charge of accumulating the standard exposure time quantum, high-velocity scanning line L2, L3 are set to the picture element signal that reads the line of each pixel in ultrashort time for exposure and short exposure time with non-destruction successively respectively.
And, in the present embodiment, as shown in Figure 7, at standard exposure in the time, picture element signal during exposure (analogue data) is read on the 1st linear memory of CH1, and on the other hand, the picture element signal after just having resetted is read on the 2nd linear memory of CH2.Then, as shown in Figure 7, the picture element signal that these read is exported to the differential amplifier 60 of the outlet side that is arranged at horizontal transport portion 58, in this differential amplifier 60, reset preceding and reset after corresponding respectively picture element signal between subtraction process carry out the detection and the noise remove of signal level.Then, be exported to 1AFE 102 and be converted to numerical data (pixel data) through the picture element signal after the subtraction process at this.On the other hand, the picture element signal former state that is read after just the resetting in the 2nd linear memory of CH2 is exported to 2AFE, and is converted to numerical data (pixel data) at this.
In addition, the picture element signal when exposing in short exposure time is read in the 3rd linear memory of CH3, exports to 3AFE 106, and is converted to numerical data (pixel data) at this; Picture element signal when exposing in the ultrashort time for exposure is read in the 4th linear memory of CH4, exports to 4AFE 108, and is converted to numerical data (pixel data) at this.
In addition, as shown in Figure 7, regularly the control of reading of the picture element signal of above-mentioned common scan line L1 and high-velocity scanning line L2, L3 is that every line according to each pixel scans common scan line L1 (the last direction among Fig. 7) successively, on this common scan line L1, accumulate resetting of electric charge, before and after accumulating the resetting of electric charge, read simultaneously the exposure of having carried out the standard exposure time pixel picture element signal and just resetted after picture element signal.For example on the 1st line, carry out reading and resetting of picture element signal then, after from linear memory, all picture element signals all being read the outside, carry out the scanning of common scan line L1 successively, when common scan line L1 arrives the 1st line once more, standard exposure time just in time the timing of process carry out the scanning of common scan line L1.With the line of such step to the pixel of the exposure area of sensor cell array 56, according to every line of each pixel, the reading and accumulate resetting of electric charge of the picture element signal when exposing usually successively.On the other hand, make by common scan line L1 when accumulating electric charge and resetting, the line of the pixel after this is resetted, on high-velocity scanning line L3 the picture element signal of the pixel of the exposure of having carried out the ultrashort time for exposure being carried out nondestructive read gets, then, on high-velocity scanning line L2 the picture element signal of the pixel of the exposure of having carried out short exposure time being carried out nondestructive read gets.With the line of such step to each pixel of sensor cell array 56, according to every line, the nondestructive read of the picture element signal when exposing successively in ultrashort time for exposure and short exposure time is got.
The internal structure of image processing system 12 then, is described according to Fig. 8～Figure 11.Herein, Fig. 8 is the block diagram of the internal structure of expression image processing system 12.Fig. 9 is the figure of the internal structure of expression non-standard view data generating unit/saturated prediction section 12c.Figure 10 (a) is the figure of the internal structure of expression the 1st non-standard image production part 70, (b) is the figure of the internal structure of expression the 1st saturated prediction section 72, (c) is the figure of the internal structure of expression the 1st composograph generating unit 73.Figure 11 is the figure that expression destroys the variation of accumulating the quantity of electric charge of reading the pixel under the mode.Figure 12 is the figure of each pixel of expression sensor cell array 56 with respect to an example of the variation of time for exposure of different irradiation light quantities and signal level (intensity level).Figure 13 is the figure of internal structure of the HDR image production part 12d of expression the 1st execution mode.
As shown in Figure 8, image processing system 12 constitutes and comprises communicator/DSP operation control part 12a, timing controller 12b, non-standard view data generating unit/saturated prediction section 12c, HDR image production part 12d, storage access arbiter 12e and output reader 12f.
Communicator/DSP operation control part 12a obtains the relevant information of the initial number got with respect to the nondestructive read of sensor cell array 56 from system controller 2a (aftermentioned), the drive control signal of the initial number of this acquisition of expression is exported to the scan line scanner 54 of shooting treatment system 10.
As shown in Figure 9, non-standard view data generating unit/saturated prediction section 12c constitutes and comprises the 1st non-standard image production part the 70, the 1st saturated prediction section the 72, the 1st composograph generating unit 73, saturated preservation memory the 74, the 2nd non-standard image production part the 75, the 2nd saturated prediction section 76 and the 2nd composograph generating unit 77 that predict the outcome.
Shown in Figure 10 (a), the 1st non-standard image production part 70 constitutes and comprises subtracter 70a.The pixel data corresponding with the time for exposure of standard exposure outside the time is called non-standard exposure pixel data herein.That is, corresponding with ultrashort time for exposure and short exposure time respectively ultrashort exposure pixel data and short exposure pixel data become non-standard exposure pixel data.
Subtracter 70a is each pixel value of short exposure image data (the short exposure pixel data of line unit) expression from one of non-standard exposure pixel data of being imported of shooting treatment system 10, deduct the location of pixels identical with this each short exposure pixel data, via storage access arbiter 12e from frame memory 14 read out just reset after pixel data be the pixel value of benchmark pixel data representation.Then, the 1st non-standard image production part 70 will be exported to the 1st saturated prediction section 72 and the 1st composograph generating unit 73 respectively by the short exposure image data that this difference constitutes.
Herein, the address information that non-standard view data generating unit/saturated prediction section 12c is imported according to timing controller 12b, to be stored in the frame memory 14 via storage access arbiter 12e by the benchmark pixel data that shooting treatment system 10 is imported, read the benchmark pixel data of this storage afterwards and use.Therefore, the benchmark image data are only obtained 1 time that begins most (standard exposure time) and are got final product.
And, the variation of accumulating the quantity of electric charge of each pixel of sensor cell array 56 is described according to Figure 11 for the action of the 1st non-standard image production part 70 is described.
As shown in figure 11, in the exposure of (standard exposure time) of 1 frame, be accumulated in each pixel of sensor cell array 56 the quantity of electric charge in time through and increase.In ultrashort time for exposure (T2 among the figure) and short exposure time (T1 among the figure),, in exposure, carry out the quantity of electric charge of accumulating that to keep each pixel that reads of inferior arbitrarily electric charge owing to get mode reading electric charges from each pixel with nondestructive read.
And resetting among Figure 11 regularly is to make the electric charge that is accumulated in the sensor cell array 56 be empty timing, and this regularly settles the standard the time for exposure.And, then as mentioned above, generate image according to difference for example with the next non-destruction of the timing of (1) shown in Figure 11～(3) ground reading electric charges.That is, according to the timing in (1) read out just reset after the quantity of electric charge and difference generation image separately at each quantity of electric charge that reads out respectively in regularly of (2) and (3).
The pixel data of the nondestructive read taking-up that obtains from shooting treatment system 10 in addition, is sneaked into the caused fixed pattern noise of difference of each pixel.Therefore can remove fixed pattern noise by calculated difference.
Return Fig. 9, the 1st saturated prediction section 72 constitutes and comprises comparator 72a shown in Figure 10 (b).
The brightness value of brightness value V1 that comparator 72a is relatively represented by the short exposure pixel data of the 1st non-standard image production part 70 inputs and the saturated prediction usefulness used via the short exposure time of communicator/DSP operation control part 12a input from system controller 2a is saturated prediction level THR1, if " V1〉THR1 " then as the saturated output " 1 " that predicts the outcome, if for " V1≤THR1 " then as the saturated output " 0 " that predicts the outcome.Promptly, when saturated when predicting the outcome to " 1 ", the pixel corresponding with this short exposure pixel data accumulate the quantity of electric charge be predicted to be the standard exposure time through out-of-date saturated, on the other hand, when saturated when predicting the outcome to " 0 ", the pixel corresponding with this short exposure pixel data accumulate the quantity of electric charge be predicted to be the standard exposure time through out-of-date unsaturation.Then, the 1st saturated prediction section 72 should saturatedly predict the outcome and export to the 1st composograph generating unit 73, and should saturatedly predict the outcome and export to saturated predicting the outcome and preserve memory 74.
Herein, for the action of the 1st saturated prediction section 72 is described, each pixel that sensor cell array 56 is described according to Figure 12 is with respect to the time for exposure of different irradiation light quantities and the variation of signal level (intensity level).
As shown in figure 12, to establish the ultrashort time for exposure be T2, establish short exposure time is that T1, accurate time for exposure of bidding are T0, considers in T0 just in time to arrive saturation level V
MaxThe gradient of (surpassing this value is saturated brightness value) increases the situation of accumulating the quantity of electric charge.When the irradiation light quantity of time per unit of establishing this moment is S0, if, can predict then whether this pixel is saturated at T0 according to irradiation light quantity S1, the S2 that the quantity of electric charge is obtained these time per units that accumulate of T2, T1.Can predict the saturation level V of sensor cell array 56 herein, in advance
Max, and if the irradiation light quantity is constant, then has dependency relation (general proportions relation) on time for exposure and output intensity level.That is, between the exposure period of standard exposure time in, as long as shooting does not exist on the object fierce brightness to change, then accumulate the quantity of electric charge and roughly be linear increase (recruitment with constant increases).
Therefore, relatively T0 becomes V constantly
MaxRecruitment (irradiation light quantity) S0 of time per unit and the irradiation light quantity of T2, T1 time per unit constantly, can be predicted as this pixel when S0 is following constantly unsaturated at T2 when them.Otherwise at least one side in T2, the T1 irradiation light quantity constantly surpasses under the situation of S0, and it is constantly saturated at T0 to be predicted as this pixel.For example as shown in figure 12, carry out making constantly and accumulate the quantity of electric charge and arrive V at T1
MaxVariation the time, T2, T1 irradiation light quantity constantly is owing to all surpass S0, so that the 1st saturated prediction section 72 is predicted as object pixels is saturated in standard exposure time T 0.And when using ultrashort exposure pixel data and short exposure pixel data two sides to predict, as shown in figure 12, the irradiation light quantity of T2 surpasses T1 becomes V constantly
MaxRecruitment (irradiation light quantity) S1 of time per unit the time, can predict that this pixel is constantly saturated at T1.That is, under short exposure time, under the saturated situation, can use ultrashort exposure pixel data to generate the HDR pixel data.
Return Fig. 9, the 1st composograph generating unit 73 constitutes to comprise and selects follower 73a shown in Figure 10 (c).
Select saturated predict the outcome of follower 73a according to the 1st saturated prediction section 72 inputs, select the short exposure pixel data and a certain side from the saturated prediction level THR1 that system controller 2a imports via communicator/DSP operation control part 12a of the 1st non-standard image production part 70 inputs, and export the data of this selection.Particularly, if saturatedly predict the outcome to " 1 " then export the short exposure pixel data, if saturatedly predict the outcome to " 0 " then export saturated prediction level THR1.Then, the 1st composograph generating unit 73 is stored in the short exposure pixel data or the saturated prediction level THR1 of selected output in the frame memory 14 via storage access arbiter 12e as the synthetic short exposure pixel data of exporting usefulness.
The saturated preservation memory 74 that predicts the outcome is saturated memories that predict the outcome that preservation the 1st saturated prediction section 72 and the 2nd saturated prediction section 76 are imported.It is for when carrying out saturated prediction according to ultrashort exposure pixel data and short exposure pixel data, absorbs difference of asynchronous time of each wire size sign indicating number (phase place) and is provided with.That is, at first carry out saturated prediction, after this stipulated time, carry out saturated prediction, so absorb its time difference based on the short exposure pixel data based on ultrashort exposure pixel data.Particularly, to be kept in the saturated preservation memory 74 that predicts the outcome based on saturated the predicting the outcome of ultrashort exposure pixel data, except based on the predicting the outcome of above-mentioned ultrashort exposure pixel data, also preserve ensuing saturated predicting the outcome (these 2 predict the outcome become final predicting the outcome) based on the short exposure pixel data.Wherein, saturated predicting the outcome preserved memory 74 as mentioned above, in reading processing, under the situation of having selected to utilize the combination that this 3 kinds of time for exposure of standard exposure time, ultrashort time for exposure and short exposure time read, can bring into play the function that absorbs the above-mentioned time difference.
The 2nd non-standard image production part 75 is different on the ultrashort exposure image data this point in the data of handling only, all the other are identical with the 1st non-standard image production part 70 structures, and it is the pixel value of benchmark pixel data representation that each pixel value of representing from each the ultrashort exposure pixel data that constitutes ultrashort exposure image data (the short exposure pixel data of line unit) deducts pixel data with this each ultrashort exposure pixel data same pixel position, that be stored in after just the resetting the frame memory 14.Then, the 2nd non-standard image production part 75 will be exported to the 2nd saturated prediction section 76 and the 2nd composograph generating unit 77 respectively by the ultrashort exposure image data that this difference constitutes.
The 2nd saturated prediction section 76 is identical with the 1st saturated prediction section 72 structures, in the comparator 76a of inside, brightness value V2 that the ultrashort exposure pixel data that comparison the 2nd non-standard image production part 75 is imported is represented and the saturated prediction level THR2 that uses from the ultrashort time for exposure that system controller 2a is imported, if " V2〉THR2 " then as the saturated output " 1 " that predicts the outcome, if for " V2≤THR2 " then as the saturated output " 0 " that predicts the outcome.Whether saturated whether saturated 2 kinds of patterns are considered in saturated prediction herein, i.e. pattern when pattern and prediction T0 when T2 predicts T1 constantly.In the present embodiment, by giving control signal, thereby can switch this 2 patterns from system controller 2a.Then, the 2nd saturated prediction section 76 is exported to the 2nd composograph generating unit 77 with any selecteed side among ultrashort exposure pixel data and the saturated prediction level THR2.
The 2nd composograph generating unit 77 is identical with the 1st composograph generating unit 73 structures, in the selection follower 77a of inside, saturated predicting the outcome according to the 2nd saturated prediction section 76 inputs, if saturatedly predict the outcome to " 1 " then export ultrashort exposure pixel data, if saturatedly predict the outcome to " 0 " then export saturated prediction level THR2.Then, the 2nd composograph generating unit 77 is stored in a side of any selected output among ultrashort exposure pixel data and the saturated prediction level THR2 in the frame memory 14 via storage access arbiter 12e as the synthetic ultrashort exposure pixel data of exporting usefulness.
The internal structure of HDR image production part 12d then, is described according to Figure 13.
Before the explanation internal structure, the operating principle of the HDR image production part 12d of present embodiment is described herein.
If above-mentioned short exposure time T1 is the 1/K1 (T1=T0/K1) of standard exposure time T 0.In addition in T1, the signal output level of the pixel of being shone by S0 can be predicted to be V
Max/ K1 (with it as above-mentioned THR1).That is, take a sample all pixels (carrying out reading of picture element signal from all pixels) behind elapsed time T1 in the back that resets, and the pixel predicted (judgement) that the signal output level of each pixel surpasses decision threshold THR1 is saturated behind elapsed time T0.And predicted (judgement) is the following pixel unsaturated behind elapsed time T0 (unsaturation) of THR1.
With above-mentioned situation is prerequisite, the following describes the generation method of the HDR image of present embodiment.
In the present embodiment, in shooting treatment system 10,, establish the short exposure pixel data that only obtains in short exposure time to be exposed herein, as non-standard exposure pixel data.In addition, in the present embodiment, be prerequisite, by the standardization of time for exposure, particularly be by the synthetic HDR image that generates of standardized linearity when the standard exposure time T 0 with the display unit that can in wide dynamic range, export.Therefore, value at the standard exposure pixel data that is predicted as T0 is under unsaturated situation, use this standard exposure pixel data, under the situation in addition, use and the short exposure pixel data that obtains when the T1 have been carried out standardized data obtain synthetic output.
Particularly,, when the time for exposure being extended to T0, predict that it becomes much values, the correction of short exposure pixel data is used for the value of this prediction on imagination ground for the short exposure pixel data.That is the short exposure pixel data behind the gamma correction when standard exposure pixel data when, synthesizing the unsaturation of T0 place and T0 place are saturated generates the HDR view data.In addition, in the present embodiment, can synthetic (K1 * V1) handle, but in order further to suppress the judgement difference that noise causes carries out synthetic to the synthetic consideration method that has adopted associated form of common linearity as simple linearity with the short exposure pixel data after proofreading and correct.That is the linearity of the standard exposure pixel data value V0 when short exposure pixel data value V1 when, having used T1 according to following formula (1) and T0 is synthetic.
Synthetic output=K1 * V1
＝K1×(V1-V
max/K1+V
max/K1)
＝K1×(V1-V
max/K1)+V
max
=K1 * (V1-THR1)+(the signal output during T0) ... (1)
In following formula (1), the ratio (normalisation coefft) of standard exposure time T 0 and short exposure time T1 be K1 (=T0/T1).
According to above-mentioned situation, the internal structure of HDR image production part 12d is described.
As shown in figure 13, HDR image production part 12d constitutes and comprises subtracter 80, multiplier 81, adder 82.
Return Fig. 8, storage access arbiter 12e is according to the read/write order for frame memory 14 from non-standard view data generating unit/saturated prediction section 12c, HDR image production part 12d and these 3 systems of output reader 12f, mediate the access request of these 3 systems, carry out access for the view data of frame memory 14.
As shown in Figure 8, frame memory 14 is memories of the various view data of Memory Reference view data, ultrashort exposure image data, short exposure image data, HDR view data etc., read when request when having, read out the pixel data that this request is represented from storage access arbiter 12e.And, writing when request when having from storage access arbiter 12e, frame memory 14 writes this pixel data that writes the request expression.
And then return Fig. 2, the internal structure of host computer system 2 is described.
Display unit 2b is made of the display device of LCD etc., has the function that can show the wide dynamic range image.And, according to indication, show from the image of the HDR view data of image processing system 12 acquisitions from system controller 2a.
The actual action of present embodiment then is described.
The following describes the action of only reading under the situation that the short exposure pixel data is used as non-standard exposure pixel data.In addition, in the present embodiment, establish the sampling time (short exposure time) that nondestructive read gets and be 1/10 of the standard exposure time.
When shooting treatment system 10 receives drive control signal, in scan line scanner 54, synchronously generate the read line control signal that reset line is selected signal and read with respect to the destruction of standard exposure time with vertical synchronizing signal 0 and horizontal-drive signal.In addition, according to initial number and horizontal-drive signal, generate the read line control signal of getting with respect to the nondestructive read of short exposure time.And then, the control signal that reads of these generations is inputed to OR logic device 54g, generate respectively with respect to the read line of each time for exposure and select signal.Then, select signal, read line to select signal (2 kinds) to export to driving pulse generator 52 reset line of these generations.Driving pulse generator 52 produces driving pulse and offers sensor cell array 56 according to from the reference timing signal of benchmark timing generator 50 with from the various selection signals of scan line scanner 54.
In image processing system 12, the standard exposure view data that 1AFE 102 is exported inputs to HDR image production part 12d, and the short exposure image data that benchmark image data that 2AFE 104 is exported and 3AFE 106 are exported inputs to non-standard view data generating unit/saturated prediction section 12c.
Non-standard view data generating unit/saturated prediction section 12c imports the benchmark image data from shooting treatment system 10, and when timing controller 12b has imported address information, according to this address information via storage access arbiter 12e with the benchmark image storage in frame memory 14.Then, non-standard view data generating unit/saturated prediction section 12c is next from shooting treatment system 10 when having imported short exposure image data, in the subtracter 70a of the 1st non-standard image production part 70, from the pixel value of each short exposure pixel data of constituting short exposure image data, deduct and be stored in pixel values in the frame memory 14 and benchmark pixel data this short exposure pixel data same pixel position, generate the short exposure pixel data of having removed fixed pattern noise.The 1st non-standard image production part 70 is exported to the 1st saturated prediction section 72 and the 1st composograph generating unit 73 respectively with it when having generated short exposure image data (line unit).
The 1st saturated prediction section 72 is imported short exposure image data from the 1st non-standard image production part 70, when communicator/DSP operation control part 12a has imported saturated prediction level THR1, in comparator 72a, relatively constitute brightness value V1 and THR1 that each short exposure pixel data of short exposure image data is represented.As mentioned above, because to establish short exposure time T1 be 1/10 of standard exposure time T 0, so as shown in figure 13, when T1, can be predicted to be V according to the signal output level of the pixel that above-mentioned S0 shone
Max/ 10.Therefore, establishing standardization COEFFICIENT K 1 herein is 10, and establishing THR1 is " V
Max/ 10 ".Therefore, comparator 72a " V1〉V
Max/ 10 " under the situation, be predicted as at standard exposure object pixels saturated and export " 1 " and predict the outcome under the time as saturated.On the other hand, if " V1≤V
Max/ 10 ", then object pixels is predicted as at standard exposure unsaturated and export " 0 " and predict the outcome under the time as saturated.The 1st saturated prediction section 72 should saturatedly predict the outcome and export to the 1st composograph generating unit 73, and should saturatedly predict the outcome and export to saturated predicting the outcome and preserve memory 74.
Owing to carry out saturated/unsaturated prediction (judgement) like this according to the short exposure pixel data, and constitute the short exposure pixel data by difference with the benchmark pixel data, so can carry out saturated/unsaturated prediction (judgement) according to the lower pixel data of the noise level of sneaking into, thus can carry out high-precision saturated/unsaturated prediction (judgement).
When having imported short exposure image data from the 1st non-standard image production part 70, when the 1st saturated prediction section 72 has been imported saturated predicting the outcome, the 1st composograph generating unit 73 is in selecting follower 73a, about each short exposure pixel data, if saturatedly predict the outcome to " 1 " then select short exposure pixel data (V1) output, if for " 0 " then select to export via the saturated prediction level THR1 that communicator/DSP operation control part 12a is imported from system controller 2a.This selects the short exposure pixel data of output as synthetic usefulness, and 12e is stored in the frame memory 14 via the storage access arbiter.
On the other hand, when in non-standard view data generating unit/saturated prediction section 12c, generated short exposure image data, and when shooting has been imported the standard exposure view data the treatment system 10, HDR image production part 12d is synthetic by combining common linearity linearity synthetic and associated form, generates the HDR view data.
HDR image production part 12d uses subtracter 80, multiplier 81 and adder 82 to carry out generating the HDR view data according to the processing of following formula (1).
Herein, because brightness value (after the difference processing)=V1, K1=10, the THR1=V of short exposure pixel data
Max/ 10, the value of standard exposure pixel data (the signal output during T0)
So following formula (1) becomes following formula (2).
Synthetic output=10 * (V1-V
Max/ 10)+V0... (2)
Promptly, HDR image production part 12d at first reads the short exposure image data of synthetic usefulness from frame memory 14 via storage access arbiter 12e, be entered in the subtracter 80, and will be input to the subtracter 80 via the saturated prediction level THR1 of communicator/DSP operation control part 12a input from system controller 2a.Then, in subtracter 80, the brightness value V1 or the THR1 (V that represent from each the short exposure pixel data that constitutes the short exposure image data imported
Max/ 10) deduct THR1 (V in
Max/ 10), the result that this is subtracted each other exports to multiplier 81.That is, under the situation of standard exposure saturated pixel in the time, the difference " V1-V of output V1 and THR1
Max/ 10 ", under the situation of standard exposure undersaturated pixel in the time, output " 0 ".
On the other hand, system controller 2a exports various synchronizing signals and carries out the request of reading of HDR view data to camera head 1.
Like this, the camera system 3 of present embodiment is in 1 imaging apparatus (sensor cell array 56), under the exposure of standard exposure time, read the shooting object is made a video recording by destruction, and between the exposure period of this standard exposure time, take owing to can be under the exposure of short exposure time fetch, so can obtain to be used for the standard exposure view data of generation of HDR image and non-standard exposure (short exposure) view data that is used for the generation of saturated/unsaturated judgement and HDR image simultaneously by nondestructive read.
In addition, camera system 3 can predict according to short exposure image data whether the pixel corresponding with short exposure image data be saturated in the time at standard exposure, so can precision judge saturated/unsaturation according to the less pixel data of The noise through the dark current noise that increases etc. in time well.And, compare the situation of carrying out saturated judgement with the standard exposure image, even for the saturation level of each pixel, the difference of sensitivity, also can suppress mistake and judge.That is, under situation about judging, because decision level is set lowlyer, so be difficult to be subjected to the influence of difference of difference, the sensitivity of saturation level with the short exposure image.And, owing to can predict the outcome the pixel data of selecting to be used to generate the HDR view data among a certain side from standard exposure pixel data and short exposure pixel data according to this, so can correctly be chosen in undersaturated standard exposure pixel data in the standard exposure time, can generate stable (wrong choice is few) HDR view data of image quality thus.
And, camera system 3 is by combining the gimmick of the synthetic and associated form of common linearity, use standard exposure pixel data and short exposure pixel data, can synthesize the short exposure pixel data under the situation saturated when T0, common linearity is synthetic can to reduce contained noise component(s) in the short exposure pixel data (HDR pixel data) that synthesizes usefulness by storage effect (low pass filter) so compare.And even just in case have wrong judgement, it is littler than V1 that the value of (V1-THR1) of formula (1) also just becomes, the influence in the time of can suppressing to amplify K1 times.
In above-mentioned the 1st execution mode, sensor cell array 56 is corresponding to certain 1 photoelectric conversion part in mode 1,2,12 and 13; Scan the destruction that benchmark timing generator 50, scan line scanner 54, driving pulse generator 52, horizontal transport portion 58 in the corresponding type imaging apparatus 100 carry out from the exposure area of sensor cell array 56 by the non-destruction of shooting treatment system 10 and read processing corresponding to certain 1 the 1st reading unit mode 1,2,3,6 and 12 or the 1st read step of mode 13 at the electric charge of standard exposure under the time; The nondestructive read that scans the electric charge under ultrashort time for exposure and short exposure time that benchmark timing generator 50, scan line scanner 54, driving pulse generator 52, horizontal transport portion 58 in the corresponding type imaging apparatus 100 carry out from the exposure area of sensor cell array 56 by the non-destruction of shooting treatment system 10 is got processing corresponding to certain 1 the 2nd reading unit mode 1～6 and 12 or the 2nd read step of mode 13.
And in above-mentioned the 1st execution mode, the saturated/unsaturated prediction processing among non-standard view data generating unit/saturated prediction section 12c is corresponding to certain 1 the saturated predicting unit in mode 1,2,3,4,6 and 12 or the saturated prediction steps of mode 13; HDR image production part 12d generates step corresponding to the HDR view data of certain 1 HDR image data creating unit in mode 2,4,6,7,8 and 12 or mode 13.
[the 2nd execution mode]
The 2nd execution mode of imaging apparatus of the present invention, camera head, camera system and image capture method is described below with reference to the accompanying drawings.Figure 14 is the figure of the 2nd execution mode of expression imaging apparatus of the present invention, camera head, camera system and image capture method.
Present embodiment obtains ultrashort exposure pixel data and short exposure pixel data both sides are used as non-standard pixel data, carry out saturated/unsaturated prediction according to ultrashort exposure pixel data and short exposure pixel data, and when being predicted as object pixels under the saturated situation at short exposure time T1, all ultrashort exposure pixel datas, short exposure pixel data and standard exposure pixel data that use has been carried out difference processing by the benchmark image data generate synthetic output (HDR pixel data), and be different with above-mentioned the 1st execution mode in this.Therefore, present embodiment is only local different on the structure of the HDR image production part 12d of Fig. 8 of above-mentioned the 1st execution mode, and structure in addition is identical with the camera system 3 of above-mentioned the 1st execution mode with action.Only explanation and the different part of above-mentioned the 1st execution mode below.
The internal structure of HDR image production part 12d of the camera head 1 of present embodiment is described according to Figure 14 below.Herein, Figure 14 (a) is the block diagram of internal structure of the HDR image production part 12d of expression present embodiment, (b) is the block diagram of the internal structure of the synthetic portion 91 of the ultrashort exposure of expression.
At first, before the explanation internal structure, the operating principle of the HDR image production part 12d of present embodiment is described.
With above-mentioned the 1st execution mode in the same manner, establish the 1/K1 that short exposure time T1 is a standard exposure time T 0 (T1=T0/K1), on the other hand, in the present embodiment, the 1/K2 that to establish ultrashort time for exposure T2 be short exposure time T1.Also have, with above-mentioned the 1st execution mode in the same manner, the signal output level of the pixel by the S0 irradiation when T1 is V
Max/ K1 (THR1), on the other hand, in the present embodiment, the signal output level of the pixel by S1 irradiation when T2 is V
Max/ K2 (with its saturated prediction level THR2 that uses as the ultrashort time for exposure).That is, take a sample all pixels behind elapsed time T2 in the back that resets, and the pixel that the signal output level of each pixel surpasses decision threshold THR2 is predicted to be behind elapsed time T1 saturated.And the following pixel of THR2 is predicted to be unsaturated behind elapsed time T1 (unsaturation).On the other hand, with above-mentioned the 1st execution mode in the same manner, reset the back behind elapsed time T1 with the sampling of all pixels, the pixel predicted (judgement) that the signal output level of each pixel surpasses decision threshold THR1 is saturated behind elapsed time T0.And the following pixel predicted (judgement) of THR1 is unsaturated behind elapsed time T0 (unsaturation).
With above-mentioned situation is prerequisite, the following describes the generation method of the HDR image of present embodiment.
In the present embodiment, in shooting treatment system 10,, establish the ultrashort exposure pixel data and the short exposure pixel data of all pixels that acquisition exposed in ultrashort time for exposure and short exposure time herein, as non-standard exposure pixel data.In addition, in the present embodiment, for ultrashort exposure pixel data and short exposure pixel data, in the time of will the time for exposure extending to T0 on imagination ground, predict that it becomes much values, ultrashort exposure pixel data and the correction of short exposure pixel data are used for the value after this prediction.
That is the short exposure pixel data behind the gamma correction when the unsaturated and T0 place in ultrashort exposure pixel data behind the gamma correction when the unsaturated and T1 place in standard exposure pixel data when, synthetic T0 place becomes unsaturation and T2 place is saturated and T1 place is saturated generates the HDR view data.In addition, in the present embodiment, with above-mentioned the 1st execution mode in the same manner, carry out synthetic to the synthetic consideration method that has adopted associated form of common linearity.That is, when being predicted as unsaturated and T0 place, T1 place when saturated, the linearity of the standard exposure pixel data value V0 when short exposure pixel data value V1 when having used T1 according to following formula (1) and T0 is synthetic.And then when being predicted as unsaturated and T1 place, T2 place when saturated, according to following formula (4) and (5), the linearity of the standard exposure pixel data value V0 when short exposure pixel data value V1 when ultrashort exposure pixel data value V2, T1 when using T0 and T0 is synthetic.
" predicted value during T1 "=K2 * V2
＝K2×(V2-V
max/K1+V
max/K1)
＝K2×(V2-V
max/K1)+V
max
=K2 * (V2-THR2)+(the signal output of T1) ... (4)
Wherein, establish standardization COEFFICIENT K 2=T1/T2.
In addition, according to following formula (1) and (4), the synthetic output of saturated situation becomes following formula (5) during T1.
Synthetic output=K1 * (predicted value during T1)
=K1 * (K2 (V2-THR2)+(the signal output of T1)-THR1)+(the signal output of T0)
=K1 * K2 * (V2-THR2)+K1 * ((the signal output of T1)-THR1))+(signal output of T0) ... (5)
According to above-mentioned situation, the internal structure of the HDR image production part 12d of present embodiment is described.
Shown in Figure 14 (a), HDR image production part 12d constitutes synthetic portion 90 of short exposure and the synthetic portion 91 of ultrashort exposure of comprising.
The synthetic portion 90 of short exposure has the structure identical with the structure shown in Figure 13 of above-mentioned the 1st execution mode, and is only different on the point of the addition result of adder 82 (below call the high-brghtness picture images data) being exported to the synthetic portion 91 of ultrashort exposure.Therefore omit its detailed description.
The synthetic portion 91 of ultrashort exposure constitutes and comprises subtracter 83, multiplier 84 and adder 85.
Subtracter 83 reads the ultrashort exposure image data of synthetic usefulness from frame memory 14 via storage access arbiter 12e, from each the ultrashort exposure pixel data (V2) that constitutes these ultrashort exposure image data or saturated prediction level (THR2), deduct from system controller 2a via the saturated prediction level (THR2) that communicator/DSP operation control part 12a is imported, this result who subtracts each other is exported to multiplier 84.That is, owing to be predicted as the ultrashort exposure pixel data of input under the situation of pixel saturated in the short exposure time T1, so (V2-THR2) in the following formula (5) is the value beyond 0; Import saturated prediction level (THR2) when being predicted as short exposure time T1 under the situation of undersaturated pixel as V2, so (V2-THR2) in the following formula (5) is 0.
Multiplier 84 will multiply each other (K1 * K2 * (V2-THR2)) via normalisation coefft K1 that communicator/DSP operation control part 12a is imported and K2 and from the value of (V2-THR2) of subtracter 83 inputs from system controller 2a, and this multiplied result is exported to adder 85.At this moment, when being predicted as short exposure time T1 under the undersaturated situation, because as mentioned above, value (V2-THR2) is 0, so also be 0 with normalisation coefft K1 and K2 multiplied result (K1 * K2 * 0=0).
Adder 85 will from the multiplication result of multiplier 84 input and with the corresponding high luminance pixel data of above-mentioned multiplication result that constitute the high-brghtness picture images data of being imported from the synthetic portion 90 of short exposure (K1 * (V1-THR1)+(the signal output of T0)) addition, this addition result is exported as synthesizing.Then, as the HDR view data, 12e is stored in the frame memory 14 HDR image production part 12d via the storage access arbiter with above-mentioned synthetic output.
The action of the reality of present embodiment then is described.
Herein, in the present embodiment, establishing the sampling time that nondestructive read gets is T2, T1, and establishing T2 is 1/10 (T2=T1/10=T0/100) of short exposure time T1, and establishing T1 is 1/10 (T1=T0/10) of standard exposure time T 0.
When shooting treatment system 10 receives drive control signal, in scan line scanner 54, synchronously generate the read line control signal that reset line is selected signal and read with respect to the destruction of standard exposure time with vertical synchronizing signal 0 and horizontal-drive signal.In addition, according to initial number and horizontal-drive signal, generate the read line control signal (2 kinds) of getting with respect to the nondestructive read of ultrashort time for exposure and short exposure time.And then, the control signal that reads of these generations is inputed to OR logic device 54g, generate respectively with respect to the read line of each time for exposure and select signal.Then, select signal, read line to select signal (3 kinds) to export to driving pulse generator 52 reset line of these generations.Driving pulse generator 52 produces driving pulse and offers sensor cell array 56 according to from the reference timing signal of benchmark timing generator 50 with from the various selection signals of scan line scanner 54.
Sensor cell array 56 is according to the driving pulse from driving pulse generator 52, scan common scan line L1, high-velocity scanning line L2 and high-velocity scanning line L3, from the line of each pixel of exposure area, the electric charge of accumulating by the exposure of standard exposure time destroyed and read (accumulating resetting of electric charge after reading), read action independently with this destruction, from the line of each pixel of exposure area to by the ultrashort time for exposure and the electric charge accumulated of the exposure of short exposure time carry out nondestructive read and get (not accumulating resetting of electric charge after reading).Then, the picture element signal that electric charge constituted before and after resetting that reads out by the scanning of common scan line L1 to differential amplifier 60 outputs via the CH1 of horizontal transport portion 58 and CH2, in differential amplifier 60, calculate these differences, this result of calculation is exported to 1AFE 102.In addition, via the CH2 of horizontal transport portion 58 picture element signal after 2AFE104 output has just resetted.On the other hand, the picture element signal that electric charge constituted that reads out by the scanning of high-velocity scanning line L2 to 3AFE 106 outputs via the CH3 of horizontal transport portion 58, the picture element signal that electric charge constituted that reads out by the scanning of high-velocity scanning line L3 to 4AFE 108 outputs via the CH4 of horizontal transport portion 58.
In image processing system 12, the standard exposure view data of 1AFE 102 outputs is input among the HDR image production part 12d, the benchmark image data of 2AFE 104 outputs, the short exposure image data of 3AFE 106 outputs and the ultrashort exposure image data of 4AFE 108 outputs are input among non-standard view data generating unit/saturated prediction section 12c.
When having imported the benchmark image data from shooting treatment system 10, and when timing controller 12b has imported address information, non-standard view data generating unit/saturated prediction section 12c according to this address information via storage access arbiter 12e with the benchmark image storage in frame memory 14.
Then, when next when shooting treatment system 10 has been imported ultrashort exposure image data, non-standard view data generating unit/saturated prediction section 12c is in the subtracter 75a of the 2nd non-standard image production part 75, from the pixel value of each ultrashort exposure pixel data of constituting ultrashort exposure image data, deduct and be stored in pixel values in the frame memory 14 and benchmark pixel data this ultrashort exposure pixel data same pixel position, generate the ultrashort exposure pixel data of the synthetic usefulness of having removed fixed pattern noise.When having generated ultrashort exposure image data (line unit), the 2nd non-standard image production part 75 should ultrashort exposure image data be exported to the 2nd saturated prediction section 76 and the 2nd composograph generating unit 77 respectively.
In addition, when following ultrashort exposure image data from shooting treatment system 10 when having imported short exposure image data, in the subtracter 70a of the 1st non-standard image production part 70, from the pixel value of each short exposure pixel data of constituting short exposure image data, deduct and be stored in pixel values in the frame memory 14 and benchmark pixel data this short exposure pixel data same pixel position, generate the short exposure pixel data of the synthetic usefulness of having removed fixed pattern noise.When having generated short exposure image data (line unit), the 1st non-standard image production part 70 is exported to the 1st saturated prediction section 72 and the 1st composograph generating unit 73 respectively with this short exposure image data.Below because the action of the 1st saturated prediction section 72 and the 1st composograph generating unit 73 is identical with above-mentioned the 1st execution mode, so the descriptions thereof are omitted.
When from the 2nd non-standard image production part 75, having imported ultrashort exposure image data, when from communicator/DSP operation control part 12a, having imported saturated prediction level THR2, the 2nd saturated prediction section 76 relatively constitutes brightness value V2 and THR2 that each ultrashort exposure pixel datas of ultrashort exposure image data is represented in comparator 76a.As mentioned above, because to establish ultrashort time for exposure T2 be 1/10 of short exposure time T1, so as shown in figure 13, when T2, the signal output level by the pixel that above-mentioned S1 shone can be predicted to be V
Max/ 10.Therefore, establishing standardization COEFFICIENT K 2 herein is 10, and THR2 is " V
Max/ 10 ".Therefore, comparator 76a " V2〉V
Max/ 10 " under the situation, be predicted as when the short exposure time object pixels saturated and export " 1 " and predict the outcome as saturated.On the other hand, if " V2≤V
Max/ 10 ", then object pixels is predicted as when short exposure time unsaturated and export " 0 " and predict the outcome as saturated.Then, the 2nd saturated prediction section 76 should saturatedly predict the outcome and export to the 2nd composograph generating unit 77, and should saturatedly predict the outcome and export to saturated predicting the outcome and preserve memory 74.
Owing to carry out saturated/unsaturated prediction (judgement) like this according to ultrashort exposure pixel data, and constitute the ultrashort exposure pixel data of synthetic usefulness by difference with the benchmark pixel data, so can carry out saturated/unsaturated prediction (judgement) according to the lower pixel data of the noise level of sneaking into, thus can carry out high-precision saturated/unsaturated prediction (judgement).
When importing short exposure image data from the 2nd non-standard image production part 75, during from saturated the predicting the outcome of the 2nd saturated prediction section 76 inputs, the 2nd composograph generating unit 77 is in selecting follower 77a, about each ultrashort exposure pixel data, if saturatedly predict the outcome to " 1 " then select ultrashort exposure pixel data (V2) to export, if for " 0 " then select to export via the saturated prediction level THR2 of communicator/DSP operation control part 12a input from system controller 2a.This selects the ultrashort exposure pixel data of output as synthetic usefulness, and 12e is stored in the frame memory 14 via the storage access arbiter.
On the other hand, when in non-standard view data generating unit/saturated prediction section 12c, having generated ultrashort exposure image data and short exposure image data, and when shooting has been imported the standard exposure view data the treatment system 10, HDR image production part 12d is synthetic by combining common linearity linearity synthetic and associated form, generates the HDR view data.
HDR image production part 12d at first uses subtracter 80, multiplier 81 and the adder 82 of the synthetic portion 90 of short exposure to carry out generating the high-brghtness picture images data according to the processing of following formula (1).
Herein, because the brightness value of short exposure pixel data (difference is handled the back)=V1, K1=10, THR1=V
Max/ 10, the value of standard exposure pixel data (the signal output during T0)
So following formula (1) similarly becomes following formula (2) with above-mentioned the 1st execution mode.
And the action of subtracter 80, multiplier 81 and the adder 82 of the synthetic portion 90 of short exposure is because identical with the action of subtracter 80, multiplier 81 and the adder 82 of above-mentioned the 1st execution mode, so omit its explanation.Wherein, the addition result of adder 82 (high-brghtness picture images data (10 * (V1-V
Max/ 10)+V0) or standard exposure view data (V0)) be exported to the synthetic portion 91 of ultrashort exposure.
The following describes the action of the synthetic portion 91 of ultrashort exposure.
Herein, because brightness value (after the difference processing)=V2, normalisation coefft K2=10, the saturated prediction level THR2=V of ultrashort exposure pixel data
Max/ 10, so following formula (5) becomes following formula (6).
Synthetic output=10 * 10 * (V2-V
Max/ 10)+10 * (V1-V
Max/ 10)+V0... (6)
HDR image production part 12d reads the ultrashort exposure image data of synthetic usefulness from frame memory 14 via storage access arbiter 12e, be entered in the subtracter 83, and will be input to the subtracter 83 via the saturated prediction level THR2 of communicator/DSP operation control part 12a input from system controller 2a.Then, in subtracter 83, the brightness value V2 or the THR2 (V that represent from each the ultrashort exposure pixel data that constitutes the ultrashort exposure image data imported
Max/ 10) deduct THR2 (V in
Max/ 10), the result that this is subtracted each other exports to multiplier 84.That is, when short exposure time T1, under the situation of saturated pixel, export the difference " V2-V of V2 and THR2
Max/ 10 ", when short exposure time T1, under the situation of undersaturated pixel, export " 0 ".
Multiplier 84 makes from system controller 2a and multiplies each other via the normalisation coefft K1 (10) of communicator/DSP operation control part 12a input and the subtraction result of K2 (10) and multiplier 83 inputs, with this multiplied result " 10 * 10 * (V2-V
Max/ 10) " or " 0 " export to adder 85.
Adder 85 makes the brightness value (10 * (V1-V of formation from the high luminance pixel data representation of the high-brghtness picture images data of the synthetic portion of short exposure 90 inputs
Max/ 10)+V0) with from the multiplication result addition of multiplier 81 input, with result " 10 * 10 * (V2-V of this addition
Max/ 10)+10 * (V1-V
Max/ 10)+V0 ", " 10 * (V1-V
Max/ 10)+V0 " and any one of " V0 " export as HDR is synthetic.That is,, will carry out standardized " 10 * 10 * (V2-V to this ultrashort exposure pixel data for being predicted to be pixel saturated when the short exposure time T1
Max/ 10)+10 * (V1-V
Max/ 10)+and V0 " export as the HDR pixel data.And, for being predicted to be pixel saturated when the standard exposure time T 0, will carry out standardized " 10 * (V1-V to this short exposure pixel data
Max/ 10)+and V0 " export as the HDR pixel data.In addition, for being predicted to be undersaturated pixel when the standard exposure time T 0, this standard exposure pixel data " V0 " itself is exported as the HDR pixel data.That is, the HDR view data become as mentioned above according to saturated by having predicted accurately/unsaturated predict the outcome select, ultrashort exposure pixel data has been carried out standardized pixel data, the short exposure pixel data has been carried out the data that standardized pixel data and standard exposure pixel data " V0 " combine.Then this HDR view data is stored in the frame memory 14 via storage access arbiter 12e.
And, under the situation of desirable transducer (do not have property difference between each pixel, noise is 0), be predicted to be that saturated pixel becomes V1=V when short exposure time T1
Max, V0=V
Max, output " 10 * 10 * (V2-V
Max/ 10)+10 * (V
Max-V
Max/ 10)+V
Max"=" 100 * V2 ".And the V2 that is predicted to be pixel saturated when short exposure time T0 is cancelled output, becomes V1=V
Max, V0=V
Max, output " 10 * (V
Max-V
Max/ 10)+V
Max"=" 10 * V1 ".On the other hand, though common transducer has the property difference and the noise of each pixel, owing to determine final signal level according to 3 pixel signal values, so can obtain noise suppression effect by storage effect.And HDR image of the present invention generates (having kept linear dynamic range enlarges) and does not directly amplify picture element signal, but the difference of amplifying signal is judged the image quality aggravation that causes so can suppress mistake.
Like this, the camera system 3 of present embodiment since can predict corresponding to the pixel of these view data according to ultrashort exposure image data and short exposure image data whether saturated in the time at standard exposure, so can precision well according to the less pixel data of The noise predict (judgement) saturated/unsaturation.And, predict the outcome according to this and can from a certain side standard exposure pixel data, short exposure pixel data and the ultrashort exposure pixel data, select to be used to generate the pixel data of HDR view data, so can correctly be chosen in undersaturated standard exposure pixel data in the standard exposure time, can generate stable (wrong choice is few) HDR view data of image quality thus.
And, camera system 3 is by combining the gimmick of the synthetic and associated form of common linearity, use standard exposure pixel data and short exposure pixel data, can synthesize the short exposure pixel data under the situation saturated when T0, and owing to can use the standard exposure pixel data, short exposure pixel data and ultrashort exposure pixel data be the ultrashort exposure pixel data under the saturated situation when synthesizing T1, so compare contained noise component(s) in synthetic short exposure pixel data (HDR pixel data) that can reduce synthetic usefulness of common linearity and the ultrashort exposure pixel data (HDR pixel data).
In above-mentioned the 2nd execution mode, sensor cell array 56 is corresponding to certain 1 photoelectric conversion part in mode 1,2,12 and 13; The destruction of scanning the electric charge of standard exposure in the time that benchmark timing generator 50, scan line scanner 54, driving pulse generator 52, horizontal transport portion 58 in the corresponding type imaging apparatus 100 carry out from the exposure area of sensor cell array 56 by the non-destruction of shooting treatment system 10 is read processing corresponding to certain 1 the 1st reading unit mode 1,2,3,6 and 12 or the 1st read step of mode 13; Scan benchmark timing generator 50 in the corresponding type imaging apparatus 100, scan line scanner 54, driving pulse generator 52, ultrashort time for exposure that horizontal transport portion 58 carries out from the exposure area of sensor cell array 56 and the nondestructive read of the electric charge in the short exposure time by the non-destruction of shooting treatment system 10 and get processing corresponding to certain 1 the 2nd reading unit mode 1～6 and 12 or the 2nd read step of mode 13.
And in above-mentioned the 2nd execution mode, the saturated/unsaturated prediction processing among non-standard view data generating unit/saturated prediction section 12c is corresponding to certain 1 the saturated predicting unit in mode 1,2,3,4,6 and 12 or the saturated prediction steps of mode 13; HDR image production part 12d generates step corresponding to the HDR view data of certain 1 HDR image data creating unit in mode 2,4,6,9,10 and 12 or mode 13.
And in above-mentioned the 1st execution mode, the example that only obtains the short exposure pixel data has been described as non-standard exposure pixel data, but be not limited thereto, can also replace the short exposure pixel data and obtain ultrashort exposure pixel data, predict saturated/unsaturation according to this ultrashort exposure pixel data (V2).At this moment, the ratio of T0 and T2 is as being 100:1, and the normalisation coefft K2 that T2 uses is 100.Therefore, following formula (2) becomes following formula (3) like this.
Synthetic output=10 * (V2-V
Max/ 100)+V0... (3)
That is,, generate synthetic output (HDR pixel data) according to ultrashort exposure pixel data (V2) and standard exposure pixel data (V0) being predicted as standard exposure in the time under the saturated situation.
And, can also except the short exposure pixel data, obtain ultrashort exposure pixel data, predict saturated/unsaturation according to them.At this moment, whether the irradiation light quantity of predicting the time per unit of ultrashort time for exposure T2 surpassed in the moment of short exposure time T1 becomes V
MaxIrradiation light quantity S1, under situation about surpassing, it is saturated when standard exposure time T 0 to be judged to be this pixel, and uses ultrashort exposure pixel data and standard exposure pixel data to generate synthetic output (HDR pixel data).Thus, even under the constantly saturated situation of T1, also can use the linear synthetic method of having used above-mentioned associated form that unsaturated T2 pixel data is constantly carried out linearity and synthesize, can generate the stable HDR view data of image quality.
And, in the above-described embodiment, constituted camera system 3 by camera head 1 and host computer system 2, but be not limited thereto, also can constitute and make camera head 1 have the function of host computer system 2 fully, can also constitute and make host computer system 2 have the function of the image processing system of camera head 1, can also constitute each inscape of separating camera head 1, for example connect structure between each inscape etc. via network.For example it is contemplated that separating the shooting treatment system is connected via network with image processing system, take the image of distant place and to its structure of handling etc.
Claims (12)
1. imaging apparatus, this imaging apparatus has the electronic shutter function of photo-electric conversion element being arranged to the time for exposure of how rectangular photoelectric conversion part and the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that this imaging apparatus has:
The 1st reading unit, it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes;
The 2nd reading unit, its between the exposure period identical with above-mentioned the 1st reading unit in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time; And
Saturated predicting unit, it is according to the non-standard exposure pixel data that above-mentioned the 2nd electric charge that uses above-mentioned the 2nd reading unit to be read is derived, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated.
2. camera head, this camera head has the electronic shutter function of photo-electric conversion element being arranged to the time for exposure of how rectangular photoelectric conversion part and the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that this camera head has:
The 1st reading unit, it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes;
The 2nd reading unit, its between the exposure period identical with above-mentioned the 1st reading unit in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time;
Saturated predicting unit, it is according to the non-standard exposure pixel data that above-mentioned the 2nd electric charge that uses above-mentioned the 2nd reading unit to be read is derived, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated; And
The high dynamic range images data generating unit, it predicts the outcome according to above-mentioned saturated predicting unit, and synthetic standard exposure pixel data and the above-mentioned non-standard exposure pixel data that uses above-mentioned the 1st electric charge and derive generates the high dynamic range images data.
3. camera head according to claim 2, it is characterized in that, above-mentioned the 2nd reading unit between the exposure period identical with above-mentioned the 1st reading unit in, from each pixel that above-mentioned photo-electric conversion element constitutes, read in the 3rd electric charge that exposure was produced of the multiple short exposure time shorter successively than the above-mentioned standard exposure time in the nondestructive read mode of getting
The non-standard exposure pixel data that above-mentioned saturated predicting unit derives according to the 3rd electric charge that uses above-mentioned the 2nd reading unit to be read, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated.
4. camera head according to claim 2, it is characterized in that, this camera head has operation processing unit, and this operation processing unit is to the 2nd electric charge that uses above-mentioned the 2nd reading unit and read and the calculation process that the non-standard exposure pixel data of deriving is stipulated
The non-standard exposure pixel data of above-mentioned saturated predicting unit after according to above-mentioned calculation process predicts whether the brightness value that above-mentioned standard exposure pixel data is represented is saturation value,
Above-mentioned high dynamic range images data generating unit predicts the outcome according to above-mentioned saturated predicting unit, and non-standard exposure pixel data and above-mentioned standard exposure pixel data after the synthetic above-mentioned calculation process generate above-mentioned high dynamic range images data.
5. camera head according to claim 4 is characterized in that this camera head has the 3rd reading unit, and the 3rd reading unit reads the electric charge after just resetting from each pixel that above-mentioned photo-electric conversion element constitutes,
Above-mentioned operation processing unit is carried out following calculation process: deduct the above-mentioned non-standard exposure pixel data of the deriving from using above-mentioned the 2nd electric charge that above-mentioned the 2nd reading unit reads corresponding with this non-standard exposure pixel data, use the electric charge after just the resetting that above-mentioned the 3rd reading unit read and the benchmark pixel data that derive.
6. camera head according to claim 2, it is characterized in that, above-mentioned high dynamic range images data generating unit is selected the 1st non-standard exposure pixel data corresponding be predicted as saturated pixel by above-mentioned saturated predicting unit from the non-standard exposure pixel data that above-mentioned the 2nd reading unit is read, from the standard exposure pixel data that above-mentioned the 1st reading unit is read, select and be predicted as the 1st corresponding standard exposure pixel data of undersaturated pixel by above-mentioned saturated predicting unit, above-mentioned the 1st standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data are synthesized, generate above-mentioned high dynamic range images data.
7. camera head according to claim 6, it is characterized in that, when the 2nd non-standard exposure pixel data that do not have the time for exposure shorter than the short exposure time corresponding with the above-mentioned the 1st non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit generates and the corresponding high dynamic range pixel data of the above-mentioned the 1st non-standard exposure pixel data according to the 2nd standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data corresponding with the above-mentioned the 1st non-standard exposure pixel data; When having the above-mentioned the 2nd non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit is according to the above-mentioned the 2nd non-standard exposure pixel data and above-mentioned the 2nd standard exposure pixel data and the above-mentioned the 1st non-standard exposure pixel data, generate and the corresponding high dynamic range pixel data of the above-mentioned the 1st non-standard exposure pixel data, and synthesize above-mentioned high dynamic range pixel data and above-mentioned the 1st standard exposure pixel data generates above-mentioned high dynamic range images data.
8. camera head according to claim 6, it is characterized in that, the brightness value of accumulating the quantity of electric charge of each pixel of above-mentioned saturated predicting unit comparison during and the saturated prediction brightness value of setting according to the time for exposure according to the exposing of above-mentioned non-standard exposure pixel data prediction in the standard exposure time, according to this comparative result predict each pixel when exposing in the above-mentioned standard exposure time whether accumulate the quantity of electric charge saturated
When not existing than the time with the 2nd non-standard exposure pixel data of shorter time for exposure of the above-mentioned the 1st non-standard exposure pixel data short exposure time that promptly the 1st non-standard exposure pixel data is corresponding, above-mentioned high dynamic range images data generating unit deducts the above-mentioned saturated prediction of using with the 1st non-standard exposure pixel data from the represented brightness value of the above-mentioned the 1st non-standard exposure pixel data be the result that the 1st saturated prediction obtains divided by the 1st normalisation coefft with brightness value with brightness value, this result who subtracts each other and above-mentioned the 1st normalisation coefft are multiplied each other, make this multiplied result add that the 2nd standard exposure pixel data corresponding with the 1st non-standard exposure pixel data generates above-mentioned high dynamic range pixel data then.
9. camera head according to claim 8, it is characterized in that, when be made as " a " the above-mentioned standard exposure time, when will the short exposure time corresponding being made as " b " with the above-mentioned the 1st non-standard exposure pixel data, will do not reach above-mentioned saturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " a/b " as the above-mentioned the 1st saturated prediction brightness value
Use above-mentioned " a/b " as above-mentioned the 1st normalisation coefft.
10. camera head according to claim 8, it is characterized in that, when having the above-mentioned the 2nd non-standard exposure pixel data, above-mentioned high dynamic range images data generating unit deducts the above-mentioned saturated prediction of using with the 2nd non-standard exposure pixel data from the represented brightness value of the above-mentioned the 2nd non-standard exposure pixel data be the result that the 2nd saturated prediction obtains divided by the 2nd normalisation coefft with brightness value with brightness value, make this subtract each other the result again and above-mentioned the 2nd normalisation coefft multiplies each other, the added luminance that this multiplied result and the above-mentioned the 1st non-standard exposure pixel data are represented, from the result of this addition, deduct the above-mentioned the 1st saturated prediction brightness value, make this result who subtracts each other multiply by above-mentioned the 1st normalisation coefft, make this multiplied result and the represented added luminance of above-mentioned the 2nd standard exposure pixel data generate above-mentioned high dynamic range pixel data.
11. camera head according to claim 10, it is characterized in that, when be made as " a " the above-mentioned standard exposure time, when will the short exposure time corresponding being made as " b " with the above-mentioned the 1st non-standard exposure pixel data, will do not reach above-mentioned saturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " a/b " as the above-mentioned the 1st saturated prediction brightness value; When the above-mentioned standard exposure time be made as " a ", when will the short exposure time corresponding being made as " c " with the above-mentioned the 2nd non-standard exposure pixel data, will do not reach above-mentioned saturated maximum accumulate brightness value that the quantity of electric charge represents divided by the result of " (ab)/(ac) " as the above-mentioned the 1st saturated prediction brightness value
Use above-mentioned " a/b " as above-mentioned the 1st normalisation coefft, use above-mentioned " (ab)/(ac) " as above-mentioned the 2nd normalisation coefft.
12. image capture method, this image capture method is used to have the camera head of photo-electric conversion element being arranged to how rectangular photoelectric conversion part and the electronic shutter function of the time for exposure of the every frame of control, described photo-electric conversion element is converted to electric charge with the light that is exposed and accumulates, it is characterized in that this image capture method has:
The 1st read step, it reads the 1st electric charge that exposure was produced that mode reads in the standard exposure time with destruction from each pixel that the above-mentioned photo-electric conversion element of the exposure area of above-mentioned photoelectric conversion part constitutes;
The 2nd read step, its with above-mentioned the 1st read step in exposure period between between identical exposure period in, reading in the time for exposure shorter than the above-mentioned standard exposure time in the nondestructive read mode of getting from each pixel that above-mentioned photo-electric conversion element constitutes is the 2nd electric charge that exposure was produced of short exposure time;
Saturated prediction steps, it is according to the non-standard exposure pixel data that uses above-mentioned the 2nd electric charge read in above-mentioned the 2nd read step to derive, predict when the exposure of above-mentioned standard exposure time above-mentioned each pixel whether accumulate the quantity of electric charge saturated; And
The high dynamic range images data generate step, it predicts the outcome according to above-mentioned saturated prediction steps, the standard exposure pixel data and the above-mentioned non-standard exposure pixel data that synthesize the electric charge when using exposing in the above-mentioned standard exposure time of reading in above-mentioned the 1st read step and derive generate the high dynamic range images data.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2006132106 | 2006-05-11 | ||
| JP2006132106 | 2006-05-11 | ||
| JP2007098182 | 2007-04-04 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101072303A CN101072303A (en) | 2007-11-14 |
| CN100539648C true CN100539648C (en) | 2009-09-09 |
Family
ID=38899271
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2007101074229A Expired - Fee Related CN100539648C (en) | 2006-05-11 | 2007-05-11 | Imaging apparatus and camera head and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100539648C (en) |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9633426B2 (en) * | 2014-05-30 | 2017-04-25 | General Electric Company | Remote visual inspection image capture system and method |
| JP5123137B2 (en) * | 2008-01-22 | 2013-01-16 | 株式会社リコー | Imaging apparatus and imaging method |
| US20100157079A1 (en) | 2008-12-19 | 2010-06-24 | Qualcomm Incorporated | System and method to selectively combine images |
| US8339475B2 (en) * | 2008-12-19 | 2012-12-25 | Qualcomm Incorporated | High dynamic range image combining |
| US8928789B2 (en) | 2010-09-30 | 2015-01-06 | Canon Kabushiki Kaisha | Solid-state imaging apparatus |
| JP6082274B2 (en) * | 2012-06-08 | 2017-02-15 | キヤノン株式会社 | Imaging apparatus and control method thereof |
| JP2014050042A (en) * | 2012-09-03 | 2014-03-17 | Toshiba Corp | Image processor and solid-state imaging device |
| CN105791717A (en) * | 2014-12-18 | 2016-07-20 | 天津炜辐医疗科技有限公司 | Method for real-time high-dynamic-range imaging digital camera device |
| KR102277178B1 (en) * | 2015-03-09 | 2021-07-14 | 삼성전자 주식회사 | Electronic Device Including The Camera Module And Method For Processing Image Of the Same |
| JP6525747B2 (en) * | 2015-06-05 | 2019-06-05 | キヤノン株式会社 | Imaging device, imaging system |
| CN108293090B (en) * | 2015-12-15 | 2020-10-20 | 株式会社理光 | Image processing apparatus and image processing method |
| JP2017112457A (en) * | 2015-12-15 | 2017-06-22 | オリンパス株式会社 | Imaging device, imaging program, imaging method |
| US9838628B2 (en) * | 2016-03-16 | 2017-12-05 | Sony Corporation | Detecting quantities beyond sensor saturation |
| CN106303311B (en) * | 2016-08-22 | 2020-08-28 | 格科微电子（上海）有限公司 | Output method of pixel photosensitive value |
| CN109997351B (en) * | 2016-12-22 | 2021-02-23 | 华为技术有限公司 | Method and apparatus for generating high dynamic range images |
| CN110036627B (en) * | 2016-12-27 | 2021-03-05 | 松下知识产权经营株式会社 | Imaging device, camera, and imaging method |
| FR3062009B1 (en) | 2017-01-17 | 2019-08-16 | Centre National De La Recherche Scientifique | ADAPTIVE GENERATION OF A DYNAMICALLY ENHANCED SCENE IMAGE OF A SCENE FROM A PLURALITY OF IMAGES OBTAINED BY NON-DESTRUCTIVE READING OF AN IMAGE SENSOR |
| US10334141B2 (en) * | 2017-05-25 | 2019-06-25 | Denso International America, Inc. | Vehicle camera system |
| EP3454547A1 (en) * | 2017-09-11 | 2019-03-13 | Canon Kabushiki Kaisha | Imaging apparatus, image processing apparatus, imaging method, image processing method, and storage medium |
| CN112019773B (en) * | 2019-05-13 | 2023-06-16 | 上海图漾信息科技有限公司 | Depth data measuring head, measuring device and method |
| WO2020186825A1 (en) | 2019-03-15 | 2020-09-24 | 上海图漾信息科技有限公司 | Depth data measuring head, measurement device and measurement method |
| CN112135063B (en) * | 2019-06-24 | 2021-11-30 | 聚晶半导体股份有限公司 | Image output method and electronic device |
| TWI723435B (en) | 2019-06-24 | 2021-04-01 | 聚晶半導體股份有限公司 | Image outputting method and electronic device |
| CN114513609A (en) * | 2020-11-17 | 2022-05-17 | 浙江大华技术股份有限公司 | Camera exposure time adjusting method, image stroboscopic detection method and device |
-
2007
- 2007-05-11 CN CNB2007101074229A patent/CN100539648C/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN101072303A (en) | 2007-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100539648C (en) | Imaging apparatus and camera head and method | |
| JP4998056B2 (en) | Imaging apparatus, imaging system, and imaging method | |
| KR100281789B1 (en) | The solid- | |
| JP4888081B2 (en) | Imaging apparatus, imaging method, imaging system, and image processing apparatus | |
| EP2161919B1 (en) | Read out method for a CMOS imager with reduced dark current | |
| US7692698B2 (en) | Signal processing apparatus and signal processing method for solid-state image pickup element and image pickup apparatus | |
| CN100394778C (en) | Signal processing apparatus and signal processing method for solid-state image pickup element and image pickup apparatus | |
| US5812703A (en) | Imaging apparatus | |
| US20080055440A1 (en) | Wide Dynamic Range Image Capturing System Method and Apparatus | |
| US20120086837A1 (en) | Image pickup apparatus | |
| WO2008027838A2 (en) | Wide dynamic range image capturing system method and apparatus | |
| CN102065219B (en) | Image sensing apparatus and control method therefor | |
| JP2006005520A (en) | Imaging apparatus | |
| JP2007194687A (en) | Imaging apparatus | |
| US7777793B2 (en) | Signal processor having A/D conversion unit and operation part | |
| JP5222068B2 (en) | Imaging device | |
| JP2007174266A (en) | Imaging apparatus | |
| US20040189836A1 (en) | System and method for compensating for noise in image information | |
| JP6160139B2 (en) | Imaging apparatus and method | |
| US7418153B2 (en) | Electronic camera | |
| US20080266423A1 (en) | Image sensing apparatus and control method therefor | |
| JP3853891B2 (en) | Electronic still camera | |
| JP4725052B2 (en) | Imaging device | |
| JP2018170703A (en) | Imaging device, imaging apparatus, and method for controlling imaging device | |
| JP7020463B2 (en) | Imaging device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090909 Termination date: 20180511 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |