JPH057336A - Method and device for image pickup - Google Patents

Abstract

PURPOSE:To prevent picture quivering, and to take a picture whose practical dynamic range is wide by providing a motion information detecting means and a picture moving means. CONSTITUTION:In a camera part, the odd frame ODD and the even frame EVEN of one frame are distinguished by a field index FI. The H period and the L period of a vertical blanking signal VBLK are an effective picture and a blanking period respectively. The charge accumulation time of an image pickup element is controlled by T pulse, and the accumulation signal of either 1/10<3> seconds or 1/60 seconds is selected by an iris gate signal. The signals of the quantity of light of 1/10<3> seconds and 1/60 seconds are outputted alternately at every field. In a camera using A CCD image pickup element, the picture is improved by utilizing positively white void and back paint-out in the field of one side. Namely, a part where the white void or the black paint-out occurs is replaced with the corresponding part of another field and the signals of both the fields are synthesized into a final video signal. If a through picture and a memory picture are combined properly in this way, a good picture free from the black paint-out and the white void can be obtained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup method and an image pickup apparatus for apparently enlarging a dynamic range of an image signal.

[0002]

2. Description of the Related Art An image pickup device is widely used as an image pickup section for a VTR with a built-in camera, a still video camera and the like.
The image pickup device using the image pickup tube and the solid-state image pickup element used in these video cameras has a narrow dynamic range as compared with the conventional silver halide photographic system. "Collapse" (common name for the part where the brightness level is extremely high or low) occurs. In such a case, the main subject is usually "black out", so
In a conventional video camera or the like, in order to adjust the exposure to this main subject, the aperture is manually opened or the backlight compensation button is operated to open the aperture by about 2 apertures,
It was adjusted so as to increase the exposure amount of the image pickup device and the like.

However, even when such backlight correction is properly performed, "whiteout" occurs in the background even if the main subject has an appropriate exposure amount, resulting in a screen with only a white background. I will end up. In other words, the narrow dynamic range of the image pickup device cannot be solved only by adjusting the light amount so that the exposure amount of the main subject becomes appropriate as in the conventional image pickup device. Therefore, for example, in an image pickup apparatus that converts a still image into an electric signal by using a line scanner or the like, conventionally, one screen is composed from a plurality of screens having different exposure amounts obtained from the same subject. .

[0004]

In the image pickup apparatus adopting the conventional structure as described above, since the positional relationship between the image pickup apparatus and the subject is always fixed, the image pickup of each screen is performed. Even if the timing is shifted over time,
No image shift occurred and no problem occurred. However, when the above method is applied to a video camera, a still video camera, or the like, the positional relationship between the video camera and the subject changes minutely due to camera shake or the like. When the images of the screens are to be imaged, the positions of the respective screens are slightly shifted, and when these screens are combined, there is a problem in that the subject is shifted in double or triple. This becomes a more serious problem in view of the tendency of camera shake during handheld shooting, especially in video cameras that have recently been made smaller and lighter.

Therefore, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is not to shift an image even in a video camera or the like, which is likely to cause a screen blur, and to have a substantial dynamics. An object of the present invention is to provide an imaging method and an imaging device capable of capturing an image with a wide range.

[0006]

In order to solve the above-mentioned problems and to achieve the object, an image pickup method of the present invention is a video signal by synthesizing a plurality of screens sequentially picked up at different exposure times. In the imaging method of apparently enlarging the dynamic range, the coordinate conversion of each of the plurality of screens is performed in response to the positional shift with time between the plurality of screens during imaging, and the plurality of screens are combined. It is characterized by performing.

The image pickup apparatus of the present invention is an image pickup apparatus having a function of apparently enlarging a dynamic range of a video signal by synthesizing a plurality of screens sequentially picked up at different exposure times. It is characterized by comprising a detection means for detecting movement information between the screens, and an image moving means for position-shifting each of the plurality of screens in plane coordinates based on the detection information of the detection means.

[0008]

As described above, since the image pickup method and the image pickup apparatus according to the present invention are configured, the image pickup apparatus according to the present invention can be practically used with no image shift even with respect to a video signal having a screen blur caused by camera shake or the like. Dynamic range expansion processing can be performed.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. FIG. 1 is a block diagram of the entire configuration when one embodiment is applied to a camera-integrated VTR.

In FIG. 1, 100 is a camera unit, and 200 is a camera unit.
Is a processing unit, and 300 is a recording unit. In the camera unit 100, the amount of light rays incident from the optical system 101 is limited by the diaphragm 102 and forms an image on the image sensor 103. The image sensor 103 is a semiconductor image sensor such as a MOS or CCD. The focus drive circuit 107, the diaphragm drive circuit 106, and the image sensor drive circuit 105 that controls the light accumulation time are
Under the control of the camera control circuit 108, the optical system 1
01, the diaphragm 102, and the image sensor 103 are driven. The camera signal processing circuit 104 is a digital circuit that performs known processing such as γ correction similar to the signal processing circuit of a normal video camera.

A video signal output from the camera section 100 is converted into a digital signal by an A / D converter 201 of the processing section 200, and pixel data, which will be described later, is converted by an arithmetic circuit 202, and a D / A converter. At 203, it is converted back to an analog signal and provided to the recording unit 300.

Reference numeral 204 is an image memory for calculation in the arithmetic circuit 202 and the screen shake detection circuit 206, and 205 is a memory control circuit for controlling the image memory. The memory control circuit 205 outputs a write / read address signal of the image memory 204 according to a timing signal from the control circuit 108 of the camera unit 100.

Further, when the screens are combined, an address signal for position adjustment is generated for each screen so as to correct the screen shake according to the output of the screen shake detection circuit. In order to combine the read common signals and match the input signals with the same form, the arithmetic circuit 202 performs enlargement interpolation processing. In this way, the gradation-combined video signal so that the screen shift does not occur is output from the arithmetic circuit 202, and the D / A converter 20
At 3, the signal is converted into an analog signal and recorded in a known VTR (video signal recording device) 301.

If this VTR is of the digital recording type, the D / A converter 203 is of course unnecessary. Next, the operation of the image sensor 103 will be described. FIG. 2 shows the camera unit 100.
FIG. 3 is a more detailed block diagram of the configuration of FIG. 3, and FIG. 3 is a timing chart of the camera unit 100 when an NTSC signal is taken as an example.
Indicates a chart. The field index (FI) signal is a signal for distinguishing between an odd (ODD) field and an even (EVEN) field which form one frame. The V _BLK signal is a vertical blanking signal, and H
The (high) period corresponds to the effective screen, and the L (low) portion corresponds to the vertical blanking period. T _{PU LSA} is a signal for controlling the charge storage time of the image sensor 103, and is a pulse for reading the pixel output to the vertical transfer CCD in the case of a CCD image sensor, for example. The iris gate signal is 1/1 as a reference video signal for automatic exposure described later.
This is a signal that specifies whether to use the accumulated signal for 000 seconds or the accumulated signal for 1/60 seconds.

In the illustrated example, 1/1000 second is accumulated during the vertical blanking period, and 1/1000 second is accumulated during the next effective period.
Output the accumulation signal for 1000 seconds. And 1/1000
Immediately after the second accumulation period, charge is accumulated for substantially 1/60 second,
The 1/60 second accumulation signal is output during the effective screen period of the next field. In this way, two types of light amount signals (1/1000 seconds and 1/60 seconds) are alternately output for each field.

In FIG. 2, a known AE control 20 receives a signal (for example, a video signal) from the camera signal processing circuit 104 via an A / D converter 201 and calculates a control signal for exposure control. A circuit, 22 is a known AF control circuit that outputs a control signal for focus control, and 24 is a 1/2 frequency divider circuit that _{divides the} vertical blanking signal V _BLK by 2. Reference numerals 26 and 27 are sample and hold circuits, 28 is an inverter, and 29 and 30 are switches for selecting whether to determine the sampling timing by the output of the 1/2 frequency dividing circuit 24 or its inverted signal by the inverter 28. . The outputs of the sample and hold circuits 26 and 27 are applied to the diaphragm drive circuit 106 and the focus drive circuit 107, respectively, and automatic exposure control and automatic focus adjustment are executed.

In the above embodiment, 1/1000 seconds and 1/6
Since it is a combination of 0 seconds and the light amount changes by about 4 steps (2 ⁴ times), for example, in the case of a camera using a CCD image sensor, the main subject is exposed based on the accumulation time of 1/60 seconds at the EVEN field. In contrast, in the EVEN field, “whiteout” is likely to occur in the background, whereas in the ODD field with a reduced amount of four-stage light, “blackout” often occurs in the main subject. Note that this example assumes the case where the exposure is adjusted to the background side at the time of backlight compensation, and of course, it may be set to a value other than 1/1000 seconds depending on the situation in that case.

In addition to the above examples, a CCD having a VOD type high speed shutter function has been put into practical use in recent years. This discharges unnecessary charges in the vertical direction of the substrate of the CCD, enables very fine shutter speed setting, and controls circuit 1 according to the brightness difference between the main subject and the background subject.
The optimum shutter speed can be set by the drive circuit 105 by the judgment of the AE control circuit 20 in 08.

In one embodiment, the screen is improved by positively utilizing "whiteout" and "blackout" in one field. In other words, the part where "whiteout" or "blackout" occurs is replaced by the corresponding part of another field ("blackout" or "whiteout" does not occur because the exposure is different), and the signal of both fields is replaced. To be the final video signal. Figure 4 shows the basic idea.
Will be described with reference to. In FIG. 4, the main subject is schematically shown by a vertically long rectangle. In FIG. 4, the through (T) image refers to the direct output of the image sensor 103, and the memory (M) image or the memory output refers to the signal of the immediately preceding field once stored in the image memory 204. In the through image, the main subject at the time of backlighting becomes "black out" in each ODD field, and EVEN
The background is "whiteout" for each field. Also,
Since the memory image is composed of signals delayed by one field period, "whiteout" and "blackout" occur at fields different from those of the through image.

Therefore, if the through image and the memory image are properly combined, a good image without "whiteout" and "blackout" can be obtained. In other words, the through image and memory image signals are compared with a predetermined threshold for each field,
If it is larger than the threshold, it is set to 1, and if it is smaller, it is set to 0, and "whiteout" or "blackout" is determined for each pixel.

FIG. 6 shows the relationship between the threshold value, the brightness value of the pixel, and the field. The horizontal axis of FIG. 6A is the brightness level,
The vertical axis represents the appearance frequency of each brightness level in one screen. Figure 6
As shown in (a), the threshold TH1 is set so that "blackout" can be determined, and the threshold TH2 is set so that "whiteout" can be determined. That is, TH1 or less is determined as "blackout", and threshold TH2 or more is determined as "whiteout". FIG. 6B shows the relationship between each field and the threshold value.
As described above, in the ODD field and the EVEN field, "whiteout" and "blackout" alternate, so the threshold value for the determination is also changed for each field.

In this way, it is possible to judge which pixel portion of which field is "blackout" or "overexposure". Therefore, by using the judgment result, a pixel having an appropriate exposure amount is used for the through image and the memory image. You can select the signal. For example, judgment A
And the determination B are ANDed, and in the ODD field, a through image signal is selected for a pixel whose AND is 1.
A selection flag as shown in FIG. 4 is obtained by selecting the signal of the memory image for the pixel whose logical product is 0 and setting the opposite relationship in the EVEN field. The picture at the bottom of FIG. 4 shows a composite image according to the selection flag.
In this figure, it is assumed that the main subject is moving at a constant velocity, and the influence of the time axis shift on the image is confirmed, but it can be seen that the moving image can be practically sufficient.

FIG. 5 is a block diagram showing the detailed structure of the circuit portion of the arithmetic circuit 202 of the processing section 200 which forms a comparison flag and a comparison with the threshold values TH1 and TH2. T
The H switching control signal is a signal in which “H” and “L” are inverted for each field like the FI signal, and is applied to the second threshold generation circuit 52 via the threshold generation circuit 53 and the inverter 51. It Depending on the switching signal, the threshold value generating circuits 52 and 53 have the threshold value TH1 or the threshold value TH1 shown in FIG.
Generates H2. The comparison circuits 54 and 55 compare the memory image and the through image with the threshold values from the threshold value generation circuits 52 and 53, respectively, and output the A signal and the B signal as the determination result.
The AND gate 56 calculates the logical product of the A signal and the B signal and outputs a selection flag signal. The switch 57 switches according to the selection flag signal and selects the signal of the memory image or the through image.

FIG. 7 shows a gradation characteristic diagram. The solid line in (a) is the characteristic diagram of a normal video camera, and the input / output is linear up to 100%, and the input (10) is higher than that.
(0 to 400%), there is a loose relationship called the KNEE characteristic. If this change point is P1,
At the time of high-speed shutter, this change point moves to the position P2. However, it is assumed that P1 is 1/60 second and P2 is 1/250 second of the change in the exposure amount of two steps. As described above, in the case of the difference of 4 steps, the relationship of (1) and (5) of FIG. Incidentally, in FIG. 7D, (1) is 1/60 seconds, (2) is 1/125 seconds, (3) is 1/250 seconds, and (4) is 1/5 seconds.
00 seconds and (5) are characteristic diagrams in the case of 1/1000 seconds. A characteristic having a desired curve is synthesized from two characteristics having different inclinations. FIG. 7B shows an example of the combined characteristic in the case where the threshold values for the “whiteout” and “blackout” judgments are different, and FIG. 7C shows the output obtained by averaging the signals of the corresponding two pixels. The respective characteristics are shown in the case of performing (the same (1)), the case of selecting one (the same (2)), and the case of performing the averaging under an appropriate count (the same (3)).

In the above example, the time resolution is substantially 30 per second, which is almost the same as that of the frame accumulation CCD image pickup device. Therefore, an example will be described in which two screens are captured in one field in order to achieve the same time resolution as that of the field accumulation CCD image pickup device. FIG. 9 shows a modified portion of the configuration example, and FIG. 8 shows a timing chart. In principle, the image sensor 1 is faster than the normal video rate.
The signal No. 03 is read out, converted on the time axis, and returned to the normal disk. The field memories 90 and 91 each have a storage capacity corresponding to one field of image information,
In memory 90, it is set to 1 in order to synchronize with 1/120 second reading.
/ 1000 seconds The accumulated signal is delayed, and in the memory 91,
Video signal of 1/120 second unit is NTS of 1/60 second unit
Double time extension processing for changing to C signal is performed. (A) to (d) in FIG. 9 correspond to the signals (a) to (d) in FIG. An XY address type MOS solid-state image pickup device can be considered as the image pickup device 103 that tests such an action.

Next, another detailed example of the control circuit 108 is shown in FIG.
It shows in 0. The master clock generator 40 generates a master clock for the control circuit 108 according to an external reference signal. The 1/1000 shutter clock generator 41 generates a high speed clock in accordance with its master clock, and the 1/60 shutter clock generator 42 generates a low speed clock in accordance with its master clock. The switch 45 switches for each field and alternately applies the outputs of the clock generators 41 and 42 to the drive circuit 105. The AE control signal generator 43 generates an AE control signal for aperture control together with the video signal from the camera signal processing circuit 104. The control signal holding circuit 44 holds the control signal for one field. The switch 46 switches for each field, and alternately applies the output of the AE control signal generator 43 and the holding signal from the control signal holding circuit 44 to the aperture control circuit 106. The switching signal generator 47 controls switching of the switches 45 and 46. The switches 45 and 46 are switched in synchronization.

In the above example, a clock generator is provided for each of the low speed and the high speed, and the clock is switched by the output signal of the switching signal generator which generates a signal for each field. It has the effect of simplifying, and is especially suitable for movies.

In the above description, screens having different exposure amounts are generated by changing the shutter speed. However, if a high-speed diaphragm device can be prepared, the diaphragm may be changed at high speed. For example, PLZT or the like may be realized by a method of electrically controlling the dimming filter.

As described above, various arithmetic processes are conceivable in one embodiment, but the following description will be given by taking an example of screen combination by simple addition which is highly effective with simple processes.

FIG. 11 is a conceptual diagram of screen shake, which shows two screens (O
The displacement of the screen (image sensor 103) with respect to the subject in the DD field screen and the EVEN field screen is shown (FIG. 11-a).

Assuming that the upper left portion of the common portion (common area) of the ODD screen and the EVEN screen is the memory read start point P, for the ODD screen, P _ODD shown in FIG. 11-b.
For point and EVEN screens, P _EVEN shown in Figure 11-c
The point becomes the reading start position from the image memory 204a, and it is possible to generate a video signal without a shift in the common area of ODD and EVEN.

A configuration diagram shown in FIG. 12 is an embodiment of the configuration diagram of FIG. 1 according to one embodiment. The input video signal is converted into a digital signal by the A / D converter 201,
It is supplied to the field memory 204a and the motion detection circuit 206. The output of the field memory 204a is supplied to the other input terminal of the motion detection circuit 206 and detects the motion information of the image from the video signal of one field time difference and generates the screen shake correction signal. This correction signal is supplied to the memory control circuit 205, and the memory control circuit 205 controls the field memory 204a based on the surface shake correction signal.

The common area portion shown in FIG. 11-a is output from the field memory 204a, and this signal is returned to the same signal form (for example, NTSC signal) as the input signal in the expansion interpolation circuit 202a at the next stage.

The smaller the common area is, the higher the enlargement ratio in this process is. Regardless of the enlargement ratio, the image signal is restored to the original form and then stored in the field memory 204b via the adder 202b. Assuming that the stored image signal is an ODD screen signal, the EVEN screen signal is output from the enlargement interpolation circuit 202a in the next field period, and the corresponding pixel data is added by the adder 202b. The added signal is supplied to the D / A converter 203 via the EVEN side terminal of the selector 207. On the other hand, this addition signal is also stored in the field memory 204b, and in the ODD period of the next field, READ-MOD.
While supplying the same screen to the D / A converter 203 via the ODD terminal of the selector 207 in the IFY-WRITE operation,
The new ODD screen information is stored in the field memory 204b. At this time, the switch 208 opens in ODD, E
When the new ODD screen is written in the memory, the adder 202b is switched to be through.

The signal thus generated is output from the D / A converter 203 as an analog signal similar to the input signal. The time chart of the above processing is shown in the timing chart of FIG. In the timing chart of FIG. 13, A is an ODD / EVEN field discrimination signal, which is the output timing of the enlarged interpolation circuit 202a. B
Indicates the opening / closing timing of the switch 208. The function of the adder 202b is changed by opening / closing the switch 208. When the switch 208 is open, the signal is passed through, and when the switch 208 is closed, the addition is performed. Work as a container. C shows the write timing to the memory 204b, the ODD screen is written as it is during the ODD period in A, and the READ- during the EVEN period in A.
By the MODIFY-WRITE operation, the EVEN screen is added to the current ODD screen, and the writing process is performed again at the same address. D indicates the timing of switching the terminals of the selector 207. The ODD screen information in the t1 period is added to the EVEN screen information in the same t2 and output in the t2 period in D, and the same information in the t3 period. Is read out again and output.

As described above, the processing is completed with the two-field period as a unit time. By the way, it is also possible to switch the line interpolation processing between ODD and EVEN to make it compatible with interlacing and reduce so-called screen interference. In order to reduce this interference, it is advisable to insert a line interpolation circuit at the point Q between the field memory 204b and the selector 207. The present invention can be applied to modifications and variations of the above embodiments without departing from the spirit of the present invention.

[0037]

As described above, according to the present invention, the dynamic range can be substantially widened. For example, even in the case of backlight, not only the main subject but also the background is appropriate. There is an effect that an image of the exposure amount can be obtained.
Further, the present invention has an effect that a practical dynamic range expanding process without image shift can be applied to a video signal having a screen blur caused by camera shake or the like. This is a great advantage for camcorders that are becoming smaller and lighter.

[Brief description of drawings]

FIG. 1 is a camera-integrated VR to which an embodiment of the present invention is applied.
It is a block diagram of the configuration of T.

FIG. 2 is a block diagram of a specific configuration of a control circuit of the camera unit shown in FIG.

FIG. 3 is an operation timing chart of the image sensor.

FIG. 4 is a conceptual diagram of image processing according to an embodiment.

5 is a block diagram of a specific configuration of the arithmetic circuit of FIG.

FIG. 6 is a diagram illustrating a method of determining a threshold value for “whiteout” and “blackout” determination.

FIG. 7 is a gradation characteristic diagram.

FIG. 8 is an operation timing chart of the image pickup device.

FIG. 9 is a block diagram of a part of the camera unit.

10 is an example of a control circuit in FIG.

FIG. 11 is a conceptual diagram of screen shake.

FIG. 12 is a diagram in which the configuration diagram of FIG. 1 is embodied.

FIG. 13 is a timing chart showing the operation of the embodiment.

[Explanation of symbols]

100 camera section 101 Optical system 102 aperture 200 Processing unit 300 recording section

Claims (2)

[Claims] 1. An image pickup method for apparently enlarging a dynamic range of a video signal by synthesizing a plurality of screens sequentially picked up at different exposure times. An image pickup method comprising: performing coordinate conversion of each of the plurality of screens in response to a position shift and then synthesizing the plurality of screens. 2. An image pickup apparatus having a function of apparently enlarging a dynamic range of a video signal by synthesizing a plurality of screens sequentially picked up at different exposure times, and detecting motion information between the plurality of screens. And an image moving unit that performs position conversion of each of the plurality of screens in plane coordinates based on the detection information of the detecting unit.