JPH05276452A - Image pickup device - Google Patents

Abstract

PURPOSE:To obtain highly precise moving images at low cost by using an imaging device having a small number of picture elements. CONSTITUTION:The image fetched through a lens 5 is halved by a half mirror 2 and formed on CCD 11 and 12. The CCD 11 is provided for a still region, a transparent plate 3 is placed over the entire face thereof, and is vertically and horizontally rotated and vibrated by a piezoelectric element 4. For example, the image for four frames passes through an A/D converter 61, afterwards, the frames are synthesized by a frame synthesizing part 71, and the image is turned to the still picture of one frame. On the other hand, the CCD 12 detects the moving image of four frames by a moving area detection part 72 during this operation, and the moving image and still picture are synthesized and outputted by an image synthesizing part 8.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image input technique, and more particularly to an image pickup apparatus for obtaining a high-definition moving image by synthesizing images from a plurality of image pickup elements.

[0002]

2. Description of the Related Art Since solid-state image pickup devices have many features such as small size, low power consumption, and no afterimage, recently, solid-state image pickup devices such as home video cameras and handy type broadcast cameras have been used. The number of imaging devices used is increasing. However, the resolution is still lower than that of an image pickup apparatus using an image pickup tube, and there are expectations for higher resolution in the future.

Since the resolution of a solid-state image pickup device is determined by the number of pixels, a conventional image pickup device has been made to have a high resolution by increasing the number of pixels in the image pickup device. As a method of increasing the number of pixels in the image pickup device, a method of making full use of a microfabrication technique and increasing the pixel density is the mainstream, but there is also a method of increasing the size of the image pickup device to increase the number of pixels.

[0004]

Recently, a 2 million-pixel solid-state image pickup device for HDTV has been developed by a method of increasing the pixel density. However, the method of increasing the pixel density makes the process difficult, and the pixel area is large. There is also a problem of reduced sensitivity due to reduction. In particular, when the decrease in sensitivity is taken into consideration, the number of pixels of 2 million is already close to the limit, and it is currently difficult to further increase the number of pixels and improve the resolution.

On the other hand, the method of increasing the size of the image pickup device and increasing the number of pixels without changing the pixel density leads to a reduction in manufacturing yield and an increase in cost. For this reason, the applications are limited to scientific purposes such as astronomy and military purposes. In addition to the above problems, in the conventional method for increasing the resolution by increasing the number of pixels in one image sensor, the driving speed of the image sensor and the A / D conversion speed at the time of digitization are increased as the number of pixels increases. The problem of speeding up is also becoming more serious.

The present invention has been made in order to solve the above problems, and provides an image pickup apparatus using an image pickup device having a small number of pixels to obtain a high-definition moving image without increasing the cost. With the goal.

[0007]

An image pickup apparatus according to the present invention comprises an image pickup device for a still region and an image pickup device for a moving region, and a still region is obtained by synthesizing a plurality of continuous frame images picked up by the image pickup device for a still region. And a moving image detection unit that extracts a moving region from an image captured by an image sensor for a moving region, and an image combining unit that combines the extracted moving region with a still region. It is a thing.

[0008]

The moving image pickup device generally picks up a moving image, but normally, even if it is a moving image, most of the screen is a still image and there are few moving parts. For example,
When the announcer is reading a manuscript in the news, most of the announcer's mouth and eyes are still images. In the conventional image pickup apparatus, since the moving area and the still area are captured by the same image pickup element, the moving area and the still area are captured at the same speed without distinction. This speed is set to a very high speed, for example, 30 frames / second in NTSC, in order to make the motion in the moving region appear continuous. However, such a high-speed image pickup is essentially unnecessary in a still area that occupies most of the screen. That is, the required imaging speed differs between the moving area and the stationary area. Also, in terms of resolution, the required values differ between the two. That is,
It is known that the visual characteristics of the human eye are inferior to the stationary area in the visual characteristics of the moving area, and the resolution required for the moving area is lower than that of the stationary area. In general, the definition of a moving image perceived by human eyes mainly exists in the resolution of a still region.

The present invention focuses on the point that the required characteristics differ between the moving area and the stationary area as described above, and uses two image pickup devices for the stationary area and for the moving area, respectively, as described below. By capturing an image with a characteristic suitable for a region, a high-definition moving image can be captured even with an image sensor having a small number of pixels.

First, in the image pickup device for the still region, the image pickup time for one frame is set to be n times as long as that of the moving image, and the resolution is increased accordingly. That is, the frame period of the image sensor is the same as in the case of a moving image, and n frame images are combined to obtain a high-definition image for one frame. As this method, for example, the following method can be applied. That is, the relative positional relationship between the subject image and the pixels is temporally changed by ½ of the pixel pitch for each frame (or field) to input n images (hereinafter, this method will be referred to as an image shift method). Called). The principle that the resolution can be increased by the image shift method will be described below.

The image pickup section of the solid-state image pickup device is composed of pixels arranged two-dimensionally, and each pixel is composed of a photosensitive section for reading the light intensity and a non-photosensitive section such as a signal transfer path. That is, as shown in FIG. 8, the photosensitive portions 101 that directly contribute to image capturing are arranged in a discrete manner, and non-photosensitive portions that do not directly contribute to image capturing are provided between the photosensitive portions 101. There is a section 102. Utilizing the presence of the non-photosensitive portion 102, the sampling point of the image is amplified to increase the definition of the input image. That is, an image input in one frame (FIG. 8A) and an image input in the next frame with the relative positional relationship between the subject image and the pixels of the image sensor shifted by ½ of the pixel pitch (FIG. 8 ( When b)) is combined, the light intensity information of the area corresponding to the non-photosensitive portion 102 of the image sensor in each frame image is sampled in the other image. As shown in (), this is equivalent to effectively reading an image with a double pixel density, and high definition can be achieved. In FIG. 8, the method of interpolating two images in the horizontal direction has been described for the sake of simplicity. However, high definition can be achieved by interpolation in both the horizontal and vertical directions, and the pixel pitch can be shifted by 1/3 each. It is possible to increase the density three times in one direction.

Next, the moving region image pickup device picks up an image at the same speed as a normal moving image pickup. In this case, the resolution is 1 / n as compared with the still area, but since the resolution of human eyes for the moving area is lower than that of the still area as described above, deterioration of the resolution is not recognized.

As described above, the present invention uses two image pickup devices for a still region and a moving region to capture an image with characteristics suitable for each region. Since the same area is imaged, the moving area imaging element also images the moving area, and conversely, the moving area imaging element also images the still area. Therefore, a method is used in which a moving area is extracted from the image captured by the moving area image sensor and is combined with the image captured by the stationary area image sensor.

The fineness of the moving image visually mainly depends on the resolution of the still region, and therefore the fineness of the input image can be visually increased n times by the above method. From this, according to the present invention, the definition of the input image can be increased without increasing the number of pixels of the image sensor.

[0015]

FIG. 1 shows a schematic configuration of the first embodiment of the present invention. In FIG. 1, 11 and 12 are CCD area sensors (hereinafter, simply referred to as CCD), 2 are half mirrors, 3 are transparent plates, 4 are piezoelectric elements, 5 are lenses, and 61 and 62 are A /
A D converter, 71 is a frame composition section, 72 is a moving area detection section, and 8 is an image composition section. In this embodiment, two CCDs 11 and 12 are used as image pickup devices, and the same subject image is formed on the image pickup surfaces of both CCDs 11 and 12 by the half mirror 2 arranged behind the lens 5. Here, CCD11
The main purpose is to image the stationary area and the CCD 12 to image the moving area. Further, the two CCDs 11 and 12 are assumed to be the same, and here, an NTSC CCD is used.

The CCD 11 inputs a still region with high precision using the image shift method. As an image shift method,
In this embodiment, a method of vibrating the transparent plate 3 in front of the CCD 11 is used. This method is described in the literature (SID '83 Technical
As described in Digest (1983)), tilting the transparent plate 3 refracts light and shifts the subject image on the imaging surface. In this embodiment, as shown in FIG. 1, piezoelectric elements 4 are attached to the upper, lower, left and right sides of the transparent plate 3, and voltage is applied to the piezoelectric elements 4 to rotate and vibrate the transparent plate 3 in the vertical and horizontal directions in synchronization with the frame period. As a result, the subject image shifts vertically and horizontally for each frame on the image pickup surface of the CCD 11. If the rotation angle, that is, the inclination of the transparent plate 3 is set so that the shift distance becomes ½ of the pixel pitch, the effective pixel density of the combined image is obtained by combining the four images captured in four frames. Is 4
Double. As shown in FIG. 1, the synthesizing process is performed by A / D
After digitization by the converter 61, frame synthesis is performed by the frame synthesis unit 71. The image combined by the frame combining unit 71 is transferred to the image combining unit 8. Thus, C
The pixels transferred from the CD 11 to the image synthesizing unit 8 are images captured over a 4-frame cycle, and the image capturing speed for one frame is 1/4 slower than that for normal moving image capturing. The density is four times higher, resulting in a high-definition image.

On the other hand, since the CCD 12 is intended to image a moving area, it takes in one frame of image at a high speed in one frame period by the normal NTSC system. The image signal from the CCD 12 is digitized by the A / D converter 62, and then only the moving area is extracted by the moving area detecting unit 72.

A method of detecting a moving area is shown in FIG. Figure 2
As shown in (a), if an object is moving from left S ₁ to right S ₂ of the screen during one frame, the signal level of the pixel in the shaded area in the figure changes. Therefore, an area in which the image signal level has changed compared to the previous frame is extracted as a moving area. In the image transferred from the CCD 11 to the image synthesizing unit 8 as a still region (FIG. 2B), the signal value of the pixel of the region corresponding to this moving region is the signal of the pixel of the region extracted by the moving region detecting unit 72. Replace with the value. At this time, since the pixel density of the image from the moving area detecting unit 72 is ¼ of the pixel density of the image transferred from the CCD 11 to the image synthesizing unit 8, as shown in FIG. Detector 72
For one pixel of the image from, the corresponding four pixels of the image transferred to the image synthesis unit 8 are collectively replaced as one set.

According to the above method, the image synthesizing unit 8 rewrites a high-definition image in the still region once in every four frames, while the moving region is rewritten every frame, and the synthesized image is rewritten in one frame in one frame. Is output at the speed of. That is, the image is output as a natural moving image at a speed according to the normal NTSC system and as an image having a definition similar to that of HDTV.

A schematic structure of the second embodiment of the present invention is shown in FIG. In FIG. 3, reference numeral 9 is a camera movement detection unit. In the present embodiment, the pixel movement due to the movement of the camera is detected and compensated by the camera movement detection unit 9. This is treated as a moving area in the first embodiment because, for example, when the camera is moved from right to left, even a stationary object moves from left to right on the screen. However, when the camera moves, the entire screen moves in the same direction, so if you detect the amount of movement for one frame and shift the entire screen in the opposite direction to the moving direction, it can be treated as a still image. Is. In the present embodiment, the camera movement detection unit 9 detects the movement amount and direction of the entire image due to the movement of the camera using a known method. Then, when four frames are combined by the frame combining unit 71 that performs frame combination, the second and subsequent frame images are shifted in the opposite direction to the combined direction and combined to compensate for pixel shift due to camera movement. As in the first embodiment, the image can be transferred to the image synthesizing unit 8 as an image of a high-resolution still area.

On the other hand, the moving area detection unit 72 shifts the entire screen in the direction opposite to the moving direction and then extracts the area in which the image signal level is changed as compared with the previous frame to extract the area in which the object has moved. You can take it out.

In this embodiment, by compensating for the movement of the camera, it is possible to output in high definition except for the area where the subject actually moved. The configuration other than the camera motion compensation processing is the same as in the first embodiment.

The schematic construction of the third embodiment of the present invention is shown in FIG. In FIG. 4, 11R, 11G, 11B, 12
R, 12G and 12B are CCDs, and DP1 and DP2 are dichroic prisms. In this embodiment, the half mirror 2 is arranged behind the lens 5, and the dichroic prism D is arranged on each optical path branched by the half mirror 2.
P1 and DP2 are arranged. With this arrangement, R,
Six subject images color-separated into G and B are picked up by six CCDs. Here, the CCDs 11R and 12R pick up a red (R) component subject image, and the CCDs 11G and 1R
2G captures a subject image of green (G) component, and CCD 11B
And 12B capture a subject image of a blue (B) component. In addition, the CCD of the set of 11R, 11G, and 11B is for the purpose of imaging the stationary region, and the CCD of the set of 12R, 12G, and 12B is the CCD.
Is intended for imaging moving areas. In the present embodiment, a still region is imaged in high definition by the method of vibrating the transparent plate 3 as in the first embodiment. However, in this embodiment, the transparent plate 3
Is placed in front of the dichroic prism DP1 and CC
The three subject images formed on the image pickup surfaces of D11R, 11G, and 11B are simultaneously shifted. Although the electric system configuration is omitted in FIG. 4, in the present embodiment, each color component is provided with an electric system configuration similar to that of the first embodiment, and a high-definition moving image is output in R, G, and B.

The schematic construction of the fourth embodiment of the present invention is shown in FIG. In FIG. 5, reference numeral 81 is an output signal synthesizer. The optical system configuration, the configuration from the CCD 11 to the frame synthesis unit 71, the configuration from the CCD 12 to the moving area detection unit 72, and the signal processing method according to the present embodiment are the same as those in the first embodiment. In this embodiment, the image information of the still area is compressed on the time axis. That is, in the first embodiment, all the images in the still area are output regardless of whether there is a change from the previous frame (4 frames before).
In this embodiment, an area having no change compared to the previous frame is not output. Therefore, only the signal of the region which has been detected as the moving region one frame before and which has become the still region in the frame is output from the frame synthesizing unit 71. On the other hand, the moving area detection unit 72 outputs a moving area signal for each frame as in the first embodiment. Then, the output signal from the frame synthesizing unit 71 and the output signal from the moving area detecting unit 72 are multiplexed by the output signal synthesizing unit 81 and output as the output signal of the present imaging device.

In the present embodiment, the receiving side needs the storage means for the still area image, but since the transmission of the image information of the still area showing most of the image becomes unnecessary, the transmission information amount can be greatly compressed. ..

A schematic configuration of the fifth embodiment of the present invention is shown in FIG. In FIG. 6, C1 and C2 are CCD cameras. In this embodiment, the still area image and the moving area image are captured by different cameras. Two CCD cameras C1,
The half mirror 2 is arranged as shown in FIG. 6 so that C2 can image the same area. Although omitted in FIG. 6, the CCD
A transparent plate 3 similar to that in the first embodiment is arranged on the front surface of the CCD image pickup surface inside the camera C1, and vibrates in synchronization with the frame period.

The CCD cameras C1 and C2 used in this embodiment are single-chip cameras, and their output signals are combined into a high-definition moving image by the same electrical system configuration as in the first embodiment.

The schematic construction of the sixth embodiment of the present invention is shown in FIG. In FIG. 7, LS is a line sensor as an image sensor. In the present embodiment, the line sensor L is used as the stationary area.
The image is taken by S. Line sensor LS is slow,
Since high-definition imaging is possible, the A / D converted signal is directly sent to the image synthesizer 8. On the other hand, the CCD 12 is used to image the moving area, as in the first embodiment. The configuration from the CCD 12 to the image synthesizer 8 and the signal processing method are the same as in the first embodiment.

It is needless to say that various modifications can be made without departing from the spirit of the present invention, since only six embodiments of the present invention have been shown above. For example, in the above-described embodiment, the CCD or line sensor LS is used as the image sensor.
However, it is possible to implement it regardless of the type of the image pickup device.

Further, in the above-mentioned embodiment, the CCD 11 is moved while shifting the image pickup of the stationary area vertically and horizontally to the half of the pixel pitch.
The images of 4 frames taken in were combined, but the CCD1
1 and the pixel of CCD12 1 / optically in either direction
If they are arranged with a shift of 2 pixel pitches, the still area imaged by the CCD 12 can be used for 1 frame out of 4 frames, and a still area image having the same resolution as that of the above-described embodiment is synthesized in a time period of 3 frames. It is also possible.

In each of the above embodiments, the images are combined by ½ the pixel pitch in one direction, but the resolution can be further increased by ⅓ of the pixel pitch.

Further, in the above embodiment, the image of the subject on the image pickup device is shifted by using the image shift method of vibrating the transparent plate 3. However, there are some methods to realize the image shift method. It has been proposed and any of the methods can be used to carry out the present invention. For example,
As a typical example thereof, as described in a document (Technical Report of the Television Society, ED736, (1986)), a solid-state image sensor is mounted on a piezoelectric element, and a field with an amplitude of 1/2 pixel pitch is used. For each method, the liquid crystal panel and the crystal plate are placed in front of the solid-state image sensor, as described in the literature (Academic Society of Electronics, Information and Communication Engineers Autumn National Conference Proceedings D-244). A method of vibrating the subject image on the image sensor by utilizing the birefringence of the crystal plate is also applicable.

In the second embodiment, the motion of the camera is compensated, but this compensation can be applied to other than the second embodiment. Further, although the present invention is implemented by using the single plate type camera in the fifth embodiment, the present invention can be implemented by using the three plate type camera. The electrical system configuration after the camera in this case is the same as that of the third embodiment.

Although the CCD camera is used in the fifth embodiment, it is obvious that the present invention can be implemented regardless of the type of the camera. Further, although the half mirror is used in the fifth embodiment so that the two cameras can image the same area, the two cameras are arranged in parallel without using the half mirror as long as the function of specifying the imaging area is provided. How to do is possible.

[0035]

As described above, according to the present invention, a frame synthesizing unit that includes a still area image sensor and a moving area image sensor and synthesizes images of a plurality of frames captured by the still area image sensor, Since the moving area detection unit that extracts the moving area from the image captured by the moving area image sensor and the image combining unit that combines the still area and the moving area are provided, it is possible to capture an image with a characteristic suitable for each area. It will be possible. That is, with respect to a still region that determines the definition of an image, it is possible to take a sufficiently high-definition image by taking time even if an image pickup device having a small number of pixels is used. Further, the moving area can be captured at the same speed as the conventional moving image capturing, and a natural moving image can be captured. From the above, in the imaging device of the present invention,
High-definition moving images can be captured even with an image sensor having a small number of pixels. Therefore, a high-definition moving image pickup device such as an HDTV can be realized at low cost. Furthermore, if the present invention is implemented using a high-resolution image sensor for HDTV,
It is possible to capture ultra-high-definition moving images that have never been seen before.

Further, according to the present invention, since the moving area is detected and taken out, when transmitting to the receiving side, it is necessary to send only the signal of the moving area and the portion which is the moving area one frame before in the stationary area. Well, the transmitted signal can be easily compressed.

[Brief description of drawings]

FIG. 1 is a schematic configuration diagram of a first embodiment of the present invention.

FIG. 2 is a diagram showing a method of detecting a moving area and synthesizing it with a stationary area in the first embodiment.

FIG. 3 is a schematic configuration diagram of a second embodiment of the present invention.

FIG. 4 is a schematic configuration diagram of a third embodiment of the present invention.

FIG. 5 is a schematic configuration diagram of a fourth embodiment of the present invention.

FIG. 6 is a schematic configuration diagram of a fifth embodiment of the present invention.

FIG. 7 is a schematic configuration diagram of a sixth embodiment of the present invention.

FIG. 8 is an explanatory diagram of increasing resolution by the image shift method.

[Explanation of symbols]

 11 CCD 12 CCD 2 Half Mirror 3 Transparent Plate 4 Piezoelectric Element 5 Lens 61 A / D Converter 62 A / D Converter 71 Frame Synthesizer 72 Motion Area Detector 8 Image Synthesizer 81 Output Signal Synthesizer 9 Camera Motion Detector DP1 dichroic prism DP2 dichroic prism C1 CCD camera C2 CCD camera LS line sensor

Claims (1)

[Claims] 1. A frame synthesizing section, which comprises a still area image sensor and a moving area image sensor, and which synthesizes a plurality of continuous frame images captured by the still area image sensor, and the moving area image sensor. An image pickup apparatus comprising: a moving area detection unit that extracts a moving area from the captured image; and an image combining unit that combines the still area obtained by the frame combining unit and the moving area.