JPH05130489A - Camera integrated moving picture coder - Google Patents

Abstract

PURPOSE:To reduce number of conventional frame memories storing picture, to decrease the circuit scale and to make the arithmetic operation processing efficient. CONSTITUTION:CMD 3, 4 are used as image pickup elements, video information passing through an optical lens 1 is subject to transmission quantity control by a translucent mirror 1 and a liquid crystal shutter on the CMD 3, 4 the result is written, a panning and a hand blur discrimination section 9 discriminates the presence of panning and hand blur of a picture based on a motion vector detected by block matching between pictures written in the CMD 3,4, an address offset section 11 compensates the panning and hand blur by compensating a picture read address based on the discrimination result, a comparator 13 discriminates a still area and a moving area of the picture and the block matching section 14 obtains the moving vector as to the area discriminated to be a moving area only.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding apparatus, and more particularly to a camera-integrated moving picture coding apparatus suitable for reducing the circuit scale.

[0002]

2. Description of the Related Art With the improvement of moving image coding technology, coding devices have been actively developed from various fields. The background to this is that the international standardization of coding methods is progressing. In particular, the MPEG method (Moving Picture Expert Gro) that is intended for storage media is used.
up), the specification has been determined.

In this MPEG system, motion compensation is performed in order to reduce redundancy in the time axis direction, a difference between images is obtained, and thereafter, DCT (discrete cosine transform), which is a kind of orthogonal transformation, is performed in order to reduce redundancy in the spatial axis direction. ) And a variable length coding method are introduced.

To reduce the redundancy in the time axis direction, the property that continuous moving images have a high correlation between preceding and following frames (or between fields) is used. That is, a motion vector is detected for each block between the image to be encoded (current image) and the images that are temporally before and after, the motion prediction is performed, the current image is predicted, and then both Calculate the prediction error between Here, each image forming the moving image is an I picture (intra-frame coded image or intra-field coded image), P picture (forward prediction image), B picture (forward, backward, bidirectional prediction image). Will be encoded after being classified into three types.

Regarding the redundancy in the direction of the spatial axis, variable length coding is performed after the prediction error signal of each image is quantized to the DCT coefficient in block units.

On the other hand, a CCD (Charge Couple)
There is a video camera as an image pickup apparatus using a solid-state image pickup device such as an ed device). With this video camera,
Analog recording is used to record moving images. The CCD used in this video camera arranges pixels having a photoelectric conversion function in a matrix and scans the pixels by a charge transfer method to read out a video signal. Pixels cannot be reduced,
There are problems that it is not suitable for high-speed driving due to unread data when transferring charges and additional noise, and that the driving circuit becomes complicated.

In order to improve this problem, a CMD (Charge M) in which a pixel is composed of a MOS phototransistor is used.
Odulation Device). This CM
D has a simple pixel structure and does not require a pixel separation region, so it is suitable for increasing the number of pixels and increasing the density. Since the pixel has a MOS structure, there is no afterimage and it is suitable for high-speed driving. However, unlike CCD, it does not operate the pixel using the external clock as it is, so it is easy to drive. It does not directly output the accumulated charge, so the output signal can be output many times unless reset. It has the characteristics that it has a non-destructive read-out function that can be taken out and that the smear suppression characteristic is very good. For the technology related to this CMD, see the Television Society Journal V.
ol. 42, No. 8 1988, Yusa, "High-sensitivity solid-state imaging technology", p. 787-793. In this CMD type solid-state image pickup device, pixels having a photoelectric conversion function are arranged in a matrix, and pixels are scanned by an XY address system to read a video signal. As a technique for speeding up this reading, the Institute of Television Engineers Technical Report Vol. 11, No. 28 1987
, Morita et al., "High-speed drive of CMD", p. 7-
12 are described.

[0008]

However, in order to detect the motion vector by the above-mentioned conventional moving picture coding system, three memories for storing the preceding picture, the current picture and the succeeding picture are required. As an example of this, FIG. 7 shows a camera-integrated video image encoding device using a CCD type solid-state image pickup device in an image pickup system. In this device, the image of the subject is captured by the optical lens 6
1, a CCD 62 for picking up an image through 1 is provided, and a frame memory 63 for storing the captured previous image and a frame memory 64 for storing the captured current image are provided in the subsequent stage, and the memory outputs thereof are the previous images in the frame memory 63. Is connected to a block matching unit 65 which outputs a motion vector by performing block matching with the current image in the frame memory 64. Further, a frame memory 66 for storing the captured post-image is provided in the subsequent stage of the CCD 62, and the memory output and the output of the frame memory 64 are block matching between the current image in the frame memory 64 and the post-image in the frame memory 66. And is connected to the block matching unit 67 that outputs a motion vector. In such a case, since three frame memories are required, the circuit scale increases, resulting in a large housing, which is not preferable.

Further, in the image, it is divided into a moving area and a still area, but even for the still area, obtaining a motion vector by block matching or the like causes a decrease in efficiency. Therefore, it is desired to detect the motion vector only in the moving area in the image to improve the processing efficiency.
Alternatively, if panning or camera shake occurs during imaging with a video camera, originally, even a stationary object is detected as a moving object, and motion vectors have to be detected for all blocks in the image, which is also inefficient. There is a problem of causing a decrease.

An object of the present invention is to solve the conventional problems described above, reduce the number of conventional frame (field) memories for storing images, reduce the circuit scale, and improve the efficiency of arithmetic processing. An object is to provide an integrated moving picture coding device.

[0011]

In order to achieve the above-mentioned object, a camera-integrated moving picture coding apparatus according to the present invention comprises optical means for forming an image of a subject and an image passed through the optical means. , And an image pickup means composed of an image pickup device capable of nondestructive read by an XY address system, and a pan / camera shake for judging the presence / absence of pan / shake of an image in the image pickup device and outputting a representative motion vector. The area determining means includes a determining means, an area determining means for determining whether the image is a still area or a moving area, and a moving vector detecting means for obtaining a moving vector only for the moving area determined by the area determining means. Is a stationary area of the image after shifting the read address from the image sensor according to the representative motion vector output from the pan / camera shake determination means,
It is characterized by determining the moving area.

[0012]

In the present invention, when the motion vector is detected, the motion vector is obtained only for the true motion area in the image, and the efficiency of the arithmetic processing is improved.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings.

I. First Embodiment FIG. 1 is a diagram showing the configuration of a camera-integrated moving image encoding apparatus, in which incident light that has passed through an optical lens 1 for forming an image of a subject is captured by a semitransparent mirror 2 for capturing a previous image. CMD
3 and CMD4 for capturing the current image.

The liquid crystal drive unit 6 for controlling the amount of light transmitted by the liquid crystal based on the timing pulse signal from the timing pulse generation unit 5 is connected to the CMD3 and CMD4. The timing pulse generator 5 is connected to a refresh signal generator 7 for refreshing the contents of CMD3 or CMD4, and the refresh signal is arranged to be input to CMD3 and CMD4, respectively.

The front image (B1 to Bn) and the rear image (B1 to Bn)
n) and block matching with each motion vector vi
The two inputs of each block matching unit 8 for outputting (1 ≦ i ≦ n) are CMD3 and CMD4, respectively.
Connected to the output of.

At the subsequent stage of the block matching unit 8, there is provided a pan / camera shake judging unit 9 for judging the presence / absence of an image pan or camera shake from each motion vector vi and outputting a representative motion vector.

The XY address designating section 10 is arranged so as to give an address designating signal to the CMD3 and the XY address offset section 11 which compensates the read address of each block of the current image of the CMD4 according to the representative motion vector. ing.

The subtractor 12 for subtracting the block of the current image from the block of the previous image which has been addressed compares the output with the threshold value Th to determine whether the block is a static region or a moving region. It is connected with 13. The output is connected to the block matching unit 14 for performing the block matching only on the moving area obtained by the judgment and outputting the moving vector.

FIG. 2 is a block diagram of a CMD. A light receiving unit 21 of the CMD has a plurality of pixels arranged in a matrix, a vertical shift register 22, a horizontal shift register 23, and a video line 24. Elements (level mix circuit, etc.) are omitted.

First, at the time of charge storage, a low potential for cutting off CMD is applied to the gate from a level mix circuit (not shown). When photogenerated holes are accumulated in the gate portion, the surface potential rises. When reading a signal, a read gate potential is applied to a selected row. Then, each column is selected by the horizontal shift register 23 via the MOS switch, and the source current from each pixel according to the amount of incident light is detected from the video line 24. After the read operation, the reset potential is applied to release the accumulated holes to the substrate.

FIG. 3 is a diagram showing an example of a liquid crystal shutter.
One pixel 31 forming the light receiving portion 21 of D is composed of a photosensitive portion 32 and a liquid crystal shutter 33 on the surface thereof.

The liquid crystal shutter 33 is operated by a liquid crystal drive unit 6 which controls the transmission of light by the liquid crystal based on a pulse signal from the timing pulse generation unit 5, and the liquid crystal shutter 33 is opened and closed by a control signal from the liquid crystal drive unit 6. To work.
The timing of opening and closing the liquid crystal shutter 33 is such that when the liquid crystal shutter 33 on the CMD 3 is open, the liquid crystal shutter 33 on the CMD 4 is closed, and conversely, the liquid crystal shutter 3 on the CMD 3 is opened.
When 3 is closed, the toggle operation is such that the liquid crystal shutter 33 on the CMD 4 is open. Therefore, by the operation of the liquid crystal shutter 33, the information of the previous image for CMD3 and the information of the current image for CMD4 are written.

FIG. 4 shows an example of an observation block for pan and camera shake detection. Five observations are arranged at the center of the image 41 displayed on the light receiving section 21 of the CMD and at positions substantially equidistant from the center. Block 42 is shown. Here, an example in which the number of the observation blocks 42 is five is shown, but the number is not limited to this, and an arbitrary number may be arranged according to a desired image.

The operation of the first embodiment will be described below with reference to FIGS.
This will be described with reference to FIG.

After passing through the optical lens 1, the image information from the subject is divided into two by the semitransparent mirror 2 and the CMD.
3 and CMD4 respectively. By the operation of the liquid crystal shutter 33 as described above, the information of the previous image for CMD3 and the information of the current image for CMD4 are written.

As shown in FIG. 4, a part of the information written in the CMD4 is selected from a plurality of arbitrary blocks in the image and sent as an observation block 42 from the CMD4 to the block matching unit 8. Further, it is also sent from the CMD 3 to the block matching unit 8 as a reference block for the observation block 42. Block matching unit 8
Are the same as the number of observation blocks 42, and the motion vector of each block can be calculated in parallel. When information is read from each CMD by using the CMD, CC
Unlike D, etc., high-speed non-destructive reading is possible, so that image information on the CMD is not lost, and information in an arbitrary block can be read by the XY address system. Unlike a charge transfer method such as that of a solid-state image sensor, it is possible to speed up block matching processing without reading unnecessary blocks.

The motion vector output from the block matching unit 8 is input to the pan / camera shake determination unit 9. In the pan / camera shake determination unit 9, depending on a plurality of motion vector states,
Whether or not there is pan or camera shake between the previous image and the current image is determined. This bread, the bread by the camera shake determination unit 9,
FIG. 5 shows a flowchart for determining camera shake. Below, FIG.
The determination operation of the presence or absence of pan or camera shake between the previous image and the current image will be described according to the flow of 1.

First, initialization is performed, and the loop counter n
Is set to "0" (step 501). Next, assuming that the pan or camera shake cannot be determined once, the number of loops N0 for detecting a motion vector in another observation block is input again (step 502). Next, the motion vectors v1, v2, ..., Vn of each observation block obtained in advance by the block matching unit 8 are input (step 503). Next, the motion vectors v1, v of each observation block
For 2, ..., vn, take the vector sum of each. When this value is equal to or greater than the preset threshold value Th, it is determined that there is no panning or blurring between images (step 504), and a flag signal FLG1 = 0 relating to panning and blurring is output (step 504). 508).

On the other hand, when the vector sum is less than the threshold value Th, it is determined that panning or camera shake may have occurred, the observation block 42 is changed to another block (step 505), and the observation block 42 is changed again. Motion vectors v1 ', v2', ...
., Vn 'are detected by block matching (step 506). Then, the vector sum is again taken from v1 'to vn' and the result is compared with the threshold Th (step 507). In this step 507, the threshold value Th
When the value is equal to or more than the above value, it is determined that there is no panning or camera shake, and the flag signal FLG1 = 0 is output (step 508). When this value is less than the threshold value Th, 1 is added to the loop counter n (step 510). ), And then compared with the initially set loop count n0 (step 511). In this step 511, when n <n0, the observation block is changed again (step 505), and then a process of detecting a motion vector is performed (step 506). On the other hand, when n = n0, panning or camera shake has occurred. Flag signal FLG
1 = 1 is output (step 512). Finally, bread
In the camera shake determination unit 9, when FLG1 = 0, the representative motion vector vt = 0, and when FLG1 = 1, the average value regarding the motion vector of each observation block is output as the representative vector vt (steps 509 and 513).

Next, the XY address offset unit 11 offsets the reading of each block of the current image according to the representative motion vector. That is, when the XY addressing unit 10 addresses the pixel position on the CMD, the address information (xi, yi)
To the current image, the information (vx, vy) of the representative vector due to the pan and camera shake of the current image is added to
The address of (x, yi + vy) is designated. Therefore, when XY addressing is performed, when the information of the pixel position of (xi, yi) in the previous image of CMD3 is read, (xi + v) in the current image of CMD4 is read.
Since the information of the pixel position of (x, yi + vy) is read out, it is equivalent to equivalently preventing pan and camera shake and compensating for the still region.

Next, each CMD 3, 4 addressed
The image stored in 1 is calculated in the subtracting unit 12, the comparator 13 calculates the sum of absolute differences A from the subtraction result for each block, and compares it with the threshold value Th '.

Here, when A <Th ', there is little change in the blocks between the images, so it is determined that this block is a still region, and the flag signal FLG2 = 0 is output.

On the other hand, when A ≧ Th ′, there are many changes in the block between the images, so it is judged that this block is the moving region, and the flag signal FLG is set.
2 = 1 is output.

The block matching unit 14 detects a motion vector for motion compensating the current image based on the flag signal FLG2. However, block matching is performed only for the moving area of FLG2 = 1. Therefore,
For static areas that originally do not require block matching,
Since the cancellation is performed, there is an advantage that the amount of calculation for detecting the motion vector is reduced and the block matching unit 14 performs high-speed processing. This block matching unit 1
As a result of the processing in 4, the motion vector obtained by block matching is output. In addition, FLG
When 2 = 0, the motion vector is forcibly set to “0” and output.

As is apparent from the first embodiment, since the CMD type image pickup device capable of high-speed nondestructive read-out by the XY address system is used as the image pickup device, a frame for storing an image ( The field) memory is not required, and the circuit scale can be reduced. Also, for motion vector detection, which is important during motion compensation, pan,
In order not to erroneously determine the original still area as a moving area due to camera shake, only the true moving area is detected by compensating for pan and camera shake, and then the motion vector detection is performed by block matching. There is an advantage that it can be realized.

Although the relationship between the previous image and the current image has been described above, the present invention is not limited to this, and the current image and the subsequent image, or the current image, the previous image and the subsequent image can be easily described. Can be implemented. If the relationship between the current image, the previous image, and the subsequent image is to be obtained, the structure of FIG.
This is done by adding one MD.

II. Second Embodiment FIG. 6 is a diagram showing the configuration of another camera-integrated moving image encoding apparatus, in which incident light that has passed through the optical lens 1 for forming an image of a subject is once written in the CMD 15 and before that. It is arranged so that it is stored in a frame (field) memory 16 for capturing an image, and subsequently the current image is captured in the CMD 15.

The liquid crystal drive unit 6 for controlling the amount of light transmitted by the liquid crystal based on the timing pulse signal from the timing pulse generation unit 5 is connected to the CMD 15. The timing pulse generator 5 is connected to the refresh signal generator 7 for refreshing the contents of the CMD 15,
It is arranged so that the refresh signal is input to the CMD 15.

The front image (B1 to Bn) and the rear image (B1 to Bn)
n) and block matching with each motion vector vi
The two inputs of each block matching unit 8 for outputting (1 ≦ i ≦ n) are connected to the outputs of the CMD 15 and the frame (field) memory 16, respectively.

At the subsequent stage of the block matching unit 8, there is provided a pan / camera shake judging unit 9 for judging the presence or absence of camera shake in the image from each motion vector vi and outputting a representative motion vector.

The XY address designating section 10 follows the frame (field) memory 16 and the representative motion vector C
It is arranged so as to give an address designation signal to the XY address offset unit 11 which compensates the read address of each block of the current image of the MD 15.

The subtractor 12 for subtracting the block of the current image from the block of the previous image that has been addressed compares the output with the threshold value Th to determine whether the block is a static area or a moving area. It is connected with 13. The output is connected to the block matching unit 14 for performing the block matching only on the moving area obtained by the judgment and outputting the moving vector. Here, the same symbols as those in the first embodiment are the same.

The operation of the second embodiment will be described below. After the image from the subject passes through the optical lens 1, C
It is taken into MD15. Immediately thereafter, the data is transferred to the frame (field) memory 16. Next, a refresh signal is sent from the refresh signal generator 7 and the CMD is sent.
The video signal captured in 15 is refreshed, and the image that has passed through the optical lens 1 again is captured in the CMD 15.
By this operation, it is clear that the previous image is stored in the frame (field) memory 16, while the current image is stored in the CMD 15. The subsequent operation is similar to that of the first embodiment described above,
The information of the previous image is sent from the frame (field) memory 16 to the block matching units 8 and 14 and the subtraction unit 12.

Such an operation is exactly the same for the current image and the previous image, or for the current image, the previous image and the subsequent image. However, if both the front and rear images are also taken into consideration, it is composed of two frame (field) memories.

As described above, in the second embodiment, like the first embodiment, the number of frame (field) memories required conventionally can be reduced and the circuit scale can be reduced. Furthermore, in this second embodiment, C
Since only one MD is provided and the frame (field) memory is provided, the semitransparent mirror can be omitted, so that the configuration of the optical system can be simplified. In addition, since only one CMD can be realized, the cost of the device can be reduced.

[0047]

As described above, according to the present invention, the number of conventional frame (field) memories for storing images can be reduced, the circuit scale can be reduced, and the efficiency of arithmetic processing can be improved. ..

[Brief description of drawings]

FIG. 1 is a configuration diagram of a camera-integrated moving image coding apparatus showing a first embodiment of the present invention.

FIG. 2 is a configuration diagram of the CMD shown in FIG.

FIG. 3 is a diagram showing an example of a liquid crystal shutter in FIG.

FIG. 4 is a diagram showing an example of an observation block for pan and camera shake detection.

5 is a determination flowchart by a pan / camera shake determination unit in FIG. 1. FIG.

FIG. 6 is a configuration diagram of a camera-integrated moving image coding apparatus showing a second embodiment of the present invention.

FIG. 7 is a configuration diagram of a camera-integrated moving image encoding apparatus using a conventional CCD type image pickup device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Optical lens 2 Semi-transparent mirror 3 CMD 4 CMD 5 Timing pulse generation part 6 Liquid crystal drive part 7 Refresh signal generation part 8 B1-Bn block matching part 9 Pan / camera shake determination part 10 XY address designation part 11 XY Address offset unit 12 Subtraction unit 13 Comparator 14 Block matching unit 15 CMD 16 Frame memory

Claims (1)

[Claims] 1. An optical means for forming an image of a subject and an image passed through the optical means are written, and XY
An image pickup means composed of an image pickup device capable of nondestructive readout by an address method; a pan / shake movement decision means for judging the presence / absence of a pan / shake of an image in the image pickup element and outputting a representative motion vector; Area determination means for determining whether it is a still area or a moving area, and a motion vector detection means for obtaining a motion vector only for the moving area determined by the area determination means, wherein the area determination means is the pan / camera shake. A camera-integrated moving image coding apparatus, characterized in that after a read address from the image pickup device is shifted according to a representative moving vector output from the judging means, a still region or a moving region of an image is judged.