JPH09130748A - Image signal processing unit and recording/reproducing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a camera-shake motion vector with a simple hardware. SOLUTION: A subtractor 5 generates a difference between motion compensation frames and an encoder 8 encodes a video signal, and the result receives processing such as variable length coding and is recorded onto a recording medium. A motion vector detection circuit 6 gives an inter-frame difference Pij and a motion vector Vhv to a camera-shake detection circuit 12. The detection circuit 12 integrates the inter-frame difference to generate an evaluation value ΣPhv and when all blocks of one pattern satisfy a relation of Vhv<TH1 and ΣPhv>TH2, it is detected that a camera-share is in existence. A motion of each block is subject to macro block processing to convert the motion vector into a motion vector for each macro block. A camera-shake motion vector of the entire pattern is generated from the motion vector for each macro block. A circuit 13 generates a correction signal from the detected camera-shake vector and a camera-shake correction circuit 4 conducts camera-shake correction processing by the correction signal.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image signal processing device for compressing image data such as a shooting output of a handy type video camera, an image signal recording device for recording compressed image data on a recording medium, and a compression from a recording medium. The present invention relates to an image signal reproducing device for reproducing image data, and more particularly to a device for correcting camera shake of a video camera.

[0002]

2. Description of the Related Art When shooting with a handy-type video camera, there is a problem that the playback screen shakes due to camera shake. To solve this problem, we detect the motion vector,
It is conceivable to correct the image data stored in the image memory based on this motion vector. The motion vector is detected by block matching, for example.
That is, the screen is divided into a number of areas (called blocks), the absolute value of the frame difference between the representative point of the previous frame located at the center of each block and the pixel data in the block of the current frame is calculated, and this frame is calculated. The absolute value of the difference is integrated for one screen, and the motion vector of the entire screen is detected from the position of the minimum value of the integrated frame difference data. The detected motion vector is converted into a correction signal, and the original image is moved by the correction signal.

For example, in FIG. 23A, an image frame La indicated by a broken line is an imaged image, and its position is corrected to an image frame Lb indicated by a broken line by camera shake correction. In the image in the corrected image frame Lb, the shaded region has no captured image, and thus the image is lost. One method of solving this problem is to enlarge the image frame to some extent, as indicated by Lc in FIG. 23B. This makes it possible to prevent image loss.

However, the enlargement of the image is performed by the process of making the reading speed of the image memory slower than the writing speed and interpolating the insufficient pixel data. Therefore, the resolution of the enlarged image is lower than that of the original image. Therefore, there is a problem that the image quality after camera shake correction is not good.

Next, another problem of the conventional camera shake correction will be described. This is a problem in which a large area movement on the screen is erroneously determined as a camera shake. 1 screen (4 × 4 = 1
6) When the method of dividing into macroblocks and detecting the motion vector in each macroblock is adopted, for example, as shown in FIG. 24A, when a person moves from right to left on the screen, The motion vector indicated by the arrow is detected. Motion vectors of other blocks that do not include a person (edge) are detected as 0.

Conventionally, a majority decision is made on the motion vector for each macroblock, and a large number of motion vectors are adopted as the motion vector for camera shake correction. However, as in the above-mentioned example, when an object having a large area is present in the center of the screen and this object moves, there is a possibility that the motion of this object is regarded as a hand shake and is erroneously determined. It is necessary to avoid such erroneous determination.

Further, there may be a plurality of motion vectors as the motion vector of the macroblock. For example, as shown in FIG. 24B, a plurality of motion vectors may be detected as a result of combining the motion vector corresponding to the motion caused by the camera shake and the motion vector caused by the motion of the object on the screen. When there is a movement of a large-area object as in the example of FIG. 24B, if a majority decision is made as in the conventional case, there is a problem that the motion vector due to camera shake cannot be accurately detected.

Further, in an image signal recording device such as a video camera integrated VTR, as an image signal to be recorded,
It is common to record the image with the camera shake corrected. However, as a result, it becomes impossible to reproduce the photographed image itself, and when camera shake correction is not properly performed, there arises a problem that the correction cannot be performed on the reproducing side.

[0009]

As described above, the conventional method of detecting a motion vector and performing camera shake correction based on the detected motion vector has various problems. Further, in addition to the above-mentioned problems, there is a problem that the hardware for detecting the motion vector is required and the scale of the hardware for camera shake correction becomes large.

The present invention is intended to reduce the scale of hardware. In general, when recording digital image data on a recording medium (magnetic tape, optical disk, etc.), motion compensation predictive coding is known as high-efficiency coding for compressing the amount of recorded data. This is to detect the difference between the input image signal and the predicted image signal, encode the difference, and further perform variable-length coding in predictive coding, where motion compensation is performed using the detected motion vector when forming the predicted signal. , Thereby reducing the difference value. Since the motion vector is used for the image stabilization of the video camera, the hardware for the motion vector detection is used by using the detected motion vector for the motion compensation predictive coding as the motion vector. It becomes possible to share.

Therefore, an object of the present invention is to provide an image signal processing apparatus and a recording / reproducing apparatus capable of sharing a motion vector detected for motion compensation predictive coding also for camera shake correction. Especially.

[0012]

According to the present invention, image data of one screen is divided into a plurality of blocks, and a motion corresponding to the position of the best matching block from the image data of one to several frames before is divided for each block. Detect the vector,
In an image signal processing device having a function of compressing image data using a motion vector, a means for detecting a motion vector of each block, a means for detecting a camera shake motion vector from the motion vector of each block, and an input image signal Means for correcting based on the shake vector, means for obtaining a motion vector obtained by correcting the motion vector of each block with the shake motion vector, means for performing motion compensation by the corrected motion vector, and shake-corrected image data and motion An image signal processing device comprising: means for performing a compression process on the compensated image data.

Further, according to the present invention, the image data of one screen is divided into a plurality of blocks, and the motion vector corresponding to the position of the best matching block is detected from the image data of one to several frames before for each block. In an image signal recording device having a function of compressing image data using a motion vector, a means for detecting a motion vector of each block, a means for detecting a shake motion vector from the motion vector of each block, and a motion by the motion vector An image signal recording apparatus comprising: means for performing compensation, means for performing a compression process using input image data and motion-compensated image data, and means for recording a camera shake motion vector together with the compressed image data. Is.

Further, according to the present invention, the image data of one screen is divided into a plurality of blocks, and each block has 1 block.
Or, from the image data of several frames before, the motion vector corresponding to the position of the best matching block is detected, the image data is compressed using the motion vector, the camera shake motion vector is detected from the motion vector, and the motion compensation is performed by the motion vector. The compressed image data is decoded in an image signal reproducing device that performs compression processing using the input image data and the motion-compensated image data, and reproduces the recording medium on which the shake motion vector is recorded together with the compressed image signal. In the image signal reproducing apparatus, the decoding means, the means for separating the camera shake motion vector from the reproduced signal, and the means for supplying the decoded image signal from the decoding means and correcting the camera shake by the camera shake motion vector are provided. is there.

Motion vectors are detected for compression of recorded data, such as motion-compensated predictive coding. This motion vector is detected in block units. By using this motion vector and the amount of change in the image data,
The camera shake motion vector is detected. The image data corrected by this camera shake motion vector is recorded. It is not limited to recording the image data subjected to camera shake correction, but the camera shake motion vector or the correction signal formed thereof is recorded on the recording medium together with the compressed image data, and the camera shake correction is performed during reproduction.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. The embodiment here is, for example, a video camera integrated with a VTR, which converts a captured image signal into a digital signal, compresses it by motion compensation predictive coding, and records it on a recording medium such as a magnetic tape. Is.

In FIG. 1, 1 is a C as an image sensor.
Indicates a CD. The image pickup output of the CCD 1 is supplied to the camera signal processing circuit 2 and converted into a video signal. This video signal is converted into a digital video signal by the A / D converter 3. For example, A / D conversion is performed at a sampling frequency of 13.5 MHz. Note that the present invention can be applied to a component signal composed of a luminance signal and two color difference signals, or a composite signal in which a luminance signal and a carrier color signal are superimposed, but in the following description, a simple description will be given. Therefore, only the processing of the luminance signal will be described.

A digital video signal from the A / D converter 3 is supplied to a camera shake correction circuit 4 to which the present invention is applied via a delay circuit 14. The camera shake correction circuit 4 is a circuit for correcting camera shake during shooting, as described later. A camera shake detection circuit 12 and a correction signal generation circuit 13 are provided for camera shake correction. The delay circuit 14 is
It is provided for phase adjustment for delaying the digital video signal for the time required for the processing for forming the image stabilization signal. The output signal of the camera shake correction circuit 4 is supplied to the subtraction circuit 5, and the output signal of the A / D converter 3 is supplied to the motion vector detection circuit 6.

The subtraction circuit 5 is supplied with the prediction signal from the motion compensation circuit 7, and the subtraction circuit 5 generates a difference between the actual video signal and the prediction signal for each pixel. This difference is supplied to the encoder 8 for ADRC encoding, and the encoded output of the encoder 8 is taken out. At the same time, the output of the encoder 8 is the ADRC decoder (local decoder).
9 and the decoded data of the difference value is extracted at its output. The decoding side may perform refresh processing by periodically inserting the intra-frame encoded image signal.

The decoded value from the ADRC decoder 9 is supplied to the adder circuit 10 and added to the prediction signal from the motion compensation circuit 7. The decoded signal obtained from the adder circuit 10 is written in the frame memory 11. The output of the frame memory 11 is supplied to the motion compensation circuit 7. The motion vector Vhv from the motion vector detecting circuit 6 is supplied to the motion compensating circuit 7 via the motion vector correcting circuit 15 to perform motion compensation. The prediction signal from the motion compensation circuit 7 is supplied to the above-mentioned addition circuit 10 and the subtraction circuit 5. In the subtraction circuit 5, the pixel difference between the actual video signal and the prediction signal is formed.

The motion vector detection circuit 6 detects a motion vector for motion compensation from the output video signal of the A / D converter 3, that is, the digital video signal before camera shake correction. In one embodiment of the present invention, the motion vector Vhv and the evaluation value Pij are supplied to the camera shake detection circuit 12, the camera shake motion vector is obtained, and this camera shake motion vector is supplied to the correction signal generation circuit 13 for correction as described later. Form a signal. By supplying this correction signal to the camera shake correction circuit 4, the camera shake can be corrected.

The structure shown in FIG. 2 is used for recording / reproducing image signals. In FIG. 2, an encoder for motion compensation predictive coding is provided at the input terminal 21 (in the example of FIG. 1,
The encoded output is supplied from the ADRC encoder 8).
This coded output is supplied to the variable-length coding encoder 22 and variable-length coded. The output of the variable length coding is supplied to the recording buffer 23. The recording buffer 23 feeds back a signal indicating the buffer capacity to the previous stage. As a result, the amount of recording data for a predetermined period is controlled to be constant.
For example, the amount of transmission data can be controlled by changing the number of quantization bits of ADRC encoding.

Although this example is a feedback control buffering circuit, a plurality of quantization bit numbers in ADRC encoding (for example, 0 bit, 1 bit, 2 bit, 3 bit, 4 bit) are prepared, The amount of generated data in a predetermined period may be estimated, and the number of quantized bits may be selected for each block so as not to exceed the target amount of data, and feedforward control buffering may be performed. The output of the recording buffer 23 is supplied to the multiplexer 24, and audio data, control data, etc. are multiplexed.

The output of the multiplexer 24 is supplied to the recording processing circuit 25. The recording processing circuit 25 performs processing such as error correction coding and channel modulation on the recording data. The recording data from the recording processing circuit 25 is recorded on the recording medium via the recording amplifier 26 and the recording side terminal R of the recording / reproducing changeover switch 27. As a recording medium,
A magnetic tape, a recordable optical disk, etc. can be used.

The reproduction data reproduced from the recording medium is supplied to the reproduction processing circuit 29 via the reproduction side terminal P of the recording / reproduction changeover switch 27 and the reproduction amplifier 28. The reproduction processing circuit 29 performs processing such as demodulation of channel modulation and error correction. The output of the reproduction processing circuit 29 is supplied to the demultiplexer 30, and the audio signal, control data and the like are separated. The reproduction data corresponding to the video signal from the demultiplexer 30 is supplied to the decoder 31 for variable length coding, and the variable length code is decoded. The reproduction data taken out to the output terminal 32 is supplied to a high efficiency encoding decoder such as an ADRC decoder.

ADRC is one of the high efficiency encodings proposed by the inventor of the present application. An example of the configuration of the ADRC encoder 8 is shown in FIG. A digital video signal is supplied from an input terminal 34 to a blocking circuit 35, and the blocking circuit 35 subdivides one screen to form data having an order for each block. The output data of the blocking circuit 35 is supplied to the detection circuit 36 and the delay circuit 37.

The detection circuit 36 detects the maximum value MAX and the minimum value MIN of a plurality of pixels in each block. Delay circuit 3
Reference numeral 7 is for delaying the digital video signal for the time required for this detection. Maximum value MAX detected
And the minimum value MIN are supplied to the subtraction circuit 38, and the dynamic range DR represented by (MAX-MIN = DR) is obtained from the subtraction circuit 38. Further, the digital video signal from the delay circuit 37 and the minimum value MIN are subtracted by the subtraction circuit 39.
The corrected pixel data with the minimum value removed is obtained at the output. Minimum value M shared by pixels in a block
Normalization is done by removing the IN.

The output of the subtraction circuit 39 and the detected dynamic range DR are supplied to the quantization circuit 40. In the quantization circuit 40, as shown in FIG. 5, a subtraction circuit 39 is performed by a quantization step (n; quantization bit number, n = 2 in FIG. 5) that divides the dynamic range DR into 2 ⁿ equal parts.
Is requantized. Pixels that are spatially close to each other have a strong correlation. Therefore, even if the number of quantization bits n is smaller than the original number of quantization bits (for example, 8 bits), the decoded image is less deteriorated and the transmission data amount is compressed. You can The encoded output DT from the quantization circuit 40 is transmitted, and
Dynamic range DR and minimum value MI for each block
N is transmitted as additional information.

FIG. 4 shows an example of the configuration of the ADRC decoder 9. The dynamic range DR and the encoded output DT from the ADRC encoder 8 are supplied to the inverse quantization circuit 42. The inverse quantization circuit 42 has a dynamic range DR
And a quantization step number determined by the number of quantization bits n are multiplied to the encoded output, and the multiplication output is converted into an integer to generate a restoration level. As shown in FIG. 5, the median value of each level range at the time of quantization is set as the restoration level (representative level) L0, L1, L2, L3.

Then, the restoration level from the inverse quantization circuit 42 and the minimum value MIN are added by the addition circuit 43 to obtain a decoded value. This decoded value is supplied to the block decomposition circuit 44, and the order of the pixel data is returned to the raster order. The decoded video signal is taken out at the output terminal 45 of the block decomposition circuit 44.

The motion vector detection circuit 6 detects a motion vector by the block matching method. this is,
As shown schematically in FIG. 6, the inspection block By of the reference frame, for example, the previous frame F _n-1 is moved within a predetermined search area, and the reference block B of the current frame Fn is moved.
The motion vector is obtained by detecting the inspection block that best matches x. Therefore, the motion vector is obtained for each block. As the evaluation value indicating the degree of matching, the frame difference is obtained by subtracting the values of the pixels at the same spatial position between the plurality of pixels in the reference block Bx and the plurality of pixels in the inspection block By. Seeking,
This sum of absolute frame differences can be used. In addition to the sum of absolute values of frame differences, the sum of squares of frame differences can be used.

FIG. 7 shows an example of the configuration of the motion vector detection circuit 6. In FIG. 7, reference numeral 51 is an input terminal for the image data of the current frame, and this image data is stored in the current frame memory 53. Reference numeral 52 is an input terminal for the image data of the previous frame, and this image data is stored in the previous frame memory 54.

The reading / writing of the current frame memory 53 and the previous frame memory 54 is controlled by the controller 55. Pixel data of the reference block of the current frame is read from the current frame memory 53, and pixel data of the inspection block of the previous frame is read from the previous frame memory 54. An address moving circuit 56 is provided in association with the previous frame memory 54. As a result of the controller 55 controlling the address moving circuit 56, the position of the inspection block changes within the search area by one pixel step.

The output of the current frame memory 53 and the output of the previous frame memory 54 are supplied to the difference detection circuit 57, which outputs 1
The difference (frame difference) for each pixel is detected. The output of the difference detection circuit 57 is converted into an absolute value by the absolute value conversion circuit 58,
This absolute value is supplied to the accumulation circuit 59. Accumulation circuit 59
, The absolute value difference generated in one block is accumulated, and the output (sum of the frame difference absolute values) is supplied to the determination circuit 60 as an evaluation value. The determination circuit 60 detects the motion vector from the sum of absolute values of the differences generated when the inspection block is moved within the search area. That is, the position of the inspection block that generates the minimum absolute difference sum is detected as a motion vector. The detected motion vector is extracted at the output terminal 61.

In one embodiment of the present invention, the motion vector thus detected for motion compensation is also used for camera shake correction. The camera shake correction motion vector represents the motion of the entire screen or a relatively large block, and as an existing method, it is known to detect a camera shake correction motion vector as shown in FIG.

That is, the current frame Fn is divided into search areas Sx, and a plurality of pixels in each search area Sx,
The absolute value of the frame difference from the representative point pixel Ry (pixel at the center position of the search area) of the previous frame F _n-1 is calculated. Further, the frame difference absolute values obtained in each search area Sx are integrated at the same spatial position, thereby creating an evaluation value table having the same size as the search area Sx. The motion vector of the entire screen is obtained by detecting the minimum value in this evaluation value table.

As described above, the motion vector obtained by the motion vector detection circuit 6 is obtained in block units, whereas the motion vector required for camera shake correction is obtained in the entire screen or in units of relatively large blocks. Is required in. Due to this difference, the camera shake detection circuit 12 receives the frame difference absolute value sum Pij and the motion vector Vhv from the motion vector detection circuit 6,
A motion vector for camera shake correction is generated.

FIG. 9 shows an example of the configuration of the camera shake detection circuit 12 and the camera shake correction circuit 4. The sum Pij of frame difference absolute values from the motion vector detection circuit 6 is supplied to the integration circuit 71. This frame difference absolute value sum Pij will be described with reference to FIG. As described with reference to FIG. 6, the sum of absolute values of the frame differences between the reference block Bx and the inspection block By is calculated at each position of the inspection block By in the search area. As an example, assuming that the search area is ± 4 in the horizontal direction and ± 3 in the vertical direction, the sum of absolute frame difference values with 9 × 7 = 63 inspection blocks is calculated.

One reference block Bx with the current frame
FIG. 10 shows the arrangement of the sums of the absolute values of the frame differences with respect to the above, which are arranged corresponding to the positions of the inspection blocks. Center position (i =
4, j = 3), the sum P43 of frame difference absolute values is obtained when the reference block Bx and the inspection block By are spatially at the same position. The sum Pij of frame difference absolute values is integrated by the integrating circuit 71 for each reference block to obtain an evaluation value Σ.
Phv is formed. As an example, the block size is (1
6 × 16) and the effective screen size of one frame is (70
With 4 pixels × 480 lines), one frame is divided into (44 × 30) blocks as shown in FIG. 11, and an evaluation value ΣPhv equal to the number of blocks is obtained.

The evaluation value ΣPhv reflects the level distribution of the image in the block. If the image in the search area has a flat level distribution, the level of the sum of absolute frame differences becomes small and the evaluation value ΣPhv
The level of will also be smaller. On the other hand, when the level distribution is not flat and includes edges and the like, the level of the evaluation value ΣPhv also becomes large. In this embodiment, the image level distribution is used as the activity, and the flat level distribution is used as the low activity. However, the activity is not limited to the spatial gradient and includes frequency components in frequency conversion.

The evaluation value ΣPhv and the motion vector Vhv are supplied to the camera shake determination circuit 72. Camera shake determination circuit 72
Performs camera shake determination as shown in the flowchart of FIG. The initial values are set to (h = 0, v = 0) (step 81), and in the next step 82, the threshold value T
It is compared with H1 and TH2 respectively. With V00 <TH1,
If ΣP00> TH2, it is determined that the camera is stationary (that is, no camera shake) (step 83).

The condition of step 82 is satisfied that the block 00 is an image (ΣP00> TH2) at a position where level changes such as edges are discontinuous, and
This means that it has been detected that there is no motion for 00 (V00 <TH1). Since the entire screen moves when camera shake occurs, it can be determined that even one still block exists is not camera shake.

When the condition of step 82 is not satisfied, the process proceeds to step 84 to check whether the value of h is the maximum value (43) in the horizontal direction. Otherwise, in step 85, the value of h is incremented by 1,
Return to step 82. In this way, the process moves from block 00 to the adjacent block 10, and this block 10
Is a static block. Then, when the processing for all blocks of v = 0 is completed, the processing moves from step 84 to step 86.

Step 86 determines whether or not the value of v is 29, and if not, the process proceeds to step 87. In step 87, the value of v is incremented by 1, and the process returns to step 82. When the value of v reaches 29, the processing of all (44 × 30) blocks is completed.
Then, if no static block is detected for all blocks, it is determined that the motion of the entire screen, that is, camera shake, and in step 88, the process of motion vector detection for each macroblock is started. In the configuration of FIG. 9, from the camera shake determination circuit 72 to the macroblocking circuit 73,
A control signal of the determination result is output.

The macroblocking circuit 73 separates the motion vector Vhv from the motion vector detecting circuit 6 into macroblocks. Here, as shown in FIG. 13, the screen of one frame is divided into four in the horizontal direction and three in the vertical direction to form 12 macroblocks having a size of (11 × 10) blocks. Not limited to the example of this macroblock, various methods of dividing one screen to obtain a plurality of macroblocks can be adopted. However, it is necessary to generate macroblocks that are not adjacent to each other. Therefore, a method of equally dividing a screen into four should not be adopted. The motion vector for each macroblock is supplied to the motion vector detection circuits 74 ₀ , 74 ₁ , ..., 74 ₁₂ .

The motion vector detection circuit 74 ₀ detects the combined motion vector MV00 by vector-adding the motion vectors of the blocks included in the macroblock MB00. Similarly, the motion vector detection circuits 74 ₁ , 74
₂ , ..., 74 ₁₂ detect the combined motion vectors MV10, MV20, ... MV32 of the respective macroblocks. The detected combined motion vectors MV00 to MV32 are supplied to the integrated vector forming circuit 75.

The integrated vector forming circuit 75 generates a motion vector for the entire screen (integrated vector) by integrating 12 combined motion vectors. This integrated vector is supplied to the correction signal generation circuit 13. The integrated vector is detected from the movement between frames and is not the same as the amount of camera shake correction. For example, when camera shake in the same direction occurs in the periods of the second and third frames in the period of three consecutive frames, the integrated vector V1 between the first frame and the next frame is obtained. , The integrated vector V between the second and third frames
2 is obtained. The correction amount for the second frame is
V1 may be used, but the correction amount for the third frame is
(V1 + V2) is required. Correction signal generation circuit 13
Generates, as an example, a correction amount obtained by integrating the integrated vector. The camera shake correction signal from the correction signal generation circuit 13 is supplied to the correction circuit 4, and the camera shake correction of the input image data is performed.

The integrated vector forming circuit 75 forms an integrated vector according to the flow shown in FIG. The motion vector of each macroblock is input (step 91), and it is checked whether or not the motion vectors of all macroblocks are the same (step 92). If they are the same, the motion vector is output as an integrated vector (step 93).
The same applies to the following cases, but the same includes some tolerance.

When step 92 is not established, the process proceeds to step 94 for determination, and it is checked whether or not the motion vectors are the same for the plurality of spatially separated macroblocks. In other words, it is checked in one frame whether macroblocks having the same motion vector are spatially separated. A motion vector satisfying step 94 is output as an integrated vector (step 9
5). "Spatially separated" means that they are adjacent in the vertical, horizontal, and diagonal directions (for example, (MB00, MB10, MB
01, MB11)) is excluded. Therefore, the spatial distance, the number of macroblocks having the same motion vector, and the like are appropriately set in consideration of the required detection accuracy and the like. The determination step 94 can eliminate an erroneous determination in which the movement of an object having a relatively large area is used as a camera shake.

When step 94 is not satisfied, the flow moves to step 96, and the numbers of macroblocks having the same vector are summed up. The number of macroblocks is weighted in the next step 97. This weighting is for giving priority to the number of blocks related to macroblocks around the screen over the number of blocks related to macroblocks near the periphery of the screen.
As an example, the number of blocks in the vicinity is multiplied by a weighting factor of 1.5, and the number of blocks in the non-neighborhood is 1
Multiply by the weighting factor of. Considering that the movement of a relatively large area object is often near the center, step 97
Are weighted.

Then the flow moves to step 98 to check if there is a maximum value in the number of weighted blocks. When there is a maximum value, the motion vector of the macroblock with the maximum value is output as an integrated vector (step 99). If the number of blocks is equal,
Or, if approximately equal, flow moves to step 100. Then, with respect to the number of macroblocks, a majority decision is made in step 100, and the motion vector of the majority is output as an integrated vector. The majority decision has a problem in terms of accuracy in generating an integrated vector, and when importance is attached to accuracy, it is possible to omit this determination and process the integrated vector as undetectable.

By the above-described processing, the motion vector is integrated into the motion vector of the entire screen by utilizing the difference between the characteristics of the moving object and the shaking motion, so that the detected shaking motion vector is the motion of the object in the screen. It is a high precision one that is not affected by.

Returning to FIG. 9, the integrated vector representing the movement of the entire screen is supplied to the correction signal generation circuit 13, and the correction signal is formed by integration. This correction signal is supplied to the camera shake correction circuit 4 to correct the camera shake.
The camera shake correction circuit 4 includes a memory 101 in which image data from the delay circuit 14 is written, a peripheral memory 102, a selector 103 for selecting the read output of the memory 101 and the peripheral memory 102, the memory 101 and the peripheral memory 102. The address control circuit 105 for controlling the address and the select signal generating circuit 106 for generating the select signal for controlling the selector 103 are provided, and the video signal subjected to the image stabilization is taken out to the output terminal 104.

The memory 101 is, for example, a frame memory, and its read address is controlled according to the correction signal. Therefore, the image data moved according to the correction signal is read from the memory 101. The data selected by the selector 103 is written in the peripheral memory 102. In response to the select signal from the select signal generating circuit 106, the selector 103 selects the image data from the memory 101 and the peripheral data stored in the peripheral memory 102 that have been shake-corrected.

In FIG. 15A, the peripheral portion (the area outside the alternate long and short dash line) 108 of one frame image (the image frame is represented by 107) taken in the memory 101 is the peripheral memory 1.
02 is stored. The width of the peripheral portion 108 is set in consideration of the range of camera shake correction, and is, for example, about 10 to 20% of the width of one frame image in the horizontal and vertical directions. In FIG. 15B, as indicated by an image frame 107a, when the image that should be in the position of FIG. 15A moves to the right, for example, toward the drawing due to camera shake, the entire image is moved to the position indicated by the broken line by the camera shake correction amount V. Will be corrected. In the case of this camera shake correction, a portion 109, which originally does not exist in the captured image and is indicated by the diagonal line on the left side of the image after movement, is shown.
Image is missing. This missing portion 109 is the peripheral memory 1
It is replaced with the image of the corresponding position stored in 02. This replacement is executed by the address control by the address control circuit 105 and the switching operation of the selector 103. Further, peripheral image data other than the missing portion 109 in the captured image data is written in the peripheral memory 102, and the content of the peripheral memory 102 is updated.

As described above, since the missing image caused by the camera shake correction is replaced with the peripheral image stored in the peripheral memory, it is possible to prevent the deterioration of the resolution of the image unlike the processing of enlarging the image.

In the above-described embodiment of the present invention, the motion vector for detecting the camera shake motion vector is obtained by processing the video signal before the camera shake correction. On the other hand, processing such as motion compensation is performed on the video signal whose camera shake is corrected by the camera shake correction circuit 4. Therefore, motion compensation cannot be performed by the motion vector detected by the motion vector detection circuit 6. to solve this problem,
A motion vector correction circuit 15 is provided. The motion vector correction circuit 15 removes the camera shake component included in the motion vector.

The processing in the motion vector correction circuit 15 will be described with reference to FIG. FIG. 16A shows motion vectors V00, V01, V10, V1 obtained in each block.
1 and so on represent that the components of the shake vector are included. If there is no camera shake, the positions of these blocks are in the positions shown by the broken lines.

Here, paying attention to the block of the motion vector V00, as shown in FIG. 16B, when the camera shake component is removed, the position of this block is the position of the broken line. For the block at the position of the broken line, the number of pixels included in the block at the position where the motion vector is detected and the other blocks in the periphery is obtained. In FIG. 16, the total number of pixels in the block is n, and at the positions indicated by the broken lines, n1, n2, n
The number of pixels represented by 3 and n4 is the number of pixels included in each block. Then, the corrected motion vector V00 'is calculated by the weighted average method represented by the following expression. The motion vectors of other blocks are similarly corrected.

V00 '= (n1 / n) .V00 + (n2 /
n) ・ V10 + (n3 / n) ・ V01 + (n4 / n) ・ V11

In order to obtain the number of pixels n1 to n4 contained in each of the plurality of blocks, the projection component on the x axis and the projection component on the y axis of the camera shake correction are used. In this way, the motion vector can be modified. The weighted average method is one method of correction, and other correction methods can be used.

FIG. 17 shows another embodiment of the present invention.
In one embodiment shown in FIG. 1, the inter-frame difference from the subtractor 5 is compressed by ADRC, while another embodiment adopts MPEG (Moving Pictures Expert Group) standard encoding. There is. MPEG is a DCT (Discrete
Cosine Transform) circuit 16a and quantizer 17a perform compression utilizing spatial correlation, and bidirectional motion-compensated interframe prediction is performed.

Bidirectional predictive coding is intraframe predictive coding, forward predictive coding and backward predictive coding.
The forward predictive coding is an interframe predictive coding that predicts a current image from a past image, and the backward predictive coding is an interframe predictive coding that predicts a current image from a future image. Further, interpolative interframe predictive coding is also performed by prediction in both the front and back directions. Corresponding to these predictive coding methods, picture types (I picture, P
Picture, B picture).

For bidirectional motion compensation, a frame memory 18 is added to the motion vector detection circuit 6,
Also, using the past, present, and future predicted images from the frame memories 11a and 11b, the backward motion compensation circuit 7r, the interpolated motion compensation circuit 7i, and the forward motion compensation circuit 7f are used.
Respectively perform motion compensation. For local decoding, the inverse quantizer 16b, the inverse DCT circuit 17b, the inverse DCT
An adder circuit 10 for adding the output of the circuit 17b and the signal passed through the switch circuit is provided. Also, the subtractor 5
There is provided a switch circuit for selecting the difference signal from the signal and the signal not passing through the subtracter 5. These switch circuits are provided with a control signal S according to the picture type described above.
1, S2, S3. Encoding control circuit 1
Control signals S1, S2, and S3 are generated from 9.

In the other embodiment of the present invention, similarly to the one embodiment, the motion vector Vhv and the evaluation value Pij from the motion vector detection circuit 6 are supplied to the camera shake detection circuit 12, the camera shake motion vector is obtained, and this camera shake is obtained. The motion vector is supplied to the correction signal generation circuit 13, a correction signal is formed, and the correction signal is supplied to the camera shake correction circuit 4 to correct the camera shake. Also, to correct the shake vector,
A camera shake vector correction circuit 15 is provided.

FIG. 18 shows the structure of the recording side according to still another embodiment of the present invention. In still another embodiment, the recording side generates a correction signal for camera shake correction, and the reproducing side shown in FIG. 19 performs camera shake correction. Therefore, FIG.
As shown in FIG. 5, the recording side is provided with the camera shake detection circuit 12 and the correction signal generation circuit 13, but the camera shake correction circuit 4 is not provided. The encoded output from the ADRC encoder 8, the motion vector, and the camera shake correction signal are supplied to the framing circuit 20 to form recording data. This recording data is recorded on the recording medium via the recording system.

As shown in FIG. 19, on the reproducing side, the reproduced data formed by the reproducing system is supplied from the input terminal 62 to the frame decomposition circuit 63, and the ADRC encoded output, the shake vector Vhv and the correction signal are separated. . The differential signal is decoded by the ADRC decoder 64, and the decoded differential signal is supplied to the adding circuit 65. The output of the adder circuit 65 is supplied to the prediction memory 66, and motion compensation similar to that on the recording side is performed by the motion vector Vhv.

The reproduced image signal from the adder circuit 65 is supplied to the camera shake correction circuit 4. In the camera shake correction circuit 4, the camera shake is corrected by the correction signal separated from the reproduction signal. The digital image signal from the camera shake correction circuit 4 is supplied to the D / A converter 67, and the reproduced image signal is taken out at the output terminal 68. In addition, the operation of camera shake correction,
You may provide the selection means which selects non-operation. Further, although the camera shake correction signal is recorded, the evaluation value ΣPij may be recorded instead of the camera shake correction signal, and the camera shake correction signal may be formed on the reproducing side by using the motion vector and the evaluation value.

Still another embodiment of the present invention in which the camera shake correction signal is recorded can be applied to the case of using MPEG.

Further, although the peripheral memory is provided in order to compensate for the loss of the image at the time of camera shake correction, the background memory may be used to compensate for the loss of the image. In the background memory, an image that hardly changes among the changes in the image between frames is selectively written. The background memory can be used to prevent uncovered background when performing motion compensation, or to thin the transmission of background data to compress the amount of transmission data. A compression encoding method based on this method has been proposed by the inventor of the present application and is described in, for example, Japanese Patent Publication No. 7-97754. Also,
By using a memory that is larger than the frame memory as the background memory, it can be used for accumulating images that do not fit within one frame, such as peripheral images due to camera shake and images due to pan and tilt.

FIG. 20 is a schematic diagram for explaining the image data stored in the background memory. For example, when a tennis player playing on the court is photographed, the images that change sequentially and the photographed image are displayed. The background image (specifically, the tennis court) is stored in the background memory. That is, the background memory stores images excluding the moving player. In this case, the area of the background memory is made wider than that of the captured image, and for example, by changing the area of the background memory according to the movement during panning and tilting of the video camera, as in the example of FIG. 20, It is possible to capture a wide range of background images.

FIG. 21 shows yet another embodiment of the present invention which uses a background memory. The components corresponding to those of the embodiment shown in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted. The image signal that has been locally decoded and motion-compensated is input from the addition circuit 10 to the camera shake correction circuit 111. The camera shake correction circuit 111 performs the camera shake correction as described above. That is,
The position of the locally decoded image data is corrected by the camera shake correction signal from the correction signal generation circuit 13. The image signal output from the image stabilization circuit 111 is the comparator 1
It is input to the background memory 112 through 13. The locally decoded image data is a block structure data.

On the other hand, the image data block at the position corresponding to the locally decoded image data is the background memory 11.
2 is read. The comparator 113 calculates a difference value for each pixel between the locally decoded image data block and the image data at the corresponding position from the background memory 112, and the absolute value of this difference value is accumulated in one block. Calculated. The accumulated value of the difference of each block is compared with a predetermined threshold value in the comparator 113. The comparison output of the comparator 13 is input to the update control circuit 114.

The update control circuit 114 determines that the image is a background image when the accumulated value of the differences is smaller than the threshold value. When the background image is determined, the data in the background memory 112 of the block at the position corresponding to the block is updated with the locally decoded image data. When updating, the background memory 112 is gradually added rather than updating the data at once.
Is preferably updated. If the accumulated value of the differences is equal to or larger than the threshold value, the data of the block is determined to include image information other than the background image and is not used for updating the background memory 112. Furthermore, when a memory larger than the frame memory is used as the background memory 112 as in the example of FIG. 20, new background data that is not filled with the data up to the previous frame is sequentially accumulated.

The background memory 11 thus accumulated
By selectively using the data stored in 2 and the data in the normal frame memory to perform motion compensation,
It is possible to compensate for missing images. The image data in the background memory 112 is supplied to the motion vector detection circuit 6'having the configuration shown in FIG.

In contrast to the configuration of the motion vector detection circuit 6 shown in FIG. 7 described above, a difference detection circuit 57 ', an absolute value conversion circuit 58', and an accumulation circuit 59 'are provided, and the accumulation circuit 5'
The output of 9'is, together with the output of the accumulator circuit 59, the judgment circuit 6
It is supplied to 0 '. The decision circuit 60 'outputs the motion vector and the memory flag to the output terminals 61 and 61', respectively.

The difference detection circuit 57 'has a background memory 112.
The difference between the pixel data at the same position and the output data of the current frame memory 53 is detected. The difference value is converted into an absolute value by the absolute value conversion circuit 58 'and accumulated by the accumulation circuit 59'. The determination circuit 60 ′ receives the cumulative calculation force from the accumulation circuit 59 (the cumulative value of the difference between the current frame and the previous frame) and the cumulative calculation force described above from the accumulation circuit 59 ′. Then, the determination circuit 60 ′ outputs a more appropriate motion vector to the output terminal 61 based on one cumulative calculation force. In addition, the determination circuit 60 'outputs the memory flag indicating the selected one of the two cumulative calculation forces to the output terminal 61'.

As shown in FIG. 21, the motion vector is supplied to the motion compensation circuit 7 via the motion vector correction circuit 15, and the memory flag is supplied to the motion compensation circuit 7. These motion vectors and memory flags are transmitted or recorded together with encoded data. One of the image data stored in the frame memory 11 and the background image data stored in the background memory 112 is selected by the memory flag, and motion compensation processing is performed on the selected image data.

Although the encoder side has been described with reference to another embodiment other than the above, the background memory is also provided on the decoder side, and the image of the frame memory and the image of the background memory are arranged on the decoder side according to the memory flag. Is selected similarly to.

[0080]

According to the present invention, the camera shake motion vector is detected from the motion vector generated for motion compensation, so that the scale of hardware can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram of one embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a recording system and a reproducing system in an embodiment of the present invention.

FIG. 3 is a block diagram of an ADRC encoder according to an embodiment of the present invention.

FIG. 4 is a block diagram of an ADRC decoder.

FIG. 5 is a schematic diagram showing an ADRC encoding process.

FIG. 6 is a schematic diagram schematically illustrating motion vector detection.

FIG. 7 is a block diagram of an example of a motion vector detection circuit.

FIG. 8 is a schematic diagram schematically illustrating camera shake motion vector detection.

FIG. 9 is a block diagram of an example of a camera shake correction circuit and a camera shake detection circuit.

FIG. 10 is a schematic diagram used for explaining a process of detecting a camera shake motion vector.

FIG. 11 is a schematic diagram used for explaining a process of detecting a camera shake motion vector.

FIG. 12 is a flowchart used for explaining a process of detecting a camera shake motion vector.

FIG. 13 is a schematic diagram illustrating an example of a process of dividing a screen into macroblocks.

FIG. 14 is a flow chart showing an operation of forming an integrated vector in one embodiment of the present invention.

FIG. 15 is a schematic diagram showing a camera shake correction operation in one embodiment of the present invention.

FIG. 16 is a schematic diagram showing a motion vector correcting operation in the embodiment of the present invention.

FIG. 17 is a block diagram of another embodiment of the present invention.

FIG. 18 is a block diagram of a recording system of still another embodiment of the present invention.

FIG. 19 is a block diagram of a reproducing system according to still another embodiment of the present invention.

FIG. 20 is a schematic diagram used for a schematic description of a background memory used in still another embodiment of the present invention.

FIG. 21 is a block diagram of still another embodiment of the present invention.

FIG. 22 is a block diagram showing the configuration of an example of a motion vector detection circuit in still another embodiment of the present invention.

FIG. 23 is a schematic diagram for explaining a conventional camera shake correction process.

FIG. 24 is a schematic diagram for explaining a conventional camera shake correction process.

[Explanation of symbols]

 5 Subtractor 6 Motion vector detection circuit 7 Motion compensation circuit 12 Hand shake detection circuit 13 Correction signal generation circuit 15 Motion vector correction circuit

Claims (8)

[Claims] 1. The image data of one screen is divided into a plurality of blocks, and a motion vector corresponding to the position of the best matching block is detected from the image data of one to several frames before for each block. In an image signal processing device having a function of compressing image data using, a means for detecting a motion vector of each block, a means for detecting a camera shake motion vector from the motion vector of each block, and a camera shake of an input image signal Vector-based correction means, motion vector correction means for obtaining motion vectors of the respective blocks, motion compensation means for the corrected motion vectors, and shake-corrected image data And means for performing compression processing based on the above-described motion-compensated image data, Image signal processing apparatus characterized by Ranaru. 2. The camera shake vector detecting means includes means for evaluating the activity of image data of each block and means for detecting the same motion vector at spatially discontinuous positions. The image signal processing device according to. 3. The camera shake vector detecting means has means for evaluating the activity of image data of each block and judging means for detecting the presence of a still block, and when the still block is detected, the camera shake correction is not performed. The image signal processing device according to claim 1, wherein 4. The evaluation value table is formed by dividing the image data of one screen into a plurality of blocks, and forming an evaluation value table based on a pixel-by-pixel difference from the image data of the block one to several frames before for each block. In the image signal processing device having the function of detecting the coordinates of the minimum value of, detecting the motion vector based on the coordinates, and compressing the image data using the motion vector, based on the evaluation value table and the motion vector. Means for detecting a camera shake motion vector, means for correcting an input image signal based on the camera shake vector, means for correcting the motion vector using the camera shake motion vector, and motion by the corrected motion vector A means for performing compensation, and means for performing compression processing using the image data that has been subjected to image stabilization and the image data that has been subjected to motion compensation. An image signal processing device comprising: 5. The image data of one screen is divided into a plurality of blocks, and a motion vector corresponding to the position of the best matching block is detected from the image data of one to several frames before for each block, In an image signal recording apparatus having a function of compressing image data by using, a means for detecting a motion vector of each block, a means for detecting a camera shake motion vector from the motion vector of each block, and motion compensation by the motion vector An image signal recording apparatus comprising: a means for performing a compression process, a means for performing a compression process using input image data and motion-compensated image data, and a means for recording the hand-movement motion vector together with the compressed image data. . 6. The evaluation value table is formed by dividing image data of one screen into a plurality of blocks, and forming an evaluation value table based on a pixel-by-pixel difference from image data of a block one to several frames before for each block. In the image signal recording apparatus having the function of detecting the minimum value of the coordinates, detecting the motion vector based on the coordinates, and compressing the image data using the motion vector, the shake motion from the evaluation value table and the motion vector A means for detecting a vector, a means for performing motion compensation using the motion vector, a means for performing a compression process using the input image data and the image data that has been motion compensated, and a camera shake motion vector together with the compressed image data. An image signal recording device comprising: 7. The image data of one screen is divided into a plurality of blocks, and a motion vector corresponding to the position of the best matching block is detected from the image data of one to several frames before for each block, The image data is compressed using, the motion vector is detected from the motion vector, motion compensation is performed by the motion vector, compression processing is performed by the input image data and the motion-compensated image data, and the In an image signal reproducing apparatus for reproducing a recording medium on which the camera shake motion vector is recorded together with an image signal, a decoding means for decoding compressed image data, a means for separating the camera shake motion vector from a reproduced signal, and a decoding means for decoding And a means for performing camera shake correction by the camera shake motion vector. An image signal reproducing device comprising: 8. The evaluation value table is formed by dividing the image data of one screen into a plurality of blocks, and forming an evaluation value table based on a pixel-by-pixel difference from the image data of the block one to several frames before for each block. The minimum value of the coordinate is detected, the motion vector is detected based on this coordinate, the image data is compressed using the above motion vector, and the camera shake motion vector is detected from the evaluation value table and the above motion vector. Motion compensation is performed by using the above-mentioned input image data and the motion-compensated image data to perform compression processing, and an image signal reproducing apparatus for reproducing a recording medium on which a camera-shake motion vector is recorded together with a compressed image signal. Decoding means for decoding data, means for separating the camera shake motion vector from the reproduced signal, and decoding by the decoding means Is supplied image signal, the image signal reproducing apparatus characterized by comprising a means for correcting the camera shake by the hand-shake motion vector.