JP3947316B2 - Motion vector detection apparatus and moving picture encoding apparatus using the same - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an encoding apparatus that encodes a moving image, and more particularly, to a motion vector detection apparatus such as a video recorder or an image transmission / reception apparatus, and a moving image encoding apparatus using the same.
[0002]
[Prior art]
Standardized by ISO / IEC for the purpose of storing image information on digital video discs (DVD), etc., and using it for video encoding systems for digital broadcasts, etc., through high-efficiency encoding processing using digital image compression technology In the case of the MPEG-2 system that supports up to a subset defined by MP @ ML (Main Profile, Main Level), in the PAL system, the maximum spatial resolution is 720 × 576 pixels in the horizontal and vertical directions (NTSC system 720 × 480 pixels), the maximum bit rate is 15 Mbit / second, and the maximum frame rate is 25 frames / second (29.97 frames / second in the NTSC system).
[0003]
An encoding apparatus for encoding a moving image based on such a standard has been proposed. For example, in “1997 IEICE General Conference C-12-33 to 37”, a configuration of a one-chip LSI and an external memory is proposed. Thus, a case where encoding / decoding is realized in real time is shown. By making the real-time encoding device into one chip, it is possible to easily realize a small-sized and lightweight portable video camera device or image transmission device. However, since it is portable and power is supplied by a storage battery, it goes without saying that power saving of the encoding device is essential to achieve long-term use.
[0004]
FIG. 5 is a block diagram showing a conventional moving picture encoding apparatus disclosed in Japanese Patent Application No. 9-236118, in which 1 is an input terminal for image information, 2 is a motion detection circuit, and 3 is a predicted image generation circuit. 4 is a coded image determination circuit, 5 is a coded image selection circuit, 6 is a DCT (Discrete Cosine Transform) circuit, 7 is a quantization circuit, 8 is an inverse quantization circuit, and 9 is an IDCT (Inverse Discrete Cosine Transform) circuit. Reference numeral 10 denotes a decoded image generating circuit, 11 is a variable length encoding circuit, 12 is an encoded information output terminal, 13 and 14 are storage media, and 15 is a storage medium control circuit.
[0005]
In the figure, image information 1 a is input from an input terminal 1. The image information 1a is sequentially input in units of frames or fields, and the input order of the frames or fields is the encoding order. In the following description, it is assumed that an image is encoded in units of frames. Therefore, the “image” in the following is an image (picture) of one frame.
[0006]
Here, in the MPEG standard, a GOP (Group of Pictures) is constituted by a plurality of continuous frames, and images constituting the GOP are divided into the following types according to the type of encoding. . That is, normally, an intra-frame (or field) encoded image (hereinafter referred to as an I image) that is encoded only with its own image information and a past I image or P image are used as reference images, and are forward-looking on the time axis. Predictive encoded images (ie, P images) encoded using motion prediction and past and future I images or P images are used as reference images, and forward and backward motion predictions are used on the time axis. These are classified into three types, that is, bi-predictive encoded images (hereinafter referred to as B images) to be encoded.
[0007]
The GOP configuration is classified according to the M value determined by the number of frames of consecutive B images. For example, when M = 3, the B image, the B image, An I image, a B image, a B image, a P image,... And a B image continue for two frames. The input moving image information is determined by encoding a block composed of 8 × 8 pixels and a macroblock composed of 16 × 16 pixels that are motion vector detection targets, and predictive coding is performed. To do.
[0008]
In FIG. 5, the storage medium 13 holds input image information corresponding to I and P images as reference image information necessary for encoding P and B images under the control of the recording medium control circuit 15. When the P image or B image is encoded, it is reproduced and output as a reference image for motion search in the previous stage of the motion detection circuit 2. Similarly, in the storage medium 14, the local decoded image information 10 a for the I image or P image encoded in the most recent past and reproduced by the local decoding process is controlled under the control of the recording medium control circuit 15. Stored and held, and reproduced and output as a reference image for motion search in the subsequent stage of the motion detection circuit 2 when a P image or B image is encoded.
[0009]
In the motion detection circuit 2, during the predictive encoding of the P image and the B image, the input image information and the decoded image information of the I image or the P image are read as reference images from the storage media 13 and 14 for each macroblock, The motion vector is detected by changing the search range and the search accuracy in multiple stages, and the input image information 1a is output as the image information 2a together with the motion vector and the prediction mode 2b. In this case, image information corresponding to the search area in the storage media 13 and 14 is accessed for each motion search at each stage. Here, the recording medium control circuit 15 accesses the storage medium 14 in accordance with the motion vector information 2c detected by the motion detection circuit 2, and reads the predicted image information 14a.
[0010]
The predicted image generation circuit 3 obtains a difference image between the input image information 2b and the predicted image information 14a in units of macroblocks, generates predicted difference image information 3b and 3c, and outputs them together with the input image information 3a.
[0011]
The encoded image determination circuit 4 and the encoded image selection circuit 5 allow the encoded image determination circuit 4 to generate prediction difference image information 3b (intra macroblock) and input image image information 3a (non-intra macroblock) in units of macroblocks. And the encoded image selection circuit 5 selects the image with the smaller code amount as the encoded image based on the comparison result, and the encoded image information 5a is selected. Output.
[0012]
The encoded image information 5a is subjected to discrete cosine transform in units of blocks by the DCT circuit 6, and the transform coefficient is quantized by the quantization circuit 7 to obtain quantized information 7a. The quantized information 7a is encoded into a bit sequence by the variable length encoding circuit 11 together with the motion vector information and the prediction mode information, and is output from the output terminal 12 as encoded information.
[0013]
On the other hand, the quantization information 7 a from the quantization circuit 7 is processed by the inverse quantization circuit 8 and the IDCT circuit 9, and the prediction difference image information 9 a is reproduced and supplied to the decoded image generation circuit 10. The decoded image generation circuit 10 performs local decoding processing of an I image or a P image in units of macroblocks. When this macroblock is an intra macroblock, the prediction difference image information 9a is converted into the local decoded image information 10a. In the case of a non-intra macroblock, the prediction difference image information 9a is added to the prediction image information 3c generated by the prediction image generation circuit 3, and this is output as local decoded image information 10a. Such locally decoded image information 10 a is stored and held in the storage medium 14.
[0014]
Here, since it is a multi-stage search, it is necessary to access the storage media 13 and 14 for each stage. However, in the case of performing the thinning readout according to the subsample or the motion search with half-pixel accuracy, It is necessary to read out all the pixel information within the search range, and the generation of the prediction image requires information on all the pixels, so access for the final stage motion search and creation of the prediction image By performing access simultaneously, the access frequency is reduced and the efficiency of memory access is improved.
[0015]
In addition, when realizing this moving image encoding apparatus, a single LSI and a memory for each application are integrated into one (for example, a frame memory), instead of using a memory for each chip set and each chip. By doing so, a moving picture coding apparatus with reduced power consumption is obtained.
[0016]
Furthermore, when encoding a B image that is not used as a reference image, it is not necessary to generate the locally decoded image information 10a, so that the processing operations of the inverse quantization circuit 8, the IDCT circuit 9, and the decoded image generation circuit 10 are performed. Is completely idle, so that wasteful power consumption is reduced. Furthermore, even when encoding is performed in real time, each circuit is not always operated, so that it is in an idle state. The power consumption is reduced by stopping the clock supply to the circuit.
[0017]
[Problems to be solved by the invention]
However, a moving image encoding processing apparatus configured by the above-described one-chip LSI and one external memory (frame memory) or the like can reduce the number of accesses by increasing the efficiency of memory access, Even in the encoding process, not all the circuit blocks used for the encoding process are always operated, but the operation of the circuit blocks not included in the encoding process target is stopped in a timely manner according to the type of image to be encoded. Even if the power consumption is reduced by operating the blocks efficiently, power saving cannot be expected if each circuit block operates simultaneously in real-time encoding processing. .
[0018]
The present invention has been made in view of such a point, and the object thereof is to realize further power saving even in the case of performing moving image encoding processing in real time, and to reduce the continuous operation time. It is an object of the present invention to provide a motion vector detection device that can be extended and a moving image encoding device using the same.
[0019]
[Means for Solving the Problems]
In order to achieve the above object, a motion vector detection device according to the present invention is provided with at least a wide range of pixel subsamples. Coarser than The motion vector search is performed with accuracy, and the motion vector coarse search means for outputting the motion vector count result as the motion information, and the vicinity neighborhood area of the motion vector obtained by the motion vector coarse search means in pixel units or A motion vector dense search means for performing motion vector search with high accuracy in half-pixel units, and a prediction difference image is generated by obtaining a difference between the predicted image information indicated by the motion vector detected by the motion vector dense search means and the input image. A predicted image generating means, an image that encodes one of the predicted difference image and the input image; Do An encoded image determination unit that performs determination, and an encoded image selection unit that selectively outputs either the prediction difference image or the input image as an image to be encoded based on a determination result of the encoded image determination unit; , Motion information from the motion vector rough search means Is less than the predetermined value Operation of the motion vector rough search means To stop System control means for controlling.
[0020]
Further, in the motion vector detection device according to the present invention, the system control means includes: (1) When the motion vector rough search means is in an operating state, When the motion information from the motion vector rough search means is smaller than a predetermined value, Judged as an image with little movement do it , Stopping the motion vector rough search means, (2) When the motion vector rough search means is in a stopped state, When the motion information from the motion vector dense search means is larger than a predetermined value, Judge whether the image is moving or not, and determine that the image is moving do it The operation of the motion vector rough search means is restored.
[0021]
The moving picture coding apparatus according to the present invention includes the motion vector detection apparatus, a DCT means for performing a discrete cosine transform on the prediction difference image or the input image selected by the coded image selection means, and an output from the DCT means. Quantization means for quantizing transform coefficients, and quantization information from the quantization means are variable-length encoded together with motion vector information and prediction conditions obtained by the motion vector detection device and generated. The Code information is temporarily stored in a memory and read out. Code remaining Monitor And a variable-length encoding unit that supplies the system control unit.
[0022]
In the moving image encoding apparatus according to the present invention, when the system control unit has the motion vector coarse search unit in a stopped state, the remaining code amount is even if the input image is a moving image. If it is smaller than a predetermined determination value set in advance, the motion vector rough search means is left in a stopped state, and if the remaining code amount is equal to or greater than the determination value, the motion vector rough search is performed. The operation of the means is restored.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing an embodiment of a moving picture coding apparatus according to the present invention. 16 is a motion vector coarse search circuit, 17 is a motion vector fine search circuit, and 18 is a system control circuit. Corresponding portions are denoted by the same reference numerals and redundant description is omitted.
[0024]
This embodiment is a moving image encoding apparatus based on the MPEG2 standard, and as described above, the PAL system is used as a sub-frame corresponding to a subset defined by MP @ ML (Main Profile, Main Level), and the spatial resolution is horizontal. , Up to 720 x 576 pixels in the vertical direction (720 x 480 pixels in the case of the NTSC system), color difference signal of CCIR 601 standard (4: 2: 2 format) as the image format The Assume that an image (4: 2: 0 format) subsampled 2: 1 in the vertical direction is handled. Therefore, the luminance signal is encoded for a maximum of 720 × 576 pixels (720 × 480 pixels in the case of the NTSC system), and the color difference signals are respectively encoded for a maximum of 360 × 288 pixels (the same applies to the NTSC system). The maximum bit rate is 15 Mbit / sec.
[0025]
As the input moving image information 1a, an interlaced image that is an interlaced scanning method or a progressive image that is a progressive scanning method is targeted, and an 8 × 8 pixel block as an encoding target and 16 × as a motion vector detection target. A macroblock having 16 pixels is defined. In addition, when an interlaced image is handled as a prediction method, prediction efficiency is improved by adaptively switching between inter-field prediction or inter-frame prediction. For example, a motion image is predicted by inter-field prediction, and a still image portion is predicted by inter-frame prediction, thereby minimizing a prediction error.
[0026]
It is also stipulated that motion prediction is performed by motion search with half-pixel accuracy.
[0027]
Further, when encoding the prediction error signal, the prediction error signal is DCT-transformed and the transform coefficient is encoded. When DCT is performed on a frame in units of macroblocks (frame DCT), or in the field When the DCT is performed separately (field DCT), the one with the smaller encoded information amount is adaptively selected. This determination may be performed after the respective encoded information amounts are actually obtained, but in general, the determination is made by comparing the power of the high-frequency component of the difference signal to suppress an increase in the amount of calculation. ing.
[0028]
When reproducing an image, a GOP is composed of a plurality of continuous frames as a structure of an encoded image for enabling random access and decoding of the image. In this case as well, the types according to the type of encoding of the images constituting the GOP are classified into three types: I image, P image, and B image. The image sequence in the GOP is represented by an M value. For example, when M = 3, the image input order (F0, F1, F2, F3, F4, F5,...) And image encoding are used. The relationship between the image type (I image, P image, B image) is F0 (B image), F1 (B image), F2 (I image), F3 (B image), F4 (B image) in the order of image input. , F5 (P image),..., And the B image continues for two frames continuously between the preceding and following I and P images or between the P images. Further, when M = 2, there is only one frame of B image between the preceding and succeeding I and P images or between P images, and when M = 1, the frame is composed of only I and P images. On the other hand, in the case of M> 3, the B image continues for (M−1) frames between the preceding and subsequent I and P images or between the P images.
[0029]
Here, as an image encoding condition in this embodiment, a case where M = 3 is described as an image sequence represented by an M value. Further, real-time processing is realized by keeping the encoding processing for each frame within one frame, and the B image at the head of the GOP (F0, F1) is the most similar to those B images of the previous GOP. Prediction based on forward prediction and backward prediction using a near past I image or P image as a reference image, or prediction based only on backward prediction is performed. In this embodiment, in order to simplify the description Assume that prediction is performed only by backward prediction.
[0030]
The operation of this embodiment will be described below.
[0031]
In FIG. 1, moving image information 1a represented in a 4: 2: 2 or 4: 2: 0 format to be encoded is input from an input terminal 1 so that the images are in the encoding order. Here, in this embodiment, the maximum M value is limited to 3. For example, when M = 3, the relationship between the display frame order and the image sequence is F0 (B image), F1 as described above. (B image), F2 (I image), F3 (B image), F4 (B image), F5 (P image), and so on. Further, the frame order of the input image information 1a is that the encoding type of the first two frames F0 and F1 is a B image, and therefore it is necessary to encode the frame F2, which is an I image, prior to these B images. The encoding order is F2 (I image), F0 (B image), F1 (B image), F5 (P image), F3 (B image), F4 (B image), and so on.
[0032]
The input moving image information 1a is input in units of frames or fields corresponding to the encoding order and in units of macroblocks. The storage medium 13 scans and stores image information corresponding to the I image or P image of the image information 1a input in units of macroblocks under the control of the recording medium control circuit 15, and stores and holds the P image or At the time of encoding the B image, image information 13a in a range in which motion search is performed is output with respect to the macroblock to be encoded, using past and future I images or P images as reference images.
[0033]
Here, at the time of encoding the B image, since forward and backward motion prediction is performed on the time axis, two frames of past and future I images or P images are necessary. It has at least a capacity of 2 frames. In addition, when motion search is performed by the motion vector rough search circuit 16, the input image stored and held in the storage medium 13 is used as a reference image. However, if the accuracy of motion search is to be roughened, the storage medium The capacity of 13 can be reduced by at least the amount of coarse search accuracy. For example, if the horizontal direction is coarsened to two pixel accuracy as the search accuracy, the capacity required for the storage medium 13 can be halved. In addition, if only the luminance signal is targeted during motion search, the capacity for storing and holding the color signal can be reduced.
[0034]
FIG. 2 is a block diagram showing the main part of one embodiment of the motion vector detection apparatus of the present invention, that is, the motion vector rough search circuit 16 and its peripheral part in FIG. 1, wherein 19 is an input image holding circuit, 20 is Reference image holding circuit, 21 is a reference image selection circuit, 22 is an absolute difference calculation circuit, 23 is an addition circuit, 24 is a data holding circuit, 25 is a determination circuit, 26 is a position information holding circuit, 27 is a motion vector determination circuit, Reference numeral 28 is a counting circuit, 29 is a data rearrangement circuit, and 30 is a timing control circuit, and parts corresponding to those in FIG.
[0035]
In the figure, a timing control circuit 30 is based on an input image 3a output from the predicted image generation circuit 3 (FIG. 1), an input image holding circuit 19, a reference image holding circuit 20, a position information holding circuit 26, a motion vector. The operation timing of the decision circuit 27, the data rearrangement circuit 29, etc. is controlled.
[0036]
Under such control, the input image 1a from the input terminal 1 (here, a P image or a B image for searching for a motion vector) is temporarily held in the input image holding circuit 19, and together with this, the storage medium control Under the control of the circuit 15, the reference image 13 a for the input image 1 a is read from the storage medium 13, processed by the data rearrangement circuit 29, and held in the reference image holding circuit 20.
[0037]
Thereafter, image information (hereinafter referred to as input MB information) 19a of one macro block of the input image 1a is read from the input image holding circuit 19 and supplied to the difference absolute value calculation circuit 22, and the reference image selection circuit 21. Thus, the image information of one macroblock of the reference image from the reference image holding circuit 20 (hereinafter referred to as reference MB information). 20a Is supplied to the difference absolute value calculation circuit 22 to calculate an absolute value (hereinafter referred to as difference absolute value) 22a for each pixel from the input MB information. These difference absolute values 22a are sequentially supplied to the adder circuit 23 and cumulatively added, and a sum of difference absolute values 22a of all pixels with respect to the input MB information 19a (hereinafter referred to as difference sum of absolute values) 23a is calculated. The first sum of absolute differences 23a for the input MB information 19a is held in the data holding circuit 24.
[0038]
When this processing is finished, the reference image selection circuit 21 reads the next reference MB information 20a whose position in the reference image holding circuit 20 is shifted, and supplies it to the difference absolute value calculation circuit 22 to supply the same input MB information. The difference absolute value 22a is calculated by calculating 19a, and the difference absolute value 22a for all the pixels is added by the adder circuit 23 to obtain the difference absolute value sum 23a. Then, the difference absolute value sum 23a and the difference absolute value sum 24a held in the data holding circuit 24 are compared by the determination circuit 25 to select the smaller value, and the selected difference absolute value sum 25a is stored as data. While being held in the holding circuit 24, it is held in the position information holding circuit 26 as the position information 26a. Here, the position information 26a includes the position of the MB (macroblock) of the input MB information 19a and the position of the MB of the reference MB information 20a of the difference absolute value sum 24a held in the data holding circuit 24 for this position. It represents a vector and is a current motion vector for the input MB information 19a.
[0039]
Thereafter, similarly, the reference image selection circuit 21 reads the reference MB information 20a whose positions are sequentially shifted from the reference image holding circuit 20, and performs the same processing on the same input MB information 19a. Thus, when all the reference MB information 20a is read from the reference image selection circuit 21 and processed, the position information 26a having the minimum difference absolute value sum 23a is obtained as the position information 26. Therefore, the motion vector determination circuit 27 determines the position information 26a at this time as the motion vector 16a for the input MB information 19a at this time, and sends it to the motion vector dense search circuit 17 and the storage medium control circuit 15 (FIG. 1).
[0040]
When the search operation of the motion vector of one input MB information 19a is completed, the other input MB information 19a whose position is shifted is read from the input image holding circuit 19, and a similar motion vector is searched for this. It is. When the same motion vector search is completed for all MBs of the input image 1a held in the input image holding circuit 19, the next input image 1a is input in accordance with an instruction from the timing control circuit 30, and the image is held. A similar motion vector search is performed by being held in the circuit 19.
[0041]
Each time the motion vector determination circuit 27 determines the motion vector 16a, the counting circuit 28 counts this, and the count value of the motion vector 16a with respect to the input MB information 19a is used as the motion information 16b to the system control circuit 18. send. This motion information 16b will be described later.
[0042]
Next, the operation of each part for each image type (I image, P image, B image) in the sequence of images shown in FIG. 3A shows the display order of images, FIG. 3B shows the encoding order, and FIG. 3C shows the storage medium 13.
First, the I image will be described. Here, the frame F2 in FIG. 3 will be described as an example.
[0043]
In an I picture, intra-picture coding (intra macroblock) is performed in units of macroblocks, so that the motion vector coarse search circuit 16, the motion vector dense search circuit 17, and the predicted image generation circuit 3 are other than the processes necessary for image transfer. Therefore, the input image information 1a is output as image information 3a as it is. Since the encoding target is an intra macroblock, the encoded image determination circuit 4 outputs a selection signal 4a indicating this image information 3a as an encoded image. In addition, when the input image is an interlaced image, the encoded image determination circuit 4 determines whether to perform DCT in a frame unit or a field unit as a block configuration, and in a macroblock unit, a high-frequency component of the difference signal And the block configuration selection signal 4b is output. Therefore, the encoded image selection circuit 5 selects the image information 3a based on these selection signals 4a from the encoded image determination circuit 4, And Based on the instruction of the block configuration selection signal 4b, a block is configured by either a frame unit or a field unit, and is output as encoded image information 5a.
[0044]
The encoded image information 5a is subjected to discrete cosine transform in units of blocks by the DCT circuit 6, and the transform system number is quantized by the quantization circuit 7 to generate quantized information 7a. The quantization information 7a is subjected to variable length coding processing by the variable length coding circuit 11 together with coding conditions such as a prediction condition at the time of coding, motion vector information, and a DCT block configuration condition, and is represented by a bit sequence. Encoded information 11a is generated. The encoded information 11a is output from the output terminal 12 as encoded information for the F2 frame.
[0045]
On the other hand, the quantization information 7a output from the quantization circuit 7 is a series of processes by the DCT circuit 6 and the quantization circuit 7 by the inverse quantization circuit 8 and the IDCT circuit 9 based on the encoding condition. The reverse processing is performed, and thereby, the prediction difference image information 9a is obtained. Here, since the decoding process is performed on the frame 2 that is the I image, the decoded image generation circuit 10 outputs the prediction difference image information 9a as the locally decoded image information 10a for the I image, and the recording medium control circuit 15 According to the instruction 15a, it is stored in the storage medium 14.
[0046]
Next, the P image will be described. Here, the frame F5 in FIG. 3 will be described as an example. In the case of a P image, the F2 frame that is the closest past I image is used as a reference image when performing a motion search of the P image.
[0047]
First, the input image information 1a of the frame F5 is supplied to the motion vector rough search circuit 16 in units of macro blocks. Also, the input image information 1a of the P image is scanned and converted into an area other than the area that the F2 frame that is the I image on the storage medium 13 stores and holds, as in the case of the I image. To do.
[0048]
The motion vector coarse search circuit 16 and the motion vector fine search circuit 17 perform motion vector detection by motion search for each supplied macroblock in units of frames and fields. Here, if motion search is performed throughout the entire frame, there is no problem at all. However, since motion search is performed in real time, the search range is limited to the surrounding area of the macroblock or the surrounding macroblocks. The time required for the search is shortened by using the motion vector that is the motion search result for as a prediction vector and limiting it to the peripheral region of the region indicated by the prediction vector.
[0049]
Further, the motion vector rough search circuit 16 performs a rough search with a plurality of pixel accuracy, for example, two pixel accuracy to determine the motion vector 16a.
[0050]
The motion vector dense search circuit 17 narrows the search range to the peripheral region indicated by the motion vector 16a, determines the motion vector 17a by motion search with integer pixel accuracy or half-pixel accuracy, and at the same time, motion search results In accordance with the prediction accuracy, the field prediction or the frame prediction is determined and set as the prediction condition 17b. In this embodiment, in order to simplify the description, the motion vector dense search circuit 17 performs motion search with half-pixel accuracy, but of course, the present invention is not limited to this.
[0051]
For each macroblock, as the processing of the motion vector rough search circuit 16, image information that matches the search accuracy and the search range is read from the storage medium 13 in the frame F2 of the I image. In this case, although a rough search is performed by thinning out the reference pixels, the motion vector 16a is output by performing a search over a wider range within a range that can be accommodated in the processing time.
[0052]
Since the motion vector dense search circuit 17 performs motion search with half-pixel accuracy, a reference image with half-pixel accuracy is required. For this reason, the motion vector 16a detected by the motion vector coarse search circuit 16 is sent to the storage medium control circuit 15, and the motion vector fine search circuit 17 responds to the instruction 15a from the recording medium control circuit 15 according to the motion vector 16a. The image information of an area of 18 × 18 pixels which is larger by a half pixel than the macroblock size at the position indicated by the motion vector 16a on the storage medium 14 (the position where the locally decoded image of the frame F2 is stored and held) is referred to as the reference image 14a. And the reference image 14a having integer pixel accuracy is interpolated to generate a reference image having half pixel accuracy. Then, the motion vector dense search circuit 17 performs a motion search with half pixel accuracy which is the search accuracy of the MPEG2 standard on the obtained reference image with half pixel accuracy, and specifies the motion vector 17a and the prediction condition 17b. To do. In this case, the motion vector search accuracy at the first stage may be supplemented by further expanding the size of the reference image.
[0053]
The predicted image generation circuit 3 obtains a difference between the input image information 1a and the predicted image information indicated by the motion vector 17a for each macroblock, generates and outputs the predicted difference images 3b and 3c, and outputs the input image information 1a. Output as image information 3a. Further, although not shown, the obtained half-pixel precision motion vector 17 a and prediction condition 17 b are supplied to the variable length coding circuit 11.
[0054]
The encoded image determination circuit 4 compares the prediction difference image information 3b and the image information 3a, determines an image with a smaller code amount when the image information is encoded as an encoded image, and determines the determination result. The selection signal 4a shown is output. Here, the selection signal 4a is an intra macroblock when the image with the smaller code amount is the image information 3a, and is a predicted encoded image (non-intra macro when the image is the prediction difference image information 3b. If the predicted image and the input image are completely matched by motion prediction, the image is not required to be encoded (skip macroblock). Further, when the input image is an interlaced image, the block configuration is determined by determining the minimum condition by comparing the amount of coding for each macroblock unit to determine whether DCT is performed in frame units or field units. And a block configuration selection signal 4b for instructing this is output.
[0055]
Therefore, the encoded image selection circuit 5 selects either the image information 3a or the prediction difference image information 3b based on the selection signal 4a, and the image information selected based on the instruction of the block configuration selection signal 4a. And is output as encoded image information 5a.
[0056]
This encoded image information 5a is converted into quantized information 7a by the DCT circuit 6 and the quantizing circuit 7 in the same manner as in the case of the I image described above, and the variable length coding circuit 11 performs prediction at the time of encoding. Along with encoding conditions such as conditions, motion vector information, and DCT block configuration conditions, variable-length encoding processing is performed to generate encoded information 11a represented by a bit sequence. The encoded information 11a is output from the output terminal 12 as encoded information for the P image of the frame F5.
[0057]
On the other hand, the quantization information 7a output from the quantization circuit 7 is subjected to inverse quantization processing by the inverse quantization circuit 8 and the IDCT circuit 9 based on the encoding conditions, similarly to the processing in the case of the I image. And the inverse DCT process is performed and the prediction difference image information 9a is reproduced. The decoded image generation circuit 10 adds the prediction difference image information 9a and the prediction image information 3c of the corresponding block from the prediction image generation circuit 3 in the case of a non-intra macroblock based on the above-described encoding conditions. Thus, the local decoded image information 10a is generated, and in the case of an intra macroblock, the prediction difference image information 9a becomes the local decoded image information 10a as it is, and in the case of a skip macroblock, the prediction difference image information 10a is predicted. The image information 3c becomes the local decoded image information 10a as it is, and is stored and held in the storage medium 14 according to the instruction 15a of the recording medium control circuit 15 as a local decoded image for the frame F5.
[0058]
next, B The image will be described. Here, the frames F0 and F3 in FIG. 3 will be described as an example. This B image encoding uses forward and backward bidirectional motion prediction on the time axis using past and future I or P images as reference images. . In the case of the F0 frame, as described above, since this is the first B image of the GOP, the prediction direction is only backward prediction. Therefore, the frame F2, which is a future I image, is used as a reference image when performing motion search. As a basic operation of predictive encoding in this case, processing similar to that for the P image of the frame F5 is performed.
[0059]
Therefore, in the case of the F0 frame, the image information 1a of the F0 frame is sequentially input to the motion vector rough search circuit 16. In this case, in the motion vector rough search circuit 16, since the prediction direction is only backward prediction, the image information stored and held in the storage medium 13 that is the input image of the frame F2 as the reference image is also stored in the motion vector dense search. The search circuit 17 uses the image information of the frame F2 stored and held in the storage medium 14, which is a locally decoded image of the frame F2, respectively, and performs the motion search with the half-pixel precision motion vector 17a and the frame prediction or the field prediction. Any one of the prediction conditions 17b is specified.
[0060]
Here, since processing for forward prediction becomes unnecessary, processing for forward prediction such as memory access processing and motion search processing for capturing a reference image may be stopped. In addition, when the prediction direction is limited to only backward prediction in the first B image of the GOP, a wide range of motion search is performed even for the B image by making it the same as the prediction processing for the P image. Also good.
[0061]
On the other hand, in the case of the B image of the frame F3, forward prediction and backward prediction are performed, respectively. In the case of forward prediction, the frame F2 is used as a reference image, and in the case of backward prediction, the frame F5 is used as a reference image. . The motion vector rough search circuit 16 reads image information matching the search range from the input image of the frame F2, which is a reference image, from the storage medium 13, and determines a motion vector 16a by motion search using the reference image.
[0062]
Further, the motion vector dense search circuit 17 reads out the image information of the frame F2 stored and held in the storage medium 14 which is a locally decoded image of the frame F2 based on the motion vector 16a as a reference image 14a, A half-pixel precision reference image is generated by interpolating between them, and a motion search is performed on the half-pixel precision reference image to specify a half-pixel precision motion vector 17a and a prediction condition 17b for forward prediction. Similarly, in the case of backward prediction, the half-pixel motion vector 17a and the prediction condition 17b are specified by the same process as in the case of forward prediction, using the frame F5 as a reference image.
[0063]
Here, by making the motion search range of the motion vector coarse search circuit 16 narrower than in the case of the P image, the backward prediction and the forward prediction are completed within the same frame.
[0064]
In the predicted image generation circuit 3, in the case of the frame F0, only the backward predicted image is used as the predicted image, the difference between the input image information 1a and the predicted image information is obtained in units of macroblocks, and the obtained predicted difference image information 3b, 3c And input image information 3a are output. In the case of the frame F3, the difference between the input image information 1a and the predicted image information is obtained for each macroblock. Here, there are three types of prediction image information of two types at the time of forward prediction and at the time of backward prediction, and bidirectional prediction image information obtained by interpolation from the respective prediction images at the time of forward prediction and at the time of backward prediction. For each predicted image information, a difference from the input image information 1a is calculated, and a sum of absolute values (hereinafter referred to as an absolute value difference sum) is obtained as a difference value for each pixel in the macroblock. Of these three types of predicted image information, the smallest absolute value difference sum is determined as the prediction direction and the predicted difference image, and the predicted difference image information 3b and 3c and the input image information 3a are output.
[0065]
The encoding process of the B image by the encoded image determination circuit 4, the encoded image selection circuit 5, the DCT circuit 6, the quantization circuit 7, and the variable length encoding circuit 11 is the same as the case of the above P image, Coding information F0 or F3 for F0 or frame F3 is generated and output from the output terminal 12.
[0066]
The processing procedure for each image type (I image, P image, B image) has been described above. Next, the encoding processing of this embodiment will be described. However, in order to simplify the description, it is assumed that the input image has a frame structure.
[0067]
The motion vector rough search circuit 16 performs a rough search of motion vectors for each macroblock of the image, and counts the generated motion vectors. Counting The result is output as motion information 16b. Here, an image in which the input image 1a almost coincides with the entire reference image ( Less than Since this is a still image type S image), there is no motion between frames. not exist It will be. In addition, in the case of an image with high motion (hereinafter referred to as a moving image type M image), the motion vector is can get There will also be macroblocks. Also, there are some movements, but most movements No The image is hereinafter referred to as a quasi-still image type T image. In the case of such a quasi-still image type T image, there are macroblocks in which a motion vector cannot be obtained in this motion portion. As a result, in the still image type S, the value of the motion information 16b is small The value of the motion information 16b increases as the motion part increases. big It will become. Therefore, by comparing the motion information 16b with a predetermined value, the image type of the input image can be roughly divided into a moving image type M, a quasi-still image type T, and a still image type S.
[0068]
The encoded image determination circuit 4 determines whether the image to be encoded is the input image 3a from the predicted image generation circuit 3 or the predicted difference image 3b. It can be used as a standard for determining whether it is M, quasi-still picture type T or still picture type S. This is output from the encoded image determination circuit 4 as determination information 4c.
[0069]
Furthermore, the moving image encoding apparatus determines the encoding condition by a fixed bit rate that fixes the generated code amount per unit time or a variable bit rate that adjusts the generated code amount together with the complexity of the image. The long encoding circuit 11 temporarily stores the generated code information in a built-in memory and reads it so as not to exceed a predetermined capacity. On the other hand, since the amount of code for an image with little motion is reduced, it is compensated by adding a redundant code to the stream in order to keep the remaining code amount in the memory above a predetermined level. Then, the variable length encoding circuit 11 monitors the remaining code amount in the memory and outputs the information as the remaining code amount 11b. Here, when a moving image type M image is encoded, the amount of code is large and the amount of code stored in the memory increases, but when encoding a still image type S image, the amount of code is small. The amount of code stored in the memory decreases, and in the case of a quasi-still image type T image, the amount of code stored in the memory may increase or decrease. Therefore, the remaining code amount 11b can be used as a guide for discriminating the image types M, T, and S.
[0070]
Next, the operation of the system control circuit 18 in FIG. 1 will be described with reference to FIG.
[0071]
Now, as shown in FIG. 4A, images arranged in the order of display: B0, B0, P1, B2, B3, P4, B5, B6, P7, B8, B9, PA,. As shown in (B),..., P1, B0, B0, P4, B2, B3, P7, B5, B6, PA, B8, B9,. Suppose that
[0072]
Here, for the P1 image, a motion vector search is performed using an I or P image (not shown) immediately before that as a reference image, and for the B0 and B0 images, the P1 image and the previous P or I image are searched. For the P4 image, the motion vector is searched for the P1 image as the reference image, and for the B2 and B3 images, the motion vector is searched for using the P1 image and the P4 image as the reference images. The B8 and B9 images are searched for motion vectors using the P7 image and the PA image as reference images.
[0073]
Further, in such an input image 1a, there is almost no movement in the entire image type, that is, a moving image type M that moves almost entirely according to a change in image content, and a quasi-still picture type T that moves only partially. Classification into still image type S. In FIG. 4B, images of P1, B0, B0, PA, B8, and B9 are moving image types M, and images of B3 and P7 are images of quasi-still image types T, P4, B2, B5, and B6, respectively. Are still image types S, respectively.
[0074]
The system control circuit 18 uses the motion information 16a from the motion vector coarse search circuit 16, the determination information 4c from the encoded image determination circuit 4, and the code remaining amount 11b from the variable length encoding circuit 11, thereby using the input image 1a. Is an image type M, T, or S, and a control signal 18a corresponding to the determination result is generated to control ON / OFF of the motion vector rough search circuit 16. Here, the OFF state refers to a state in which no power supply voltage is supplied.
[0075]
Now, as shown in FIG. 4B, if the input image 1a is a P1 image and the motion vector coarse search circuit 16 is in the ON state, the motion vector coarse search circuit 16 has the storage medium 13 as described above. A reference image is read out from this, and a motion vector rough search is performed on the P1 image. At the same time, the number of motion vectors generated is counted, and at the end of this search, the count result is used as motion information 16b for system control. Send to circuit 18. The system control circuit 18 determines that the P1 image is an image of the moving image type M based on the motion information 16b, and also refers to the remaining code amount 11b from the variable length encoding circuit 11, and FIG. As indicated by an arrow from the “P1” portion of (B), a control signal 18a for turning on the motion vector coarse search circuit 16 is output. As a result, the motion vector rough search circuit 16 is maintained in the ON state, and then the next B0 image is input as the input image 1a to perform a rough search of the motion vector, and the motion vector is counted by searching. When this search is completed, the count result is sent to the system control circuit 18 as motion information 16b.
[0076]
Similarly, every time the system control circuit 18 determines that the motion vector coarse search image from the motion information 16b and the remaining code amount 11b is an image of the moving image type M, the motion vector rough search circuit 16 is used for the next image. In the ON state, the motion vector is roughly searched and counted.
[0077]
Thereafter, as shown in FIG. 4B, the motion vector coarse search circuit 16 performs a coarse search of the motion vectors of the P4 image and counts them, and the system control circuit 18 uses the motion information 16b and the remaining code amount 11b. If the P4 image is determined as an image of the still image type S, the system circuit 18 moves the motion vector coarse search circuit 16 as indicated by an arrow from the “P4” portion of FIG. 4B in FIG. A control signal 18a for turning off is output. As a result, the motion vector rough search circuit 16 is set to the OFF state, and is kept OFF until the next time the control signal 18a instructing the ON state is supplied. As a result, as shown by hatching in FIG. 4D, the motion vector coarse search circuit 16 is applied to the images of B2, B3, P7, B5, and B6 that are still image types S and quasi-still image types T. Is maintained in the OFF state, and the motion vector is not roughly searched.
[0078]
As described above, even in the case of the still image type S image or the quasi-still image type T, in the case of an image with few moving parts, the arrangement relationship of the macroblocks between the image and its reference image is almost one to one. It is possible to search for a motion vector immediately within a narrow search range. For this reason, the motion vector rough search circuit 16 is not indispensable for such an image type image, and as described above, even if it is set to the OFF state, it does not pose any problem.
[0079]
On the other hand, during the period in which the motion vector rough search circuit 16 is held in the OFF state, the system control circuit 18 takes in the remaining code amount 11b from the variable length coding circuit 11 and monitors the remaining code amount in the memory. For each input image 1a, a request signal 18b is sent to the encoded image determination circuit 4 to acquire the determination information 4c, and an image obtained by searching for a motion vector by the motion vector dense search circuit 17 is a moving image type M, a quasi-still image. It is determined whether the type is T or still image type S, and a control signal 18a corresponding to the determination result is output. Here, the motion vector coarse search circuit 16 is held in the OFF state during the period of the images B2, B3, P7, B5, and B6.
[0080]
Further, since the previous PA image is also determined that the previous B6 image is a still image type S image, the motion vector rough search circuit 16 is held in the OFF state and the motion vector is not roughly searched. However, since the determination information 4c from the encoded image determination circuit 4 for the PA image represents the moving image type M, the system control circuit 18 switches the motion vector rough search circuit 16 to the ON state by the control signal 18a. . As a result, from the B8 image, the system controller 18 next causes the motion vector coarse search circuit 16 to operate. OFF Until the control signal 18a is output, the motion vector coarse search circuit 16 operates to repeat the coarse search of the motion vectors of each image and the count of the searched motion vectors.
[0081]
As shown in FIG. 4C, when the motion vector rough search circuit 16 is held in the OFF state, even if the determination information 4c from the encoding determination circuit 4 does not indicate the moving image type M, When the remaining code amount 11b from the variable length coding circuit 11 indicates that the remaining code amount in the memory exceeds the threshold value, the system control circuit 18 receives the input of the input image 1a of the moving image type M. It is determined that the motion vector rough search circuit 16 is turned on.
[0082]
Similarly, when the motion vector rough search circuit 16 is held in the OFF state, if the determination information 4c from the encoding determination circuit 4 indicates the moving image type M, the system as described above. The control circuit 18 normally switches the motion vector coarse search circuit 16 to the ON state. At this time, the control circuit 18 further indicates that the remaining code amount in the memory of the variable length coding circuit 11 is smaller than the above threshold value. When 11b represents, the motion vector rough search circuit 16 is held in the OFF state.
[0083]
As described above, in this embodiment, in the moving image encoding apparatus such as the MPEG1 / MPEG2 standard, not all the circuit blocks used for moving image encoding processing are always operated, Depending on the conditions, the operation of the circuit block that is not the target of the encoding process is stopped in a timely manner, and even if the target image is an image equivalent to a still image or if there is movement only in a part of the area in the frame, By monitoring the amount of generated code, the motion vector coarse search circuit is stopped while minimizing the deterioration of the image quality. Power saving can be easily achieved by stopping the motion vector coarse search circuit during the digitization process, not only for stationary use, but especially for mobile phones with built-in power sources such as storage batteries Video rate compression as well as such as video recorders and image transmitting / receiving apparatus becomes suitable to a playback device.
[0084]
In addition, even if a single-chip LSI is used, whether it is a built-in or external memory such as a frame memory, according to this embodiment, the access to the frame memory for the reference image is stopped by It is possible to reduce the occurrence of useless fluttering on the control line and the control signal line, and it is possible to easily reduce not only power consumption but also power supply noise and radiation noise.
[0085]
In this embodiment, only the case where M = 3 is shown as an image sequence having a GOP structure, but it is needless to say that the same effect can be obtained even when M = 2 and 1. Further, although only the case where DCT is used as the encoding process and decoding process in this embodiment is shown, of course, the present invention is not limited to this, and other encoding methods and decoding methods are used. However, it goes without saying that the same effect can be obtained by applying the present invention.
[0086]
In this embodiment, only the case where the operation of the motion vector coarse search circuit is controlled in units of frames is shown. However, the present invention is not limited to this, and it is performed in units of macroblocks, fields, GOPs or multiple frames. Needless to say, the same effect can be obtained. In addition, since the encoding process is performed in real time, it is possible to prevent excessive control by using information of a plurality of past frames.
[0087]
Furthermore, in this embodiment, the case where processing is performed by hardware has been described, but of course, the present invention is not limited to this, and in the case of software, the CPU occupancy rate can be reduced by reducing the arithmetic processing required for motion vector rough search processing. This is an effective means for reduction.
[0088]
Further, in this embodiment, the case where the result of the motion vector coarse search is used as the motion vector count information has been described, but there is no problem even if the result of the motion vector dense search is used. However, it is possible to increase the processing speed in real time by using the result of the motion vector rough search detected in the initial stage.
[0089]
【The invention's effect】
As described above, according to the present invention, when applying to the MPEG1 / MPEG2 standard for moving images taken by a home video camera as a target image to be compressed and encoded in real time, the image is taken. By setting the motion vector search range that matches the characteristics of the motion of the target object, it is possible to easily suppress the deterioration of motion vector detection accuracy, and when the target image is an image equivalent to a still image or a frame Even if there is motion only in a part of the area, monitoring the amount of generated code can stop the motion vector detection circuit while minimizing image degradation, although there is an increase in the amount of code Therefore, when applied to a portable image processing device with a built-in power supply, etc., continuous operation for the reduced power consumption is possible. Prolonged becomes possible between.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an embodiment of a moving picture encoding apparatus according to the present invention.
FIG. 2 is a block diagram showing a main part of an embodiment of a motion vector detection device according to the present invention used in FIG. 1;
FIG. 3 is a timing chart showing the relationship between the input image and the storage timing in the storage medium in the embodiment shown in FIG. 1;
FIG. 4 is a timing chart for explaining the operation of the embodiment shown in FIG. 1;
FIG. 5 is a block diagram illustrating an example of a conventional moving picture encoding apparatus.
[Explanation of symbols]
1 Image information input terminal
3. Predictive image generation circuit
4. Encoded image determination circuit
5 Coded image selection circuit
6 DCT circuit
7 Quantization circuit
8 Inverse quantization circuit
9 IDCT circuit
10 Decoded image generation circuit
11 Variable length coding circuit
12 Encoding information output terminal
13,14 storage media
15 Storage medium control circuit
16 Motion vector coarse search circuit
17 Motion vector dense search circuit
18 System control circuit

Claims (4)

For each frame or field image, for each frame or field image, frame or intra-field coding that encodes only the own image information, and forward motion prediction on the time axis using a previously coded image as a reference image. Forward-predictive coding that uses and encodes past and future frames or intra-field coded images or forward-predicted coded images as reference images using forward and backward motion prediction on the time axis In a motion vector detection device for performing an encoding process according to any encoded image condition of bidirectional predictive encoding
A motion vector rough search means for performing a motion vector search at least over a wide range and with a coarser accuracy than a pixel sub-sample, and outputting a count result of the searched motion vectors as motion information;
A motion vector fine search means for performing a motion vector search with high accuracy in a pixel unit or a half pixel unit in the vicinity neighborhood of the motion vector obtained by the motion vector rough search means;
A predicted image generation unit that generates a predicted difference image by obtaining a difference between predicted image information indicated by the motion vector detected by the motion vector dense search unit and an input image;
An encoded image determination means for determining whether one of the prediction difference image and the input image is an image to be encoded;
Based on the determination result of the encoded image determination unit, an encoded image selection unit that selectively outputs either the prediction difference image or the input image as an image to be encoded;
And a system control unit that controls to stop the operation of the motion vector coarse search unit when the motion information from the motion vector coarse search unit is smaller than a predetermined value . In claim 1,
The system control means includes
(1) When the motion vector rough search means is in an operating state and the motion information from the motion vector rough search means is smaller than a predetermined value, it is determined that the image has almost no motion, and the motion vector Stop the operation in the coarse search means,
(2) When the motion vector coarse search means is in a stopped state, whether or not the image on which the motion vector fine search is performed by the motion vector fine search means based on the determination information of the encoded image determination means is an image with motion When the motion vector is determined to be an image having motion, the motion vector coarse search means is caused to return the motion vector detecting device. The motion vector detection device according to claim 1 or 2,
DCT means for performing discrete cosine transform on the prediction difference image or the input image selected by the encoded image selection means;
Quantization means for quantizing the transform coefficient output from the DCT means;
The quantization information from the quantization means is variable-length encoded together with the motion vector information obtained by the motion vector detection device and the prediction condition, and the generated code information is temporarily stored in a memory and read out, An encoding device comprising: variable-length encoding means for monitoring the remaining code amount of the memory and supplying the code to the system control means. In claim 3,
When the motion vector rough search means is in a stopped state, the system control means may be an image with motion,
(3) If the remaining code amount is smaller than a predetermined determination value set in advance, the motion vector rough search means is left in a stopped state,
(4) When the remaining code amount is equal to or greater than the determination value, the operation of the motion vector rough search means is restored.

Images