JP2009005417A - Signal encoding method, signal encoding apparatus and signal recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform encoding, approximately in real time, at the optimal encoding rate according to complexity of an image in response to input of video signals, and further to perform encoding in such a manner as to reflect characteristics of the visual sense of the human being. <P>SOLUTION: An image analysis circuit 60 determines information on picture characteristics such as luminance of an image of an input video signal, chromaticity, flatness, a moving amount of a moving image, a differential between images and the like and sends it to an encoding control circuit 30. The encoding control circuit 30 relates encoding difficulty obtained based on the information on picture characteristics form the image analysis circuit 60 and an encoding bit rate based on a predetermined relational expression or based on a predetermined table and sends a resultant encoding bit rate to an encoder circuit 40. The encoder circuit 40 encodes the input video signal in accordance with the encoding bit rate obtained for every pre-set time interval to produce encoded data. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a signal encoding method, a signal encoding apparatus, and a signal recording medium, and in particular, a system for encoding and recording a moving image video signal on a recording medium such as an optical disk, a magnetic disk, and a magnetic tape, and transmission The present invention relates to a signal encoding method, a signal encoding device, and a signal recording medium used in a system or the like that transmits a video signal of a moving image through a channel.

  Conventionally, for example, a video transmission system such as a video conferencing system or a videophone system transmits a video signal to a remote location, and a video signal is recorded on an image recording medium such as an optical disk, a magnetic disk, or a magnetic tape. In a system for reproducing a video signal of a moving image, etc., in order to efficiently use a transmission path (or image recording medium), so-called high-efficiency encoding is performed on the video signal using line correlation or frame correlation of the video signal. The system reduces the redundancy in the space axis direction and the time axis direction and transmits only significant information to increase transmission efficiency.

For example, in the encoding process in the spatial axis direction (hereinafter referred to as intra-frame encoding process), for example, line correlation of the video signal is used, and the times t ₁ , t ₂ , t ₃ ,... Shown in FIG. When transmitting each image PC1, PC2, PC3... Constituting the moving image in FIG. 1, the image data to be transmitted is encoded one-dimensionally within the same scanning line, for example, or the image is divided into a plurality of blocks. Then, the image data of each block is two-dimensionally encoded to perform data compression and improve transmission efficiency.

  Further, in the encoding process in the time axis direction (hereinafter referred to as inter-frame encoding process), for example, by so-called predictive encoding using the inter-frame correlation of the video signal, that is, as shown in FIG. Image data PC12, PC23..., Which are image data differences (so-called prediction errors) for the corresponding pixels between the images PC1 and PC2, PC2 and PC3..., Are obtained, and these image data PC12, PC23. By transmitting the data, data compression is performed and transmission efficiency is improved.

  Thus, it is possible to transmit a video signal with a remarkably small amount of data compared to the case of transmitting all image data of the images PC1, PC2, PC3.

  Further, in the predictive coding in the above-described interframe coding processing, motion compensation prediction is used, for example, in units of macroblocks in order to further increase the efficiency. That is, for example, when a person in the center of the screen moves, the movement of the moving object in the screen is detected, and the position of the image data used for prediction is corrected in the previous image by the amount of the movement. Encoding efficiency can be improved by performing encoding. However, a lot of data still needs to be sent to the part where the object has moved and appeared from behind. Therefore, encoding efficiency can be further improved by performing motion compensation prediction not only in the above-described direction but also in the backward direction or a combination of both.

  Specifically, as shown in A of FIG. 20, the frame data F0, F1, F2, and the 0th, 1st, 2nd, 3rd,..., Frame data of the video signal VD of the moving image to be transmitted. In the macro block of F3..., When there is a change in the image represented by the motion vectors X0, X1, X2, X3. Every second frame), that is, the second, fourth,... Frames are designated as interpolation frames, and transmission interpolation is performed with respect to these interpolation frames by predetermined interpolation frame processing as shown in FIG. Frame data F2X, F4X,... Are generated. Further, the frame data F1, F3,... Are subjected to a predetermined encoding process for the remaining non-interpolated frames to generate transmission non-interpolated frame data F1X, F3X,.

  For example, the difference SP2 (prediction error) between the frame data F3 and the frame data F2 subjected to motion compensation, the difference SP3 between the frame data F1 and the frame data F2 subjected to motion compensation, and the frame data F1 and F3 subjected to the motion compensation are interpolated. The difference SP4 between the obtained frame data and the frame data F2 is obtained for each macroblock, and the difference between the macroblock SP1 of the frame data F2 and these differences is compared. Then, among these data SP1 to SP4, data with the smallest data generation amount is set as transmission interpolation data F2X in units of macroblocks, and transmission interpolation data F4X... For each interpolation frame is generated in the same manner. Further, the non-interpolated frame data F1, F3,... Are subjected to, for example, DCT conversion processing, variable length coding processing, and the like to generate transmission non-interpolated frame data F1X, F3X,.

  These transmission non-interpolated frame data F1X, F3X,... And transmission interpolation data F2X, F4X,..., Together with the motion vectors X0, X1, X3. To transmit.

  On the other hand, the receiving-side apparatus sends transmission data DATA (transmission non-interpolated frame data F1X, F3X..., Transmission interpolation data F2X, F4X..., Motion vectors X0, X1, X3. ) Is subjected to a decoding process corresponding to the encoding process on the transmission side to reproduce the frame data F0, F1, F2, F3. As a result, encoding efficiency can be further improved by performing motion compensation prediction not only in the forward direction but also in the backward direction or a combination of both.

  Here, an image encoding device and an image decoding device having the above-described functions will be described with reference to FIG.

  The image encoding apparatus 70 shown in FIG. 21 converts a preprocessing circuit 71 that separates an input video signal VD into a luminance signal and a color difference signal, and converts the luminance signal and the color difference signal from the preprocessing circuit 71 into digital signals, respectively. Analog / digital (hereinafter referred to as A / D) conversion circuits 72a and 72b; a frame memory group 73 for storing luminance data and color difference data (hereinafter referred to as image data) from the A / D conversion circuits 72a and 72b; A format conversion circuit 74 that reads image data from the frame memory group 73 according to a block format, and an encoder 75 that performs high-efficiency encoding on the block image data from the format conversion circuit 74 are provided.

  The preprocessing circuit 71 separates the input video signal VD into a luminance signal and a color difference signal, and the A / D conversion circuits 72a and 72b convert the luminance signal and the color difference signal into luminance data and color difference data each consisting of 8 bits. The frame memory group 73 stores these luminance data and color difference data.

  The format conversion circuit 74 reads image data (luminance data, color difference data) stored in the frame memory group 73 in accordance with the block format, and the encoder 75 reads the read image data by predetermined high-efficiency encoding. Encode and output a bitstream.

  The bit stream is supplied to the image decoding apparatus 80 via a transmission path 90 and a transmission medium 90 including an image recording medium such as an optical disk, a magnetic disk, and a magnetic tape.

  As shown in FIG. 21, the image decoding apparatus 80 includes a decoder 81 corresponding to the encoder 75, a format conversion circuit 82 for converting image data reproduced by the decoder 81 into a frame format, A frame memory group 83 for storing image data from the format conversion circuit 82; D / A conversion circuits 84a and 84b for converting luminance data and color difference data read from the frame memory group 83 into analog signals; And a post-processing circuit 85 that generates an output video signal by mixing luminance signals and color difference signals from the / A conversion circuits 84a and 84b.

  The decoder 81 decodes the bit stream by decoding corresponding to the high-efficiency encoding of the encoder 75 and reproduces the image data in the block format, and the format conversion circuit 82 converts the image data into the frame format. And stored in the frame memory group 83.

  The D / A conversion circuits 84a and 84b convert the luminance data and color difference data read from the frame memory group 83 into luminance signals and color difference signals, respectively. The post-processing circuit 81 mixes these luminance signals and color difference signals. Thus, an output video signal is generated.

  Specifically, the pre-processing circuit 71 and the A / D conversion circuits 72a and 72b convert the luminance signal and the color difference signal into digital signals as described above, and the number of pixels in the vertical and horizontal directions with respect to the color difference signal. After reducing the data amount so as to be ½ of the luminance signal, time axis multiplexing is performed, and the obtained luminance data and color difference data are supplied to the frame memory group 73.

  Then, as described above, the luminance data and the color difference data are read from the frame memory group 73 according to the block format. That is, for example, image data for one frame is divided into N slices as shown in FIG. 22A, and each slice includes M macroblocks as shown in B of FIG. Each macroblock is made up of luminance data Y [1], Y [2], Y of four luminance blocks adjacent to each other vertically and horizontally, as shown in C of FIG. [3], Y [4] and color difference data Cb [5], Cr [6] of a color difference block composed of 8 × 8 pixels in a range corresponding to these four luminance blocks. From the frame memory group 73, image data continues in units of macroblocks in the slice, and Y [1], Y [2], Y [3], Y [4], Cb [5], in the macroblock. Luminance data and color difference data are read out so as to be continuous in the order of Cr [6]. The image data read according to the block format in this way is supplied to the encoder 75.

  As shown in FIG. 23, the encoder 75 includes a motion vector detection circuit 101. The motion vector detection circuit 101 detects a motion vector of image data supplied in a block format in units of macroblocks. That is, the motion vector detection circuit 101 detects the motion vector of the current reference image in units of macroblocks using the front original image and / or the rear original image stored in the frame memory group 102. Here, in the detection of the motion vector, the motion vector is determined so that the absolute value sum of the inter-frame difference in the macro block unit is minimized. The detected motion vector is supplied to the motion compensation circuit 113 and the like, and the sum of absolute values of inter-frame differences in units of macroblocks is supplied to the intraframe / forward / backward / bidirectional prediction determination circuit 103.

  Based on this value, the intra-frame / forward / backward / bidirectional prediction determination circuit 103 determines the prediction mode of the reference block, and based on the determined prediction mode, the intra-frame / front / backward / backward / The predictive coding circuit 104 is controlled so as to switch bidirectional prediction. The predictive coding circuit 104 includes adder circuits 104a, 104b, and 104c and a changeover switch 104d. The predictive encoding circuit 104 predicts the input image data itself in the intra-frame encoding mode and the predictive encoding circuit in the forward / backward / bidirectional prediction mode. A difference for each pixel of input image data with respect to an image (hereinafter referred to as difference data) is selected, and the selected data is supplied to the DCT circuit 105.

  The DCT circuit 105 performs DCT conversion of input image data or difference data in units of blocks using the two-dimensional correlation of the video signal, and supplies the obtained coefficient data to the quantization circuit 106.

  The quantization circuit 106 quantizes the coefficient data using a quantization step size (quantization scale) determined for each macroblock or slice, and the obtained quantization data is a variable length coding (VLC) circuit 107. And supplied to the inverse quantization circuit 108. By the way, the quantization step size used for the quantization is determined to a value that does not cause the transmission buffer memory 109 to fail by feeding back the remaining buffer capacity of the transmission buffer memory 109, which will be described later. This quantization step size is also a variable length. This is supplied to the encoding circuit 107 and the inverse quantization circuit 108.

  The variable length coding circuit 107 performs variable length coding of the quantized data together with the quantization step size, the prediction mode, and the motion vector, and supplies the data to the transmission buffer memory 109 as transmission data.

  The transmission buffer memory 109 temporarily stores the transmission data, and then reads out the transmission data at a constant bit rate, thereby smoothing the transmission data and outputting it as a bit stream, and also according to the amount of residual data remaining in the memory. The unit quantization control signal is fed back to the quantization circuit 106 to control the quantization step size. As a result, the transmission buffer memory 109 adjusts the amount of data generated as a bit stream, and maintains an appropriate remaining amount of data (data amount that does not cause overflow or underflow) in the memory. For example, when the remaining amount of data in the transmission buffer memory 109 increases to an allowable upper limit, the transmission buffer memory 109 increases the quantization step size of the quantization circuit 106 with the quantization control signal, thereby Reduce the amount of data. On the other hand, when the remaining amount of data in the transmission buffer memory 109 is reduced to the allowable lower limit, the transmission buffer memory 109 reduces the quantization step size of the quantization circuit 106 by using the quantization control signal, so that the quantized data data is reduced. Increase the amount.

  In this way, the bit stream output from the buffer memory 109 has a constant bit rate and, as described above, via the transmission path and the transmission medium 90 composed of an image recording medium such as an optical disk, a magnetic disk, or a magnetic tape, This is supplied to the image decoding device 80.

  On the other hand, the inverse quantization circuit 108 inversely quantizes the quantized data supplied from the quantization circuit 106, and reproduces coefficient data (quantization distortion added) corresponding to the output of the DCT circuit 105 described above. The coefficient data is supplied to an inverse discrete cosine transform (IDCT: Inverse Discrete Cosine Trasform) circuit 110.

  The IDCT circuit 110 performs IDCT conversion on the coefficient data and reproduces image data corresponding to the input image data in the intra-frame coding mode, and a difference corresponding to the output of the prediction coding circuit 104 in the forward / backward / bidirectional prediction mode. Data is reproduced and supplied to the adder circuit 111.

  In the forward / backward / bidirectional prediction mode, the adder circuit 111 is supplied with motion compensated predicted image data from a motion compensation circuit 113 described later, and adds the motion compensated predicted image data and difference data. As a result, the image data corresponding to the input image data is reproduced.

  The image data reproduced in this way is stored in the frame memory 112. That is, the inverse quantization circuit 108 to the addition circuit 111 constitute a local decoding circuit, and based on the prediction mode, the quantized data output from the quantization circuit 106 is locally decoded, and the obtained decoded image is subjected to forward prediction. An image or a backward predicted image is written in the frame memory 112. The frame memory 112 is composed of a plurality of frame memories, the bank of the frame memory is switched, and a single frame is output as forward prediction image data or output as backward prediction image data depending on the image to be encoded. Is done. Further, in the case of bi-directional prediction, the forward prediction image data and the backward prediction image data are averaged and output, for example. These predicted image data are exactly the same as the images reproduced by the decoder 81, which will be described later, and the next processed image is subjected to forward / rearward / bidirectional predictive coding based on the predicted image.

That is, the image data read from the frame memory 112 is supplied to the motion compensation circuit 113. The motion compensation circuit 113 performs motion compensation on the predicted image data based on the motion vector, and the motion compensated predicted image data. Is supplied to the prediction encoding circuit 104 and the addition circuit 111.
Next, the decoder 81 will be described with reference to FIG.

  The bit stream is input to the decoder 81 shown in FIG. 24 via the transmission medium 90 shown in FIG. This bit stream is input to the variable length decoding (IVLC) circuit 202 via the reception buffer 201. The variable length decoding circuit 202 reproduces quantized data, a motion vector, a prediction mode, a quantization step size, and the like from the bit stream. The quantized data and the quantization step size are supplied to the inverse quantization circuit 203, the motion vector is supplied to the motion compensation circuit 207, and the prediction mode is supplied to the addition circuit 205.

  The operations of the inverse quantization circuit 203 to the addition circuit 205 are the same as those of the local decoding circuit of the encoder 75 shown in FIG. 23, and the operations of the frame memory group 206 and the motion compensation circuit 207 are the frame memory 112 of the encoder 75, respectively. Similar to the motion compensation circuit 113, decoding is performed based on the quantized data, the motion vector, the prediction mode, and the quantization step size. As a result, reproduced image data is output from the adder circuit 205, and this reproduced image data is sent to the format conversion circuit 82 in FIG.

  Patent documents 1 and 2 are known as prior art.

JP-A-6-334968 International Publication No. 94/24822

  As described above, in the conventional apparatus, the encoding bit rate of the bit stream generated by the encoder 75 is constant according to the transfer rate of the transmission medium 90, and the data generation amount, that is, the encoder 75 The quantization step size of the quantization circuit 106 was controlled. In other words, for example, when images with complicated patterns continue, the quantization step size is increased to suppress the amount of data generated, and conversely, when simple patterns continue, the quantization step size is decreased to reduce the data generation amount. By increasing the generation amount, a fixed rate is maintained without causing an overflow or underflow of the buffer memory 109.

  Therefore, in the conventional apparatus, when complex images are continuous, the quantization step size is increased and the image quality is deteriorated. When simple images are continuous, the quantization step size is decreased and the image quality is uniform throughout. Could not get.

  Also, when recording a bitstream on an image recording medium with a limited data capacity, a high rate that does not impair the image quality of the complex image in order to avoid extreme image quality degradation for images with complex patterns. A fixed rate had to be applied to the whole, resulting in a reduction in recording time.

  Furthermore, even if the complexity of the images is the same, the subjective impression of the encoded image quality may be poor depending on the design. This is due to human visual characteristics. For example, there is a problem that coding noise such as block noise and mosquito noise is conspicuous in a dark portion, a red portion, and a flat portion of an image.

  Here, prior to the actual encoding of the input signal, the allocated code amount per predetermined time is obtained by encoding the same input signal, and the predetermined time is within a range not exceeding the total usable data determined by the recording medium or the like. It has been proposed to perform the actual encoding by calculating the encoding bit rate for each. This is a so-called two-pass encoding method, in which one sequence of input signals, for example, one movie is encoded to calculate the encoding bit rate, so that one sequence or more is required before actual encoding is started. There is a problem that it takes more than two sequences to generate a bit stream of encoded data for final transmission or recording.

  Further, the above is a case of handling a video signal of a moving image, but there is a similar problem when an input signal is an audio signal or a signal such as caption data or character data.

  The present invention has been made in view of such circumstances, and can obtain encoded data with uniform and good quality throughout, enabling long-time recording and encoding an input signal almost in real time. An object of the present invention is to provide a signal encoding method, a signal encoding device, and a signal recording medium. It is another object of the present invention to enable encoding so that encoding noise is not conspicuous reflecting human visual characteristics when the input signal is a video signal.

  According to the present invention, the encoding difficulty level for each predetermined time standardized by a general input signal of the type to which the input signal belongs is related to the allocated code amount, and the encoding difficulty level for the predetermined time period of the input signal is determined. The allocated code amount for each predetermined time that is related to the encoding difficulty level of the input signal for each predetermined time is determined, and the input signal is encoded for each predetermined time based on the allocated code amount. The above-described problems are solved by generating encoded data by converting the data.

  The encoding difficulty level may be obtained based on the data amount per predetermined time of encoded data generated by encoding at least a part of an input signal.

  In addition, when the input signal is a video signal, the encoding difficulty may be obtained based on image characteristic information for each predetermined time of the input video signal. As the image characteristic information at this time, Average value of luminance signal as luminance information of image every predetermined time, average value of chromaticity signal as chromaticity information every predetermined time, variance value of luminance signal as flatness information, macro as motion information of image The average value of the motion vector amount of the block every predetermined time, and the inter-image difference information can include the sum of the inter-picture prediction residuals of the motion vector of the macro block every predetermined time.

  The encoding difficulty level for each predetermined time standardized by the general input signal of the type to which the input signal belongs is associated with the assigned code amount, and the above relationship with respect to the encoding difficulty level for the predetermined time period of the input signal. Based on the assigned code amount for each predetermined time, the input signal is encoded for each predetermined time to generate encoded data. As a result, in accordance with the input of the signal, it is possible to perform encoding with an optimum allocated code amount corresponding to the complexity of the signal in almost real time.

  According to the present invention, the encoding difficulty level for each predetermined time standardized by a general input signal of the type to which the input signal belongs is related to the allocated code amount, and the encoding difficulty level for the predetermined time period of the input signal is determined. The allocated code amount for each predetermined time that is related to the encoding difficulty level of the input signal for each predetermined time is determined, and the input signal is encoded for each predetermined time based on the allocated code amount. Since encoded data is generated by encoding, encoding can be performed with an optimum allocated code amount corresponding to the complexity of the signal in almost real time according to the input of the signal.

  Further, the encoding difficulty level is obtained based on the data amount per predetermined time of the encoded data generated by encoding at least a part of the input signal, so that even if a complex signal continues, quantization is performed. Even if the step size is not extremely increased, uniform high-quality encoded data can be obtained throughout.

  Further, when the input signal is a video signal, the encoding difficulty is obtained based on image characteristic information for each predetermined time of the input video signal, and the image characteristic information at this time is used as luminance information of the image. Average value of luminance signal every predetermined time, average value of chromaticity signal every predetermined time as chromaticity information, variance value of luminance signal as flatness information, macro block motion vector amount as image motion information By using the average value for each predetermined time, the sum of the inter-picture prediction residuals of the motion vector of the macroblock for each predetermined time as the inter-image difference, etc., at a coding rate in which human visual characteristics are reflected by the image characteristic information Can be encoded.

  Hereinafter, several preferred embodiments of a signal encoding method, a signal encoding apparatus, and a signal recording medium according to the present invention will be described with reference to the drawings.

  Here, in the following embodiments, an example of an image encoding method, an image encoding apparatus, and an image recording medium when encoding an image signal or a video signal will be described, but the present invention is not limited to this. The present invention can also be applied to encoding of audio signals, and can also be applied to encoding of subtitle data and character data signals.

First Embodiment An image encoding apparatus to which a first embodiment of the present invention is applied, as shown in FIG. 1, for example, encodes an input video signal to generate first encoded data. The encoding circuit 10, the image analysis circuit 60 for obtaining statistical properties or image characteristics of the input video signal, the amount of data of the first encoded data from the first encoding circuit 10 per predetermined time, and An encoding control circuit 30 for obtaining an encoding rate for each predetermined time based on statistical data or image characteristic information from the image analysis circuit 60 and the total amount of usable data, and encoding from the encoding control circuit 30 And a second encoding circuit 40 that encodes the input video signal every predetermined time based on a rate to generate second encoded data.

  The image analysis circuit 60 includes an intra-frame information analysis circuit 61 and an inter-frame information analysis circuit 62 as shown in FIG. The intra-frame information analysis circuit 61 calculates, for example, statistical information on luminance, chromaticity, and flatness as the image characteristic information of the input image, and the inter-frame information analysis circuit 62 calculates, for example, the image characteristic information of the input image. Calculate statistical information on the amount of motion of moving images.

  As a specific example of the image characteristic information, as the statistical information about the luminance of the input image, for example, an average value (L) of the luminance signal Y every predetermined time is calculated, and the statistical information about the chromaticity of the input image As the information, for example, an average value (R) of the chromaticity signal Cr every predetermined time is calculated, and as the statistical information of the flatness of the input image, for example, the variance value (V) of the luminance signal Y every predetermined time As the statistical information of the motion amount of the input image, for example, an average value (M) of the motion vector amount per predetermined time is calculated.

  Further, as shown in FIG. 1, the first encoding circuit 10 includes a frame memory group 12 that stores input image data that is an input video signal, and image data stored in the frame memory group 12. Based on the motion vector detection circuit 11 for detecting the motion vector of the input image data, the frame memory 22 for storing the predicted image data, and the frame memory 22 based on the motion vector from the motion vector detection circuit 11. A motion compensation circuit 23 that performs motion compensation on the predicted image data that has been output; a prediction encoding circuit 14 that predictively encodes input image data based on the motion-compensated prediction image data from the motion compensation circuit 23; A difference or the like that is a prediction error from the predictive encoding circuit 14 is encoded, for example, a discrete cosine transform (hereinafter referred to as DCT: Discrete Cosine Transform). ) To generate coefficient data, quantize the coefficient data from the DCT circuit 15 with a constant quantization step size, and generate quantized data; and the quantizing circuit 16 A variable length coding (VLC) circuit 17 that performs variable length coding on the quantized data from the output and outputs variable length code data, and a coefficient obtained by dequantizing the quantized data from the quantization circuit 16 An inverse quantization circuit 18 for reproducing data, and an IDCT circuit 20 for decoding the coefficient data from the inverse quantization circuit 18, for example, inverse discrete cosine transform (hereinafter referred to as IDCT) and reproducing the difference. And the difference from the IDCT circuit 20 and the motion compensated predicted image data from the motion compensation circuit 23 are added to obtain predicted image data for the next input image data. Form, and an adder circuit 21 supplied to the frame memory 22 the predicted image data.

  As shown in FIG. 1, the second encoding circuit 40 includes a delay unit 43 that delays input image data, a frame memory 52 that stores predicted image data, and the motion vector detection circuit 11. A motion compensation circuit 53 for performing motion compensation on the predicted image data read out from the frame memory 52 based on the motion vector, and the delay unit based on the motion compensated predicted image data from the motion compensation circuit 53. A predictive encoding circuit 44 that predictively encodes input image data delayed at 43, a DCT circuit 45 that encodes a difference from the predictive encoding circuit 44, for example, DCT transforms to generate coefficient data, and the above A quantization scale setting circuit 33 for setting a quantization step size based on an encoding rate from the encoding control circuit 30; and a coefficient from the DCT circuit 45 Data is quantized with the quantization step size from the quantization scale setting circuit 33 to generate quantized data, and the quantized data from the quantization circuit 46 is variable-length encoded, Variable length encoding circuit 47 for outputting variable length code data, transmission buffer memory 49 for temporarily storing variable length code data from the variable length encoding circuit 47 and outputting it at a constant bit rate, and the quantization circuit An inverse quantization circuit 48 that dequantizes the quantized data from 46 to reproduce coefficient data, and an IDCT circuit 50 that decodes the coefficient data from the inverse quantization circuit 48, for example, performs IDCT conversion to reproduce the difference. And the difference from the IDCT circuit 50 and the motion compensated predicted image data from the motion compensation circuit 53 are added to generate predicted image data for the next input image data. And, an adding circuit 51 supplied to the frame memory 52 the predicted image data.

  In this image coding apparatus, the first coding circuit 10 performs coding processing on input image data, for example, prediction coding processing, DCT conversion processing, quantization processing with a constant quantization step size, variable length Encoding processing is performed, and encoding is performed by the second encoding circuit 40 based on the encoding bit rate obtained by the encoding control circuit 30. The encoding control circuit 30 has a data amount per predetermined time of the variable length code data that is the first bit stream obtained by the first encoding circuit, and the luminance of the input video signal obtained by the image analysis circuit 60. The average value (L) and variance (V) of the signal Y every predetermined time, the average value (R) of the chromaticity signal Cr every predetermined time, and the average value (M) of the motion vector amount every predetermined time, for example, Based on the total data volume determined by the data capacity of the image recording medium 55 such as an optical disk, magnetic disk, magnetic tape or the like, or the bit rate (transfer rate) of the transmission path, the encoding bit rate per predetermined time is determined. Seeking. Further, the second encoding circuit 40 performs the predictive encoding process, the DCT conversion process, the quantization process, and the variable length encoding process on the input image data again to obtain the variable length code data that is the second bit stream. When generating, quantization is performed with a quantization step size based on the encoding bit rate.

  Here, a specific example of the operation of the first encoding circuit 10 constituting the image encoding apparatus of the first embodiment will be described with reference to the flowchart of FIG.

  That is, in step ST1 of the flowchart shown in FIG. 2, the quantization circuit 16 of the first encoding circuit 10 in FIG. 1 quantizes the coefficient data supplied from the DCT circuit 15 with the quantization step size set to 1, for example. Then, the quantized data is generated, and the counter 31 of the encoding control circuit 30 determines the data amount of the variable length code data (first bit stream) obtained by variable length encoding the quantized data for a predetermined unit time. For example, the number of generated codes (y) is obtained for each macroblock by counting for each macroblock. Further, the image analysis circuit 60 obtains the average value (L) and variance (V) of the luminance signal Y in the macroblock and the average value (R) of the chromaticity signal Cr, and the motion vector amount of the macroblock The average value (M) is obtained.

  In the next step ST2, the bit rate calculation circuit 32 assigns each macroblock based on the values y, L, V, R, and M and the total usable data (B) described above. The code amount (b) is obtained. In this case, specifically, the encoding difficulty (difficulty) (d) is obtained based on the values y, L, V, R, and M, and the encoding difficulty (d) and the usable data are obtained. The allocated code amount (b) is obtained based on the total amount (B).

  In step ST3, the quantization circuit 46 of the second encoding circuit 40 quantizes the coefficient data supplied from the DCT circuit 45 with the quantization step size based on the allocated code amount, and generates quantized data. It is like that.

  The operation of the first encoding circuit 10 will be described more specifically with reference to FIG.

  The input image data is temporarily stored in the frame memory group 12. Then, it is read from the frame memory group 12 according to the block format as described in the prior art.

  The motion vector detection circuit 11 reads necessary image data from the frame memory group 12 in units of the macroblocks described above, and detects a motion vector. That is, the motion vector detection circuit 11 detects the motion vector of the current reference image in units of macroblocks using the front original image and / or the rear original image stored in the frame memory group 12. Here, for the detection of the motion vector, for example, a motion vector having the smallest absolute value sum of inter-frame differences in units of macroblocks is set as the motion vector. The detected motion vector is supplied to the motion compensation circuits 23 and 53, and the absolute value sum of the inter-frame difference in units of macroblocks is supplied to the intra-frame / forward / backward / bidirectional prediction determination circuit 13.

  The intra-frame / forward / backward / bidirectional prediction determination circuit 13 determines the prediction mode of the reference block based on this value, and based on the determined prediction mode, the intra-frame / forward / backward / bidirectional prediction is performed in units of blocks. The predictive coding circuit 14 is controlled so as to perform switching.

  As shown in FIG. 1 described above, the predictive coding circuit 14 includes adder circuits 14a, 14b, 14c and a changeover switch 14d, and in the intra-frame coding mode, the input image data itself is forward / backward / bidirectionally predicted. In the mode, a difference for each pixel of input image data (hereinafter referred to as difference data) with respect to each predicted image is selected, and the selected data is supplied to the DCT circuit 15.

  The DCT circuit 15 DCT-transforms input image data or difference data supplied from the changeover switch 14d in units of blocks using the two-dimensional correlation of the video signal, and supplies the obtained coefficient data to the quantization circuit 16.

  The quantization circuit 16 quantizes the coefficient data supplied from the DCT circuit 15 with a constant quantization step size, for example, a quantization step size of 1, and converts the obtained quantized data into the variable length coding circuit 17 and the inverse quantum. To the circuit 18.

  The variable length coding circuit 17 performs variable length coding on the quantized data together with the quantization step size, prediction mode, motion vector, and the like, and supplies the obtained variable length code data to the coding control circuit 30 as a first bit stream. To do.

  As shown in FIG. 1 described above, the encoding control circuit 30 includes a counter 31 that counts the data amount of the variable length code data from the variable length encoding circuit 17 every predetermined time, and the data amount from the counter 31 and And a bit rate calculation circuit 32 for obtaining an allocated code amount per predetermined time based on the total usable data amount. Then, the counter 31 counts the data amount of the first bit stream every predetermined time, for example, every macro block, obtains the generated code amount for each macro block, and supplies this generated code amount to the bit rate calculation circuit 32. To do.

  The bit rate calculation circuit 32 generates the generated code amount for each macroblock, the average value (L) and variance (V) of the luminance signal Y of the macroblock, and the average value (R) of the chromaticity signal Cr of the macroblock. Based on the average value (M) of the motion vector amount of the macroblock and the total usable data amount, the assigned code amount assigned to each macroblock, that is, the average coding rate for each macroblock time is obtained, and this assigned code The quantity is supplied to the quantization scale setting circuit 33 of the second encoding circuit 40.

  Specifically, the bit rate calculation circuit 32 first calculates the average bit rate SQBR of the sequence based on the time SQT of one sequence to be encoded and the total available data B as shown in the following equation 1. Calculate

SQBR = B / SQT Equation 1
Here, the one sequence assumes all frames recorded on one image recording medium, for example, for one movie or program. In addition, each sequence is used when the recording medium is divided and used. All frames recorded for each divided area may be one sequence.

The bit rate calculation circuit 32 sets the total number of macro blocks in the one sequence as N, sets the total usable data amount as B, and sets the i (i = 0, 1, 2,... N−1) -th macro block. The generated code amount is yi, the average value of the luminance signal Y of the i-th macroblock is Li, the variance is Vi, the average value of the chromaticity signal Cr is Ri, the average value of the motion vector amount is Mi, Coefficients α _i , β _i , γ _i , and θ _i that reflect the subjective impression of the picture pattern on the encoding difficulty di are obtained from the characteristics of f ₁ , f ₂ , f ₃ , and f ₄ , as shown in Equation 6. In addition, the encoding difficulty level di is obtained by the product of these coefficients and yi.

α _i = f ₁ (Li) Equation 2
β _i = f ₂ (Ri) Equation 3
γ _i = f ₃ (Vi) Equation 4
θ _i = f ₄ (Mi) Equation 5
d _i = α _i × β _i × γ _i × θ _i × y _i Expression 6
Here, specific examples of the characteristics of the functions f ₁ , f ₂ , f ₃ , and f ₄ will be described with reference to FIGS.

FIG. 3 shows the characteristics of the function f ₁ representing the coefficient α with respect to the average value L of the luminance signal Y. In the L on the horizontal axis in FIG. 3, 0 is black and MAX is the brightest value. In general, human eyes can easily understand coding noise in dark images. However, at darkness below a certain level, it is too dark to perceive coding noise. Based on this feature, α = 1 in the vicinity of L = 0, L is at the level of L that is most easily perceived of deterioration, α has a peak, and at more than L, α is gradually reduced. When α is 1 or more, the encoding difficulty level d is increased. Conversely, when α is 1 or less, the encoding difficulty level d is decreased. Note that the average value of the luminance signal Y for the i-th macroblock is Li, and the coefficient α _i for this i-th macroblock is f ₁ (Li).

Figure 4 shows the mean value characteristic of the function f ₂ of the R and the coefficient β of the chroma signal Cr. As for R on the horizontal axis in FIG. 4, 0 is gray and MAX is the most red value. In general, the human eye has a feature that makes red deterioration easy to understand. Based on this feature, the larger R is, the larger β is 1 or more, and the encoding difficulty d is increased. Note that the average value of the chromaticity signal Cr for the i-th macroblock is Ri, and the coefficient β _i for this i-th macroblock is f ₂ (Ri).

FIG. 5 shows the characteristics of the function f ₃ of the variance value V of the luminance signal Y and the coefficient γ. As for V on the horizontal axis in FIG. 5, 0 is completely flat, and MAX has the most random pattern. In general, the human eye can easily understand the coding noise in the flat portion of the design, and conversely, the deterioration is difficult to understand in the portion where the design is messy. Based on this feature, as the value of V increases, γ is set to a smaller value of 1 or less, and the encoding difficulty d is decreased. variance value of the luminance signal Y for the i-th macro block is Vi, the coefficient gamma _i of the i-th macro block is f 3 _(Vi).

Figure 6 shows the characteristic of the function f ₄ of the mean value M and the coefficient of the motion vector amount theta. M on the horizontal axis in FIG. 6 is a case where 0 is stationary, and MAX has the largest movement (fast). Generally, when the movement of an image is large (fast), the human eye cannot keep up with the movement, and it is difficult to understand the deterioration of the image. Based on this feature, θ is set to a smaller value of 1 or less as M increases, and the encoding difficulty d is reduced. When the macro block is intra-coded, θ = 1. the average value of the motion vector amount for the i-th macroblock in Mi, coefficient theta _i for this i-th macro block is f 4 _(Mi).

The coefficient α, β, γ, θ obtained in this way is multiplied by the generated code amount y to obtain the encoding difficulty level d. For the i-th macroblock, coefficients α _i , β _i , γ _i , and θ _i are respectively obtained and multiplied by the generated code amount y _i as shown in the above equation 6 to obtain the encoding difficulty level di.

  The bit rate calculation circuit 32 calculates the allocated code amount bi according to the following equation (7), where bi is the allocated code amount for the i-th macroblock. Here, a and c are constants and are values determined according to the average bit rate SQBR of one sequence.

bi = a × di + c Expression 7
Note that the allocated code amount bi may be obtained by table lookup without using a linear relational expression such as Expression 7.

  Next, the relational expression 7 will be described.

  Equation 7 is obtained empirically through coding experiments on many moving image sequences, for example, many movies, evaluating the image quality, and thinking and error. In general, the distribution of di in one sequence is as shown in FIG.

  In FIG. 7, the horizontal axis indicates the encoding difficulty d, and the vertical axis indicates the appearance probability h (d) in one sequence of the encoding difficulty d, that is, when the predetermined time unit is, for example, a macroblock unit. , Shows the frequency of appearance of a macroblock with encoding difficulty level d divided by the total number of macroblocks in one sequence, and is surrounded by the distribution curve and the horizontal axis in the figure. The area of the shaded area is 1.

  Here, whether the average encoding difficulty level of several sequences is high or low can be determined by the average value of the encoding difficulty levels d of the respective sequences. For example, FIG. 8 is a diagram illustrating encoding difficulty distribution curves A, B, and C for three types of movies having an average encoding difficulty level of low, medium, and high. The encoding difficulty levels of the i-th macroblock of each movie corresponding to the distribution curves A, B, and C are respectively represented by dAi (i = 0 to N), dBi (i = 0 to M), and dCi (i = 0 to K), there is the following relationship.

  In this case, based on the distribution curve C of the movie having the highest average encoding difficulty level, the relationship of the above equation 7 is created. As a procedure, within the range of the total data amount B that can be used, the allocated code amount b necessary for obtaining sufficient image quality is empirically related to the encoding difficulty level d. The relational expression thus created can be applied to movies of other types of distribution curves A and B.

  More specifically, the allocated code amounts bAi, bBi, bCi of the i-th macroblocks of movies A, B, C are calculated as follows.

bAi = a × dAi + c (9)
bB i = a × dB i + c (10)
bC i = a × dC i + c (11)
In addition, the allocated code amounts, BA, BB, and BC for each sequence of movies A, B, and C are calculated as follows.

At this time, the relationship is as follows.
BA <BB <BC (Formula 15)
That is, when a video signal for one sequence, for example, a movie, is recorded on a recording medium having a constant total recording capacity, by creating the relationship of Equation 7 based on the distribution curve C, the curves A, B, Any movie corresponding to C can be recorded without exceeding the capacity of the recording medium.

  FIG. 7 shows a distribution curve of the coding difficulty d of a typical sequence (for example, a movie) which is a model obtained by coding experiments on many moving image sequences or a reference. The average coding difficulty of this model can be considered to cover most moving image sequences in the world. Here, FIG. 9 shows the relationship between the encoding difficulty level d and the allocated code amount b when the average bit rate SQBR in an arbitrary sequence is set to a predetermined value. In FIG. 9, a function b (d) represents b (d) = a × d + c, which is a generalization of Equation 7, and the allocated code amount bi for the i-th macroblock is the i-th macroblock. Of course, it is obtained by bi = a × di + c according to the encoding difficulty level di for the block.

  FIG. 10 shows the relationship between the encoding difficulty level d and the allocated code amount b for each predetermined time when the average bit rate SQBR in the sequence is an independent variable. FIG. 10 shows, as specific examples, cases where the average bit rate SQBR is 7 Mbps, 6 Mbps, 5 Mbps, 4 Mbps, and 3 Mbps.

  These relational expressions corresponding to Expression 7 shown in FIG. 9 and FIG. 10 can be applied to variable bit rate coding of most sequences in the world within the total amount B of usable data.

  A sum of bi for all macroblocks constituting one slice is an allocated code amount of the slice. Also, the sum of bi for all macroblocks constituting one frame is the allocated code amount of that frame. Thus, for example, the bit rate calculation circuit 32 increases the assigned code amount for a frame with a complex picture, and conversely reduces the assigned code amount for a frame with a simple picture.

  On the other hand, the inverse quantization circuit 18 inversely quantizes the quantization data supplied from the quantization circuit 16 with a quantization step size of 1, and adds coefficient data (quantization distortion is added) corresponding to the output of the DCT circuit 15. The coefficient data is supplied to the IDCT circuit 20.

  The IDCT circuit 20 performs IDCT conversion on the coefficient data, reproduces input image data corresponding to the output of the predictive coding circuit 14 in the intra-frame coding mode, and reproduces difference data in the forward / backward / bidirectional prediction mode. , And supplied to the adder circuit 21.

  In the forward / backward / bidirectional prediction mode, the adder circuit 21 is supplied with predicted image data subjected to motion compensation from the motion compensation circuit 23, and adds the predicted image data and the difference data supplied from the IDCT circuit 20. Thus, the image data corresponding to the input image data is reproduced.

  The image data reproduced in this way is stored in the frame memory 22 as predicted image data. That is, the inverse quantization circuit 18 to the addition circuit 21 constitute a local decoding circuit, locally decode the quantized data output from the quantization circuit 16 based on the prediction mode, and perform forward prediction on the obtained decoded image. An image or a backward predicted image is written in the frame memory 22. The frame memory 22 is composed of a plurality of frame memories, and bank switching of the frame memory is performed. Depending on the image to be encoded, for example, a single frame is output as forward predicted image data or as backward predicted image data Is output. Further, in the case of forward / backward / bidirectional prediction, the forward prediction image data and the backward prediction image data are averaged and output, for example. These predicted image data are exactly the same image data as those reproduced by an image decoding apparatus to be described later, and the next processed image is subjected to forward / backward / bidirectional predictive coding based on this predicted image. Is called.

  Next, the operation of the second encoding circuit 40 will be described. The circuits other than the quantization scale setting circuit 33, the delay unit 43, the quantization circuit 46, and the transmission buffer memory 49 that constitute the second encoding circuit 40 are circuits that constitute the first encoding circuit 10 described above. Since the same operation is performed, the description is omitted.

  The delay unit 43 delays the input image data for a time until the encoding control signal is output from the encoding control circuit 30, for example. As a delay time of the delay unit 43, a time of one sequence or more is required in the case of the conventional two-pass method, but in this embodiment, a bit rate switching unit time, for example, described later. The time may be about 1 GOP (group of pictures). Then, the predictive encoding circuit 44 and the DCT circuit 45 perform predictive encoding processing and DCT conversion processing according to the prediction mode supplied from the intra-frame / forward / backward / bidirectional prediction determination circuit 13 to the delayed input image data. The coefficient data is generated.

  The quantization scale setting circuit 33 compares the generated code amount generated in a certain macroblock detected from the buffer feedback from the transmission buffer 49 and the allocated code amount for each macroblock.

  The quantization scale setting circuit 33 approaches the generated code amount of each macroblock in the second encoding to the assigned code amount of each set macroblock, so that the generated code amount in the macroblock is assigned to each macroblock. When the code amount is larger, the quantization step size of the next macroblock is set to be large in order to suppress the generated code amount of the next macroblock, and the generated code amount in the macroblock is smaller than the allocated code amount for each macroblock. Reduces the quantization step size of the next macroblock in order to increase the amount of generated codes. In the above description, the allocated code amount and the generated encoding amount are controlled to be close to each macroblock. However, in addition to this, for each slice, for each frame, or for each GOP (group of pictures) as described later. Control may be performed.

  Further, when the buffer feedback from the transmission buffer 49 indicates that the overflow of the transmission buffer 49 is close, the quantization scale setting circuit 33 performs quantization regardless of the comparison result between the allocated code amount and the generated code amount. If the overflow is suppressed by increasing the step size, and the buffer feedback from the transmission buffer indicates that the underflow of the transmission buffer 49 is close, it does not depend on the comparison result between the allocated code amount and the generated code amount. The underflow may be suppressed by reducing the quantization step size.

  In the above description, the generated code amount and the assigned code amount are compared for each macro block, and the quantization step size is switched for each macro block. However, the switching is performed for each slice, each frame, or each GOP. Can also be done.

  In the above description, the generated code amount is detected from the accumulated amount of the transmission buffer 49, but it can also be obtained directly from the output of the variable length encoding circuit 47. The quantization scale setting circuit 33 supplies the quantization step size set in this way to the quantization circuit 46.

  The quantization circuit 46 quantizes the coefficient data supplied from the DCT circuit 45 with the quantization step size supplied from the quantization scale setting circuit 33 described above, and generates quantized data.

  Then, the variable length coding circuit 47 converts the quantized data supplied from the quantization circuit 46 into the quantization step size from the quantization scale setting circuit 33, the intra-frame / forward / backward / bidirectional prediction determination circuit 13 and the like. Variable length coding is performed together with the prediction mode, the motion vector from the motion vector detection circuit 11 and the like, and the obtained variable length code data is supplied to the transmission buffer memory 49 as a second bit stream.

  Here, FIG. 11 is a flowchart schematically showing the operation of the second encoding circuit 40 of the image encoding device.

  As shown in FIG. 11, when image data is input via the delay unit 43 in step ST11, in step ST12, the quantization scale setting circuit 33 assigns the frame to be currently encoded. The code amount is read from the encoding control circuit 30, and the process proceeds to step ST13.

  In step ST13, the predictive coding circuit 44 to the variable length coding circuit 47 perform predictive coding processing and DCT conversion processing on the image data, and also obtain coefficient data by a quantization step size based on the allocated code amount of the macroblock. After quantization, variable length coding is performed, and the process proceeds to step ST14.

  In step ST14, for example, it is determined whether or not the encoding process has been completed for all frames (sequences) to which the same screen size and the same transfer rate are applied. If applicable, the process ends. If not, step ST11. Return to. Thus, variable-rate coding in which the coding rate is changed in units of frames is realized, and even if images (frames) with complicated patterns are continuous, the quantization step size for these images is large as in the conventional device. Therefore, uniform image quality can be obtained throughout.

  On the other hand, the inverse quantization circuit 48 inversely quantizes the quantized data supplied from the quantization circuit 46 with the quantization step size used in the quantization circuit 46 described above, and a coefficient corresponding to the output of the DCT circuit 45. Data (quantized distortion is added) is reproduced, and this coefficient data is supplied to the IDCT circuit 50. That is, the inverse quantization circuit 48 to the addition circuit 51 constituting the local decoding circuit locally decode the quantized data output from the quantization circuit 46, and use the obtained decoded image as a forward prediction image or a backward prediction image as a frame. Write to memory 52. The image data stored in the frame memory 52 is used as a predicted image for the next processed image.

  The transmission buffer memory 49 may be provided as necessary. In such a case, the transmission buffer memory 49 temporarily stores the variable length code data and then reads out the variable length code by reading it at a constant bit rate. Data is smoothed and output as a bit stream. The bit stream output from the transmission buffer memory 49 is multiplexed with, for example, an encoded audio signal, a synchronization signal, etc., and further added with an error correction code to perform predetermined modulation suitable for transmission or recording. After being added, for example, it is transmitted to an image decoding apparatus via a transmission path, or recorded on an image recording medium 55 made up of an optical disk, a magnetic disk, a magnetic tape or the like as shown in FIG. That is, in the second encoding circuit 40, variable rate coding is performed by increasing the allocated code amount bi for a complex image in advance and decreasing the allocated code amount bi for a simple image, for example. Therefore, it is not necessary to apply a high fixed rate throughout the image in order to avoid extreme image quality degradation with respect to an image with a complicated pattern as in the conventional apparatus, and the recording time of the image recording medium 55 is increased. can do.

  When the transmission buffer memory 49 is not provided or when a small-capacity memory is provided, the output bit stream is supplied to the image recording medium 55 at a variable bit rate. However, the recording rate can be variably controlled and the maximum recording rate can be controlled. By using a recording device having a bit rate higher than the maximum bit rate of the output bit stream, one sequence of video signals is recorded in a range that does not overflow the total recording capacity of the image recording medium 55 or the total usable data amount B. It is possible to allocate the total recording capacity of the image recording medium 55 to a large amount for a complex image and a small amount to a simple image so as to obtain a good image quality as a whole. be able to.

  As is clear from the embodiment of the present invention described above, a standard quantization value is set in advance in the first encoding circuit 10 in order to calculate an encoding difficulty level every predetermined time. However, the code amount is obtained by quantizing the DCT coefficient only by adaptive quantization without performing control based on the buffer occupancy, and adaptive quantization is added to the amount of encoded information when a fixed standard quantization value is used. Image characteristic information such as image brightness, flatness, chromaticity, and image motion is calculated every predetermined time, and the bit rate calculation circuit can use the image characteristic information and the amount of encoded information. Based on the total amount of data, an allocated code amount allocated every predetermined time, for example, an average encoding rate for each frame time is obtained.

  Here, as the brightness of the image, for example, the average value (L) of the luminance signal Y every predetermined time, and as the flatness, for example, the variance value (V) of the luminance signal Y every predetermined time, As the chromaticity, for example, an average value (R) per predetermined time of the chromaticity signal Cr, and as the image motion, for example, an average value (M) per predetermined time of the motion vector amount of the macroblock, respectively. Used.

  In the second encoding circuit 40, a target code amount is determined with respect to the generated code amount for each predetermined time obtained by the temporary encoding (first encoding), and the quantization step size or the quantization value is set. Control and perform image coding. That is, at each predetermined time, the quantization step size is reduced at a portion where the noise of dark images is easily perceived so that a code amount corresponding to human visual characteristics is allocated, and the quantization step is performed at a flat portion of the screen. The size is reduced, the quantization step size is reduced as the red level is increased, and the quantization step size is increased as the motion vector amount is increased (motion is faster).

  In this way, by performing encoding that reflects human visual characteristics, if the subjective impression of encoded image quality is bad depending on the design, for example, even in the dark part, red part, flat part, etc. of the image, The conspicuous encoding noise (block noise, mosquito noise) can be avoided in advance.

  Here, assuming that the generated code amount in the i-th macroblock is b′i in the second encoding circuit 40, when the sequence has been encoded, the sum total for all b′i is as follows: Equation 16 must be satisfied.

  For this reason, in order to reduce the number of failure cases, it is effective to refrain from the total amount of codes allocated from the encoding control circuit 30 to be less than the actual B, for example, about 95% of B. In some cases, it may be effective to divide the length of one sequence into, for example, four parts, and to perform encoding so that ¼ of B is the target code amount in each small sequence.

  The signal encoding method or apparatus according to the embodiment of the present invention as described above is a so-called one-pass encoder, and can encode an input signal almost in real time. In contrast, a so-called two-pass encoder, that is, all the encoding difficulty levels di for one sequence are calculated in advance in the first encoding, and all the encoding difficulty levels di and usable data are calculated. An encoding method in which an allocated code amount allocated every predetermined time is obtained based on the total amount, and bit rate control is performed according to the code amount distribution obtained in the first encoding in the second encoding. In apparatuses and apparatuses, a waiting time or delay time of at least one sequence is required from the input of a signal to the output.

  Therefore, according to the present embodiment as described above, it is possible to generate a bit rate for transmission in real time (real time) for most sequences in the world as compared with the two-pass encoder. Saving time.

  Note that one-pass encoding as in the embodiment of the present invention may be combined with conventional two-pass encoding.

  That is. The relationship between the encoding difficulty level and the allocated code amount prepared by the encoding control circuit 30 can be applied to most moving image sequences, but cannot correspond to some special sequences. When it is difficult to satisfy the condition, after calculating all the coding difficulty levels di for one sequence by the first pass coding, that is, the first coding circuit, the bit rate arithmetic circuit can use di. Based on the total amount of data, an allocated code amount allocated every predetermined time, for example, an average encoding rate for each frame time is obtained, and the second encoding circuit responds to the code amount distribution obtained in the first encoding. Thus, bit rate control may be performed.

Second Embodiment By the way, in the first embodiment described above, the allocated code amount per predetermined time, that is, the average coding rate per predetermined time is obtained for each macro block with the macro block as the predetermined time. However, the present invention is not limited to this. For example, the same applies when the frame is set to a predetermined time. In this case, the parameter group of yi, Li, Ri, Vi, and Mi in the above formulas 1 to 5 is calculated with one frame time as a predetermined time. Based on these parameter groups, the allocated code amount bi can be obtained for each frame by the above equation (7).

  The calculation method of Li, Ri, Vi, and Mi in the second embodiment may be an average value of all macroblocks in one frame, or a part of macros specified in one frame. It is good also as an average value of a block. The latter method is effective when dealing with an image in a letterbox format, that is, a format in which a 16: 9 image is displayed on a 4: 3 screen and the upper and lower frames are masked with black bands. In this case, Li, Ri, Vi, and Mi are calculated for macroblocks that constitute the central effective image portion excluding the upper and lower black belt mask portions of the image. This method is effective because the properties of the effective image portion at the center of the frame can be accurately grasped and the calculation of the mask portion of the black belt can be omitted.

  In the second encoding circuit 40 according to the second embodiment, it is necessary to calculate the allocated code amount for each macroblock in the quantization scale setting circuit 33 as compared with the first embodiment described above. In this case, the quantization scale setting circuit 33 calculates the allocated code amount for each macroblock from the supplied allocated code amount bi for each frame, for example, the allocated code amount bi for each frame is the number of macroblocks in one frame. Divided by. Buffer feedback control from the transmission buffer 49 is the same as that in the first embodiment.

  In the second embodiment, since the allocated code amount bi is obtained based on the frame, the memory amount necessary to store all the bi is smaller than that in the first embodiment. It is possible to save.

Third Embodiment In the above-described embodiment, the allocated code amount per predetermined time, that is, the average coding rate per predetermined time is obtained for each frame with the frame as the predetermined time. It is not limited to. For example, a GOP (Group of Picture) in a so-called MPEG (Moving Picture Expert Group) may be set as the predetermined time. Note that the above-mentioned MPEG is a moving image that is being studied in the WG (Working Group) 11 of the SC (Sub Committee) 29 in the JTC (Joint Technical Committee) 1 of the so-called ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission). It is a common name for image coding.

  That is, in the third embodiment, the MPEG GOP is composed of at least one so-called I picture and a plurality of P pictures or B pictures (non-I pictures). Specifically, for example, as shown in FIG. 12, if the encoding control circuit 30 is composed of one I picture, four P pictures in a three-picture cycle, and ten B pictures, The allocated code amount is obtained for each GOP. Here, an I picture is an image that is encoded in a field or in a frame, and a P picture is an image that can be predicted only from the forward direction and is encoded between fields or between frames. Is an image that can be predicted from the forward direction, the backward direction, and from both directions, and is encoded between fields or between frames.

  Then, in the first encoding circuit 10, for example, as shown in FIG. 13, any two consecutive pictures in the GOP with the number of pictures constituting the GOP as a cycle are temporarily set as an I picture and a P picture, For example, assuming that the quantization step size is 1, the image data of these I picture and P picture is subjected to predictive coding processing, DCT conversion processing, and variable length coding processing to generate variable length code data. Data is supplied to the encoding control circuit 30. Here, the reason why the two pictures are the I picture and the P picture is to check the complexity of the picture and the correlation between the frames, and the complexity of the picture can be known from the generated code amount of the I picture. The correlation between frames can be known from the generated code amount of the P picture. In general, since a plurality of consecutive frames have similar images, the tendency of the GOP pattern can be seen from two extracted pictures.

  The encoding control circuit 30 counts the data amount bitIj of the I picture and the data amount bitPj of the P picture for each GOP, and configures these data amounts bitIj, bitPj and GOP as shown in the following Expression 17, for example. Based on the number of P pictures N, the generated code amount GOPyj (j = 0, 1, 2,...) Is obtained for each GOP.

GOPyj = bitIj + N × bitPj Expression 17
Then, the encoding control circuit 30 obtains the allocated code amount allocated for each GOP based on the generated code amount GOPyj for each GOP and the total usable data amount, and the allocated code amount is second encoded. Supply to circuit 40.

Specifically, the total number of GOPs is M, the total usable data amount is B, the generated code amount of the j (j = 0, 1, 2,... M−1) th GOP is GOPyj, and the jth In the GOP, the average value of the luminance signal Y is Lj, the variance is Vj, the average value of the chromaticity signal Cr is Rj, the average value of the motion vector amount is Mj, and the functions f ₁ , f ₂ , f ₃ , coefficients reflect the characteristics of f ₄ the subjective impression of the pattern of the image to the coding difficulty _{GOPdj α j, β j, γ} j, seeking theta _j, as shown in equation 22, by the product thereof with GOPyj, The encoding difficulty GOPdj is obtained. As the characteristics of the functions f ₁ , f ₂ , f ₃ and f ₄ , the same characteristics as those shown in FIGS. 3 to 6 can be applied.

  The bit rate calculation circuit 32 calculates the allocated code amount GOPbj for the j-th GOP according to the following equation (23). Here, a and c are constants, and are values determined in accordance with the average bit rate SQBR shown in Equation 1 in one sequence.

α _j = f ₁ (Lj) Equation 18
β _j = f ₂ (Rj) Equation 19
γ _j = f ₃ (Vj) Equation 20
θ _j = f ₄ (Mj) Equation 21
GOPdj = α _j × β _j × γ _j × θ _j × GOPyj Equation 22
GOPbj = a × GOPdj + c (Equation 23)
Note that the allocated code amount GOPbj may be obtained by table lookup without using a linear relational expression such as Expression 23 above.

  Thus, the encoding control circuit 30 increases the allocated code amount GOPbj for, for example, a GOP having a complex picture or a low correlation between frames, and conversely a simple picture or frame. For GOPs having a high correlation between them, the allocated code amount GOPbj is reduced.

  Next, as shown in FIG. 14, for example, when the image data is input via the delay unit 43 in step ST21, the second encoding circuit 40 determines that the currently input image data is GOP in step ST22. If it is applicable, the process proceeds to step ST23. Otherwise, the process proceeds to step ST24.

  In step ST23, the second encoding circuit 40 reads the allocated code amount for the GOP currently being encoded from the encoding control circuit 30, and proceeds to step ST24.

  In step ST24, the second encoding circuit 40 performs predictive encoding processing and DCT conversion processing on the image data, and after quantizing the coefficient data with a quantization step size based on the allocated code amount, Go to step ST25.

  Here, the quantization scale setting circuit 33 converts the allocated code amount for each frame from the supplied allocated code amount for each GOP to the picture type (I picture, P picture, B picture) in actual encoding, that is, 12 is set in consideration of the picture type shown in FIG. Specifically, the allocated code amount for the I picture is increased, the allocated code amount for the B picture is decreased, and the allocated code amount for the P picture is set in the middle. The subsequent processing of the quantization scale setting circuit 33 is the same as that in the embodiment in which the allocated code amount is obtained for each frame described above.

  Next, in step ST25, it is determined whether or not the encoding process has been completed for all frames (sequences) to which the same screen size and the same transfer rate are applied. If applicable, the process ends. Return to ST21. In this way, variable rate coding in which the coding rate is changed in units of GOP is realized, and even if images (frames) with complicated patterns are continuous, the quantization step size for these images is large as in the conventional device. Therefore, uniform image quality can be obtained throughout. Further, in this embodiment, since the allocated code amount for each GOP is obtained based on two pictures, high-speed processing is possible as compared with the above-described embodiment. Needless to say, the allocated code amount of each GOP may be obtained based on the data amount of all the pictures in the GOP.

Fourth Embodiment Next, a fourth embodiment of an image encoding device to which the present invention is applied will be described with reference to FIG.

  The fourth embodiment and the first to third embodiments have the following two differences. That is, first, there is no first encoding circuit 10 in FIG. 1, and the method of obtaining the encoding difficulty is different, and secondly, the operation of the bit rate calculation circuit 32 is different. Hereinafter, these differences will be described.

  The image coding apparatus according to the fourth embodiment shown in FIG. 15 has an image analysis circuit 60 that obtains statistical properties of an input video signal, statistical data from the image analysis circuit 60, and the total amount of usable data. A coding control circuit 30 for obtaining a coding rate for each predetermined time based on the coding rate, and generating the coded data by coding the input video signal for each predetermined time based on the coding rate from the coding control circuit 30 Encoding circuit 40.

  The image analysis circuit 60 obtains information or image characteristic information based on the statistical properties of the input video signal. As this image characteristic information, for example, an average value (L) for each predetermined time of the luminance signal Y is calculated, Further, statistical information on the chromaticity of the input image, for example, an average value (R) of the chromaticity signal Cr every predetermined time is calculated, and statistical information on the flatness of the input image, for example, every predetermined time of the luminance signal Y The variance value (V) of the input image is calculated, the statistical information of the motion amount of the input image, for example, the average value (M) of the motion vector amount per predetermined time is calculated, and further, the difference information between images, for example, the motion vector image An absolute value sum (E) for each predetermined time of the inter prediction residual is calculated.

  That is, as the inter-picture prediction residual of the motion vector, a sum E of absolute values of differences between the luminance signal Yj of the encoding target macroblock and the luminance signal Ri of the macroblock referred to by the motion vector is obtained.

For E, a sum of squares may be used instead of the sum of absolute values.

  In this image encoding device, for example, as shown in FIG. 16, in step ST31, image characteristic information that is statistical property information of an input video signal representing the difficulty of encoding (difficulty), here, a luminance signal of a macroblock The average value L and variance V of Y, the average value R of the chromaticity signal Cr, the average value M of the motion vector amount of the macroblock, and the absolute value sum E of the inter-picture prediction residuals of the motion vector are calculated every predetermined time, For example, it is calculated for each macro block. If the macroblock is an intra coding mode, that is, a so-called I picture, the absolute value sum of the average value separation residuals in the macroblock is calculated rather than calculating the absolute value sum of the inter-picture prediction residuals of the motion vector. Is preferably calculated.

  That is, the absolute value sum E of the difference between the luminance signal Yi of the encoding target macroblock and the average value Yav of the macroblock luminance signal Yi is used as the absolute value sum of the average value separation residuals in the block of the intra coding mode. Is obtained by the following equation 25.

For E in Equation 25, the sum of squares, that is, the variance value V may be used instead of the absolute value sum.

  In the next step ST32, the bit rate calculation circuit 32 calculates the average value and variance of the luminance signal for each macroblock, the average value of the chromaticity signal, the average value of the motion vector amount, and the inter-picture prediction residual of the motion vector. Alternatively, the assigned code amount assigned to each macroblock is obtained based on the average value separation residual of the luminance signal and the total usable data amount.

  In the next step ST33, the quantization circuit 46 of the encoding circuit 40 quantizes the coefficient data supplied from the DCT circuit 45 with a quantization step size based on the assigned code amount so as to generate quantized data. It has become.

Here, a specific operation of the bit rate calculation circuit 32 will be described. The bit rate calculation circuit 32 sets N as the total number of macroblocks in the sequence, Li is the average value of the luminance signal Y of the i (i = 0, 1, 2,..., N−1) th macroblock, and Vi is the variance. The average value of the chromaticity signal Cr is Ri, the average value of the motion vector amount is Mi, and the subjective impression of the picture of the image is encoded from the characteristics of the functions f ₁ , f ₂ , f ₃ , and f _4. coefficient to reflect to _{di α i, β i, γ} i, determine the θ _i. Further, based on the absolute value sum E of the inter-picture prediction residual of the motion vector, the characteristic of the function f _6, the estimated value of the generated code amount of the macroblock, i.e. corresponding to yi in the first embodiment Things y'i are estimated empirically. Then, as shown in Equation 31 below, the encoding difficulty level di is obtained by the product of each coefficient α _i , β _i , γ _i , θ _i and the estimated value y′i of the generated code amount.

α _i = f ₁ (Li) Equation 26
_{β i = f 2 (Ri)} ··· formula 27
γ _i = f ₃ (Vi) Equation 28
θ _i = f ₄ (Mi) Equation 29
y ′ _i = f ₆ (Ei) Equation 30
d _i = α _i × β _i × γ _i × θ _i × y ′ _i Expression 31
As specific examples of the characteristics of the functions f ₁ , f ₂ , f ₃ , and f ₄ , those shown in FIGS. 3 to 6 can be used. Further, FIG. 17 shows an example of the function _{f 6.} That is, FIG. 17 shows an estimated value y ′ of the generated code amount for each macroblock with respect to the inter-picture prediction residual of the motion vector. From the experimental results, E and y ′ are empirically proportional to each other, and the generated code amount estimate y′i for the inter-picture prediction residual Ei of the motion vector of the i-th macroblock is f ₆ (Vi )

When the encoding target macroblock is in the intra coding mode, it is preferable to use the absolute value sum or variance value (V) of the average value separation residual of the luminance signal in the macroblock. In this case, for example, Based on the variance Vi of the luminance signal Y, an estimated value y′i of the generated code amount of the macroblock is estimated from the characteristic of the function f ₅ .

y ′ _i = f ₅ (Vi) Equation 32
A specific example of this function f ₅ (Vi) is shown in FIG.
FIG. 18 shows an estimated value y ′ of the generated code amount for each macroblock with respect to the variance V of the luminance signal Y. The estimated value y of the generated code amount for the variance Vi of the luminance signal Y of the i-th macroblock. 'i becomes f ₅ (Vi).

  Similarly to the first embodiment, the bit rate calculation circuit 32 calculates the allocated code amount bi of the i-th macroblock for the encoding difficulty level di as shown in the following Expression 33. Note that a linear relational expression such as Expression 33 may not be used, and the table lookup may be performed.

bi = a × di + c Expression 33
This relational expression 33 is empirically obtained through coding experiments on many moving image sequences, for example, many movies, evaluating the image quality, and thinking and error, and can be applied to most sequences in the world. It is a general relational expression. About how to obtain it, it is made by the same method as Expression 7 explained in the first embodiment.

  Thus, for example, the bit rate calculation circuit 32 increases the assigned code amount for a frame with a complex picture, and conversely reduces the assigned code amount for a frame with a simple picture.

  Since the operation of the encoding circuit 40 is the same as that of the second encoding circuit 40 described in the first embodiment, a description thereof will be omitted.

  The present invention is not limited to the above-described embodiment. For example, in the above-described embodiment, the transform coding is DCT, but the so-called Strato transform, Haar transform, wavelet transform, etc. Good. The signals to be handled are not limited to image signals and video signals, and can be applied to encoding of audio signals, for example, and can also be applied to encoding of caption data, character data signals, and the like. For example, when the present invention is applied to encoding of an audio signal, encoding is performed using auditory statistical information and a generated code amount per predetermined time obtained by performing the first encoding as necessary. What is necessary is just to ask for difficulty.

  As is clear from the above description, according to the embodiment of the present invention, the relationship between the encoding difficulty level and the allocated code amount for each predetermined time standardized in advance by a general input signal of the type to which the input signal belongs is related. And determining the encoding difficulty level of the input signal for each predetermined time, determining the allocated code amount for the predetermined time with respect to the encoding difficulty level for the predetermined time of the input signal, and determining the allocation code Since the encoded data is generated by encoding the input signal every predetermined time based on the amount, the optimal allocated code amount corresponding to the complexity of the signal is almost real time according to the input of the signal. Can be encoded.

  Further, the encoding difficulty level is obtained based on the data amount per predetermined time of the encoded data generated by encoding at least a part of the input signal, so that even if a complex signal continues, quantization is performed. Even if the step size is not extremely increased, uniform high-quality encoded data can be obtained throughout.

  Further, when the input signal is a video signal, the encoding difficulty is obtained based on image characteristic information for each predetermined time of the input video signal, and the image characteristic information at this time is used as luminance information of the image. Average value of luminance signal every predetermined time, average value of chromaticity signal every predetermined time as chromaticity information, variance value of luminance signal as flatness information, macro block motion vector amount as image motion information By using the average value for each predetermined time, the sum of the inter-picture prediction residuals of the motion vector of the macroblock for each predetermined time as the inter-image difference, etc., at a coding rate in which human visual characteristics are reflected by the image characteristic information Can be encoded.

It is a block diagram which shows the circuit structure of the principal part of the image coding apparatus to which the 1st Embodiment of this invention is applied. It is a flowchart for demonstrating operation | movement of the 1st encoding circuit which comprises the image coding apparatus of 1st Embodiment. It is a characteristic diagram of the function f ₁ of the mean value L and the coefficient α of the luminance signal Y. It is a characteristic diagram of the function f ₂ of the mean value R and the coefficient β of the chroma signal Cr. It is a characteristic diagram of the function f ₃ of the variance V and the coefficient γ of the luminance signal Y. It is a characteristic diagram of the function f ₄ of the mean value M and the coefficient of the motion vector amount theta. It is a distribution map which shows the appearance probability h (d) of the macroblock of the encoding difficulty d in a sequence. It is a distribution diagram which shows the appearance probability h (d) of the macroblock of the encoding difficulty level d in case an average encoding difficulty level is low, medium, and high. It is a distribution map which shows the appearance probability h (d) of the macroblock with respect to the encoding difficulty d, and the allocation code amount b. It is a distribution diagram which shows the allocation code amount b when making average bit rate SQBR into an independent variable with respect to the encoding difficulty d. It is a flowchart for demonstrating operation | movement of the 2nd encoding circuit which comprises the image coding apparatus of 1st Embodiment. It is a figure which shows each picture for demonstrating the structure of GOP in MPEG. It is a figure which shows each picture for demonstrating the encoding control signal for every GOP. It is a flowchart for demonstrating operation | movement of the 2nd encoding circuit which comprises the image coding apparatus used as the 3rd Embodiment of this invention. It is a block diagram which shows the circuit structure of the principal part of the image coding apparatus to which the 4th Embodiment of this invention is applied. It is a flowchart for demonstrating operation | movement of the principal part of the image coding apparatus of this 4th Embodiment. It is a characteristic diagram of the function f ₆ estimates y'i generated code amount of the macroblock units to the image prediction residuals of the motion vector. It is a characteristic diagram of the function f ₅ estimates y'i of generated code amount of variance V macroblock of the luminance signal Y. It is a figure which shows the image for demonstrating the principle of predictive encoding. It is a figure which shows the image for demonstrating the principle of motion compensation prediction encoding. It is a block diagram which shows the structural example of an image coding apparatus and an image decoding apparatus. It is a figure which shows the structure of a macroblock and a slice. It is a block diagram which shows an example of the circuit structure of the conventional encoder. It is a block diagram which shows an example of the circuit structure of the conventional decoder.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 1st encoding circuit, 14,44 Predictive encoding circuit, 15,45 DCT circuit, 16,46 Quantization circuit, 17,47 Variable length encoding circuit, 30 encoding control circuit, 31 counter, 32 bit rate Arithmetic circuit, 33 Quantization scale setting circuit, 40 Second encoding circuit, 43 Delay device, 60 Image analysis circuit, 61 In-frame information analysis circuit, 62 Inter-frame information analysis circuit

Claims (11)

Relating the encoding difficulty per predetermined time and the allocated code amount standardized in advance by a general input signal of the type to which the input signal belongs,
Obtain the encoding difficulty level of the input signal every predetermined time,
The allocated code amount for each predetermined time that is related to the encoding difficulty level for each predetermined time of the input signal is obtained,
A signal encoding method, wherein encoded data is generated by encoding the input signal every predetermined time based on the allocated code amount. 2. The signal encoding method according to claim 1, wherein the encoding difficulty level is obtained based on a data amount per predetermined time of encoded data generated by encoding at least a part of an input signal. The input signal is a video signal,
2. The signal encoding method according to claim 1, wherein the encoding difficulty level is obtained based on image characteristic information for each predetermined time of the input video signal. 4. The signal encoding method according to claim 3, wherein the image characteristic information is obtained by calculating at least one of image brightness, chromaticity, flatness, image motion, and inter-image difference every predetermined time. . The image characteristic information includes an average value of a luminance signal as luminance information of the image every predetermined time, an average value of the chromaticity signal as chromaticity information every predetermined time, a variance value of the luminance signal as flatness information, an image Obtaining at least one of the average value of the motion vector amount of the macroblock as the motion information for each predetermined time and the total sum of the motion vector of the macroblock for the predetermined time as the inter-image difference. 4. A signal encoding method according to claim 3, wherein: An encoding control means for relating an encoding difficulty level and an allocated code amount at predetermined time intervals that are standardized in advance by a general input signal of a type to which the input signal belongs;
An encoding difficulty output means for obtaining the encoding difficulty for each predetermined time of the input signal and sending it to the encoding control means;
Encoding data is generated by encoding the input signal every predetermined time based on the assigned code amount obtained in association with the encoding difficulty level from the encoding difficulty output unit by the encoding control unit. And a signal encoding device. The encoding means associates the encoding difficulty obtained based on the data amount per predetermined time of the encoded data generated by encoding at least a part of the input signal and the allocated code amount. The signal encoding apparatus according to claim 6, wherein: The input signal is a video signal,
7. The signal coding apparatus according to claim 6, wherein the coding control means uses the coding difficulty obtained based on image characteristic information for each predetermined time of the input video signal. The encoding control means uses, as the image characteristic information, information obtained by calculating at least one of image brightness, chromaticity, flatness, image motion, and inter-image difference every predetermined time. The signal encoding apparatus according to claim 8. The coding control means includes, as the image characteristic information, an average value of a luminance signal as luminance information of the image every predetermined time, an average value of the chromaticity signal as chromaticity information every predetermined time, and as flatness information Of the variance value of the luminance signal, the average value of the motion vector amount of the macroblock as the motion information of the image per predetermined time, and the sum of the prediction residual between the images of the macroblock motion vector as the inter-image difference per predetermined time 9. The signal coding apparatus according to claim 8, wherein at least one obtained is used. The encoding difficulty level for each predetermined time standardized by a general input signal of the type to which the input signal belongs and the allocated code amount are related, and the above-mentioned relationship is set for the encoding difficulty level for each predetermined time of the input signal. A signal recording medium comprising: recorded encoded data obtained by determining an allocated code amount for each predetermined time and encoding the input signal for each predetermined time based on the allocated code amount.

Images