JP4399794B2 - Image coding apparatus and image coding method - Google Patents

Description

  The present invention relates to an image coding apparatus for performing image coding of moving images such as MPEG2 (Moving Picture Experts Group Phase 2), and in particular, suppresses visual deterioration during image coding. The present invention relates to a possible image encoding device.

  In recent years, services that distribute information through various transmission paths such as satellite waves, terrestrial waves, and telephone lines using information compressed by high-efficiency coding for digitized image signals have been put into practical use. Yes. In such a service, when distributing information such as moving images and sounds, MPEG2 which is an international standard is used as a high-efficiency encoding method for moving images and sounds. MPEG2 is an encoding method that compresses the information amount of an image signal by using a correlation between adjacent pixels (spatial direction) of an image signal and a correlation between adjacent frames or adjacent fields (time direction).

  Image encoding in the MPEG2 standard is processed by the following algorithm. First, temporally continuous image frames are divided into a reference frame and a prediction frame. By encoding the reference frame using only the spatial correlation, the original image can be restored using only the encoded data of the frame. On the other hand, the prediction frame is encoded using the correlation in the time direction and the correlation in the spatial direction from the reference frame, thereby realizing higher encoding efficiency than the reference frame. Note that the encoded data of the prediction frame is recovered by the recovered reference frame and the encoded data of the prediction frame.

  Next, a specific encoding system used in MPEG2 image encoding will be described with reference to FIG. In FIG. 4, numbers are assigned to the picture types as necessary so that they can be identified. An I picture (I frame), which is a reference frame indicated as “I” in FIG. 4A, is periodically present and is information serving as a reference for decoding processing. On the other hand, in the prediction frame, a P picture (P frame) encoded by only prediction from a temporally previous (past) reference frame, indicated by “P” in FIG. There is a B picture (B frame) that is predictively encoded from two reference frames before and after (past and future) in time, which are indicated as “B” in FIG. Note that arrows in FIG. 4A indicate prediction directions related to the P picture and the B picture. The P picture itself is a prediction frame and is also used as a reference frame for other P pictures and B pictures.

  An I-picture image signal is divided into processing units called macroblocks of horizontal 16 pixels × vertical 16 pixels based on the luminance signal. The divided macroblock data is further divided into two-dimensional blocks of 8 pixels × 8 pixels and subjected to DCT (Discrete Cosine Transform) processing which is a kind of orthogonal transform.

  Since the signal after the DCT processing shows a value according to the frequency component of the two-dimensional block, the component is concentrated in a low band in a general image. In addition, information degradation of high frequency components has a property that is visually less noticeable than information degradation of low frequency components. Therefore, the amount of information is compressed by finely quantizing the low frequency components and coarsely quantizing the high frequency components, and variable length coding the coefficient component and the continuous length of coefficient 0 having no component.

  Similarly to the I picture, the P picture image signal is also divided into units of macroblocks of horizontal 16 pixels × vertical 16 pixels based on the luminance signal. In the P picture, a motion vector between the reference frame and each macroblock is calculated. Motion vector detection is generally obtained by block matching. In this block matching, the difference between each pixel of the macroblock and each pixel obtained by blocking the reference frame where the horizontal / vertical position where the macroblock exists by the motion vector value is moved into 16 horizontal pixels × 16 vertical pixels. The absolute value sum (or the sum of squared differences) is obtained, and the value of the motion vector taking the minimum value is output as the detected motion vector.

  Each pixel of the macro block is subjected to a difference from each pixel of the two-dimensional block cut out by the motion vector. When an accurate motion vector is detected, the information amount of the difference block is significantly smaller than the information amount of the original macroblock, so that coarser quantization processing than that of the I picture is possible. Actually, it is selected whether to encode a differential block or a non-differential block (intra block) (prediction mode determination), and DCT similar to an I picture is selected for the selected block. A variable length encoding process is performed to compress the amount of information.

  In addition, the same processing as that for the P picture is performed for the B picture, but the I and P pictures that are the reference frames exist before and after the time frame, and motion vectors are detected between the reference frames. Is called. In the B picture, prediction options are prediction from a previous reference frame (forward prediction), prediction from a subsequent reference frame (backward prediction), and an average value (average (average (2)) of two prediction blocks. There are three types of (average) prediction), and prediction mode determination is performed from among four types of schemes including a scheme for decoding only by intra blocks.

  B pictures can be predicted from temporally preceding and following reference frames, so that prediction efficiency is further improved than P pictures. Therefore, in general, a B picture is quantized more coarsely than a P picture. The block selected as the B picture is subjected to the same encoding process as the I and P pictures.

  In the decoding process of a B picture, a prediction process from a later reference frame is also performed, and thus this reference frame needs to be encoded before the B picture. For this reason, in the encoding process, as shown in FIG. 4B, the image signal recorded and input is arranged such that the B picture is arranged after the I picture or P picture which is the reference frame of the B picture. Then, the order is rearranged and encoded. That is, during the encoding process, the order of the input order of the original image is rearranged in view of the encoding order during the decoding process. On the other hand, in the decoding process, as shown in FIG. 4C, the decoded image is reproduced in the order of the input image signals by performing the reverse rearrangement with respect to the order of FIG. It becomes possible.

  Next, a general encoding device and decoding device for realizing MPEG2 image encoding will be described. First, a general encoding apparatus in the prior art will be described. FIG. 5 is a block diagram illustrating an example of a general image encoding device according to the related art. In FIG. 5, the digital image signal (input image signal) input from the input terminal 201 is supplied to and stored in the input image memory 202, and is delayed to be rearranged in the order of encoding according to the encoding syntax. Is done. The digital image signal output from the input image memory 202 is subjected to macroblock cutout processing in the two-dimensional block conversion circuit 203.

  Macroblock data relating to the reference frame is supplied to the orthogonal transform circuit 205 via the subtractor 204, where DCT processing is performed in units of horizontal 8 pixels × vertical 8 pixels to calculate DCT coefficients. The DCT coefficients are further collected in units of macroblocks of horizontal 16 pixels × vertical 16 pixels based on the luminance signal, and sent to the quantization circuit 206. In the quantization circuit 206, for example, the quantization process is performed by dividing by a different value for each DCT coefficient by a quantization matrix having a different value for each frequency component. The quantized DCT coefficient is sent to the encoding circuit 214, and variable length or fixed length encoding is performed by referring to an address corresponding to the coefficient of the encoding table 215. Then, the multiplexer 216 multiplexes the encoded data after the processing in the encoding circuit 214 and the additional information indicating the location of the macroblock in the screen from the two-dimensional block conversion circuit 203, and the image stream After being stored in the buffer 218 once, it is output from the output terminal 219 as a bit stream (output image bit stream).

  Also, the DCT coefficients quantized by the quantization circuit 206 are subjected to inverse quantization processing and inverse DCT processing by the inverse quantization circuit 212 and the inverse orthogonal transform circuit 213, and the quantized DCT coefficients are decoded. The data is supplied to and stored in the reference image memory 209 via the adder 210 and the deblocking circuit 211. The image stored in the reference image memory 209 is used at the time of predictive frame encoding processing.

  On the other hand, for the predicted frame, a motion vector between images is obtained by the motion vector detection circuit 207 between the macroblock data cut out from the input image memory 202 and the image stored in the reference image memory 209. The motion vector obtained in the motion vector detection circuit 207 is supplied to the motion compensation prediction circuit 208, where a prediction block is cut out from the reference image from the reference image memory 209. The motion compensated prediction circuit 208 selects an optimal prediction mode according to the plurality of extracted prediction blocks, and sends a difference signal from the input image block to be encoded to the orthogonal transformation circuit 205. The difference signal is processed in the same manner as each block of the reference frame described above, the DCT coefficient is quantized, and the image stream buffer 218 is output from the multiplexer 216 as an output image bit stream together with the motion vector and the prediction mode. Then, it is output from the output terminal 219.

  Regarding the control of the code amount, the code amount control circuit 217 compares the code amount of the bit stream output from the multiplexer 216 with the target code amount (target code amount) to obtain the target code amount. In order to make it closer, the quantization fineness (quantization scale) of the quantization circuit 206 is controlled. Then, for the above-described three types of picture types (frame types) having different information amounts, the target code amount for each frame is calculated using the nature and appearance frequency of each picture type for the set encoding bit rate. .

  Also, the target code amount is limited so that a buffer overflow / underflow does not occur in a stream buffer (called a VBV (Video Buffer Verifier) buffer) virtually simulated by a decoding device. In addition, the quantization scale is calculated by calculating the quantization scale value for the target code amount for each frame type by utilizing the fact that the scale and the output code amount are generally inversely proportional. Done. Then, by controlling the quantization scale in a direction approaching the target code amount for each block, control is performed to suppress the encoded stream within the target code amount.

  Next, a general decoding device in the prior art will be described. FIG. 6 is a block diagram illustrating an example of a general decoding device according to the related art. In FIG. 6, first, an image bit stream (image stream) input from the input terminal 101 is stored in the image stream buffer 102. Note that a virtually simulated buffer value is written in the image bit stream, and the following decoding process is performed after the image bit stream is stored in the image stream buffer 102 by the buffer value. By doing so, it is possible to prevent the decoding process from stopping due to the failure of the buffer. In the image bit stream output from the image stream buffer 102, the variable length decoding circuit 103 separates additional information such as a quantization scale, a prediction mode, and a motion vector, and decodes the quantized DCT coefficient. .

  The decoded DCT coefficients are processed in the same manner as the inverse quantization circuit 212 and the inverse orthogonal transform circuit 213 in the encoding circuit (image encoding apparatus shown in FIG. 5), and the inverse quantization circuit 105 and the inverse orthogonal circuit are processed. In the transform circuit 111, inverse quantization processing and inverse DCT processing are performed, and the intra block or the difference block is decoded and supplied to the adder 107. In the case of a prediction block, a reference image signal read from the reference image memory 109 in the motion compensated prediction circuit 106 based on the prediction mode and the motion vector value decoded by the variable length decoding circuit 103 (of the process). A prediction block is cut out from a previously stored image signal of an I picture or a P picture. As a result, the decoded intra block or difference block and the prediction block cut out by the motion compensation prediction circuit 106 are added by the adder 107, and the image signal of the macro block is restored.

  The macroblock data (macroblock image signal) restored by the addition processing in the adder 107 is supplied to the deblocking circuit 108 and returned to the image signal in the order of image scanning. At this time, in the case of an I or P picture, it is written in the reference image memory 109, and in the case of a B picture, it is once stored in the output frame memory 110 and then output as an image signal (output image signal). Note that the I or P picture image data stored in the reference image memory 109 is once stored in the output frame memory 110 in accordance with the image output timing as shown in FIGS. In the same manner as described above, an image signal (output image signal) is output.

  In addition, in an image encoding technique that allows variation in the code amount for each predetermined frame as in the MPEG2 standard, as a method for controlling an output image bitstream to be a target code amount, generally, for example, the following Such a method is adopted.

For example, a target code amount for each frame is assigned to a target code amount for a certain period of time according to the information amount of each picture type encoded image. Specifically, when the code amount required for each previously encoded picture type is Bits and the average value of the quantization scale value of each picture type is AvgQ, an approximation of the complexity of each picture type C can be calculated by the product of the code amount and the average quantization scale value.
C (T) = Bits (T) * AvgQ (T)
(T: Picture type)

At this time, using the approximate value C of the complexity of each picture type described above, each picture type has a target code amount TotalBits given within a predetermined time (for example, F frame) assumed in coding control. When the number of frames used is Fnum, the target code amounts Budget (I), Budget (P), and Budget (B) for frames when the picture types to be encoded are I, P, and B pictures, respectively,
Budget (I) = {TotalBits * C (I)} / {Fnum (I) * C (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}
Budget (P) = {TotalBits * C (P)} / {Fnum (I) * C (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}
Budget (B) = {TotalBits * C (B)} / {Fnum (I) * C (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}
Is calculated.

  The target code amount calculated in this way is a target code amount for a future input image set based on the encoding difficulty level of the past input image. Therefore, according to the control of the code amount according to the target code amount, it is possible to perform a good control when the scene with a small change in difficulty does not change. However, it is slow to follow scene changes. For example, fade-in, scene change, movement due to screen movement, change in resolution, etc., may result in disordered control. There is a risk that the quality of the image is greatly degraded.

On the other hand, in order to cope with a scene change, a control method is also known in which a scene of a fade-in / out or scene change is determined by referring to a change in luminance component of an input image. In this control method, for example, the change in the scene is recognized by measuring the change rate between frames of the sum of the luminance signal (DC component: DC component) within the frame, and the target code of each frame corresponding to the change rate is recognized. A quantity correction is made. In addition, this control method is disclosed by the following patent document 1, for example.
JP-A-8-317387 (FIGS. 1, 3, 4, paragraphs 0015 to 0036)

  For example, according to the technique disclosed in Patent Document 1, it is possible to perform coding control for a scene change in which only the luminance component of the input image changes (for example, a scene with a large luminance change that is visually noticeable). It is possible to improve the efficiency. However, there are many cases in which the luminance component does not change during fade-in / out or scene change. For example, when two scenes shot in the same scene are switched, or fade-in from the gray level instead of the black level When a scene change such as “/ out” occurs, it is impossible to accurately detect the scene change by the technique disclosed in Patent Document 1 described above.

  In addition, in the technique disclosed in Patent Document 1 described above, when a scene change is detected, a target code amount corresponding to the variation is sequentially allocated to a frame to be encoded. When a complex image appears due to fade-in, the code amount is allocated too much during the fade-in, and a sufficient code amount is allocated to the complex image immediately after the fade-in ends. There is a risk that it will not be possible to do so.

  In addition, there is generally a method called 2-pass encoding as a method for measuring the difficulty of encoding an input image in the future and controlling the code amount based on the result. ing. However, according to this method, since the same image is required twice as a material (input image) to be encoded, image data once recorded on another recording medium is used as an input image to be encoded. There is a restriction that must be done.

  In particular, when a signal from a camera is input and encoded, or when image data that has been subjected to editing processing is directly encoded, it is difficult to perform processing by two-pass encoding. Therefore, when the input image is directly encoded, the code amount assignment is changed by predicting or judging the change of the scene without performing the process by the two-pass encoding, and the stable code amount is controlled. There is a need.

  In view of the above problems, the present invention provides an image encoding device capable of realizing good image encoding control in which image quality deterioration is suppressed in a scene where a video changes smoothly (for example, a fade scene). The purpose is to provide.

In order to achieve the above object, according to the present invention, an image encoding device for encoding an image signal,
Calculates the frame complexity and DC Ingredients for each of a plurality of frames temporally continuous in the image signal to be subjected to the encoding, the intraframe information amount calculating and outputting the calculation result as a frame data amount Means,
An intra-frame information amount holding means for holding the intra-frame complexity for each of the plurality of frames output from the intra-frame information amount calculation means;
Based on the intra-frame complexity and the DC component of the plurality of frames held in the intra-frame information amount holding means, by verifying temporal changes of the intra-frame complexity and the DC component of the plurality of frames, Scene detection means for determining whether or not a scene displayed by the plurality of frames is in a fade state;
Based on the in-frame complexity of the plurality of frames held in the in-frame information amount holding unit and the determination result of whether or not the fade state is in the fade state by the scene detection unit , The prediction value calculated using the gradient of the intra-frame complexity of a plurality of frames is set as the intra-frame complexity of the subsequent frame of the plurality of frames. An intra- frame information amount prediction means for setting an intra-frame complexity of a frame to an intra-frame complexity of the subsequent frame ;
While holding the approximate value of the complexity calculated from the code amount of the frame of each picture type encoded last and the quantization scale average value, the frame corresponding to the frame of each picture type encoded last The intra-frame complexity is held as the latest measured complexity, and the intra-frame complexity of the subsequent frame predicted by the intra-frame information amount prediction unit and the determination result of the fade state by the scene detection unit Based on the above, if not in the fade state, only the approximate value of the complexity of the I picture is corrected using the intra-frame complexity of the subsequent frame and the latest measured complexity of the I picture. Each of the picture types using the approximate value of the complexity after the correction, the target code amount sum in a certain time, and the number of frames in which each picture type is used. While setting the target code amount, in the fade state, the assumption related to the target code amount using the intraframe complexity of the subsequent frame and the latest measured complexity of each picture type. For all the numbers of frames, the approximation value of the complexity is converted for each frame, and the target code amount of each picture type is set using the target code amount sum of the assumed number of frames related to the target code amount. Code amount control means,
An image encoding device is provided.

In order to achieve the above object, according to the present invention, there is provided an image encoding method executed by an image encoding apparatus that encodes an image signal,
An intra-frame information amount calculation step of calculating an intra-frame complexity and a DC component for each of a plurality of temporally continuous frames of the image signal to be encoded and outputting the calculation result as an intra-frame information amount When,
An intra-frame information amount holding step for holding the intra-frame complexity for each of the plurality of frames output in the intra-frame information amount calculation step;
Based on the intra-frame complexity and the DC component of the plurality of frames held in the intra-frame information amount holding step, by verifying temporal changes of the intra-frame complexity and the DC component of the plurality of frames, A scene detection step for determining whether or not a scene displayed by the plurality of frames is in a fade state;
Based on the in-frame complexity of the plurality of frames held in the in-frame information amount holding step and the determination result of whether or not the fade state is in the scene detection step, The prediction value calculated using the gradient of the intra-frame complexity of a plurality of frames is set as the intra-frame complexity of the subsequent frame of the plurality of frames. An intra-frame information amount prediction step in which an intra-frame complexity of a frame is set to an intra-frame complexity of the subsequent frame;
While holding the approximate value of the complexity calculated from the code amount of the frame of each picture type encoded last and the quantization scale average value, the frame corresponding to the frame of each picture type encoded last The intra-frame complexity is held as the latest actually measured complexity, and the intra-frame complexity of the subsequent frame predicted in the intra-frame information amount prediction step and the determination result of whether or not the fade state is in the scene detection step Based on the above, if not in the fade state, only the approximate value of the complexity of the I picture is corrected using the intra-frame complexity of the subsequent frame and the latest measured complexity of the I picture. Using the approximation of the complexity after the correction, the target code amount total within a certain time, and the number of frames in which each picture type is used, While setting the target code amount of the picture type, in the fade state, the target code amount is set using the intraframe complexity of the subsequent frame and the latest measured complexity of each picture type. The target code amount for each picture type is obtained by converting the approximate value of the complexity for each frame and using the target code amount sum of the assumed number of frames related to the target code amount. A code amount control step for setting
An image encoding method is provided.

  The image coding apparatus according to the present invention refers to a scene (for example, fade scene (fade-in and fade-out) in which video is surely changed smoothly by referring to the amount of information in a frame such as a frequency component and a DC component of an input image. )), And the time continuity of the information amount in the frame is used to predict the information amount in the frame of the future frame. Based on the prediction result, the code amount such as the target code amount For example, in a fade scene, it is possible to realize good image coding control in which image quality deterioration is suppressed.

  Hereinafter, an image coding apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of an image coding apparatus according to an embodiment of the present invention. Note that the image encoding device shown in FIG. 1 has components common to the above-described conventional image encoding device (see FIG. 5), and here, description of these common components is provided. Is omitted.

  The image encoding apparatus shown in FIG. 1 has a frequency component that represents an intra-frame information amount (activity (intra-frame complexity of the screen)) and a DC component that represents information related to each pixel such as a luminance average value. ) To estimate the complexity of the input frame after a certain time. That is, the image encoding apparatus shown in FIG. 1 detects a change in activity related to a plurality of frames, thereby estimating the state of the scene and predicting a future input image, and based on the prediction result, the target code By assigning the amount, for example, for an input image generated in real time, such as a camera, image encoding processing with less delay and less deterioration with respect to changes in the material in the input image is realized. On the other hand, it is possible to realize good control of the image code amount. The configuration and operation of the image encoding device shown in FIG. 1 will be specifically described below.

  The image coding apparatus according to the present invention shown in FIG. 1 includes an activity calculation circuit 301, an activity holding memory 302, a scene detection circuit 303, and an activity prediction circuit 304 in addition to the conventional image coding apparatus shown in FIG. have. These components are provided to perform processing related to activities related to the present invention. In addition to the function of the code amount control circuit 217 shown in FIG. 5, the code amount control circuit 305 further has a function for performing processing based on the processing result relating to the activity described above. Since the other constituent elements have the same functions as those of the constituent elements provided in the conventional coding apparatus shown in FIG.

  An input image signal (digital image signal) input from the input terminal 201 of the image encoding device is supplied to the input image memory 202 and also to the activity calculation circuit 301. The activity calculation circuit 301 can calculate the complexity of the image of each frame. The activity calculated by the activity calculation circuit 301 includes, for example, the level of the image of each frame calculated from the size of a difference between adjacent pixels (for example, a difference in arbitrary pixel information such as luminance and color difference signals). The value of the high frequency component in the vertical direction can be used.

Specifically, the parameter indicating the activity (activity FAct) can be calculated by accumulating the absolute difference values of the adjacent pixels on the upper, lower, left, and right sides of the input image as in the following expression.
FAct = Σ {abs (Pixel [x, y] -Pixel [x + 1, y]) + abs (Pixel [x, y] −Pixel [x, y + 1])}
/ (Width * Height * α)
However, Σ represents the integration from x to 0 to Width-2 and y from 0 to Height-2. Α is a constant, abs is an absolute value, Width is the horizontal width of the entire image, Height is the vertical height of the entire image, and Pixel is arbitrary pixel information.

  As described above, the activity calculation circuit 301 outputs, for example, an activity calculated by normalizing the sum of differences of each pixel information in accordance with the screen size Width × Height.

Furthermore, in the activity calculation circuit 301, for example, the luminance average value FDC of the input image is simultaneously calculated and output as in the following equation.
FDC = Σpixel [x, y] / (Width * Height)

  These values FAct (activity) and FDC (luminance average value) are output from the activity calculation circuit 301 and then stored in the activity holding memory 302. The activity holding memory 302 is configured to accumulate the activity FAct and the luminance average value FDC output from these activity calculation circuits 301. For example, the activity holding memory 302 stores the activity FAct and luminance average value FDC for the past N frames. At the same time, when storing a new activity FAct and luminance average value FDC, the oldest value (the value before N + 1 frame) is discarded.

  For example, an encoding system is configured with an encoding syntax (period M = 3) as shown in FIGS. 4A to 4C, and when I and P pictures are encoded, The calculated activity FAct and luminance average value FDC of the past N frames including its own frame are stored, and the previous frame and the previous frame are uncoded frames. On the other hand, in the case of encoding a B picture, the calculated activities of three future frames (one intraframe coded I / P picture) and past N-3 frames with respect to its own frame Is stored.

  On the other hand, the scene detection circuit 303 reads out the activity FAct and the luminance average value FDC stored in the activity holding memory 302 and detects the continuity of these values between frames, thereby detecting fade-in / out, etc. I do. Here, a fade-in / out detection algorithm in the scene detection circuit 303 will be described with reference to FIGS.

  FIG. 2 is the first page of a flowchart illustrating an example of a fade-in / out detection algorithm of the scene detection circuit of the image encoding device according to the embodiment of the present invention. Note that the activity and luminance average value of the latest frame (that is, the Nth frame) stored in the activity holding memory 302 are FAct (N) and FDC (N), and the previous frame (that is, N−1). The activity and luminance average value of the first frame) are FAct (N−1) and FDC (N−1), and the activity and luminance average value of the frame N frames before (ie, the 0th frame) are FAct (0) and It will be described as FDC (0). N is a constant, and P is a constant smaller than N-1.

  In the scene detection circuit 303, in order to detect a fade-in / out scene, a change in activity FAct between N frames is measured, and it is checked whether or not the activity FAct is changing uniformly. Specifically, first, variables I and J are set to J = 0 and I = J + 1, respectively (steps S1 and S2), and the magnitudes of FAct (J) and FAct (N) are compared (steps). S3).

  Here, if FAct (J) ≦ FAct (N) (“Yes” in step S3), that is, if the activity of the latest frame is greater than or equal to the activity before the NJ frame, N− It is checked whether the change in activity from before the J frame to the latest frame is in a broad monotonic relationship. That is, whether the relationship of FAct (I-1) ≦ FAct (I) ≦ FAct (N) is satisfied for all the variables I while incrementing the variable I from the initial value J + 1 to N−1. It is examined (steps S4 to S6). If the relationship of FAct (I-1) ≦ FAct (I) ≦ FAct (N) is established for all variables I, the activities from NJ frames before to the latest frame are broadly defined. It is determined that there is a possibility that the scene from the J frame to the latest frame (N frame) is in a fade-in state because of a monotonically increasing relationship (step S7: provisional fade-in determination).

  If the relationship FAct (I-1) ≦ FAct (I) ≦ FAct (N) does not hold for an arbitrary variable I in step S4, the scene from J frame to the latest frame (N frame) fades in. Therefore, the process proceeds to step S8, and the variable J is incremented by one (J = J + 1). The above-described fade determination is performed until the variable J exceeds the constant M (“No” in step S9). On the other hand, if the variable J exceeds the constant M (“Yes” in step S9), the final determination is made. Furthermore, regarding the input image to be determined (the input image of N frames stored in the activity holding memory 302), it is determined that the fade has not been detected (step S14).

  On the other hand, if FAct (J) ≦ FAct (N) is not satisfied (“No” in step S3), that is, FAct (J)> FAct (N), and the latest frame activity is NJ frames before. If the activity is smaller than the current activity, it is checked whether or not the change in activity from before the NJ frame to the latest frame is in a monotonic decrease relationship in a broad sense. In this case, whether the relationship of FAct (I-1) ≧ FAct (I) ≧ FAct (N) is satisfied for all the variables I while the variable I is incremented from the initial value J + 1 to N−1. It is checked whether or not (steps S10 to S12).

  Then, as in the case of checking whether or not there is a broad monotonic increase relationship in steps S4 to S6 described above, for all variables I, FAct (I-1) ≧ FAct (I) ≧ FAct (N ) Relationship is established, the activity from the NJ frame to the latest frame is in a monotonically decreasing relationship, and the scene from the J frame to the latest frame (N frame) may be in a fade-out state. Is determined (step S13: provisional fade-out determination). On the other hand, if the broad monotonic decrease relationship is not established, the process proceeds to step S8, where the variable J is incremented by one and the fade determination is performed again. On the other hand, if the variable J exceeds the constant M, It is determined that no fade has been detected.

  As described above, the scene detection circuit 303 performs a fade determination for a scene from NJ frames before (however, 0 ≦ J ≦ M) to the latest frame, and as a result, the activity has a monotonically increasing relationship. The temporary fade-in determination (step S7), the temporary fade-out determination (step S13) in which the activity has a monotonous decrease relationship in a broad sense, and the fade not detected (step S14) are performed.

  In addition, when a temporary fade-in determination or a temporary fade-out determination is made by the above-described processing, it is desirable to further determine whether or not the scene is a fade scene by examining a change in the luminance average value FDC. FIG. 3 is the second page of the flowchart illustrating an example of the fade-in / out detection algorithm of the scene detection circuit of the image encoding device according to the embodiment of the present invention. The temporary fade detection (step S21) shown in FIG. 3 represents a state in which the determination result of the temporary fade-in determination (step S7) or the temporary fade-out determination (step S13) is obtained in FIG.

  When the scene detection circuit 303 obtains the determination result of the temporary fade-in determination or the temporary fade-out determination (step S21), the scene detection circuit 303 further determines the determination target section (the latest frame from the J frame) where the determination result of the temporary fade detection is obtained. Observe the transition of the DC component during the interval up to the frame. In this determination, for example, it is verified whether the luminance average value FDC in the determination target section is in an increasing tendency or a decreasing tendency state.

  Specifically, first, the variable I is set to I = J + 1 (where J is a constant that identifies the determination target section from which the above-described provisional fade detection determination result is obtained) (step S22), and FDC (J ) And FDC (N) are compared (step S23).

  Here, when FDC (J) ≦ FDC (N) (“Yes” in step S23), that is, when the luminance average value of the latest frame is greater than or equal to the luminance average value before the NJ frame. It is checked whether or not the change in the average brightness value from before the NJ frame to the latest frame tends to increase. That is, while the variable I is incremented from the initial value J + 1 to N-1, the relationship of FDC (I-1)-. Beta..ltoreq.FDC (I) .ltoreq.FDC (N) +. Beta. It is checked whether or not it exists (steps S24 to S26). Note that the predetermined value β in the above expression is a value set to allow an error in the change of the luminance average value (fluctuation only between the luminance average values within ± β between adjacent frames). For example, the predetermined value β is preferably set to a value of about 1/10 of the DC difference value FDC (N) -FDC (J) from the N-J frame before to the latest frame.

  If the relationship of FDC (I-1) −β ≦ FDC (I) ≦ FDC (N) + β holds for all variables I, the frame from N−J frame to the latest frame The change in the luminance average value tends to increase, and the scene from the J frame to the latest frame (N frame) is determined to be a fade scene (step S27: fade detection). In the fade detection in step S27, the fade-in detection or the fade-out detection is determined according to the determination result of the temporary fade-in determination or the temporary fade-out determination in step S21. If the relationship of FDC (I-1) −β ≦ FDC (I) ≦ FDC (N) + β is not established for an arbitrary variable I in step S24, the scene related to the determination target section is a fade scene. (Step S32: Fade not detected).

  On the other hand, if FDC (J) ≦ FDC (N) is not satisfied (“No” in step S23), that is, FDC (J)> FDC (N), and the luminance average value of the latest frame is N−J. If it is smaller than the average luminance value before the frame, it is checked whether or not the change in activity from before the NJ frame to the latest frame tends to decrease. That is, while the variable I is incremented from the initial value J + 1 to N−1, the relationship of FDC (I−1) + β ≧ FDC (I) ≧ FDC (N) −β is established for all variables I. It is checked whether or not it is present (steps S28 to S30). As in the case of checking whether or not there is an increasing tendency in step S24 described above, in this step S28, it is desirable to set the predetermined value β described above in order to allow an error in the change of the luminance average value. .

  When the relationship of FDC (I-1) + β ≧ FDC (I) ≧ FDC (N) −β is established for all variables I, the frame from the previous N−J frame to the latest frame The change in the average luminance value tends to decrease, and the scene from the J frame to the latest frame (N frame) is determined to be a fade scene (step S31: fade detection). In the fade detection in step S31, the fade-in detection or the fade-out detection is determined according to the determination result of the temporary fade-in determination or the temporary fade-out determination in step S21. Further, if the relationship of FDC (I-1) + β ≧ FDC (I) ≧ FDC (N) −β is not established with respect to an arbitrary variable I in step S28, the scene related to the determination target section is a fade scene. (Step S32: Fade not detected).

  In the detection algorithm using the average luminance value shown in FIG. 3, the reason why both the increasing tendency and decreasing tendency of the average luminance value FDC are verified is, for example, that black characters appear on a white background (or a white ground) When the average brightness value fades from an image with a high average brightness value (or disappears), such as a black character disappears), a white character appears on a black background (or white character on the black ground) This is because it is possible to consider both fades in which an image whose luminance average value is increased from an image whose luminance average value is low (such as disappears).

  With the above detection algorithm, the scene detection circuit 303 can detect a fade scene with reference to the activity FAct and further the luminance average value FDC. When the scene detection circuit 303 detects a fade scene, the scene detection circuit 303 sends a value for specifying the fade scene to the activity prediction circuit 304 (for example, the value N for specifying the latest frame described above). And a flag indicating whether an increase (fade in) or a decrease (fade out) in activity is detected.

  The activity prediction circuit 304 predicts an activity related to a future frame in accordance with information from the scene detection circuit 303. For example, in the case of an I / P picture, the activity prediction circuit 304 encodes from the frame (N-2 frame) two frames before the current frame (N frame) including two unencoded B frames. A predicted activity value (predicted activity value PredAct) is generated until a frame (N + L frame) after a certain time (for example, after L frames), which is a control target related to the quantity control. As the predicted activity value, the measured activity value can be used for the activities from the N-2 frame to the N frame, while the activity value is predicted for the activities from the N + 1 frame to the N + L frame. The result is used.

  For a frame that the scene detection circuit 303 determines is not a fade scene, for example, the activity prediction circuit 304 assumes that there is no scene change, and the predicted activity value PredAct of each frame includes the latest frame (N frames). ) Can be output.

  On the other hand, when the scene detection circuit 303 determines that the scene is a fade scene, the predicted activity value PredAct (K) of the K frame (where the variable K is N + 1 ≦ K ≦ N + L) is J frame, N frame. , N-1 frames are predicted from the actually measured activity values.

That is, if FAct (J) ≦ FAct (N),
PredAct (K) = FAct (J)
+ Min [{FAct (N) -FAct (J)} * (KJ) / (JN), {FAct (N) -FAct (N-1)} * (KJ)]
Is predictable.

If FAct (J)> FAct (N),
PredAct (K) = FAct (J)
−min [{FAct (J) -FAct (N)} * (KJ) / (JN), {FAct (N-1) -FAct (N)} * (KJ)]
Is predictable.

  In the above two formulas, a small value of the gradient from the J frame to the N frame and the gradient from the N-1 frame to the N frame is used as the prediction gradient. Further, the band of the predicted activity value PredAct (K) calculated by the above formula is limited by the minimum assumed value MinAct and the maximum assumed value MaxAct (that is, MinAct <PredAct (I) <MaxAct).

  Similarly, in the case of a B picture, if the encoding syntax related to the current frame is taken into consideration, the frame to be encoded is an N-3 frame, including the encoded N-1 frame or N-2 frame. Since actual measurement values up to N frames are available, actual activity values can be used from N-3 frames to N frames, while from N + 1 frames to N + L frames, as in the case of I / P pictures, The activity value is predicted, and the prediction result is used.

  The code amount control circuit 305 corrects / calculates the target code amount using the predicted activity value PredAct generated / output by the activity prediction circuit 304. In the code amount control circuit 305, an approximation value C (T) = Bits of the complexity calculated from the code amount Bits (T) of each picture type encoded last and the quantization scale average value AvgQ (T) (T) * AvgQ (T) (where T is a picture type) is held, and the actual activity value SuvAct (T) of each picture type encoded last is held. Yes.

  The code amount control circuit 305 is supplied with a flag indicating whether or not it is a fade scene from the scene detection circuit 303. When the flag supplied from the scene detection circuit 303 indicates that the scene is not a fade scene, the code amount control circuit 305 uses the prediction activity value PredAct to approximate the I picture complexity C (I). Only correct. If the scene is not a fade scene, PredAct is the measured activity value of the frame of the I picture or the measured activity value of the frame closest to the next I picture.

That is, the correction formula for the approximate value C (I) of the complexity of the I picture is
Crev (I) = C (I) * PredAct (NextI) / SuvAct (I)
Can be expressed as Note that NextI indicates a frame in which the next I picture determined by the encoding syntax appears.

Further, the target code amount Budget of each picture type I, P, B can be set as follows using Crev (I) converted by the above formula.
Budget (I) = {TotalBits * Crev (I)} / {Fnum (I) * Crev (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}
Budget (P) = {TotalBits * C (P)} / {Fnum (I) * Crev (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}
Budget (B) = {TotalBits * C (B)} / {Fnum (I) * Crev (I) + Fnum (P) * C (P) + Fnum (B) * C (B)}

  On the other hand, when the flag supplied from the scene detection circuit 303 indicates that the scene is a fade scene, the code amount control circuit 305 approximates the complexity for all the assumed number of frames related to the target code amount. The value C (T) is converted for each frame to obtain the target code amount. At this time, assuming that the picture to be encoded is N, the approximate value C (T (W)) of complexity from N frames to N + L frames (where T (W) is the picture type of the W frame and W is N to N + L) ) Is corrected.

For example, if the picture type T (W) is an I picture,
Crev (W) = C (I) * PredAct (W) / SuvAct (I)
It can be corrected as follows.

For example, when the picture type T (W) is a P picture,
Crev (W) = C (P) * γ + C (P) * {PredAct (W) / SuvAct (P)} * (1-γ)
Where γ is a constant (0 <γ <1)
It can be corrected as follows.

For example, when the picture type T (W) is a B picture,
Crev (W) = C (B) * Δ + C (B) * {PredAct (W) / SuvAct (B)} * (1-Δ)
Where Δ is a constant (0 <Δ <1)
It can be corrected as follows.

Using the correction value Crev (W) of the approximate value of the complexity related to each picture type T (W), the target code amount is finally obtained by the following equation.
Budget (N) = TotalBits * Crev (N) / ΣCrev (W)
Where Σ is the sum of W = N to N + L

  The target code amount correction value finally obtained in this manner is supplied from the code amount control circuit 305 to the quantization circuit 206. The quantization circuit 206 uses the correction value of the target code amount as a new target code amount, as in the conventional image encoding process, so that the code amount of the image bitstream approaches the new target code amount. The quantization scale is controlled.

  With the above operations, fade scenes are reliably detected by referring to the input image activity and average brightness, and future frame activity (predicted activity value) is predicted using the time continuity of the activity. In addition, based on the prediction result, it is possible to correct the code amount control parameter such as the target code amount, and as a result, it is possible to suppress image quality deterioration particularly in a fade scene.

  In addition, according to the present invention, in particular, even for a scene in which a complicated video appears by fading in from a uniform scene such as black or gray, the tendency of the fade scene is recognized in advance, and the frame Since it is possible to set a target code amount for each, it is possible to assign a target code amount in a state where an amount of information that will be required in the future for a complex video is reserved in advance. As a result, the conventional technique has a disadvantage that it becomes difficult to assign an appropriate code amount to a complex image immediately after the fade-in, and even when the image quality is deteriorated. According to the present invention, it is possible to realize good code amount control particularly for a complex image immediately after the end of fade-in.

  In the above-described embodiment, image coding control for a fade scene has been mainly described. However, for a scene in which the tendency of a video changes smoothly (for example, a scene change due to a screen pan or an effect), The image coding control according to the present invention can be applied, and a good code amount control can be realized as in the case where the image coding control is applied to the above-described fade scene.

  In the above-described embodiment, hardware elements such as a circuit and a schematic block are described as examples of components of the image encoding device according to the present invention. Similar to the apparatus, these hardware elements can be realized by software (program) executable by a computer.

  The image coding apparatus according to the present invention can realize good image coding control in which image quality deterioration is suppressed in a scene (for example, a fade scene) in which a video changes smoothly. The present invention can be applied to an image encoding technique for performing image encoding of a moving image.

It is a block diagram which shows an example of the image coding apparatus in embodiment of this invention. It is the 1st page of the flowchart which shows an example of the fade-in / out detection algorithm of the scene detection circuit of the image coding apparatus in the embodiment of the present invention. It is the 2nd page of the flowchart which shows an example of the fade-in / out detection algorithm of the scene detection circuit of the image coding apparatus in embodiment of this invention. It is a figure which shows typically the arrangement | sequence of the image at the time of the process in MPEG2 image coding based on the prior art, and an output, (A) is a figure which shows the encoding system used by MPEG2 image coding, ) Is a diagram showing rearrangement of the encoding order at the time of MPEG2 image encoding, and (C) is a diagram showing the stream arrival order and the decoded image output order at the time of MPEG2 image decoding. It is a block diagram which shows an example of the general image coding apparatus which concerns on a prior art. It is a block diagram which shows an example of the general decoding apparatus concerning a prior art.

Explanation of symbols

101, 201 Input terminal 102, 218 Image stream buffer 103 Variable length decoding circuit 104, 215 Encoding table 105, 212 Inverse quantization circuit 106, 208 Motion compensation prediction circuit 107, 210 Adder 108, 211 Deblock circuit 109, 209 Reference image memory 110 Output frame memory 111, 213 Inverse orthogonal transformation circuit 112, 219 Output terminal 202 Input image memory 203 Two-dimensional block transformation circuit 204 Subtractor 205 Orthogonal transformation circuit 206 Quantization circuit 207 Motion vector detection circuit 214 Coding circuit 216 Multiplexers 217 and 305 Code amount control circuit 301 Activity calculation circuit 302 Activity holding memory 303 Scene detection circuit 304 Activity prediction circuit

Claims (2)

An image encoding device for encoding an image signal,
Intraframe information amount calculation means for calculating an intraframe complexity and a DC component for each of a plurality of temporally continuous frames of the image signal to be encoded and outputting the calculation result as an intraframe information amount When,
An intra-frame information amount holding means for holding the intra-frame complexity for each of the plurality of frames output from the intra-frame information amount calculation means;
Based on the intra-frame complexity and the DC component of the plurality of frames held in the intra-frame information amount holding means, by verifying temporal changes of the intra-frame complexity and the DC component of the plurality of frames, Scene detection means for determining whether or not a scene displayed by the plurality of frames is in a fade state;
Based on the in-frame complexity of the plurality of frames held in the in-frame information amount holding unit and the determination result of whether or not the fade state is in the fade state by the scene detection unit , The prediction value calculated using the gradient of the intra-frame complexity of a plurality of frames is set as the intra-frame complexity of the subsequent frame of the plurality of frames. An intra- frame information amount prediction means for setting an intra-frame complexity of a frame to an intra-frame complexity of the subsequent frame;
While holding the approximate value of the complexity calculated from the code amount of the frame of each picture type encoded last and the quantization scale average value, the frame corresponding to the frame of each picture type encoded last The intra-frame complexity is held as the latest measured complexity, and the intra-frame complexity of the subsequent frame predicted by the intra-frame information amount prediction unit and the determination result of the fade state by the scene detection unit Based on the above, if not in the fade state, only the approximate value of the complexity of the I picture is corrected using the intra-frame complexity of the subsequent frame and the latest measured complexity of the I picture. Each of the picture types using the approximate value of the complexity after the correction, the target code amount sum in a certain time, and the number of frames in which each picture type is used. While setting the target code amount, in the fade state, the assumption related to the target code amount using the intraframe complexity of the subsequent frame and the latest measured complexity of each picture type. For all the numbers of frames, the approximation value of the complexity is converted for each frame, and the target code amount of each picture type is set using the target code amount sum of the assumed number of frames related to the target code amount. Code amount control means,
An image encoding apparatus having the same. An image encoding method executed by an image encoding device that encodes an image signal,
An intra-frame information amount calculation step of calculating an intra-frame complexity and a DC component for each of a plurality of temporally continuous frames of the image signal to be encoded and outputting the calculation result as an intra-frame information amount When,
An intra-frame information amount holding step for holding the intra-frame complexity for each of the plurality of frames output in the intra-frame information amount calculation step;
Based on the intra-frame complexity and the DC component of the plurality of frames held in the intra-frame information amount holding step, by verifying temporal changes of the intra-frame complexity and the DC component of the plurality of frames, A scene detection step for determining whether or not a scene displayed by the plurality of frames is in a fade state;
Based on the in-frame complexity of the plurality of frames held in the in-frame information amount holding step and the determination result of whether or not the fade state is in the scene detection step, The prediction value calculated using the gradient of the intra-frame complexity of a plurality of frames is set as the intra-frame complexity of the subsequent frame of the plurality of frames. An intra-frame information amount prediction step in which an intra-frame complexity of a frame is set to an intra-frame complexity of the subsequent frame;
While holding the approximate value of the complexity calculated from the code amount of the frame of each picture type encoded last and the quantization scale average value, the frame corresponding to the frame of each picture type encoded last The intra-frame complexity is held as the latest actually measured complexity, and the intra-frame complexity of the subsequent frame predicted in the intra-frame information amount prediction step and the determination result of whether or not the fade state is in the scene detection step Based on the above, if not in the fade state, only the approximate value of the complexity of the I picture is corrected using the intra-frame complexity of the subsequent frame and the latest measured complexity of the I picture. Using the approximation of the complexity after the correction, the target code amount total within a certain time, and the number of frames in which each picture type is used, While setting the target code amount of the picture type, in the fade state, the target code amount is set using the intraframe complexity of the subsequent frame and the latest measured complexity of each picture type. The target code amount for each picture type is obtained by converting the approximate value of the complexity for each frame and using the target code amount sum of the assumed number of frames related to the target code amount. A code amount control step for setting
An image encoding method.

Images