JP2009153102A - Motion vector detecting device, inter-frame encoding device, moving image encoding device, motion vector detection program, inter-frame encoding program and moving image encoding program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform real-time processing without reducing detection accuracy of a motion vector even when the motion is large, and to perform real-time processing even when there is a frame that requires a long time for encoding processing. <P>SOLUTION: The present invention relates to a motion vector detecting device for obtaining a motion vector by determining positions of similar blocks on a reference frame for the unit of a block obtained by longitudinally and laterally dividing an input frame. The motion vector detecting device includes: a motion search means for detecting the motion vector within a search range specified on the reference frame; and a search range control means for determining a search range of each block and indicating the search range to the motion search means so that a total search ranges of all blocks within the frame does not exceed an allowable search range amount determined for the frame. Furthermore, it may also be preferable to include a computational complexity control means for determining computational complexity for each of frames so as not to exceed computational complexity determined for a plurality of frames and indicating the determined computational complexity to an encoding means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a motion vector detection device, an inter-frame encoding device, a moving image encoding device, a motion vector detection program, an inter-frame encoding program, and a moving image encoding program. It can be applied.

  When transmitting moving images in digital format through various communication paths for video on demand, video conferencing, digital TV broadcasting, etc., the amount of uncompressed data captured using a video capture device is enormous. It is difficult to send. It is necessary to reduce the amount of data by deleting a part that is redundant to some extent. In video conferences, digital TV broadcasts, and the like, captured images need to be compressed (encoded) in real time and transmitted.

  Conventionally, this type of moving image real-time encoding apparatus includes intra-frame encoding (compressed in blocks using local similarity of pixels in a frame) and inter-frame encoding (divides a frame into blocks, A motion vector indicating how much each block has moved between frames is obtained, and an encoding output using this motion vector is used together, and one of the encoded outputs is selected according to the image content.

  In some cases, the search range of each block when obtaining a motion vector is constant, but in order to reduce the time required for the motion search of the block, the search range is adjusted from the size of the motion vector of the already encoded peripheral block. In some cases.

Note that techniques that speed up processing other than the motion vector search function in consideration of real-time encoding are described in Patent Document 1 and Patent Document 2.
Japanese Patent Laid-Open No. 10-210475 JP 2000-232644 A

  Depending on the performance of the hardware, it is necessary to narrow the motion vector search range in advance. Even when the search range is adjusted based on the motion vector size of the peripheral blocks that have already been encoded, the maximum range of the search range needs to be narrowed depending on the performance of the hardware.

  However, if the search range is narrowed, there is a problem in that a moving image with a large amount of motion cannot find a matching block well, resulting in poor image quality. In the case where the search range is narrowed, if a scene with large motion continues for a long time, images with poor image quality continue, and if a period with motion and a period without motion continue, the image quality becomes unstable.

  Of course, if processing is performed without narrowing the search range, processing delays may accumulate and real-time processing may become difficult.

  Therefore, a motion vector detection device, an inter-frame encoding device, a motion vector detection program, and an inter-frame encoding program that enable real-time processing without reducing detection accuracy and image quality even when the motion is large is desired. In addition, a moving picture coding apparatus and a moving picture coding program that can obtain a coded image with stable image quality by real-time processing are desired for a plurality of frames.

  The first aspect of the present invention is a motion vector detection device that obtains a motion vector by obtaining a position of a similar block on a reference frame in block units obtained by dividing an input frame vertically and horizontally. Motion search means for detecting a motion vector in the search range; and (2) determining the search range of each block so that the sum of the search ranges of all the blocks in the frame does not exceed the search range allowable amount determined for the frame. And a search range control means for instructing the motion search means.

  The second aspect of the present invention is a difference between an input block obtained by dividing an input frame vertically and horizontally and a reference block corresponding to the input block in the reference frame whose position is determined according to a motion vector detected by a built-in motion vector detection device. In the inter-frame encoding apparatus that encodes the above, the apparatus of the first aspect of the present invention is applied as the motion vector detection apparatus.

  According to a third aspect of the present invention, there is provided a motion vector detection program for obtaining a motion vector by obtaining a position of a similar block on a reference frame in units of blocks obtained by dividing an input frame vertically and horizontally, and comprising: (1) a reference frame; (2) a motion search means for detecting a motion vector in the specified search range above, and (2) the sum of the search ranges of all the blocks in the frame does not exceed the search range tolerance determined for the frame. A search range is determined and functioned as search range control means for instructing the motion search means.

  The fourth aspect of the present invention is a difference between an input block obtained by dividing an input frame vertically and horizontally and a reference block corresponding to the input block in the reference frame whose position is determined according to a motion vector detected by a built-in motion vector detection program. In the inter-frame encoding program described so as to be executable by a computer, the program of the third aspect of the present invention is applied as the motion vector detection program.

  According to a fifth aspect of the present invention, in the moving image encoding apparatus that encodes a moving image for each frame, (1) encoding means for encoding each frame and capable of switching a processing calculation amount required for encoding; (2) Depending on the contents of each frame for each frame in the frame unit so that the predetermined number (any value of 2 or more) does not exceed the total amount of calculation predetermined for the continuous frame unit And (3) a calculation amount control unit for controlling encoding by the encoding unit so as to follow the calculation amount allocated to each frame.

  A sixth aspect of the present invention is a moving picture coding program installed in a computer constituting a moving picture coding apparatus for coding a moving picture for each frame, and (1) encoding each frame. Encoding means capable of switching the amount of processing calculation required for encoding, and (2) a predetermined number (any value greater than or equal to 2) exceeds a predetermined total calculation amount for a continuous frame unit Calculation amount distribution means for allocating the calculation amount to each frame in the frame unit according to the contents of each frame, and (3) the encoding means so as to follow the calculation amount allocated to each frame. It is made to function as a calculation amount control means for controlling the encoding according to.

  According to the motion vector detection device, the inter-frame coding device, the motion vector detection program, and the inter-frame coding program of the present invention, real-time processing can be performed without degrading detection accuracy and image quality even when the motion is large. Can do.

  Further, according to the moving image encoding apparatus and the moving image encoding program, it is possible to obtain encoded images with stable image quality for a plurality of frames by real-time processing.

(A) First Embodiment Hereinafter, a first example in which a motion vector detection device, an interframe coding device, a motion vector detection program, and an interframe coding program according to the present invention are applied to a real-time coding device for moving images. Embodiments will be described in detail with reference to the drawings.

  Here, all or a part of the motion vector detection device and the inter-frame coding device of the first embodiment can be constructed by hardware, but can also be constructed by a CPU and a program executed by the CPU. The program may be introduced into the apparatus via a recording medium, or may be introduced into the apparatus via communication processing (for example, download) (second and third described later). The embodiment is also the same).

(A-1) Configuration of First Embodiment FIG. 2 is a block diagram showing an overall configuration of a moving image real-time encoding device 10 to which the inter-frame encoding device of the first embodiment is applied.

  In FIG. 2, the input buffer 11 records each frame of the captured moving image (one image constituting the moving image). The intra-frame coding unit 12 acquires a frame from the input buffer 11 and performs intra-frame coding (compression in units of blocks using local similarity of pixels in the frame). On the other hand, the inter-frame coding unit 13 The frame is divided into blocks, and a motion vector (see FIG. 3) indicating how much each block has moved between frames is obtained and encoded. The output buffer 14 stores the output from the encoding unit 12 or 13.

  Here, each frame of the moving image input to the input buffer 11 includes identification information indicating whether to perform intraframe encoding (for example, I frame) or interframe encoding (for example, P frame). Shall. However, the input buffer 11 may determine which encoding method to apply based on, for example, the sum of squared differences of pixel values at corresponding positions that follow each other.

  FIG. 1 is a block diagram showing a detailed configuration of the interframe encoder 13 (interframe encoder of the embodiment).

  In FIG. 1, the inter-frame encoding unit 13 includes a block dividing unit 21, a motion search unit 22, a motion compensation unit 23, a transform quantization inverse quantization inverse transform unit 24, an encoding unit 25, a reference buffer management unit 26, and a search. A range control unit 27 is included. Here, the motion search part 22 and the search range control part 27 comprise the motion vector detection apparatus of embodiment.

  The block dividing unit 21 divides one frame into a plurality of blocks having a predetermined size and gives the same to the motion search unit 22.

  The motion search unit 22 searches for a block most similar to the input block from the reference frame (frame reconstructed after inversely transforming the already encoded one) stored in the reference buffer management unit 26. The position shift is output to the motion compensation unit 23 and the search range control unit 27 as a motion vector. In the case of this embodiment, the search range by the motion search unit 22 is a range determined by the search range control unit 27.

  The motion compensation unit 23 extracts the block of the reference frame that is most similar from the reference frame using the motion vector (the extracted block is appropriately called a prediction block), the transform quantization inverse quantization inverse transform unit 24, and the search range This is given to the control unit 27.

  The search range control unit 27 includes a difference between the prediction block obtained from the motion compensation unit 23 and the original block (input block), a motion vector obtained from the motion search unit 22, and a predetermined search range. The motion search range is determined based on the total allowable amount and some threshold values. A method for determining the motion search range will be described later in detail.

  The transform quantization inverse quantization inverse transform unit 24 performs a discrete cosine transform (DCT transform) on the difference between the prediction block and the original block, performs quantization, and gives this quantized data to the encoding unit 25. . In addition, the transform quantization inverse quantization inverse transform unit 24 performs inverse quantization on the above-described quantized data, and then performs inverse DCT transform to return the difference data to the reference buffer management unit 26. is there.

  The reference buffer management unit 26 adds the difference data obtained from the transform quantization inverse quantization inverse transform unit 24 to the corresponding block of the stored reference frame to form a new reference frame. As described above, the constructed reference frame is used as a reference frame at the time of encoding the next frame.

  The encoding unit 25 performs entropy encoding on the quantized data obtained by the transform quantization inverse quantization inverse transform unit 14.

(A-2) Overview of Search Range Control Method in the First Embodiment In the first embodiment, the search range control unit 27 that can suppress the calculation amount without narrowing the maximum search range is provided. Has great features. Below, the outline | summary of the control method of the search range which the search range control part 27 is applying is demonstrated.

The motion search by the motion search unit 22 is an operation of searching for a block similar to the input block from the reference frame. For example, assuming that the reference frame is R and the input block is O, this operation is performed by the position (x + 1, y + m) that minimizes the absolute difference sum (Sum of Absolute Difference; hereinafter referred to as SAD) shown in Equation (1). ). It can be said that the smaller the SAD, the more similar.

  In the formula (1), abs (xx) is the absolute value of xx. R (x + l + i, y + m + j) is a value at the position (x + l + i, y + m + j) of the reference frame R. For example, the luminance value (or R (red), G (green), B (blue) value, or C ( Color component values such as cyan), M (magenta), and Y (yellow). Since the range of i changes by the block width and the range of j changes by the block height, it can be seen that R (x + l + i, y + m + j) represents a block of the reference frame R. O (i, j) is a value at the position (i, j) of the input block O, and is, for example, a luminance value (or RGB value, CMY value, etc.). Since the range of i changes by the block width and the range of j changes by the block height, it can be seen that O (i, j) represents the input block O itself.

  The range of x is from 0 to the image width X, the range of y is from 0 to the image height Y, and (x, y) is the reference point of the search, that is, the upper left pixel of the block O on the image It is a relative position with the position as the reference point (0, 0). The range of i can take from x to the block width, and the range of j can take from y to the block height.

  l is a parameter that defines the search range in the x-axis direction, and m is a parameter that defines the search range in the y-axis direction. If the search range information in the x-axis direction is L and the search range information in the y-axis direction is M, the values of l and m are changed in the range of −L ≦ l ≦ L and −M ≦ m ≦ M. 1) A position (x + l, y + m) where the SAD of the equation is the minimum is searched. This minimum position (x + 1, y + m) represents a motion vector.

  The search ranges L and M (in other words, −L ≦ l ≦ L, −M ≦ m ≦ M) may be constant in each block in the frame, and may be between predetermined maximum ranges and minimum ranges. It can also be made variable.

  The meaning of the various parameters will also be clarified in the description of the relationship between the search range and the input block using FIG. 4 described later.

  In the first embodiment, such a method of determining the search range is not adopted, and the total allowable amount of the search range is determined in each frame, and the sum of the search ranges of the blocks in the frame is the total allowable search range. The size of the search range in each block is controlled so as not to exceed the capacity.

  By doing so, the amount of calculation required for the search in one frame can be limited to a certain value or less. The total allowable amount of the search range can be regarded as a pool of search ranges. In each block, the required search range is extracted from this pool and searched based on the value.

  Further, in the first embodiment, based on the difference between the reference block and the block obtained as a result of the motion search and the magnitude of the motion vector, the prediction search range for the peripheral blocks of the block is determined, and the unprocessed block This is reflected in the determination of the search range. The predicted search range is a predicted value for determining a search range that an unprocessed block should take.

(A-3) Operation of the First Embodiment Hereinafter, the operation of the interframe coding unit 13, particularly the operation related to the detection of the motion vector will be described.

  In the motion search unit 22, the block position most similar to the input block (determined based on a certain evaluation criterion (for example, the above-mentioned SAD)) within the search range designated by the search range control unit 27 is determined. Search from the reference frame and calculate the motion vector. The motion compensation unit 23 obtains a block (hereinafter referred to as a reference block) in the reference frame obtained from the motion vector. The search range control unit 27 determines a search range. Basically, the search range is extracted from the search range allowable TotalRange that can be said to be a pool of search ranges, and the search is executed based on the search range. The search range control unit 27 performs control so that the search range within the search range allowable amount TotalRange is not used up.

  The search range allowable amount TotalRange can be expressed, for example, by the number of blocks × appropriate search range (not necessarily so), and the appropriate search range is the maximum search range (maxRange; the maximum value that is twice the above-described L and M) ) And the minimum search range (minRange; minimum value twice as large as L and M described above). For example, when the maximum search range is 16 (the number of pixels on one side), the minimum search range is 4, the image size is QVGA (320 × 240), and the block size is 16 × 16 pixels, the search range allowable TotalRange is 8 × ( 320 × 240 ÷ 16 ÷ 16) = 2400. Here, 8 is an appropriate search range, and (320 × 240 ÷ 16 ÷ 16 = 300) is the number of blocks. The search range allowable amount TotalRange takes a value within the range of (number of blocks−1) × minRange + maxRange ≦ TotalRange ≦ number of blocks × maxRange, for example. In the case of the above example, 1212 (= 4 × 299 + 16) ≦ TotalRange ≦ 4800 (= 16 × 300). The minimum value of the range (number of blocks −1) × minRange + maxRange is the maximum search range maxRange for the first searched block that affects the subsequent search in order to improve accuracy. It became like the formula.

  FIG. 4 is an explanatory diagram showing the relationship between the search range and the input block.

  FIG. 4 basically shows a reference frame. In the reference frame, a block that can be a reference block ranges from a block 0 having 13 × 13 pixels having (0,0) as an upper left coordinate to a block 1023 having 13 × 13 pixels having (31,31) as an upper left coordinate. There are 1024 pieces.

  In FIG. 4, a solid line block of 13 × 13 pixels whose upper left coordinate is (9, 10) is an input block O (i, j). In FIG. 4, the block size is changed from the above-described example due to the size restriction of the drawing. In the input block O (i, j), i takes a value in the range 9-21, and j takes a value in the range 10-22.

  If L and M that define the search range are each 4, the upper left coordinate (9, 10) of the input block O (i, j) that is the motion vector detection target is the center, and ± L (= ± 4) in the x direction. ), And a broken line rectangle shifted by ± M (= ± 4) in the y direction is a search range, and a reference block of 13 × 13 pixels with a coordinate located in the search range as an upper left coordinate is input block O The SAD with (i, j) is determined. In the case of the input block of FIG. 4, the one-dot chain line block R (9−2 + i, 10−3 + j) whose upper left coordinate is (9−2, 10−3) and the upper left coordinate is (9−3, 10 + 2). A two-dot chain line block R (9-3 + i, 10 + 2 + j), a three-dot chain line block R (9 + 2 + i, 10 + 3 + j) whose upper left coordinates are (9 + 2, 10 + 3), and the like are reference blocks.

  The search range control unit 27 determines the search range by the following procedures T1 to T4, for example. FIG. 5 is a flowchart showing processing portions related to procedures T2 and T3, and shows processing for one input block.

(Procedure T1)
A search range total allowable amount (allowable amount per frame) TotalRange is set (initialized) in a parameter (remaining search range allowable amount) RemainRange indicating the remaining search range allowable amount. That is, RemainRange = TotalRange.

  When the processing of a certain block is completed, after confirming that the search is not executed for all the blocks in the frame, it is confirmed whether or not the input block to be processed is the first block in the frame.

  If the input block to be processed is the first block in the frame, the procedure T2 is executed, and if the input block to be processed is the second and subsequent blocks, the procedure T3 is executed.

(Procedure T2)
When the input block is the first in the frame, the search range control unit 27 sets a predetermined value as the search range. For example, the parameter Range (= L, M) representing the search range is set as the maximum allowable search range maxRange. As a result, the motion search unit 22 performs a search within the maximum search range maxRange.

  In addition, the search range control unit 27 decreases the remaining search range allowable amount RemainRange by the search range Range from the current value. That is, RemainRange = RemainRange-Range.

(Procedure T3)
When the input block is the second or later in the frame, the search range Range is based on the predicted search range of the peripheral block (see FIG. 6 described later) for which the motion vector has already been determined and the remaining search range allowance RemainRange. To decide. The peripheral blocks used for determining the search range Range are, as illustrated in FIG. 6, an upper left block BA, an upper block BB, an upper right block BC, and an adjacent block to the left of the input processing target block. BD. However, as will be described later in procedure T4, it is assumed that predicted search ranges a, b, c, and d are obtained for each of the peripheral blocks BA, BB, BC, and BD.

  The search range Range is calculated according to the formula (2), for example.

Range = Max (Min (Int (WA2 (wa, rr)), maxRange), minRange) (2)
In equation (2), wa applied the function WA1 (a, b, c, d) to the prediction search ranges a, b, c, d (minRange to maxRange) of the neighboring blocks BA, BB, BC, BD. As the function WA1 (a, b, c, d), a weighted average value as shown in the equation (3) can be applied. The weighting coefficients α, β, γ, and φ in equation (3) are set to 0 when there are those that cannot be obtained in the surrounding blocks. rr is a value obtained by dividing the remaining search range allowable amount RemainRange by the number of remaining blocks as shown in the equation (4). The function WA2 (x, y) in the equation (2) is a weighted average value of the two values x and y as shown in the equation (5). Int (x) represents a value obtained by rounding off x. Max is a function that selects the larger of the two arguments, and Min is a function that selects the smaller of the two arguments.

WA1 (a, b, c, d) = αa + βb + γc + φd
However, 0 ≦ α, β, γ, φ ≦ 1, α + β + γ + φ = 1
(For example, α, β, γ, φ = 0.25) (3)
rr = RemainRange / number of remaining blocks (4)
WA2 (x, y) = δx + σy
However, 0 ≦ δ, σ ≦ 1, δ + σ = 1
(For example, δ, σ = 0.5) (5)
Equation (2) is assigned to the remaining blocks determined by dividing the search range candidate wa determined from the prediction search ranges of the neighboring blocks BA, BB, BC, and BD and the remaining search range allowable amount RemainRange by the number of remaining blocks. Basically, the weighted average value WA2 (wa, rr) with the search range candidate rr per one swinging block is set as the search range candidate. Since the value WA2 (wa, rr) may take a value other than an integer value, rounding is performed. The functions Max and Min are functions for causing the candidate Int (WA2 (wa, rr)) after the rounding process to fall within the range between the minimum search range minRange and the maximum search range maxRange.

  That is, by obtaining the search range Range as described above, the average search that can be currently assigned while reflecting the influence of the peripheral blocks that have already been processed in the determination of the search range Range of the block currently being processed. The search range Range is determined so as to be as close as possible to the range rr.

  However, as an exception, when the value rr (= RemainRange / number of remaining blocks) is smaller than the minimum search range minRange, the minimum search range Range is set as the minimum search range minRange. Otherwise, a search range smaller than the minimum search range minRange may be assigned. For example, when the minimum search range is 4, if the remaining search range allowable amount RemainRange is 100 and the number of remaining blocks is 26, the search range Range = minimum search range minRange.

  When the search range Range for the current input block is determined, the remaining search range allowable amount Remain is reduced by that amount. That is, RemainRange = RemainRange-Range.

(Procedure T4)
When the motion search for the input block to which the determined search range Range is applied is completed, the above-described predicted search range used for determining the search range used in the motion search of the next block is obtained. For example, the predicted search range is obtained by the following method.

  When the SAD between the reference block obtained by the motion search and the block being processed is large (in other words, the possibility that the search is not successful) is small, the search range of the unprocessed peripheral blocks is increased. In order to achieve this, the prediction search range is increased so that the search for the unprocessed peripheral blocks can be performed better. Conversely, when the SAD is small and the motion vector is large, in order to reduce the search range of unprocessed peripheral blocks, the prediction search range is reduced, and the amount of calculation in the unprocessed peripheral blocks is somewhat reduced. Such a determination method is described in a program as follows.

IF (SAD> g and magnitude of motion vector <m)
Prediction search range = Min (Range + p, MaxRange)
ELSE IF (SAD <f and magnitude of motion vector> n)
Prediction search range = Min (Range-q, MinRange)
ELSE
Prediction search range = Range
Here, g and f are integers satisfying 0 <g, f <(maximum value of SAD), respectively.

The maximum value of SAD is 16 × 16 × 255 (= 65280) when the maximum luminance value is 255 in a block of 16 × 16 pixels. p and q are positive integers and there is no particular range, but when adding or subtracting from the search range Range, a value that is less likely to exceed the maximum search range MaxRange or cross the minimum search range MinRange is applied. .

(A-3) Effects of the First Embodiment According to the first embodiment, the total search range of each block is equal to or less than a certain value using the search range allowable amount TotalRange and the remaining search range allowable amount RemainRange. Thus, the amount of calculation by searching in each frame can be suppressed. For example, in the case of a search method with a constant motion search range, a calculation amount of a (maxRange × number of blocks) (a is an arbitrary positive number) is required. Although it is considered that the calculation amount of the same a (maxRange × number of blocks) is required, in this embodiment, the calculation amount of a × TotalRange is required, and the search range permission is expressed as TotalRange <(maxRange × number of blocks). When the capacity TotalRange is determined, the amount of calculation can be reduced as compared with the conventional method, and a search using the maximum search range maxRange can be performed in a necessary part. In other methods, there is no compensation that can realize the calculation amount of a × TotalRange unless the maximum search range maxRange is reduced.

(A-4) Modified Embodiment of First Embodiment In the first embodiment, an interframe coding apparatus (interframe coding section 13) is applied to the coding apparatus having the configuration shown in FIG. However, the application target is not limited to the configuration shown in FIG. 2, and can be widely applied to an encoding device using an inter-frame encoding device.

  In addition, the motion vector detection apparatus according to the first embodiment is not limited to the moving picture encoding apparatus. For example, when an image processing apparatus that performs processing for extracting and tracking a moving body incorporates a motion vector detection device, the present invention is also applied to such a motion vector detection device. can do.

  In the first embodiment, the search range allowable amount TotalRange is fixed, but the search range allowable amount TotalRange may be changed. For example, when the remaining search range allowance RemainRange after determining the search range of all the blocks of a certain frame is larger than the threshold, the search range allowance TotalRange of the next frame is set to the reference search range allowance TotalRange. A predetermined amount may be added.

  In the first embodiment, the peripheral blocks that affect the determination of the search range are shown as four blocks on the left upper left, right above, right upper right, and left adjacent, but the peripheral blocks are limited to this. It is not something. For example, it may be three blocks right above, diagonally right above, and adjacent to the left.

  In the first embodiment, the index for evaluating the degree of approximation between the input block and the reference block is SAD, but another index may be used. For example, it may be the square root of the sum of squared differences.

(B) Second Embodiment Next, an embodiment of a moving picture coding apparatus and a moving picture coding program according to the present invention (hereinafter referred to as a second embodiment) will be described in detail with reference to the drawings.

  Here, all or part of the moving picture coding apparatus according to the second embodiment can be constructed by hardware, but can also be constructed by a CPU and a program executed by the CPU.

  In the first embodiment described above, the calculation amount of each block is adjusted within one frame. However, this second embodiment does not adjust the calculation amount within one frame. This is an attempt to adjust the amount of calculation required to encode each frame in units of multiple frames, considering the variation in the amount of calculation required for each frame, and the amount of calculation in units of multiple frames taking into account real-time processing. This is an attempt to allocate a large amount to a frame that requires a large amount of calculation and a small amount to a frame that requires a small amount of calculation.

(B-1) Configuration of Second Embodiment As described above, the moving image encoding apparatus according to the second embodiment assigns a calculation amount required for encoding each frame in units of a plurality of frames. FIG. 7 is a block diagram showing the overall configuration of the moving picture encoding apparatus according to the second embodiment.

  In FIG. 7, the moving picture encoding apparatus according to the second embodiment includes a calculation amount control buffer 31, a calculation amount prediction unit 32, a calculation amount control unit 33, and an encoding unit 34.

  The calculation amount control buffer 31 is a storage device that takes in and stores a plurality of frames for performing calculation amount control. The calculation amount control is control for assigning the calculation amount to each frame so that the sum of the calculation amounts required for encoding each frame in the buffer does not exceed a predetermined total calculation amount.

  FIG. 8 is an explanatory diagram of a two-surface configuration in the calculation amount control buffer 31. The calculation amount control buffer 31 includes two buffers 31-1 and 31-2 that can be accessed in parallel. In the calculation amount control buffer 31, when one buffer 31-1 (or 31-2) is applied for image input, the other buffer 31-2 (or 31-1) is applied for encoding processing. By performing switching control in this manner, image (frame) input and encoding processing can be performed simultaneously. Here, the capacity of each of the buffers 31-1 and 31-2 is selected as a capacity capable of storing a predetermined number (two or more) of frame images.

  The calculation amount prediction unit 32 roughly calculates (predicts) a calculation amount considered necessary for encoding each frame. The calculation amount prediction unit 32 predicts the calculation amount using the variance of pixel values and the like for a frame to be encoded by the intraframe encoding method. For example, the range that the variance can take is divided into a plurality of ranges, a table in which the calculation amount of the range is associated with each range is prepared, and the calculation amount of the range to which the obtained variance belongs is set as the predicted calculation amount . Further, for example, a conversion function for converting the variance into the calculation amount may be determined in advance, and the prediction calculation amount may be obtained by inputting the variance obtained in the conversion function. The calculation amount prediction unit 32 predicts the calculation amount using the absolute value sum (SAD described above) of the difference from the pixel value of the reference frame for a frame to be encoded by the inter-frame encoding method. For example, the range that can be taken by the sum of absolute values of differences is divided into a plurality of ranges, and a table in which the amount of calculation of the range is associated with each range is prepared. The calculation amount is set as a predicted calculation amount. Alternatively, for example, a conversion function for converting the sum of absolute values of differences into a calculation amount may be determined in advance, and the prediction calculation amount may be obtained by inputting the sum of absolute values of the differences obtained in this conversion function.

  The calculation amount control unit 33 assigns the calculation amount of each frame based on the calculation amount necessary for each frame predicted by the calculation amount prediction unit 32. The calculation amount control unit 33 controls the encoding unit 34 to perform encoding on the frame based on the assigned calculation amount.

  The encoding unit 34 performs encoding for each frame while adjusting the calculation amount under the control of the calculation amount control unit 33. For example, in the case of intra-frame encoding, the encoding unit 34 adjusts the amount of calculation of encoding by adjusting the number of prediction modes in the intra-frame encoding to be performed. For example, Japanese Unexamined Patent Application Publication No. 2006-148419 discloses that the calculation amount of encoding varies depending on the number of modes. For example, in the case of inter-frame encoding, the encoding unit 34 adjusts the calculation amount of encoding by adjusting the search range of a motion vector applied to the frame.

(B-2) Operation of Second Embodiment Next, the operation of the moving image encoding apparatus of the second embodiment will be described with reference to the drawings.

  The frame (image) input to the moving image encoding apparatus is input to the calculation amount control buffer 31, and is recorded in one of the buffers 31-1 and 31-2 that are two screens. When the buffer that is recording the input frame is clogged, the recorded input buffer is switched for encoding processing, and the encoding processing buffer is switched for input. During the encoding process using the encoding process buffer, a newly input frame is recorded in the input buffer in parallel with the encoding process (see FIG. 8). Here, the number of frames (hereinafter referred to as NumOfFrame) entering one buffer 31-1 or 31-2 of the calculation amount control buffer 31 is determined in advance. A large number of storage frames allows more flexible control and higher coding efficiency, but on the other hand, the delay between the input of moving images and the output of moving image codes increases, and the memory usage also increases. Increase. The number of frames NumOfFrame may be determined in consideration of the trade-off between the two.

  FIG. 9 is a flowchart showing encoding processing in a period from when one buffer 31-1 (or 31-2) is switched for encoding processing to when the buffer 31-1 is switched for input next time. is there.

  When the buffer is switched, the calculation amount prediction unit 32 predicts the calculation amount required for encoding each frame stored in the encoding processing buffer of the calculation amount control buffer 31 (step S1).

  For example, in the case of a frame for which intraframe coding is performed, prediction is performed based on the complexity of the frame. As a simple index for capturing the complexity, for example, dispersion of pixel values of the frame can be cited. Here, as an intra-frame coding method, based on several prediction modes, a pixel value of a block can be predicted, and a coding method using a difference from a pixel of the original image can be raised, and the image is complicated. The more it is necessary, the more complicated prediction mode must be tried. Therefore, a parameter that reflects the complexity of the image is applied as a parameter for predicting the calculation amount.

  Also, for example, in the case of inter-frame coding, the amount of calculation in coding changes depending on how much the change is compared with the reference frame, so the amount of calculation required for coding is predicted due to the difference from the reference frame. To do. As an index for capturing the difference from the reference frame, for example, the sum of absolute values of differences from the pixel values of the reference frame for which motion compensation is not performed can be given. In the case of inter-frame coding, the difference from the corresponding range in the reference frame is encoded. Generally speaking, the larger the difference from the reference frame, the wider the range for searching for the corresponding range. If search is not performed (increasing the amount of calculation), the encoding efficiency cannot be increased. Therefore, the amount of calculation required for encoding is predicted based on the difference from the reference frame.

  The calculation amount control unit 33 allocates the calculation amount to each frame based on the predicted value of the calculation amount in the encoding process of each frame (step S2).

  The total amount of calculation (TotalComputation) for the number of NumOfFrame frames is determined in advance before encoding starts. Total Computation takes into account the processing time in the calculation amount prediction unit 32 and the calculation amount control unit 33, and the time required for encoding the number of NumOfFrame frames is the number of frames corresponding to the number of NumOfFrame frames. Set so that the time required for input in 2 is not exceeded. By doing this, even if there is a delay for inputting the number of NumOfFrame frames, the real-time property of the encoding can be maintained (when the input buffer is clogged, the encoding processing is performed). The encoding of the frame in the buffer has been completed).

Next, a method for assigning the amount of calculation in the encoding process to each frame will be described. Various allocation methods are conceivable, but the method considered to be the simplest method is a method of calculating according to the equation (6).

  In Expression (6), AmountOfComputation [i] represents the amount of calculation allocated to the i-th frame (i is 0 to NumOfFrame-1). α [i] is a parameter for absorbing a difference between encoding methods that take different values depending on whether intra-frame encoding or inter-frame encoding. α [i] takes a predetermined value within the range of 0 <α [i] <1 when the i-th frame is intra-frame coding, and is 1 when inter-frame coding. . This α [i] is a value obtained by the calculation amount of the intraframe encoding process with respect to the calculation amount of the interframe encoding process, depending on the implementation. PredComp [i] is the calculation amount of the i-th frame predicted by the calculation amount prediction unit 32. Therefore, the expression (6) is obtained by multiplying the prediction calculation amount PredComp [i] by α [i] to absorb the difference of the encoding method and then dividing the total calculation amount TotalComputation of the i-th frame. This indicates that the allocation calculation amount is MountOfComputation [i].

  The calculation amount control unit 33 switches the processing target frame while incrementing the parameter I from 0, and at the time of the encoding process of the I-th frame, based on the allocation calculation amount AmountOfComputation [I] of the I-th frame, The control unit 34 is controlled (steps S3 to S8).

  When the encoding unit 34 takes out the I-th frame from the calculation amount control buffer 31 under the control of the calculation amount control unit 33 (assuming that the processing of the (I-1) -th frame is finished), a new one is generated. When the frame is a frame that is encoded according to the intraframe encoding method, the number of prediction modes to be applied is obtained by the function NumberOfMode (AmountOfComputation [I]), and the number of modes in which intraframe prediction is performed is limited. The value output by the function NumberOfMode () with respect to the input value is determined in advance according to the performance of the intraframe encoding process. When the I-th frame extracted from the calculation amount control buffer 31 is a frame that is encoded according to the inter-frame encoding method, the encoding unit 34 applies a motion vector search range to be applied by the function SearchRange (AmountOfComputation [I]). And performs motion search within the search range to perform interframe coding. The value output by the function SearchRange () with respect to the input value is also determined in advance according to the performance of the interframe coding process.

(B-3) Effect of Second Embodiment As described above, according to the second embodiment, the processing time required for the encoding process of the frame for NumOfFrame is determined in advance so as not to exceed this. Since the calculation amount is allocated based on the predicted calculation amount of each frame, the calculation amount is reduced in the frame that seems to require less calculation amount, and the calculation amount that is not used is required more. It can be allocated to frames, and an improvement in encoding efficiency can be expected.

  If encoding is performed with a constant calculation amount, extra calculation is performed in a frame that does not require calculation amount, and sufficient encoding efficiency cannot be expected in a frame that requires calculation amount. In addition, in the method in which the calculation amount is simply varied according to the predicted calculation amount of the frame, the calculation amount is not constant, and when frames with a large calculation amount are continuous, real-time processing becomes impossible, and the frame It becomes necessary to perform thinning.

  In addition, although a delay for buffering NumOfFrame is generated, the condition of a constant amount of calculation for the frame of NumOfFrame is satisfied, so real-time encoding is possible.

(C) Third Embodiment Next, another embodiment (hereinafter referred to as a third embodiment) of a moving image encoding apparatus and a moving image encoding program according to the present invention will be described in detail with reference to the drawings. To do.

  Here, all or part of the moving picture coding apparatus according to the third embodiment can be constructed by hardware, but can also be constructed by a CPU and a program executed by the CPU.

  In the third embodiment, as in the second embodiment, the calculation amount is not adjusted within one frame, but the calculation amount necessary for encoding each frame is adjusted in units of a plurality of frames. It is.

(C-1) Configuration of the Third Embodiment FIG. 10 is a block diagram showing the overall configuration of the video encoding apparatus of the third embodiment, which is the same as FIG. 7 according to the second embodiment. Corresponding parts are denoted by the same reference numerals.

  As shown in FIG. 10, in the case of the moving picture encoding apparatus according to the third embodiment, the encoding amount in each frame is fed back from the encoding unit 34 to the calculation amount predicting unit 32. The prediction unit 32 predicts the calculation amount using the fed back code amount. Note that since the calculation amount prediction method for each frame has been changed, the calculation amount determination method for each frame by the calculation amount control unit 33 is also different from that of the second embodiment.

(C-2) Operation of the Third Embodiment Next, the operation of the video encoding apparatus according to the third embodiment will be described.

  The basic operation flow of the video encoding apparatus according to the third embodiment is the same as that of the second embodiment. That is, the input buffer and the encoding processing buffer are switched in units of a predetermined number of frames. When a new switching occurs, the calculation amount prediction unit 32 predicts the calculation amount in the encoding process for each frame, and the calculation amount control unit 33 calculates the total calculation amount for a predetermined number of frames as the predicted calculation amount of each frame. Accordingly, the encoding unit 34 performs encoding processing corresponding to the allocated calculation amount for each frame under the control of the calculation amount control unit 33.

  However, the third embodiment differs from the second embodiment in how to calculate the predicted calculation amount in the calculation amount prediction unit 32 and how to calculate the calculation amount in the calculation amount control unit 33. FIG. 11 shows how to calculate the predicted calculation amount in the calculation amount prediction unit 32 and how to control the calculation amount in the calculation amount control unit 33 in the third embodiment in terms of an operation program. In FIG. 11, the frame is expressed by the term “screen”.

  In the first row R1 in FIG. 11, when the number of frames that are a processing unit is N, the parameter I that defines each frame in the processing unit is changed from 0 by 1 increment, while the parameter I This represents that the process is repeated, and when the parameter I reaches N, the process is terminated. Therefore, when the parameter I becomes a new value (when a new frame is designated), the processing of the second row R2 to the sixth row R6 is executed for the new value. Yes.

  The process in the second row R2 is a calculation amount prediction process performed by the calculation amount prediction unit 32. The “code amount of the previous screen (previous frame)” is a value fed back from the encoding unit 34, and a difference between the actual code amount and the standard code amount is obtained, and a predetermined parameter α (second The second term on the right-hand side multiplied by a different meaning from that of the embodiment of FIG. 4 represents the encoding efficiency. The larger the coding efficiency of the second term, the worse the coding efficiency on the previous screen. In order to increase the amount of calculation on the current screen (current frame determined by the parameter I), this coding efficiency is set to the predicted amount of calculation. reflect. In the case of the first screen of encoding (frame where I is 0), an appropriate initial value is used. Also, if the previous screen is not the same encoding method as the current screen (if the current screen is an interframe encoding method and the previous screen is an intraframe encoding method, or the current screen is an intraframe encoding method and the previous screen is a frame) In the case of an inter-coding system), an appropriate parameter is applied to the amount of calculation of the previous screen.

  The processes of the third line R3 to the sixth line R6 are processes performed by the calculation amount control unit 33.

  The process in the third line R3 is a process for obtaining a calculation amount to be actually allocated for the current screen. The calculation amount actually allocated for the current screen is obtained by weighted addition of the “predicted calculation amount” and the average calculation amount S / (N−I) allocated per screen of the remaining amount S that can be allocated. . The initial value of the remaining amount S that can be allocated is a total calculation amount that is predetermined for N frames.

  In the process of the fourth line R4, the calculation amount for the current screen allocated in the process of the third line R3 is the remaining amount S that can be allocated (or a predetermined multiple of the value obtained by dividing the remaining amount S by the number of remaining frames ( If the value exceeds 1)), the allocation calculation amount for the current screen is replaced with an average calculation amount S / (N−I) allocated per screen of the remaining amount S that can be allocated. It is processing. Thereby, an allocatable amount for an unallocated screen is left as it is.

  The process of the fifth line R5 is a process of controlling the encoding process of the encoding unit 34 based on the allocation calculation amount for the current screen determined by the process of the third line R3 or the process of the fourth line R4.

  The process of the sixth line R6 is a process of reducing the remaining amount S that can be allocated by the amount of allocation calculation for the current screen.

(C-3) Effects of the Third Embodiment According to the third embodiment, as in the second embodiment, the calculation amount is reduced and not used in a frame that does not require much calculation amount. Thus, it is possible to allocate the minutes to frames that require a larger amount of calculation, and it is possible to expect an improvement in encoding efficiency.

  According to the third embodiment, compared with the second embodiment, it is possible to reduce the amount of calculation required for predicting the calculation amount of the encoding process in each frame.

(D) Other Embodiments In the description of each of the above embodiments, various modified embodiments have been mentioned, and further modified embodiments as exemplified below can be given.

  The difference between the reference frame and the input frame used for the interframe coding method in each of the above embodiments is not limited to one frame. Further, the reference frame is not limited to a frame past the input frame.

  In the first embodiment, the calculation amount (the size of the search range) is assigned to each block process in one frame. In the second and third embodiments, each frame process in a plurality of frames is processed. Although the calculation amount allocation is shown in Fig. 5, these technical ideas may be used together. For example, with respect to a frame in accordance with the interframe coding scheme to which the calculation amount is allocated by the method of the second or third embodiment, the calculation amount is assigned to each block process in the frame by the method of the first embodiment. Anyway.

It is a block diagram which shows the detailed structure of the inter-frame coding apparatus (inter-frame coding part) which concerns on 1st Embodiment. It is a block diagram which shows the whole structure of the real-time coding apparatus of the moving image to which the inter-frame coding apparatus of 1st Embodiment was applied. It is explanatory drawing of the detection method of a motion vector. It is explanatory drawing which shows the relationship between a search range and an input block. It is a flowchart which shows the partial determination procedure of the search range of the motion in 1st Embodiment. It is explanatory drawing of the peripheral block in 1st Embodiment. It is a block diagram which shows the whole structure of the moving image encoder which concerns on 2nd Embodiment. It is explanatory drawing of the 2 surface structure of the calculation amount control buffer in 2nd Embodiment. It is a flowchart which shows operation | movement of the moving image encoder which concerns on 2nd Embodiment. It is a block diagram which shows the whole structure of the moving image encoder which concerns on 3rd Embodiment. It is explanatory drawing which shows the operation | movement program of the calculation amount estimation part and calculation amount control part in the moving image encoder of 3rd Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 13 ... Interframe encoding part, 21 ... Block division part, 22 ... Motion search part, 23 ... Motion compensation part, 24 ... Transform quantization Dequantization inverse transformation part, 25 ... Encoding part, 26 ... Reference buffer management part , 27 ... search range control unit, 31 ... calculation amount control buffer, 32 ... calculation amount prediction unit, 33 ... calculation amount control unit, 34 ... encoding unit.

Claims (10)

In a motion vector detection device that obtains a motion vector by obtaining the position of a similar block on a reference frame in units of blocks obtained by dividing an input frame vertically and horizontally,
Motion search means for detecting a motion vector in the indicated search range on the reference frame;
Search range control means for determining the search range of each block and instructing the motion search means so that the sum of the search ranges of all the blocks in the frame does not exceed the search range allowable amount determined for the frame. A motion vector detection apparatus characterized by the above.   The search range control means determines the search range of each block based on the number of remaining blocks in which the search range in the same frame is not determined and the remaining search range allowable amount that can be allocated to the number of remaining blocks. The motion vector detection device according to claim 1, wherein: Based on the difference between the reference block and the input block obtained by the motion search by the motion search means and the size of the obtained motion vector, the prediction range determination means for determining the prediction search range,
The search range control means determines the search range of each block by using also the predicted search range for the surrounding blocks of the input block for which motion search has already been performed. Motion vector detection device. An interframe code that encodes the difference between an input block obtained by dividing an input frame vertically and horizontally and a reference block corresponding to the input block in the reference frame whose position is determined according to a motion vector detected by a built-in motion vector detection device. In the conversion device,
An inter-frame coding apparatus, wherein the motion vector detection apparatus according to any one of claims 1 to 3 is applied. A motion vector detection program that obtains a motion vector by obtaining a position of a similar block on a reference frame in units of blocks obtained by dividing an input frame vertically and horizontally,
Computer
Motion search means for detecting a motion vector in the indicated search range on the reference frame;
As search range control means for determining the search range of each block and instructing the motion search means so that the sum of the search ranges of all the blocks in the frame does not exceed the search range allowable amount determined for the frame. A motion vector detection program characterized by functioning.   The search range control means determines the search range of each block based on the number of remaining blocks in which the search range in the same frame is not determined and the remaining search range allowable amount that can be allocated to the number of remaining blocks. The motion vector detection program according to claim 5, wherein: A program that causes the computer to function as a prediction range determination unit that determines a prediction search range based on the degree of difference between a reference block obtained by motion search by the motion search unit and an input block and the size of the obtained motion vector Including parts,
The said search range control means determines the search range of each block also using the said prediction search range about the surrounding block of an input block in which the motion search has already been performed. Motion vector detection program. A computer that encodes a difference between an input block obtained by dividing an input frame vertically and horizontally and a reference block corresponding to the input block in the reference frame whose position is determined according to a motion vector detected by a built-in motion vector detection program. In an interframe coding program written to be executable,
An interframe coding program characterized by applying the motion vector detection program according to any one of claims 5 to 7. In a video encoding device that encodes a video for each frame,
Encoding means for encoding each frame, and capable of switching a processing calculation amount required for encoding;
The amount of calculation for each frame in the frame unit according to the content of each frame so that the predetermined number (any value of 2 or more) does not exceed the total amount of calculation predetermined for the continuous frame unit Calculation amount distribution means for allocating
And a calculation amount control means for controlling encoding by the encoding means so as to follow the calculation amount allocated to each frame. A moving image encoding program installed in a computer constituting a moving image encoding device for encoding a moving image for each frame,
Computer
Encoding means for encoding each frame, and capable of switching a processing calculation amount required for encoding;
The amount of calculation for each frame in the frame unit according to the content of each frame so that the predetermined number (any value of 2 or more) does not exceed the total amount of calculation predetermined for the continuous frame unit Calculation amount distribution means for allocating
A moving picture encoding program, which functions as a calculation amount control unit for controlling encoding by the encoding unit so as to follow a calculation amount allocated to each frame.

Images