JP4019030B2 - Image coding order determination device, image coding order determination method, image coding order determination program, and recording medium recording the program - Google Patents

Description

  The present invention relates to an image coding order determination device and image coding method that can reduce a calculation load when determining the coding order in image coding in which a plurality of images are coded using inter-frame predictive coding. The present invention relates to an encoding order determination method, an image encoding order determination program, and a recording medium on which the program is recorded.

  When encoding a plurality of images such as a moving image, if the correlation between the images is high, it is possible to realize high encoding efficiency by applying the inter-frame prediction encoding by regarding the image as a frame. International standard video coding system In H.264 (see Non-Patent Document 1), the frame encoding mode includes an I frame that is encoded without using correlation between frames, a P frame that is predicted from one previously encoded frame, and a previously encoded frame. B frames that can be predicted from the two frames are defined.

  A plurality of frames of decoded images are stored in the reference image memory, and P frames and B frames can be predicted by selecting a reference image from the memory. This reference picture can be selected in units of macroblocks. A method of predicting from the past two frames in the B frame is called bidirectional prediction, and a predicted image is created by weighted averaging of image information of each frame.

Here, the frame encoding order (decoding order) and the input order (output order) may be different. For example, the time in the input moving image may be predicted from a past frame, or predicted from a future frame. May be.
ITU-T Rec.H.264 / ISO / IEC 11496-10, "Advanced Video Coding", Final Committee Draft, Document JVT-E022, September 2002.

  When encoding a plurality of images using inter-frame predictive encoding, the encoding efficiency of the plurality of images depends on the encoding order of the images. In inter-frame predictive coding, the difference from the reference image stored in the reference image memory is used as the encoding target. Therefore, the larger the correlation between the reference image stored at the time of encoding and the encoding target image, the higher the prediction. Efficiency is improved and coding efficiency can be improved.

  As a method for determining the coding order, a method is considered in which coding is performed once in the coding order of all patterns and the order with the smallest code amount is obtained. However, in this method, since the number of patterns in the encoding order is a permutation of the number of images, there is a problem that the required amount of computation increases rapidly as the number of images to be encoded increases.

  For example, the encoding order is a (1 × 2 × 3) pattern when the number of images is 3, and a (1 × 2 × 3 × 4) pattern when the number of images is 4. Therefore, when the number of images is changed from 3 to 4, the encoding time increases four times.

  Alternatively, an evaluation function indicating correlation between frames instead of code amount may be set, and a coding order may be determined so that the sum of the values is minimized. At this time, an evaluation function is set such that the smaller the evaluation value, the smaller the code amount when encoding is performed. However, even in this method, it is necessary to obtain the sum of evaluation values between all frames in the coding order of all patterns. Since the number of patterns in the encoding order is a permutation of (number of images-1), there is a problem that the required amount of calculation increases rapidly when the number of images to be encoded increases.

  An object of the present invention is to solve the above problems and to reduce the calculation load when determining the encoding order when encoding a plurality of images using inter-frame predictive encoding.

In order to solve the above-described problems, the present invention provides a code for M frames out of N frames of image information in an image encoding apparatus that encodes image information of a plurality of frames using interframe predictive encoding. An image coding order determination device for determining a coding order,
Intra-frame evaluation value calculation means for calculating an evaluation value E (i, i) for evaluating the generated code amount using image information of a frame with frame number i (i = 1, 2,..., N);
All combinations for selecting two frames of frame number i (i = 1, 2,..., N) and frame number j (j = 1, 2,..., N; where j ≠ i) from the N frames. Inter-frame evaluation value calculation means for calculating an evaluation value E (i, j) for evaluating the amount of generated code using difference information from the image information of the frame number i of the frame image information of frame number j When,
Evaluation value storage means for storing evaluation values calculated by the intra-frame evaluation value calculation means and the inter-frame evaluation value calculation means;
Of the N frames, a value F (i) (i = 1, 1) of an encoding order frame number F indicating the encoding order of frames determined by a predetermined method for a certain (M−1) frames. 2,..., M-1) frame addition initial setting means;
Target frame setting means for setting a frame number value k of a frame to be added as an encoding target to a value from 1 to N other than the encoding order frame number F;
A target frame order setting means for setting the frame of frame number k to the b-th (b is a preset value c to M; where 1 ≦ c ≦ M−1) encoding order;
Provisional encoding order T (i) (i = 1, 2,...) Obtained by adding the frame of frame number k set by the target frame order setting means to the encoding order set in encoding order frame number F. , M), provisional encoding order setting means,
An evaluation total initial setting means for setting the evaluation value E (T (1), T (1)) to the evaluation total value S;
J of T (j) is sequentially changed from 2 to M in the order of the frame numbers set in the provisional encoding order, and the evaluation value E (T (i), T (j)) (i is 1 to j− Evaluation sum addition means for repeatedly adding any one of 1) to the evaluation total value S;
The evaluation total value S calculated by sequentially changing the value b from the value c to M by the target frame order setting means, the provisional encoding order setting means, the evaluation total initial setting means, and the evaluation total addition means. The target frame for determining the provisional encoding order T (i) (i = 1, 2,..., M) with the value b having the smallest evaluation total value S as the encoding order of M frames to be obtained. Order determining means .

In the above invention, the predetermined method for setting the value F (i) of the encoding order frame number F by the frame addition initial setting means is the calculation of the intra-frame evaluation value for each of the (M−1) frames. And a method for determining an evaluation total value to be small by using the evaluation value calculated by the inter-frame evaluation value calculation means, or a method for selecting the vicinity of a frame having a predetermined initial frame number. To do.

Further, the present invention provides an image for determining an encoding order of M frames among image information of N frames in an image encoding method for encoding image information of a plurality of frames using inter-frame predictive encoding. A coding order determination method comprising:
A frame for calculating an evaluation value E (i, i) for evaluating a generated code amount using image information of a frame with frame number i (i = 1, 2,..., N) and storing it in the evaluation value storage means Internal evaluation value calculation step,
All combinations for selecting two frames of frame number i (i = 1, 2,..., N) and frame number j (j = 1, 2,..., N; where j ≠ i) from the N frames. An evaluation value E (i, j) for evaluating the amount of generated code is calculated using the difference information from the image information of the frame number i of the frame image information of the frame number j, and the evaluation value storage means An inter-frame evaluation value calculation step to be stored;
Of the N frames, a value F (i) (i = 1, 1) of an encoding order frame number F indicating the encoding order of frames determined by a predetermined method for a certain (M−1) frames. 2,..., M-1), a frame addition initial setting step;
A target frame setting step of setting a frame number value k of a frame to be added as a coding target to a value from 1 to N other than the coding order frame number F;
A target frame order setting step for setting the frame of frame number k to the b-th (b is a preset value c to M; where 1 ≦ c ≦ M−1) encoding order;
Provisional encoding order T (i) (i = 1, 2,...) Obtained by adding the frame of frame number k set in the target frame order setting step to the encoding order set in encoding order frame number F. , M), a provisional encoding order setting step;
An evaluation total initial setting step for setting the evaluation value E (T (1), T (1)) to the evaluation total value S;
J of T (j) is sequentially changed from 2 to M in the order of the frame numbers set in the provisional encoding order, and the evaluation value E (T (i), T (j)) (i is 1 to j− An evaluation total addition step of repeatedly adding any one of 1) to the evaluation total value S;
The evaluation total value S calculated by sequentially changing the value b from the value c to the M by the target frame order setting step, the provisional encoding order setting step, the evaluation total initial setting step, and the evaluation total addition step. Among them, the provisional encoding order T (i) (i = 1, 2,..., M) with the value b having the smallest evaluation total value S is determined as the encoding order of M frames to be obtained. And a frame order determining step.

  The operation of the present invention is as follows. In-frame evaluation value calculation step, inter-frame evaluation value calculation step, frame addition initial setting step, target frame setting step, target frame order setting step, provisional encoding order setting step, evaluation total initial setting step, evaluation total By executing the addition step and the target frame order determination step, the evaluation value of all M frames is the smallest when one frame is added to the (M-1) frames for which the encoding order has been set in advance. In this way, the encoding order of the frames to be added can be determined.

  In this method, every time the location where a frame is added is changed, a calculation for adding the evaluation values of all M frames is required. Therefore, the calculation for adding the evaluation values is performed (M−c) × (M−1) times. . Conventionally, since the number of additions of only the permutation of (M-1) is necessary, according to the present invention, the number of calculations can be greatly reduced as compared with the conventional method.

  In the target frame order setting step, the value c at the time of setting the encoding order of the frame of frame number k can be set to a value from 1 to (M−1). As the value c is increased, the number of calculations can be reduced, but the number of encoding order candidates to be evaluated is reduced.

  The evaluation total addition step changes the value i to any value from 1 to j−1 for the frame number T (j), and the evaluation value E (T (i), T (j)) is minimized. The frame evaluation total addition step for searching for the value to be added and adding the minimum evaluation value E (T (i), T (j)) to the evaluation total value S is performed by sequentially changing j from 2 to M. It can be implemented by repeated execution.

  In this evaluation sum addition step, the evaluation value E (T (i ), T (j)) and adding to the evaluation total value S, the value i can be determined so as to be the minimum value as the evaluation total value S.

  As another determination method of the value i in the evaluation value E (T (i), T (j)) to be added in the evaluation sum addition step, for example, T (i) adds or subtracts 1 to T (j) The value i may be set to be a value.

  The frame evaluation total addition step executes a referenceable frame number setting step of setting L values R (d) (d = 1, 2,..., L) as referenceable frame numbers R, and the T Only when (i) is equal to one of the values included in the set referenceable frame number R, the evaluation value E (T (i), T (j)) can be used as a search candidate. it can. If this method is used, the range in which the value i is changed in the frame evaluation total addition step can be reduced. Therefore, the number of calculations can be further reduced.

  In the frame addition initial setting step, for the i-th frame, the evaluation value E (i, j) (j ≠ i) calculated in the inter-frame evaluation value calculation step is j. The frame neighborhood evaluation search step for obtaining the value j1 (i), the j value j2 (i) when the evaluation value is the second smallest, and the evaluation value E (i, i) and the evaluation value E ( i, j1 (i)) and the evaluation value E (i, j2 (i)) are summed to obtain a frame neighborhood evaluation addition step for obtaining an evaluation total value ES (i). The value m of i when the evaluation total value ES (i) (1 ≦ i ≦ N) is the smallest is obtained, and m, j1 (m), j2 (m) and the coding order frame number F are obtained. It can be implemented by setting.

  By using this method, when three encoding order frame numbers F are determined in the frame addition initial setting step, an evaluation value E (i, i) and two evaluation values E (i, j) ( The values i and j can be determined so that the sum of j ≠ i) is minimized. Thereby, the value of the evaluation total value S can be made smaller than the method of determining the initial value of the encoding order frame number F without using the evaluation value.

  In addition to this, as a coding order frame number F set in the frame addition initial setting step, for example, an initial value i is determined and set to a value obtained by repeatedly adding or subtracting 1 around that value. It may be used. For example, if five values are set, F = (i, i-1, i + 1, i-2, i + 2).

  The intra-frame evaluation value calculating step includes an intra-frame encoding step for intra-frame encoding of image information of the i-th frame, and a decoded image for generating a decoded image from the encoded data generated in the intra-frame encoding step. The evaluation value E (i, i) using the creation step, the difference calculation step for calculating difference information between the image information of the i-th frame and the created decoded image, and the difference information calculated in the difference calculation step. can be implemented by performing the step of calculating i).

  In this method, the evaluation value E (i, i) is calculated using difference information between the image information once encoded and decoded and the original image. In the case of lossy encoding, the original image and the decoded image are different, and the larger the difference, the larger the encoding distortion. Therefore, if the evaluation function is set so that the evaluation value becomes smaller as the difference is smaller, the value i having a smaller difference can be searched by searching for E (i, i) having a smaller evaluation value.

  As an evaluation method of the magnitude of the difference, there is an absolute value sum SADR of differences or a square sum SSD of differences. SADR or SSD is calculated by the following formula (1) or formula (2), respectively. o (i) (x, y) indicates the original image of frame number i at the pixel position (x, y), and r (i) (x, y) indicates the decoded image at the pixel position (x, y).

SADR = Σx _{, y} | r (i) (x, y) -o (i) (x, y) | Equation (1)
SSD = Σx _{, y} (r (i) (x, y) −o (i) (x, y)) ² ...... Formula (2)
Such an SADR or SSD may be set as an evaluation value, and an evaluation function J that takes into account the amount of generated codes for prediction residuals and prediction information may be set as an evaluation value. J is calculated by the following formula (3). R represents a generated code amount, and λ represents a predetermined coefficient. The SSD in equation (3) may be SADR.

J = SSD + λ × R (3)
As another method for obtaining the evaluation value E (i, i) in the intra-frame evaluation value calculation step, for example, a calculation method using an average value or variance of intra-frame image information or a correlation coefficient between adjacent pixels. Can also be used.

  The inter-frame evaluation value calculation step includes a difference encoding step for encoding difference information from the image information of the i-th frame of the image information of the j-th frame, decoding the encoded difference information, and the i-th frame A decoded image creating step for creating a decoded image by adding to the image information, a difference calculating step for calculating difference information between the image information of the jth frame and the created decoded image, and the difference calculating step The step of calculating the evaluation value E (i, j) using the calculated difference information can be performed. In addition, as a method for obtaining the evaluation value E (i, j) in the inter-frame evaluation value calculation step, for example, there is a calculation method using an average value or variance of absolute values of inter-frame image information.

  In this inter-frame evaluation value calculation step, for example, as a method of calculating the evaluation value E (i, j), a position (x0, y0) where the difference is calculated at frame number i with respect to the pixel position (x, y) of frame number j If x0 and y0 are real number precision, the absolute value SADO of the difference between two frames may be used as the evaluation function.

  SADO is calculated by the following equation (4). o (j) (x, y) indicates image information of the pixel position (x, y) in the frame number j, and o (i) (x0, y0) indicates an image of the pixel position (x0, y0) in the frame number i. Indicates information.

SADO = Σx _{, y} | o (j) (x, y) -o (i) (x0, y0) | (4)
These evaluation values may be calculated for each area obtained by dividing the screen, for example, for each macro block. In this case, T (i) in the evaluation value E (T (i), T (j)) to be added to the evaluation total value S in the evaluation total addition step may be different for each region.

  According to this method, in the inter-frame evaluation value calculation step, the evaluation value E (i, j) is calculated using difference information between the image information once encoded and decoded and the original image. Here, the larger the difference between frames, the greater the coding distortion. Therefore, if the evaluation function is set so that the evaluation value becomes smaller as the difference is smaller, the value i having a smaller difference can be searched by searching for E (i, j) having a smaller evaluation value. The SADR, SSD, or J described above can be used as a method for evaluating the magnitude of the difference.

  According to the present invention, when determining the encoding order when encoding a plurality of images using inter-frame predictive encoding, the encoding order is obtained by obtaining the sum of evaluation values in all the encoding order patterns. Compared with the method of determining the number of times, the number of times of calculating the total evaluation value can be reduced, and the calculation load can be reduced.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of the configuration of an image coding order determination apparatus according to the present invention. Reference numeral 1 denotes an image coding order determination device that determines the coding order of M frames of N frame image information. In the image coding order determination apparatus 1, 11 is an original image storage memory for storing original images, and 12 is a switching unit for switching between an intra-frame evaluation value calculation mode and an inter-frame evaluation value calculation mode.

  13 is an intra-frame evaluation value calculation unit that calculates an evaluation value E (i, i) using image information of frame number i (1 ≦ i ≦ N), and 14 is a frame number for image information of a frame of frame number i. An inter-frame evaluation value calculation unit 15 for calculating an evaluation value E (i, j) using difference information from image information of a frame with frame number i of image information of j (j ≠ i), 15 is a calculated evaluation An evaluation value storage memory in which values are stored.

  16 is a frame addition initial setting unit for setting a value F (i) (i is 1 to M−1) of an encoding order frame number F indicating the encoding order of frames for (M−1) frames; Is a target frame setting unit for setting the frame number value k of a frame to be added as a coding target to a value from 1 to N other than the coding order frame number F.

  18 is a target frame order setting unit that sets the frame of frame number k to the b-th encoding order (b is one of preset values c to M), and 19 is the encoding order frame number F. The provisional encoding order T (i) (i is 1 to M) is set by adding the frame of the frame number k for which the encoding order is set by the target frame order setting unit 18 to the set encoding order. The provisional encoding order setting unit.

  Also, 20 is an evaluation total initial setting unit that sets the evaluation value E (T (1), T (1)) as the evaluation total value S, and 21 is the order of frame numbers set in the provisional encoding order. ) And the evaluation value E (T (i), T (j)) (i is any one from 1 to j-1) is repeated to the evaluation total value S for all of j from 2 to M The evaluation sum adding unit 22 adds the value b which is the smallest value among the evaluation total values S obtained by the evaluation sum adding unit 21 for the provisional encoding order for all the values c to M. And the encoding order is determined as the encoding order for the M frames.

  FIG. 2 is a diagram showing an example of the configuration of the frame addition initial setting unit 16 shown in FIG. In the frame addition initial setting unit 16, reference numeral 161 denotes a frame neighborhood evaluation search unit, 162 denotes a frame neighborhood evaluation addition unit, 163 denotes a frame neighborhood evaluation accumulation memory, and 164 denotes a frame neighborhood evaluation total search unit.

  For example, when M = 4, the frame neighborhood evaluation search unit 161 evaluates the evaluation value E (i, i) of the evaluation value E (i, j) (j ≠ i) stored in the evaluation value storage memory 15 for the i-th frame. The process of obtaining the value j1 (i) of j when j) is minimum and the value j2 (i) of j when the evaluation value E (i, j) is the second smallest are all i from 1 to N Run about.

The frame neighborhood evaluation adding unit 162 sums up the evaluation value E (i, i), the evaluation value E (i, j1 (i)), and the evaluation value E (i, j2 (i)) for the i-th frame. Thus, the process for obtaining the evaluation total value ES (i) is executed for all i from 1 to N. That is, ES (i) is
ES (i) = E (i, i) + E (i, j1 (i)) + E (i, j2 (i))
(I = 1, 2,..., N).

  The frame neighborhood evaluation accumulation memory 163 accumulates the evaluation total value ES (i) obtained by the frame neighborhood evaluation addition unit 162. The frame neighborhood evaluation total search unit 164 obtains the value m of i when the evaluation total value ES (i) (1 ≦ i ≦ N) is the smallest, and sets the encoding order frame number to m, j1 (m), j2 ( m).

  3 and 4 are diagrams for explaining the outline of processing according to the embodiment of the present invention. Assume that the encoding order of M (M = 4 in this example) frames with good encoding efficiency is determined from the image information of N frames as shown in FIG. As shown in FIG. 3B, the intra-frame evaluation value calculation unit 13 uses the image information of each frame to evaluate the evaluation values E (1, 1) and E (2, 2) for evaluating the generated code amount. ,..., E (N, N) are calculated, and the result is stored in the evaluation value storage memory 15. Further, as shown in FIG. 3C, the inter-frame evaluation value calculation unit 14 calculates the frame number i (i = 1, 2,..., N) and the frame number j (j = 1, 2, .., N; provided that the evaluation value E (i, j) is calculated for all combinations for selecting two frames of j ≠ i), and the result is stored in the evaluation value storage memory 15.

As shown in FIG. 3D, the frame addition initial setting unit 16 encodes the encoding order indicating the encoding order of frames for (M−1) frames (three in this example) out of N frames. The values F (1), F (2) and F (3) of the frame number F are set. In this encoding order setting, for the i-th frames from 1 to N, the evaluation value E (i, j) (j ≠ i), the j value j1 (i) when the evaluation value is minimum, and the evaluation The value j2 (i) of j when the value E (i, j) is the second smallest is obtained, and for each i-th frame,
ES (i) = E (i, i) + E (i, j1 (i)) + E (i, j2 (i))
And the value m of i when the total evaluation value ES (i) (1 ≦ i ≦ N) is the smallest is obtained, and F (1) = m, F (2) = j1 (m), F ( 3) = j2 (m).

  Next, as shown in FIG. 3E, the target frame setting unit 17 sets the frame number k of the frame to be added as the encoding target from 1 to N other than the value F (i) of the encoding order frame number F. Set to value. The target frame order setting unit 18 sets the frame of frame number k to the b-th encoding order (b is one of preset values c to M). For example, when c = 2, b is either 2, 3 or 4.

  As shown in FIG. 4A, the provisional encoding order setting unit 19 performs frame processing for the encoding orders F (1), F (2), and F (3) set in the encoding order frame number F. A provisional encoding order T (1), T (2), T (3), and T (4) is set by adding the frame of number k to the b-th.

  For each of the provisional encoding orders (orders 1 to 3), the evaluation total initial setting unit 20 and the evaluation total addition unit 21 calculate the evaluation total value S as shown in FIG. 4B.

S = E (T (1), T (1)) + E (T (1), T (2)) + min (E (T (1), T (3)) + E (T (2), T (3) ))) + Min (E (T (1), T (4)), E (T (2), T (4)), E (T (3), T (4)))
Here, min (x, y, z) represents the minimum value in x, y, z.

  The evaluation total value S is calculated for each provisional encoding order, and the provisional encoding order of the evaluation total value S that is the smallest among them is output as the encoding order of the M frames to be obtained, as shown in FIG. .

  FIG. 5 shows a coding sequence determination process flow by the image coding sequence determination apparatus 1. First, the intra-frame evaluation value calculation unit 13 executes the intra-frame evaluation value calculation step S1 and the inter-frame evaluation value calculation unit 14 executes the inter-frame evaluation value calculation step S2. The evaluation value E (i, i) and the evaluation value E ( i, j) is calculated and the result is stored in the evaluation value storage memory 15. The intra-frame evaluation value calculation step S1 and the inter-frame evaluation value calculation step S2 are repeatedly executed for all combinations of N frames (step S3). Here, the order of the intra-frame evaluation value calculation step S1 and the inter-frame evaluation value calculation step S2 may be switched.

  Thereafter, the frame addition initial setting unit 16 executes a frame addition initial setting step S4, and sets the value of the encoding order frame number F for (M−1) frames. Further, the target frame setting unit 17 executes the target frame setting step S5, and sets the frame number value k of the frame to be added as the encoding target to a value from 1 to N other than the encoding order frame number F. . Note that the order of the frame addition initial setting step S4 and the target frame setting step S5 may be switched.

  Next, the target frame order setting unit 18 executes the target frame order setting step S6, and the frame number k is designated as the b-th encoding order (b is any value from a preset value c to M). Set to. The provisional encoding order setting unit 19 executes provisional encoding order setting step S7, and with respect to the encoding order set in the encoding order frame number F, the frame number k set in the target frame order setting step S6. A provisional encoding order T (i) (i is from 1 to M) with frames added is set.

  The evaluation total initial setting unit 20 executes the evaluation total initial setting step S8, and sets the evaluation value E (T (1), T (1)) as the evaluation total value S. Thereafter, the evaluation sum addition step S9 is executed by the evaluation sum addition unit 21, T (j) is changed in the order of the frame numbers set in the provisional encoding order, and the evaluation values E (T (i), T (j )) (I is any one from 1 to j−1) and j is repeatedly added to the evaluation total value S from 2 to M.

  The processing from the target frame order setting step S6 to the evaluation sum adding step S9 is repeated for all the encoding orders of the target frames (step S10), and the target frame order determination unit 22 executes the target frame order determination step S11. , The value of the target frame order b in which the evaluation total value S for each provisional coding order obtained in the evaluation sum addition step S9 is the smallest is determined, and the process is terminated.

  An example of the processing flow of the evaluation total addition step S9 is shown in FIG. First, j = 2 is set (step S91). Next, the value i is determined to be any value from 1 to j−1 (step S92). The evaluation value E (T (i), T (j)) is added to the evaluation total value S (step S93), and j is incremented by 1 until j is equal to M (step S94) (step S95). ), Steps S92 to S94 are repeatedly executed, and when j becomes equal to M, the process is terminated.

  FIG. 7 shows an example of another processing flow of the evaluation total addition step S9 shown in FIG. First, j = 2 is set (step S101). Next, the frame evaluation total addition step S102 (FIGS. 8, 9, and 10) is executed. Thereafter, it is determined whether j is equal to M (step S103). If j is not equal to M, j is incremented by 1 (step S104), the process returns to step S102, and the process is repeated in the same manner. If j is equal to M, the process is terminated.

  FIG. 8 shows an example of the processing flow of the frame evaluation total addition step S102. In this example, first, E (T (1), T (j)) is set as the minimum evaluation value ME (step S111). It is determined whether j = 2 (step S112). If j is 2, the process proceeds to step S118. If j is not 2, i = 2 is set (step S113).

  Next, the magnitudes of E (T (i), T (j)) and ME are compared (step S114). If E (T (i), T (j)) <ME, E (T ( i) and T (j)) are set to ME (step S115). If E (T (i), T (j)) <ME, the process proceeds to step S116.

  In step S116, it is determined whether i is equal to j-1. If i is not equal to j-1, the value of i is incremented by 1 (step S117), and the process returns to step S114. If i is equal to j-1, ME is added to the evaluation total value S (step S118). , Terminate the process.

  FIG. 9 shows an example of another processing flow of the frame evaluation total addition step S102. In this example, first, the maximum value ∞ is set as the minimum value ME of the evaluation values (step S121), and i = 1 is set (step S122). Thereafter, it is determined whether E (T (i), T (j)) <ME (step S123). If E (T (i), T (j)) <ME, E (T (i), T (j)) is set to ME (step S124), and E (T (i), T ( j)) If not <ME, go to step S125. In step S125, it is determined whether i is equal to j-1 (step S125). If i is not equal to j-1, 1 is added to the value of i (step S126), and the process returns to step S123. If it is equal to j-1, ME is added to the evaluation total value S (step S127), and the process is terminated.

  In the example of FIG. 8 and FIG. 9, the setting method of the initial value of the minimum evaluation value is different. As described above, the flow for obtaining the minimum value of the evaluation value may be a plurality of patterns depending on the initial value setting method. Therefore, the processing flow of the frame evaluation total addition step S102 may be other than the flow shown in FIGS.

  An example of still another processing flow in the frame evaluation total addition step S102 is shown in FIG. The frame evaluation total addition step S102 shown in FIG. 10 is an example of processing adopted in Example 2 described later.

  First, the referenceable frame number setting step S131 is executed to set the value of the referenceable frame number. Next, the maximum value ∞ is set as the minimum value ME of the evaluation values (step S132), i = 1 is set (step S133), and the following processing is repeated.

  It is determined whether T (i) is included in the referenceable frame number (step S134). If T (i) is not included in the referenceable frame number, the process proceeds to step S137. If T (i) is included in the referenceable frame number, it is determined whether E (T (i), T (j)) <ME (step S135). If E (T (i), T (j)) <ME, E (T (i), T (j)) is set to ME (step S136), and E (T (i), T (J)) If not <ME, go directly to step S137.

  In step S137, it is determined whether i is equal to j−1. If i is not equal to j−1, the value of i is incremented by 1 (step S138), and the process returns to step S134. When equal to −1, ME is added to the evaluation total value S (step S139), and the process is terminated.

  The flow for obtaining the minimum value of the evaluation values after the referenceable frame number setting step (step S131) is not limited to the processing flow shown in FIG.

  FIG. 11 shows an example of the processing flow in the frame addition initial setting step S4 shown in FIG. In FIG. 11, first, i = 1 is set (step S141). Next, frame neighborhood evaluation search step S142 is executed. Here, for the i-th frame, for example, the evaluation value E (i, j), j value j1 (i) when the evaluation value is minimum, and j value j2 (i) when the evaluation value is the second smallest ) Subsequently, frame neighborhood evaluation addition step S143 is executed. Here, for the i-th frame, the evaluation value E (i, i), the evaluation value E (i, j1 (i)), and the evaluation value E (i, j2 (i)) are summed to obtain an evaluation total value. Find ES (i).

  Next, it is determined whether i is equal to N (step S144). When i is not equal to N, 1 is added to the value of i (step S145), and the process returns to step S142. When i is equal to N, the process proceeds to execution of the frame neighborhood evaluation total search step S146.

  In frame neighborhood evaluation total search step S146, the value m of i when evaluation total value ES (i) is the smallest is obtained, and then in step S147, m and j1 (m ) And j2 (m) are set, and the process ends.

  A method for determining the encoding order of a two-dimensional image of N frames will be described. An encoding order is determined when the two-dimensional image is divided into all A macroblocks and encoded.

  Assume that the frame addition initial setting unit 16 determines the encoding order for the first three frames in the frame addition initial setting step S4. Further, in the intra-frame evaluation value calculation step S1 by the intra-frame evaluation value calculation unit 13 and the inter-frame evaluation value calculation step S2 by the inter-frame evaluation value calculation unit 14, the value J calculated by Expression (3) is used as the evaluation value. Shall.

  In the intra-frame evaluation value calculation step S1, the international standard video encoding method Assume that H.264 intra prediction is used. In the intra-frame evaluation value calculation step S1 and the inter-frame evaluation value calculation step S2, the prediction residual is DCT quantized and further dequantized and IDCT is added to the prediction image to add the prediction error to the prediction image. create.

  The same parameters are used for quantization in all macroblocks of N frames. The evaluation value is obtained for each macroblock, and the evaluation value of the macroblock a at the frame number i calculated in the intraframe evaluation value calculation step S1 is E (i, a, i). Let E (i, a, j) be the evaluation value related to the difference from the frame number i of the macro block a in the frame number j calculated in the inter-frame evaluation value calculation step S2. The value c in the target frame order setting step S6 is set to 2.

  An example of a method in which the referenceable frame number setting step is not performed will be described as the first embodiment. In the evaluation total addition step S9, the frame evaluation total addition step S102 of FIG. 7 is executed, and the frame evaluation total addition step S102 is executed according to the flow shown in FIG. In the present embodiment, a procedure for determining the encoding order of all frames after the fourth frame after the encoding order of the first three frames is determined in the frame addition initial setting step S4 will be described.

  First, according to the flow of FIG. 5, the intra-frame evaluation value calculation step S1 by the intra-frame evaluation value calculation unit 13 and the inter-frame evaluation value calculation step S2 by the inter-frame evaluation value calculation unit 14 are executed for all N frame combinations. Subsequently, the frame addition initial setting unit 16 executes a frame addition initial setting step S4 (FIG. 11).

  According to the flow of FIG. 11, encoding order frame numbers F (F (1), F (2), F (3)) for three frames are obtained. First, the frame vicinity evaluation search unit 161 executes frame vicinity evaluation search step S142. In step S142, for example, the evaluation value E (i, i, i) is obtained by summing up the evaluation values E (i, a, i) for all the macroblocks for the frame number i. Next, for frame numbers j from 1 to N other than i, evaluation values E (i, a, j) are summed up for all macroblocks for each frame to obtain evaluation values E (i, j). Then, a j value j1 (i) when the evaluation value E (i, j) is the smallest and a j value j2 (i) when the second smallest value are obtained.

  Next, the frame vicinity evaluation adding unit 162 executes frame vicinity evaluation adding step S143. Here, the evaluation values E (i, i) and E (i, j1 (i)) are obtained from the two values j1 (i) and j2 (i) of j for each i obtained in the frame neighborhood evaluation search step S142. And E (i, j2 (i)) are summed to obtain an evaluation value ES (i). The frame neighborhood evaluation search step S142 and the frame neighborhood evaluation addition step S143 are executed by changing the value i from 1 to N (steps S144 and S145).

  Next, the frame neighborhood evaluation total search unit 164 executes the frame neighborhood evaluation total search step S146 to obtain the value m of i when the evaluation value ES (i) is the smallest (step S146). Then, (m, j1 (m), j2 (m)) is set to the encoding order frame number F (F (1), F (2), F (3)) (step S147).

  Next, in the target frame setting step S5 of FIG. 5, the target frame setting unit 17 sets a value from 1 to N other than the encoding order frame number F as the value k. Here, the smallest value is set. Thereafter, a frame having the frame number as the value set here is added, and the encoding order is determined.

  A method for determining the coding order by adding the frame number k when the coding order frame number F is (M−1) frames will be described below. First, in the target frame order setting step S6 of FIG. 5, the encoding order of the frame with frame number k is set to b-th. In the provisional encoding order setting step S7 of FIG. 5, the provisional encoding order setting unit 19 inserts the frame number k at the b-th of the encoding order frame number F, and the provisional encoding order T (T (i), i Set from 1 to M). In the evaluation total initial setting step S8 of FIG. 5, the evaluation total initial setting unit 20 sets the evaluation total value S to E (T (1), T (1)).

  In the evaluation total addition step S9 of FIG. 5, the evaluation total addition unit 21 calculates the evaluation value E (T (i), T (j)) frame by frame in the provisional encoding order T according to the flow of FIG. It adds to the evaluation total value S.

  The evaluation values E (T (i), T (j)) to be added are determined and added according to the flow shown in FIG. 9 as the frame evaluation total addition step S102. Here, the frame evaluation total addition step S102 is executed for each macroblock. That is, the evaluation value E (T (i), a, T (j)) is determined for the macroblock a, and the addition process is repeatedly executed by changing a from 1 to A. In the evaluation total addition step S9, an evaluation total value S obtained by adding the evaluation values for all (M-1) frames other than T (1) set in the provisional encoding order T is obtained.

  The processes from the target frame order setting step S6 to the evaluation sum adding step S9 are executed by changing the value b (step S10 in FIG. 5). In the target frame order determining step S11, the target frame order determining unit 22 The value of b that minimizes the evaluation total value S is searched, and the order in which the frame number k is inserted at the searched position is determined as the encoding order. The encoding order of the M frames obtained in this way is set to the encoding order frame number F.

  The above-described target frame setting step S5 to target frame order determination step S11 are repeatedly executed (N-3) times while changing the value k to determine the encoding order for all N frames.

  In the present embodiment, the encoding order of the first three frames is determined by the frame addition initial setting unit 16, but the encoding order of the number of frames of the second frame or the fourth frame or more may be determined by the frame addition initial setting unit 16. Good. When the coding order of the first (k + 1) frame is determined in the frame addition initial setting step S4, the frame neighborhood evaluation search unit 161 determines that the evaluation value E (i, j) is the smallest value j. A total of k values of the j value jk (i) when the kth value is smaller than the j1 (i) value are obtained. The frame neighborhood evaluation adding unit 162 evaluates E values (i, i) and E (i, j1 (i)) to E for k values j1 (i) to jk (i) for each i. An evaluation value ES (i) is obtained by summing up to (i, jk (i)).

  The processes of the frame neighborhood evaluation search unit 161 and the frame neighborhood evaluation addition unit 162 are executed by changing the value i from 1 to N. Next, the frame neighborhood evaluation total search unit 164 obtains the value m of i when the evaluation value ES (i) is the smallest. Based on the obtained value m, (m, j1 (m), j2 (m),..., Jk (m)) are encoded order frame numbers F (F (1), F (2),. (K)).

  As described above, according to the first embodiment, compared with the method of verifying the coding order of all patterns, the number of additions of evaluation values necessary for determining the coding order can be reduced, and the calculation load is reduced. Can be reduced.

  As a second embodiment, an example of a method for performing the referenceable frame number setting step will be described. The number of referable frames is set to L (L <N). In the evaluation total addition step S9 of FIG. 5, the frame evaluation total addition step S102 of FIG. 7 is executed, and the frame evaluation total addition step S102 is executed according to the flow shown in FIG.

  When the encoding order frame number F is L or less, the operation is exactly the same as in the first embodiment. The processing when the encoding order frame number F is greater than L will be described below. The processing from the intra-frame evaluation value calculation step S1 to the evaluation total initial setting step S8 shown in FIG. 5 is the same processing as in the first embodiment.

  The operation in the evaluation total addition step S9 in FIG. In the evaluation total addition step S9, according to the flow of FIG. 7, the evaluation total addition unit 21 calculates the evaluation value E (T (i), T (j)) for each frame in the provisional coding order T in the order of the evaluation total value. Add to S. The evaluation value E (T (i), T (j)) to be added is determined and added according to the flow shown in FIG. 10 as the frame evaluation total addition step S102. In referenceable frame number setting step S131 in FIG. 10, referenceable frame number R (R (d), d is 1 to L) is set.

  When determining the evaluation value E (T (i), T (j)) of the j-th frame in the tentative encoding order, the referenceable frame number R is determined by (T (j -D), d is set from 1 to L). If j is L or less, the referenceable frame number R is set to the first j values in the provisional encoding order T.

  The frame evaluation total addition step S102 is executed for each macro block. That is, the evaluation value E (T (i), a, T (j)) is determined for the macroblock a, and the addition process is repeatedly executed by changing a from 1 to A. The candidate for T (i) is selected so as to coincide with any of the referenceable frame numbers R (R (d), d is 1 to L).

  In the evaluation total addition step S9, an evaluation total value S obtained by adding the evaluation values for all (M-1) frames other than T (1) set in the provisional encoding order T is obtained. The other steps thereafter are processed in the same manner as in the first embodiment, and the encoding order for all N frames is determined. As described above, according to the second embodiment, the number of times of adding the inter-frame evaluation value can be reduced, and the calculation load can be further reduced as compared with the first embodiment.

  In addition, according to the second embodiment, it is possible to determine the encoding order when an image is encoded by the image encoding order determination apparatus including the reference image memory for the number of frames smaller than the encoding target frame number N. . The number of referenceable frame numbers set in the referenceable frame number setting step S131 of the second embodiment may be set to the same value as the number of frames in the reference image memory of the image encoding order determination device.

  In the referenceable frame number setting step S131 in the second embodiment, when determining the evaluation value E (T (i), T (j)) of the jth frame in the provisional encoding order, the referenceable frame number is used. R is set to (T (j−d), d is 1 to L) in the provisional encoding order T, but other methods may be used.

For example, the following method is mentioned. Not limited to these methods, any method may be used in the referenceable frame number setting step S131 as long as L referenceable frame numbers can be set. For example, the following methods d1 and d2 may be used.
Method d1: Set to L values in descending order of T (e) (e is 1 to j) in the provisional encoding order T.
Method d2: Set to T (e) (e is from 1 to L) in the provisional encoding order T.

  According to the “method d1”, it is possible to determine the encoding order using images having similar frame numbers. In addition, according to “method d2”, the encoding order can be determined using a frame with an early encoding order.

  According to this embodiment, when transmitting a plurality of images encoded in the encoding order determined under the condition that the number of reference image memories is limited to L, encoded data is transmitted via a transmission path in which a transmission error occurs. According to the “method d2”, even if a transmission error occurs in the frame received immediately before, the transmission error is propagated as long as the current frame refers to the frame with the earlier coding order. Can be prevented.

In the first embodiment and the second embodiment, the values k set in the target frame setting step are set in order from the smallest value, but other methods may be used. Assuming that the frame number when the Mth frame is added is k, the following two methods k1 and k2 can be cited as examples of value setting methods when the target frame setting step S5 is executed next. Not limited to these methods, in the target frame setting step S5, the value k may be set according to any method as long as values from 1 to N other than the encoding order frame number F can be set.
Method k1: Set the value to either k + 1 or k-1.
Method k2: A value obtained by adding 1 to the maximum frame number of the encoding order frame number F or a value obtained by subtracting 1 from the minimum frame number is set.

  The above-described processing for determining the image coding order can be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or provided via a network. It is.

It is a figure which shows an example of a structure of the image coding order determination apparatus of this invention. It is a figure which shows an example of a structure of a frame addition initial setting part. It is a figure explaining the process outline | summary of embodiment of this invention. It is a figure explaining the process outline | summary of embodiment of this invention. It is a figure which shows the example of the encoding sequence determination processing flow which concerns on this invention. It is a figure which shows the processing flow (example 1) of an evaluation total addition step. It is a figure which shows the processing flow (example 2) of an evaluation total addition step. It is a figure which shows the processing flow (example 1) of a frame evaluation total addition step. It is a figure which shows the processing flow (example 2) of a frame evaluation total addition step. It is a figure which shows the processing flow (example 3) of a frame evaluation total addition step. It is a figure which shows the example of the processing flow of a frame addition initial setting step.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image coding order determination apparatus 11 Original image storage memory 12 Switching part 13 Intra-frame evaluation value calculation part 14 Inter-frame evaluation value calculation part 15 Evaluation value storage memory 16 Frame addition initial setting part 17 Target frame setting part 18 Target frame order setting Unit 19 provisional coding order setting unit 20 evaluation total initial setting unit 21 evaluation total addition unit 22 target frame order determination unit 161 frame neighborhood evaluation search unit 162 frame neighborhood evaluation addition unit 163 frame neighborhood evaluation accumulation memory 164 frame neighborhood evaluation total search unit

Claims (11)

An image coding order determination device for determining the coding order of M frames out of N frame image information in an image coding device that encodes image information of a plurality of frames using inter-frame predictive coding. There,
Intra-frame evaluation value calculation means for calculating an evaluation value E (i, i) for evaluating the generated code amount using image information of a frame with frame number i (i = 1, 2,..., N);
All combinations for selecting two frames of frame number i (i = 1, 2,..., N) and frame number j (j = 1, 2,..., N; where j ≠ i) from the N frames. Inter-frame evaluation value calculation means for calculating an evaluation value E (i, j) for evaluating the amount of generated code using difference information from the image information of the frame number i of the frame image information of frame number j When,
Evaluation value storage means for storing evaluation values calculated by the intra-frame evaluation value calculation means and the inter-frame evaluation value calculation means;
Of the N frames, a value F (i) (i = 1, 1) of an encoding order frame number F indicating the encoding order of frames determined by a predetermined method for a certain (M−1) frames. 2,..., M-1) frame addition initial setting means;
Target frame setting means for setting a frame number value k of a frame to be added as an encoding target to a value from 1 to N other than the encoding order frame number F;
A target frame order setting means for setting the frame of frame number k to the b-th (b is a preset value c to M; where 1 ≦ c ≦ M−1) encoding order;
Provisional encoding order T (i) (i = 1, 2,...) Obtained by adding the frame of frame number k set by the target frame order setting means to the encoding order set in encoding order frame number F. , M), provisional encoding order setting means,
An evaluation total initial setting means for setting the evaluation value E (T (1), T (1)) to the evaluation total value S;
J of T (j) is sequentially changed from 2 to M in the order of the frame numbers set in the provisional encoding order, and the evaluation value E (T (i), T (j)) (i is 1 to j− Evaluation sum addition means for repeatedly adding any one of 1) to the evaluation total value S;
The evaluation total value S calculated by sequentially changing the value b from the value c to M by the target frame order setting means, the provisional encoding order setting means, the evaluation total initial setting means, and the evaluation total addition means. Among these, the provisional encoding order T (i) (i = 1, 2,..., M) with the value b having the smallest evaluation total value S is determined as the encoding order of the M frames to be obtained. An image coding order determining apparatus comprising: an order determining unit. In the image coding order determination apparatus according to claim 1,
The predetermined method for the frame addition initial setting means to set the value F (i) of the encoding order frame number F is the intra-frame evaluation value calculation means and the inter-frame evaluation for each of the (M−1) frames. A method for determining an evaluation total value to be small using an evaluation value calculated by a value calculating means, or a method for selecting a neighborhood of a frame having a predetermined initial frame number.
An image coding order determination apparatus characterized by the above. In an image encoding method for encoding image information of a plurality of frames using inter-frame predictive encoding, an image encoding order determination method for determining an encoding order of M frames among image information of N frames There,
A frame for calculating an evaluation value E (i, i) for evaluating a generated code amount using image information of a frame with frame number i (i = 1, 2,..., N) and storing it in the evaluation value storage means Internal evaluation value calculation step,
All combinations for selecting two frames of frame number i (i = 1, 2,..., N) and frame number j (j = 1, 2,..., N; where j ≠ i) from the N frames. An evaluation value E (i, j) for evaluating the amount of generated code is calculated using the difference information from the image information of the frame number i of the frame image information of the frame number j, and the evaluation value storage means An inter-frame evaluation value calculation step to be stored;
Of the N frames, a value F (i) (i = 1, 1) of an encoding order frame number F indicating the encoding order of frames determined by a predetermined method for a certain (M−1) frames. 2,..., M-1), a frame addition initial setting step;
A target frame setting step of setting a frame number value k of a frame to be added as a coding target to a value from 1 to N other than the coding order frame number F;
A target frame order setting step for setting the frame of frame number k to the b-th (b is a preset value c to M; where 1 ≦ c ≦ M−1) encoding order;
Provisional encoding order T (i) (i = 1, 2,...) Obtained by adding the frame of frame number k set in the target frame order setting step to the encoding order set in encoding order frame number F. , M), a provisional encoding order setting step;
An evaluation total initial setting step for setting the evaluation value E (T (1), T (1)) to the evaluation total value S;
J of T (j) is sequentially changed from 2 to M in the order of the frame numbers set in the provisional encoding order, and the evaluation value E (T (i), T (j)) (i is 1 to j− An evaluation total addition step of repeatedly adding any one of 1) to the evaluation total value S;
The evaluation total value S calculated by sequentially changing the value b from the value c to the M by the target frame order setting step, the provisional encoding order setting step, the evaluation total initial setting step, and the evaluation total addition step. The target frame for determining the provisional encoding order T (i) (i = 1, 2,..., M) with the value b having the smallest evaluation total value S as the encoding order of M frames to be obtained. An image coding order determination method comprising: an order determination step. The image encoding order determination method according to claim 3,
The predetermined method for setting the value F (i) of the encoding order frame number F in the frame addition initial setting step is the intra-frame evaluation value calculation step and the inter-frame evaluation for each of the (M−1) frames. A method for determining the total evaluation value to be small by using the evaluation value calculated in the value calculation step, or a method for selecting a neighborhood of a frame having a predetermined initial frame number.
An image coding order determination method characterized by the above. The image coding order determination method according to claim 3 ,
In the evaluation total addition step,
For the frame number T (j), the value i is changed to any value from 1 to j−1, and a value that minimizes the evaluation value E (T (i), T (j)) is searched. The frame evaluation total addition step of adding the minimum evaluation value E (T (i), T (j)) to the evaluation total value S is performed repeatedly by sequentially changing j from 2 to M. A method for determining an image encoding order. In the image coding order determination method according to claim 5,
In the frame evaluation total addition step,
A referenceable frame number setting step for setting L values R (d) (d = 1, 2,..., L) as referenceable frame numbers R;
The evaluation value E (T (i), T (j)) is used as a search candidate only when the T (i) is equal to one of the values included in the set referenceable frame number R. A method for determining an image encoding order. The image coding order determination method according to claim 3 ,
In the frame addition initial setting step,
For the i-th frame, the evaluation value E (i, j) (j ≠ i) calculated in the inter-frame evaluation value calculation step is the j value j _p (i) when the evaluation value is the pth smallest, a frame neighborhood evaluation search step for obtaining p from 1 to M-2;
For the i-th frame, the evaluation value E (i, i) and the evaluation value E (i, j _p (i)) are summed with all values from 1 to M−1 to obtain an evaluation total value ES ( a frame neighborhood evaluation addition step for obtaining i) for all i from 1 to N;
The value m of i when the evaluation total value ES (i) (1 ≦ i ≦ N) is the smallest is obtained, and m and j _p (m) (p = 1, 2,... M-2) is set in the order of m, j ₁ (m), j ₂ (m),..., J _M-2 (m) . The image coding order determination method according to claim 3 ,
In the intra-frame evaluation value calculation step,
An intra-frame encoding step for intra-frame encoding the image information of the i-th frame;
A decoded image creating step for creating a decoded image from the coded data created in the intra-frame coding step;
A difference calculating step for calculating difference information between the image information of the i-th frame and the generated decoded image;
And a step of calculating the evaluation value E (i, i) using the difference information calculated in the difference calculating step. The image coding order determination method according to claim 3 ,
In the inter-frame evaluation value calculation step,
A difference encoding step for encoding difference information from the image information of the i-th frame of the image information of the j-th frame;
A decoded image creating step of decoding the encoded difference information and adding it to the image information of the i-th frame to create a decoded image;
A difference calculating step for calculating difference information between the image information of the jth frame and the generated decoded image;
And a step of calculating the evaluation value E (i, j) using the difference information calculated in the difference calculation step. An image coding order determination program for causing a computer to execute the image coding order determination method according to any one of claims 3 to 9 . A recording medium on which an image encoding order determination program for causing a computer to execute the image encoding order determination method according to any one of claims 3 to 9 is recorded.

Images