KR20090037475A - Multi-pass video encoding - Google Patents

Abstract

Some embodiments of the invention provide a multi-pass encoding method that encodes several images (e.g., several frames of a video sequence). The method iteratively performs an encoding operation that encodes these images. The encoding operation is based on a nominal quantization parameter, which the method uses to compute quantization parameters for the images. During several different iterations of the encoding operation, the method uses several different nominal quantization parameters. The method stops its iterations when it reaches a terminating criterion (e.g., it identifies an acceptable encoding of the images).

Description

Multi-pass video encoding method {MULTI-PASS VIDEO ENCODING}

The video encoder encodes a sequence of video images (eg, video frames) by using various encoding schemes. Video encoding schemes typically encode a video frame or a portion of a video frame (eg, a set of pixels of a video frame) in terms of intraframe or interframe. An intra frame encoded frame or set of pixels is encoded independently of another frame or set of pixels of another frame. An inter frame encoded frame or set of pixels is encoded with reference to a set of pixels of one or more other frames or other frames.

When compressing video frames, some encoders implement a 'rate controller' which provides a 'bit budget' for the video frame or set of video frames to be encoded. Bit budget describes the number of bits allocated for encoding a video frame or set of video frames. By effectively allocating bit budgets, the rate controller seeks to produce the best quality compressed video stream in terms of certain constraints (eg, target bitrate, etc.).

To date, various single-pass and multi-pass rate controllers have been proposed. Single-pass rate controller provides a bit budget for an encoding scheme that encodes a continuous video image in one pass, while a multi-pass rate controller provides a bit budget for an encoding scheme that encodes a continuous video image in multiple passes. To provide.

Single-pass rate controllers are useful for real time encoding situations. On the other hand, the multi-pass rate controller optimizes the encoding of a particular bitrate based on a set of constraints. To date, only some rate controllers take into account the spatial or temporal complexity of a frame or set of pixels within a frame in controlling the bitrate of its encoding. In addition, most multi-pass rate controllers do not properly search the solution space to find an encoding solution that uses the optimal quantization parameter for the frame and / or set of pixels within the frame in terms of the desired bitrate.

Therefore, there is a need in the art for rate controllers using new techniques for controlling the bitrate for encoding a set of video images while taking into account the spatial or temporal complexity of the video image and / or a portion of the video image. There is also a need in the art for a multi-pass controller to properly identify an encoding solution and identify the encoding solution using an optimal set of quantization parameters for the video image and / or a portion of the video image.

Some embodiments of the present invention provide a multi-pass encoding method for encoding a plurality of images (eg, a plurality of video sequence frames). This method repeatedly performs an encoding operation for encoding these images. This encoding operation is based on the nominal quantization parameter that the method uses to calculate the quantization parameter for the images. In repetition of several different encoding operations, the method uses several different nominal quantization parameters. The method stops these iterations when it reaches an end criterion (eg, when identifying an acceptable encoding of the images).

Some embodiments of the present invention provide a method for encoding a video sequence. This method identifies a first attribute that quantifies the complexity of the first image of the video. The method also identifies a quantization parameter for encoding the first image based on the identified first attribute. The method then encodes the first image based on the identified quantization parameter. In some embodiments, the method performs these three operations on several images of the video.

Some embodiments of the present invention encode a video image sequence based on the "visual masking" attribute of the video image and / or a portion of the video image. Visual masking of an image or part of an image is an indicator of how much coding artifacts can be allowed in the image or part of the image. In order to represent the visual masking properties of an image or part of an image, some embodiments calculate visual masking intensities that quantify the luminance energy of the image or part of the image. In some embodiments, luminance energy is measured as a function of average luma or pixel energy of an image or portion of an image.

Instead of, or with, the luminance energy, the visual masking intensity of the image or portion of the image may also quantify the active energy of the image or portion of the image. Active energy expresses the complexity of an image or part of an image. In some embodiments, the activation energy includes a spatial component that quantifies the spatial complexity of the image or portion of the image, and / or a movement component that quantifies the amount of distortion that can be tolerated or masked due to movement between the images.

Some embodiments of the present invention provide a method for encoding a video sequence. This method identifies the visual-masking properties of the first image of the video. The method also identifies quantization parameters for encoding the first image based on the identified visual-masking attributes. The method then encodes the first image based on the identified quantization parameter.

New features of the invention are set forth in the appended claims. However, for the sake of illustration, some embodiments of the invention are described in the following figures.

In the following detailed description of the invention, various details, examples, and examples of the invention are described and described. However, it will be apparent to those skilled in the art that the present invention is not limited to the described embodiments and that the present invention may be practiced without some specific details and described examples.

I. Justice

This section provides definitions of some of the symbols used herein.

는 프레임 시퀀스를 인코딩하는 데에 바람직한 비트레이트인 목표 비트레이트를 나타낸다. 통상적으로, 이러한 비트레이트는 bit/sec(초 당 비트) 단위로 표현되며, 바람직한 최종 파일 사이즈, 시퀀스에서의 프레임 개수, 및 프레임레이트로부터 계산된다.

Denotes a target bitrate, which is the preferred bitrate for encoding the frame sequence. Typically, this bitrate is expressed in bits / sec (bits per second) and is calculated from the desired final file size, the number of frames in the sequence, and the frame rate.

는 패스 p의 끝에서의 인코딩된 비트 스트림의 비트레이트를 나타낸다.

Denotes the bitrate of the encoded bit stream at the end of pass p.

는 패스 p의 끝에서의 비트레이트의 오차 퍼센트를 나타낸다. 몇몇의 경우, 이러한 퍼센트는 100 x

로서 계산된다.

Denotes the error percentage of the bitrate at the end of pass p. In some cases, this percentage is 100 x

Is calculated as

ε represents the error tolerance at the final bitrate.

는 제1 QP 검색 단계에 대한 비트레이트의 오차 허용치를 나타낸다.

Denotes an error tolerance of the bitrate for the first QP search step.

QP represents a quantization parameter.

는 프레임 시퀀스에 대한 패스 p 인코딩에 이용되는 명목상의 양자화 파라미터를 나타낸다.

의 값은 제1 QP 조정 단계에서 본 발명의 멀티-패스 인코더에 의해 목표 비트레이트에 도달하도록 조정된다.

Denotes a nominal quantization parameter used for pass p encoding for a frame sequence.

Is adjusted to reach the target bitrate by the multi-pass encoder of the present invention in the first QP adjustment step.

는 패스 p에서의 프레임 k에 대한 양자화 파라미터(QP)인 마스킹된 프레임 QP를 나타낸다. 몇몇의 실시예에는 이 값을 명목상의 QP 및 프레임-레벨 시각적 마스킹에 의해 계산한다.

Denotes the masked frame QP, which is the quantization parameter QP for frame k in pass p. In some embodiments this value is calculated by nominal QP and frame-level visual masking.

는 프레임 k 및 패스 p의 (매크로블록 인덱스 m을 가지는) 개별적인 매크로블록에 대한 양자화 파라미터(QP)인 마스킹된 매크로블록 QP를 나타낸다. 몇몇의 실시예는

를

및 매크로블록 레벨 시각적 마스킹을 이용하여 계산한다.

Denotes the masked macroblock QP, which is the quantization parameter (QP) for the individual macroblocks (with macroblock index m) of frame k and pass p. Some embodiments

To

And macroblock level visual masking.

는 프레임 k에 대한 마스킹 강도라 칭하는 값을 나타낸다. 마스킹 강도

는 프레임에 대한 복잡도의 측정치이며, 몇몇의 실시예에서, 이 값은 어떻게 시각적인 코딩 아티팩트(artifact)/잡음이 나타날지를 결정하고 프레임 k의

를 계산하는 데에 이용된다.

Denotes a value called masking strength for frame k. Masking strength

Is a measure of the complexity of the frame, and in some embodiments, this value determines how visual coding artifacts / noises will appear and

It is used to calculate

는 패스 p의 기준 마스킹 강도를 나타낸다. 기준 마스킹 강도는 프레 임 k의

를 계산하는 데에 이용되고, 제2 단계에서 본 발명의 멀티-패스 인코더에 의해 목표 비트레이트에 도달하도록 조정된다.

Denotes the reference masking intensity of pass p. Reference masking strength of frame k

It is used to calculate, and adjusted to reach the target bitrate by the multi-pass encoder of the present invention in the second step.

는 프레임 k의 인덱스 m을 가지는 매크로블록에 대한 마스킹 강도를 나타낸다. 마스킹 강도

는 매크로블록에 대한 복잡도의 측정치이고, 몇몇의 실시예에서 이 값은 시각적인 코딩 아티팩트/잡음이 어떻게 나타날지를 결정하고

를 계산하는 데에 이용된다. AMQP_p는 패스 p의 프레임들에 대한 평균 마스킹된 QP를 나타낸다. 몇몇의 실시예에서, 이 값은 패스 p의 모든 프레임들에 대한 평균

로서 계산된다.

Denotes the masking strength for the macroblock with index m of frame k. Masking strength

Is a measure of complexity for the macroblock, and in some embodiments this value determines how visual coding artifacts / noise will appear and

It is used to calculate AMQP _p represents the average masked QP for the frames of pass p. In some embodiments, this value is an average over all frames of pass p.

Is calculated as

II. survey

Some embodiments of the present invention provide an encoding method that achieves the best visual quality in encoding a frame sequence at a predetermined bitrate. In some embodiments, this method uses a visual masking process that assigns the quantization parameter QP to all macroblocks. This assignment is based on the fact that coding artifacts / noise in brighter or spatially complex areas of an image or video frame are less visible than coding artifacts / noise in darker or flat areas.

을 통하여 시각적인 마스킹 프로세스를 제어한다. 이하에 더 기술될 바와 같이, 명목상의 양자화 파라미터를 조정하는 것과 마스킹 알고리즘을 제어하는 것은 각각의 화상(즉, 통상적인 비디오 인코딩 스킴에서의 각 프레임) 및 각각의 화상 내의 각각의 매크로블록에 대한 QP 값을 조정한다.In some embodiments, this visual masking process is performed as part of the multi-pass encoding process of the present invention. This encoding process adjusts the nominal quantization parameter and sets the reference masking intensity parameter so that the final encoded bit stream reaches the target bitrate.

To control the visual masking process. As will be described further below, adjusting the nominal quantization parameter and controlling the masking algorithm are QP for each picture (ie, each frame in a typical video encoding scheme) and each macroblock in each picture. Adjust the value.

을 전역적으로 조정한다. 다른 실시예에서, 이 프로세스는 비디오 시퀀스를 세그먼트들로 나누고 각 세그먼트에 대하여 명목상의 QP 및

이 조정된다. 이하의 설명은 멀티-패스 인코딩 프로세스가 채용된 프레임의 시퀀스를 언급한다. 본 기술 분야에서 통상의 기술을 가진 자라면 이러한 시퀀스는 몇몇의 실시예에서는 전체 시퀀스를 포함하지만, 다른 실시예에서는 시퀀스 중 어느 한 세그먼트만을 포함함을 인식할 것이다.In some embodiments, the multi-pass encoding process includes a nominal QP for the entire sequence and

Globally. In another embodiment, this process divides the video sequence into segments and for each segment a nominal QP and

This is adjusted. The following description refers to a sequence of frames in which a multi-pass encoding process is employed. Those skilled in the art will recognize that such a sequence includes the entire sequence in some embodiments, but only one segment of the sequence in other embodiments.

에서 수행되는 제1 검색 단계, 및 (3) 패스

내지 패스

에서 수행되는 제2 검색 단계이다.In some embodiments, the method has three encoding steps. These three steps are (1) an initial analysis step performed in pass 0, (2) pass 1 to pass

A first search step performed in (3), and (3) pass

To pass

The second search step is performed at.

의 값 또한 식별한다.In an initial analysis step (ie pass 0), the method identifies an initial value for the nominal QP (QP _Nom (1) to be used in pass 1 of encoding). During the initial analysis phase, the method uses the reference masking intensity used in all passes of the first search phase.

It also identifies the value of.

번의 반복(즉,

패스들)을 수행한다. 각 패스 p 중 각각의 프레임 k에 대하여, 프로세스는 프레임 k 내의 개별적인 매크로블록 m에 대한 특정 양자화 파라미터

및 특정 양자화 파라미터

를 이용함으로써 프레임을 인코딩하며, 여기서

는

를 이용하여 계산된다.In the first retrieval step, the method comprises

Repetitions (i.e.

Passes). For each frame k of each pass p, the process may specify specific quantization parameters for the individual macroblock m in frame k.

And specific quantization parameters

Encode the frame by using

Is

Calculated using

는 패스들마다 변화하는 명목상의 양자화 파라미터

로부터 유도되기 때문에 패스들마다 변한다. 다시 말하면, 제1 검색 단계 중에 각각의 패스 p의 끝에서, 프로세스는 패스

에 대한 명목상의

을 계산한다. 몇몇의 실시예에서, 명목상의

는 이전의 패스(들)로부터의 명목상의 QP 값(들) 및 비트레이트 오차(들)에 기초한다. 다른 실시예에서, 명목상의

값은 제2 검색 단계에서 각각의 패스의 끝에서 서로 다르게 계산된다.In the first search step, the quantization parameter

Is the nominal quantization parameter that varies from pass to pass

Since it is derived from, it changes from pass to pass. In other words, at the end of each pass p during the first search phase, the process passes

Nominal for

Calculate In some embodiments, nominal

Is based on the nominal QP value (s) and bitrate error (s) from the previous pass (s). In another embodiment, nominal

The value is calculated differently at the end of each pass in the second search step.

번의 반복(즉,

개의 패스들)을 수행한다. 제1 검색 단계에서와 같이, 프로세스는 프레임 k 내의 개별적인 매크로블록 m에 대한 특정 양자화 파라미터

및 특정 양자화 파라미터

를 이용함으로써 각각의 패스 p 중에 각각의 프레임 k를 인코딩하 는데, 여기서

는

로부터 유도된다.In the second retrieval step, the method is characterized by

Repetitions (i.e.

Pass). As in the first retrieval step, the process performs specific quantization parameters for the individual macroblock m in frame k.

And specific quantization parameters

Encode each frame k in each pass p by using

Is

Derived from.

는 패스들마다 변한다. 그러나, 제2 검색 단계 중에, 이러한 파라미터는 패스들마다 변하는 기준 마스킹 강도

를 이용하여 계산되기 때문에 변화하는 것이다. 몇몇의 실시예에서, 기준 마스킹 강도

는 이전 패스(들)로부터의 값(들) 및 비트레이트(들)에서의 오차에 기초하여 계산된다. 다른 실시예에서, 이러한 기준 마스킹 강도는 제2 검색 단계에서의 각각의 패스 끝에서 다른 값이 되도록 계산된다.Also, as in the first search step, the quantization parameter

Changes from pass to pass. However, during the second search phase, this parameter varies with reference masking intensity from pass to pass.

Because it is calculated using In some embodiments, reference masking strength

Is calculated based on the value (s) from the previous pass (s) and the error in the bitrate (s). In another embodiment, this reference masking intensity is calculated to be a different value at the end of each pass in the second search step.

을 무시하고 상술한 제2 검색 단계를 생략함으로써 시각적인 마스킹 없이 소정의 목표 비트레이트에 근접하게 비트스트림을 인코딩하는 데에 이용된다.Although the multi-pass encoding process has been described with a visual masking process, one of ordinary skill in the art will recognize that the encoder does not need to use both of these processes together. For example, in some embodiments, the multi-pass encoding process is

By ignoring the above and skipping the above-described second search step, it is used to encode the bitstream close to the desired target bitrate without visual masking.

The visual masking and multi-pass encoding process is further described in sections III and IV herein.

III. Visual masking

) 및 프레임 마스킹 강도

를 이용하여 각각의 프레임에 대 한 마스킹된 프레임 양자화 파라미터(MQP)를 계산한다. 그 다음 이 프로세스는 각각의 매크로블록에 대한 마스킹된 매크로블록 양자화 파라미터

를, 프레임 및 매크로블록-레벨 마스킹 강도(φ_F 및 φ_MB)에 기초하여, 계산한다. 시각적인 마스킹 프로세스가 멀티-패스 인코딩 프로세스에 채용된다면, 몇몇의 실시예에서의 기준 마스킹 강도(

)는 상술하였고 후술될 바와 같이 제1 인코딩 패스 중에 식별된다.Given a nominal quantization parameter, the visual masking process first begins with the reference masking intensity (

A) and frame masking strength

Calculate the masked frame quantization parameter (MQP) for each frame. The process then performs a masked macroblock quantization parameter for each macroblock.

Is calculated based on the frame and macroblock-level masking intensities φ _F and φ _MB . If a visual masking process is employed in the multi-pass encoding process, the reference masking intensity (in some embodiments)

) Is identified during the first encoding pass as described above and will be described later.

A. Frame-Level Masking Strength Calculation

1. The first approach

To calculate the frame-level masking intensity φ _F (k), some embodiments use the following equation:

From here,

는 bxb 영역을 이용하여 계산된 프레임 k에서의 평균 픽셀 강도이며, b는 1 이상의 정수이다(예를 들면, b=1 또는 b=4).ㆍ

Is the average pixel intensity in frame k calculated using the bxb region, and b is an integer of at least 1 (e.g., b = 1 or b = 4).

는 프레임 k의 모든 매크로블록에 대한

의 평균이다.ㆍ

For all macroblocks in frame k

Is the average.

는 인덱스 m을 가지는 매크로블록의 모든 4x4 블록에 대한 함수

에 의해 주어진 값들의 합이다.ㆍ

Is a function of all 4x4 blocks of the macroblock with index m

Is the sum of the values given by.

, C, D 및 E는 상수이고/거나 지역적 통계에 적응된다.ㆍ

, C, D and E are constant and / or adapted to local statistics.

을 의미한다.Power (a, b) is

Means.

의 의사 코드는 다음과 같다:function

The pseudo code for is as follows:

2. The Second Approach

Another embodiment calculates frame-level masking intensity differently. For example, Equation 1 above essentially calculates the frame masking intensity as follows:

는

와 같고,

는 프레임의 모든 매크로블록에 대한 평균 매크로블록 SAD(

) 값인

와 같으며, 여기서 평균 매크로블록 SAD는 매 크로블록의 모든 4x4 블록에 대한 (

에 의해 주어진) 평균 제거된 4x4 픽셀 편차의 절대값의 합과 같다. 이

는 코딩되고 있는 프레임 내의 픽셀의 영역의 공간적 전개의 총계를 측정한다.In Equation 1, the frame

Is

Is the same as

Is the average macroblock SAD (for all macroblocks in a frame)

) Value

Where the average macroblock SAD is (for all 4x4 blocks of each macroblock)

Equal to the sum of the absolute values of the mean removed 4x4 pixel deviations, given by. this

Measures the total amount of spatial evolution of the region of pixels in the frame being coded.

Another embodiment extends the active measure to include the amount of temporal evolution in the pixel region over a plurality of consecutive frames. Specifically, these examples calculate the frame masking strength as follows.

는 다음의 수학식 3에 의해 주어진다.In this equation,

Is given by the following equation (3).

는 프레임들 간의 움직임에 의한, 허용될 수 있는(즉, 마스킹될 수 있는) 왜곡의 양을 정량화한다. 몇몇의 이들 실시예에서, 프레임의

는 프레임 내에 정의된 픽셀 영역의 움직임 보상된 오차 신호의 절대값 합의 정수배와 같다. 다른 실시예에서,

는 아래의 수학식 4에 의해 제공된다.In some embodiments,

Quantifies the amount of distortion that can be tolerated (ie, masked) by motion between frames. In some of these embodiments, the frame

Is an integer multiple of the absolute value sum of the motion compensated error signals of the pixel region defined within the frame. In another embodiment,

Is provided by Equation 4 below.

"는 (상술한 바와 같이) 프레임의 평균 매크로블록 SAD(

) 값을 나타내고,

(0)은 현재 프레임에 대한

이며, 음수 j는 현재 프레임 이전의 순시(time instant)를 인덱싱하고 양수 j는 현재 프레임 이후의 순시를 인덱싱한다. 그러므로,

는 현재 프레임 이전의 2개의 프레임의 평균 프레임 SAD를 나타내고,

는 현재 프레임 이후의 3개의 프레임의 평균 프레임 SAD를 나타낸다.In Equation 4, "

"Is the average macroblock SAD (

) Value,

(0) for the current frame

Negative j indexes the instant instant before the current frame and positive j indexes the instant instant after the current frame. therefore,

Denotes the average frame SAD of the two frames before the current frame,

Denotes an average frame SAD of three frames after the current frame.

Also, in Equation 4, variables N and M represent the number of frames before the current frame and the number of frames after the current frame, respectively. Instead of simply selecting values N and M based on a particular number of frames, some embodiments calculate values N and M based on a specific time interval before and after the current frame time. Correlating motion masking in a temporal period is advantageous over correlating motion masking to a set number of frames. This is because correlating motion masking with temporal intervals directly conforms to the observer's time-based visual perception. On the other hand, if such masking is correlated to the number of frames, there is a problem in that the display interval is changed because different displays display video at different frame rates.

)은 현재 프레임의 프레임 SAD를 나타낸다.In Equation 4, " W " represents, in some embodiments, a weighting factor that decreases as frame j advances from the current frame. Also, in this equation, the first sum represents the amount of motion that can be masked before the current frame, the second sum represents the amount of motion that can be masked after the current frame, and the last expression (

) Represents the frame SAD of the current frame.

In some embodiments, the weight coefficients are adjusted to account for the scene transitions. For example, some embodiments describe upcoming scene transitions within the foresight range (ie, within M frames), but do not describe any frames after the scene transition. For example, these embodiments may set the weighting factor to zero for a frame within the forecast range after the scene change. In addition, some embodiments do not describe frames before or during scene transitions within the recall range (ie, within M frames). For example, these embodiments may set the weighting factor to zero for frames within the recall range associated with the previous scene or before the previous scene transition.

3. Modifications to the Second Approach

에 미치는 영향을 제 한하기a) past or future frames

Limit the impact on

를 표현한다:Equation 4 is essentially

Expresses:

,

,

는

와 같고,

는

와 같으며,

는

와 같다.here

Is

Is the same as

Is

Is the same as

Is

Same as

의 계산을 수정하여

와

중 어느 것도 부적절하게

의 값을 제어하지 않도록 한다. 예를 들면, 몇몇의 실시예는 초기에 PFA가

와 같고, FFA가

와 같도록 정의한다.Some embodiments

By modifying the calculation of

Wow

None of them are inappropriately

Do not control the value of. For example, some embodiments initially show that PFA

Is equal to the FFA

Defined as

These embodiments then determine if the PFA is greater than the scalar product of the FFA. If so, these embodiments set the PFA to be equal to the upper limit PFA threshold (eg, a scalar product of the FFA). In addition to setting the PFA equal to the upper limit PFA threshold, some embodiments may perform a combination of setting the FFA to zero and setting the CFA to zero. Other embodiments may set both or one of PFA and CFA to a weighted combination of PFA, CFA, and FFA.

Similarly, after initially defining PFA and FFA values based on a weighted sum, some embodiments also determine whether the FFA value is greater than the scalar product of the PFA. If so, these embodiments set the FFA equal to the upper limit FFA limit value (eg, the scalar product of the PFA). In addition to setting the FFA equal to the upper limit FFA limit value, some embodiments may perform a combination of setting the PFA to zero and setting the CFA to zero. Other embodiments may set both or both of FFA and CFA to a weighted combination of FFA, CFA, and PFA.

를 제어하는 것을 방지한다.Potential subsequent adjustments of PFA and FFA values (after initial calculation of these values based on weighted sums) result in one of these values being inappropriately

To control it.

및

가

에 미치는 영향을 제한하기b)

And

end

Limit the impact on

를 표현한다.Equation 3 is essentially

Express

,

,

는 scalar^*(scalar^*

)^β와 같고,

는

와 같다.here

Scalar ^* (scalar ^*

) ^β ,

Is

Same as

의 계산을 수정하여

및

중 어느 것도

의 값을 부적절하게 제어하지 않도록 한다. 예를 들면, 몇몇의 실시예는 초기에

를 scalar^*(scalar^*

)^β와 같도록 정의하고,

를

와 같도록 정의한다.Some embodiments

By modifying the calculation of

And

None of

Do not improperly control the value of. For example, some embodiments initially

Scalar ^* (scalar ^*

) equal to ^β ,

To

Defined as

These embodiments then determine if SA is greater than the scalar product of the TA. If so, these embodiments set the SA equal to the upper SA limit value (eg, the scalar product of the TA). In addition to setting the SA to be equal to the upper SA limit, some embodiments in this case may set the TA value to zero or a weighted combination of TA and SA.

Likewise, after initially defining SA and TA values based on exponential equations, some embodiments also determine whether the TA value is greater than the scalar product of SA. If so, these embodiments set the TA equal to the upper limit TA limit value (eg, a scalar product of SA). In addition to setting the TA equal to the upper TA limit, in this case some embodiments may set the SA value to 0 or a weighted combination of SA and TA.

를 제어하는 것을 방지한다.Potential subsequent adjustments of SA and TA values (after initial calculation of these values based on exponential equations) result in one of these values being improperly adjusted.

To control it.

B. Calculation of Macroblock-Level Masking Intensity

1. The first approach

는 다음과 같이 계산된다:In some embodiments, macroblock-level masking strength

Is calculated as follows:

here,

는 프레임 k, 매크로블록 m의 평균 픽셀 강도이다.ㆍ

Is the average pixel intensity of frame k, macroblock m.

및 C는 상수이고/거나 지역적 통계에 적응된다.ㆍ

And C are constant and / or adapted to local statistics.

2. The Second Approach

Equation 5 above essentially calculates the macroblock masking strength as follows:

는

와 같고,

는

와 같다. 이

는 코딩되고 있는 매크로블록 내의 픽셀 영억에서의 공간적인 전개의 양을 측정한다.In Equation 5, the macroblock

Is

Is the same as

Is

Same as this

Measures the amount of spatial evolution in pixel permanents within the macroblock being coded.

As in the case of frame masking intensity, some embodiments may extend the active measure of macroblock masking intensity to include the amount of temporal evolution in the pixel region over a plurality of consecutive frames. Specifically, these examples will calculate the macroblock masking intensity as follows:

Where Mb_Activity_Attribute is given by:

The calculation of Mb_Temporal_Activity_Attribute for the macroblock may be similar to the calculation of Mb_Temporal_Activity_Attribute for the above-described frame. For example, in some of these embodiments, Mb_Temporal_Activity_Attribute is provided by the following equation:

The variables in Equation 8 are defined in section III.A. In equation (5), macroblock m of frame i or j may be a macroblock at the same position as that in macroblock m of the current frame, or frame i or j initially predicted to correspond to macroblock m of the current frame It may be a macroblock in.

The Mb_Temporal_Activity_Attribute provided by Equation 8 may be modified in a manner similar to the modification (described in section III.A.3 above) of the frame Temporal_Activity_Attribute provided by Equation 4. In detail, Mb_Temporal_Activity_Attribute provided by Equation 8 may be modified to limit an improper effect of a macroblock of a past or future frame.

의 (상기 섹션 Ⅲ.A.3)에 기술된 수정과 유사한 방식으로 수정될 수 있다. 상세히는, 수학식 7이 제공한 Mb_Activity_Attribute는 Mb_Spatial_Activity_Attribute 및 Mb_Temporal_Activity_Attribute의 부적절한 영향을 제한하도록 수정될 수 있다.Similarly, Mb_Activity_Attribute provided by Equation 7 is a frame provided by Equation 3

It may be modified in a manner similar to the modification described in (section III.A.3 above). In detail, Mb_Activity_Attribute provided by Equation 7 may be modified to limit inappropriate effects of Mb_Spatial_Activity_Attribute and Mb_Temporal_Activity_Attribute.

C. Calculation of Masked QP Values

및

) 및 기준 마스킹 강도의 값

에 기초하여, 시각적인 마스킹 프로세스는 2개의 함수

및

를 이용함으로써 프레임 레벨 및 매크로블록 레벨에서 마스킹된 QP 값을 계산할 수 있다. 이들 2 함수의 의사 코드는 다음과 같다:Value of masking intensity (

And

) And the value of the reference masking strength

Based on this, the visual masking process has two functions

And

By using, it is possible to calculate the masked QP value at the frame level and the macroblock level. The pseudo code for these two functions is as follows:

및

는 소정의 상수이거나 지역적 통계에 적응될 수 있다.In the above function,

And

May be a constant or adapt to local statistics.

Ⅳ. Multi-pass encoding

1 shows a process 100 conceptually illustrating a multi-pass encoding method of some embodiments of the present invention. As shown in this figure, process 100 has three steps, which are described in the following three sub-sections.

A. Analysis and initial QP selection

As shown in FIG. 1, the process 100 first begins with an initial value of the reference masking intensity φ _{R (1} ) and an initial nominal quantization parameter during the initial analysis phase of the multi-pass encoding process (ie, during pass 0 ₎ . The value QP _{Nom (1} ) is calculated (105). The initial reference masking intensity φ _{R (1)} is used during the first search phase, while the initial nominal quantization parameter QP _{Nom (1)} is used during the first pass of the first search phase (ie, multi-pass encoding). During pass 1 of the process).

(k)가 각 프레임에 대하여 계산된 다음, 기준 마스킹 강도, φ_R(1), 는 패스 0의 끝에서 avg(

(k))와 같도록 설정된다. 기준 마스킹 강도 φ_R에 대한 다른 결정 또한 가능하다. 예를 들면, 값

(k)의 중앙치 또는 다른 산술 함수, 예를 들면 값

(k)의 가중된 평균으로서 계산될 수 있다.At the beginning of pass 0, φ _{R (0)} may be some arbitrary value or one value selected based on experimental results (eg, the middle value of a typical range of φ _R values). During the analysis of the sequence, masking intensity

(k) is calculated for each frame, then the reference masking intensity, φ _{R (1)} , is avg (

(k)). Other determinations of the reference masking strength φ _R are also possible. For example, a value

median of (k) or other arithmetic function, eg value

It can be calculated as the weighted average of (k).

There are several approaches to initial QP selection that vary in complexity. For example, the initial nominal QP may be chosen at any value (eg, 26). Alternatively, a value known to produce an acceptable quality for the target bitrate may be selected based on the coding experiment.

The initial nominal QP value may also be selected from the lookup table based on spatial resolution, frame rate, space / temporal complexity, and target bitrate. In some embodiments, these initial nominal QP values may be selected from a table using distance measurements that depend on each of these parameters, or may be selected using weighted distance measurements of these parameters.

Since this initial nominal QP value is also selected during fast encoding using a rate controller (without masking), it can be set to the adjusted average of the frame QP values, where the average is the bitrate percentage of pass 0. It was adjusted based on the ratio error E ₀ . Similarly, the initial nominal QP can also be set to a weighted and adjusted average of the frame QP values, where the weight for each frame is determined by the percentage of macroblocks in this frame that are not coded as omitted macroblocks. do. Alternatively, since the initial nominal QP is chosen during fast encoding using the rate controller (while masking), so long as the influence of the change in the reference masking intensity from φ _{R ( 0)} to φ _{R (1)} is taken into account , An adjusted average of the frame QP values or an adjusted and weighted average.

B. First Search Phase: Nominal QP Adjustment

After reference numeral 105, the multi-pass encoding process 100 enters a first search step. In a first search step, process 100 performs N ₁ encoding of the sequence, where N ₁ represents the number of passes across the first search step. During each pass of the first step, the process uses varying nominal quantization parameters with constant reference masking intensity.

및 프레임 k 내의 각각의 개별적인 매크로블록 m에 대한 특정 양자화 파라미터

를 계산한다(107). 소정의 명목상의 양자화 파라미터

및 기준 마스킹 강도

에 대한 파라미터

및

의 계산은 섹션 Ⅲ에 기술되었다(여기서,

및

는 섹션 Ⅲ에서 기술하였던 함수

및

를 이용함으로써 계산된다.). 제1 패스(즉, 패스 1) 내지 참조번호(107)에서, 명목상의 양자화 파라미터 및 제1 단계 기준 마스킹 강도는 초기 분석 단계(105) 중에 계산되었던 파라미터 QP_Nom(1) 및 기준 마스킹 강도 φ_R(1)이다.In detail, during each pass p of the first search step, process 100 performs specific quantization parameters for each frame k.

And specific quantization parameters for each individual macroblock m in frame k

Calculate (107). Any nominal quantization parameter

And reference masking strength

Parameters for

And

The calculation of is described in section III (where

And

Is the function described in section III.

And

Calculated by using In the first pass (i.e., pass 1) to reference numeral 107, the nominal quantization parameter and the first stage reference masking intensity are determined by the parameter QP _{Nom (1)} and the reference masking intensity φ _R which were calculated during the initial analysis step 105. ₍₁₎

Following reference numeral 107, the process encodes 110 the sequence based on the quantization parameter value calculated at reference numeral 107. The encoding process 100 then determines 115 whether it should end. Different embodiments have different criteria for terminating the entire encoding process. Examples of escape conditions that completely terminate the multi-pass encoding process include:

<ε, 여기서 ε는 최종 비트레이트에서의 오차 허용치이다.ㆍ

<ε, where ε is the tolerance in the final bitrate.

가 QP 값의 유효 범위의 상한 또는 하한 경계에 있다.ㆍ

Is at the upper or lower boundary of the effective range of QP values.

The number of passes exceeded the maximum number of allowable passes P _MAX .

Some embodiments may use all of these escape conditions, while others may use only a few of them. However, other embodiments may use other escape conditions to terminate the encoding process.

If it is determined that the multi-pass encoding process ends (115), the process 100 skips the second search step and moves to reference numeral 145. At 145, the process saves the bitstream from the last pass p as the final result and then terminates.

On the other hand, if it is determined that the process should not end (115), it is determined whether the first search step should be terminated (120). Similarly, different embodiments have different criteria for terminating the first search step. Examples of escape conditions that terminate the first search phase of the multi-pass encoding process include:

이

와 동일하고

이다(이 경우, 비트레이트에서의 오차는 명목상의 QP를 수정함으로써 임의로 더 낮추어질 수 없다).ㆍ

this

Same as

(In this case, the error in the bitrate cannot be arbitrarily lowered by correcting the nominal QP).

ε, 여기서 ε_C는 제1 검색 단계에서의 비트레이트에서의 오차 허용치이다.ㆍ

ε, where ε _C is the error tolerance in the bit rate in the first search step.

And the number of paths exceeds P _1, where P ₁ is less than P _MAX.

이다.And the number of paths exceeds P _2, where P ₂ is less than P _1,

to be.

Some embodiments may use all of these escape conditions, while others may use only a few of them. However, other embodiments may use other escape conditions to end the first search step.

을 정의한다). 몇몇의 실시예에서, 명목상의

는 다음과 같이 갱신된다. 패스 1의 끝에서, 이들 실시예는If it is determined that the multi-pass encoding process ends the first search step (120), the process 100 proceeds to the second search step to be described in the next sub-section. On the other hand, if the process determines that it should not terminate the first search step (120), it updates 125 the nominal QP for the next pass of the first search step (i.e.,

To define). In some embodiments, nominal

Is updated as follows: At the end of pass 1, these embodiments

를 정의하며, 여기서

는 상수이다. 그 다음 패스 2 내지 패스

에서의 패스의 각각의 끝에서, 이들 실시예는

, Where

Is a constant. Then pass 2 to pass

At each end of the pass in, these embodiments

를 정의하며, 여기서

는 이하 더 기술될 함수이다. 또한, 상기 수학식에서, q1 및 q2는 패스 p까지의 모든 패스들 중에서 가장 낮은 대응하는 비트레이트 오차를 가지는 패스 번호이고, q1, q2, 및 p는 다음과 같은 관계를 가진다:

, Where

Is a function to be described further below. Further, in the above equation, q1 and q2 are pass numbers with the lowest corresponding bitrate errors among all passes up to pass p, and q1, q2, and p have the following relationship:

The following is pseudo code for InterpExtrap function. Note that if x is not between x1 and x2, this function is an extrapolation function. Otherwise, this function is an interpolation function.

값을 계산할 수 있음을 인식할 것이다.The nominal QP value is typically rounded to an integer value and taken anhydrous to be within the valid range of the QP value. Those skilled in the art will appreciate that other embodiments may be nominally different from the approaches described above.

It will be appreciated that the value can be calculated.

및 프레임 k 내의 개별적인 매크로 블럭 각각에 대한 특정 양자화 파라미터

를 계산한다(107). 그 다음, 프로세스는 이들 새롭게 계산된 양자화 파라미터에 기초하여 프레임의 시퀀스를 인코딩한다(110). 그 다음 참조번호(110)로부터, 프로세스는 상술하였던 참조번호(115)로 이동한다.At 125, the process moves back to 107 and starts the next pass (i.e. p: = p + 1), in which a specific quantization for each frame k for the current pass p. parameter

And specific quantization parameters for each individual macroblock in frame k

Calculate (107). The process then encodes 110 a sequence of frames based on these newly calculated quantization parameters. Then from reference numeral 110, the process moves to reference numeral 115 described above.

C. Second Search Step: Adjust Baseline Masking Strength

If process 100 determines that the first search step should be terminated (120), the process moves to reference numeral 130. In a second search step, process 100 performs N ₂ sequence encodings, where N ₂ represents the number of passes throughout the second search step. During each pass, the process uses the same nominal quantization parameter and the changing reference masking intensity.

In reference numeral 130, the process 100 calculates the reference masking intensity φ _{R (p + 1)} for the next pass, pass p + 1, which is pass N ₁ +1. In pass N ₁ +1, process 100 encodes the sequence of frames at reference numeral 315. Different embodiments calculate the reference masking strength φ _{R (p + 1)} at the end of pass p in different ways (130). Two alternative approaches are described below.

의 비트레이트(들) 및 값(들)의 오차에 기초하여 기준 마스킹 강도

를 계산한다. 예를 들면, 패스 N₁의 끝에서, 몇몇의 실시예는

를 정의한다.Some embodiments may be from previous pass (s).

Reference masking strength based on the bitrate (s) and error of the value (s) of

Calculate For example, at the end of pass N ₁ , some embodiments

Define.

At the end of pass N ₁ + m (m is an integer greater than 1), some embodiments

Define. Alternatively, some embodiments

Where q1 and q2 are the previous passes that provided the best error.

에 대한 몇몇의 값을 계산하기 위한 한가지 방식은 함수

의 의사코드를 참조하여 다음에 기술된다.Other embodiments calculate the reference masking intensity at the end of each pass of the second search step by using the AMQP as defined in section I. AMQP for a given nominal QP and

One way to calculate some values for is a function

It is described next with reference to the pseudo code of.

은 의사코드가 이 서브섹션의 끝에 주어지는 함수

에 의해 주어진 검색 프로세저를 통해 구한다.Some embodiments using AMQP calculate a predetermined AMQP for pass p + 1 based on the error in the bitrate (s) and value (s) of AMQP from the previous pass (s). Then corresponds to this AMQP

Is a function whose pseudocode is given at the end of this subsection.

Obtained through the search procedure given by.

를 계산하는데, 여기에서

일 때,

이고,

일때,

이다.For example, some embodiments at the end of pass N ₁

Where

when,

ego,

when,

to be.

를 정의한다. 패스 N₁+m의 끝에서(여기서 m은 1보다 큰 정수), 몇몇의 실시예는

및

을 정의한다.These examples then

Define. At the end of pass N ₁ + m (where m is an integer greater than 1), some embodiments

And

Define.

의 몇몇의 디폴트 값이 주어지면, 소정의 AMQP에 대응하는

은 몇몇의 실시예에서 다음의 의사 코드를 가지는 Search 함수를 이용하여 구할 수 있다.Given AMQP and

Given some default values of, corresponding to a given AMQP

In some embodiments, may be obtained by using a Search function having the following pseudo code.

In the pseudo code, the numbers 10, 12 and 0.05 can be replaced with appropriately selected thresholds.

및 프레임 k 내의 개별적인 매크로블록에 대한 특정 양자화 파라미터

을 계산한다(132). 소정의 명목상의 양자화 파라미터

및 기준 마스킹 강도

에 대한 파라미터

및

의 계산은 섹션 Ⅲ에 기술되었다(여기서

및

은 상기 섹션 Ⅲ에 기술에 기술되었던 함수

및

를 이용함으로써 계산된다). 제1 패스 내지 참조번호(132) 중에, 기준 마스킹 강도는 바로 이전의 참조번호(130)에서 계산되었던 것이다. 또한, 제2 검색 단계 중에, 명목상의 QP는 제2 검색 단계 전반에 걸쳐 상수를 유지하고 있다. 몇몇의 실시예에서, 제2 검색 단계 전반에 걸친 명목상의 QP는 제1 검색 단계 중에 최상의 인코딩 솔루션(즉, 가장 낮은 비트레이트 오차를 가지는 인코딩 솔루션)을 생성하는 명목상의 QP이다.After calculating the reference masking intensity for the next pass (pass p + 1) through the encoding of the frame sequence, process 100 moves to reference 132 to determine the next pass (i.e. p: = p + 1). To start. For each frame k and each macroblock m during each encoding pass p, the process performs specific quantization parameters for each frame k.

And specific quantization parameters for individual macroblocks in frame k

Compute (132). Any nominal quantization parameter

And reference masking strength

Parameters for

And

The calculation of is described in section III (where

And

Is the function that was described in the description in section III above.

And

Is calculated by using). During the first pass through reference 132, the reference masking intensity was that calculated at the previous reference 130. In addition, during the second search phase, the nominal QP maintains a constant throughout the second search phase. In some embodiments, the nominal QP throughout the second search step is the nominal QP that produces the best encoding solution (ie, the encoding solution with the lowest bitrate error) during the first search step.

After reference numeral 132, the process encodes the frame sequence using the quantization parameter calculated at reference numeral 130 (135). After reference numeral 135, the process determines 140 whether to terminate the second search step. Another embodiment uses different criteria for ending the second search step at the end of pass p.

Examples of such criteria are as follows.

<ε, 여기서 ε는 최종 비트레이트의 오차 허용치이다.ㆍ

<ε, where ε is the tolerance of the final bitrate.

• The number of passes exceeds the maximum number of passes allowed.

Some embodiments may use all of these escape conditions, while others may use only a few of them. However, other embodiments may use other escape conditions to end the first search step.

If process 100 determines that it should not end the second search step (140), it returns to reference number 130 to recalculate the reference mask intensity for the next encoding pass. From reference numeral 130, the process moves to reference numeral 132 to calculate the quantization parameter and then to reference numeral 135 to encode the video sequence by using the newly calculated quantization parameter.

On the other hand, if the process determines that it has finished the second search step (140), it moves to reference numeral 145. At reference numeral 145, process 100 stores the bitstream from last pass p as the final result and then terminates.

V. Decoder Input Buffer Underflow Control

Some embodiments of the present invention provide a multi-pass encoding process that examines various encodings of a video sequence to find a target bitrate, in order to identify an optimal encoding solution for the use of the input buffer used by the decoder. In some embodiments, this multi-pass process follows the multi-pass process 100 of FIG. 1.

Due to several factors, such as the size of the encoded image, the rate at which the decoder receives the encoded data, the size of the decoder buffer, and the variation in the speed of the decoding process, the use of the decoder input buffer ("decoder buffer") may result in an encoded image. It will vary somewhat during the decoding of the sequence (e.g., frames).

Decoder buffer underflow means a situation in which the decoder prepares to decode the next image before the image reaches the decoder side completely. The multi-pass encoder of some embodiments simulates the decoder buffer and re-encodes selected segments in the sequence to prevent decoder buffer underflow.

2 illustrates a codec system 200 of some embodiments of the present invention. The system includes a decoder 205 and an encoder 210. In this figure, encoder 210 has several components that allow this encoder to simulate the operation of the same components of decoder 205.

Specifically, the decoder 205 includes an input buffer 215, a decoding process 220, and an output buffer 225. Encoder 210 simulates these modules by holding simulated decoder input buffer 230, simulated decoding process 235, and simulated decoder output buffer 240. In order not to obscure the description of the present invention, FIG. 2 is simplified to show the decoding process 220 and the encoding process 245 as one block. In addition, in some embodiments, the simulated decoding process 235 and the simulated decoder output buffer 240 are not used for buffer underflow management and are shown in this figure for illustrative purposes only.

The decoder maintains an input buffer 215 to remove variations in rate and arrival time of the incoming encoded image. If the decoder has run out of data (underflow) or fills the input buffer (overflow), it will appear that the decoding is broken because the picture decoding is stopped or the incoming data is discarded. Such cases are undesirable.

To remove the underflow condition, the encoder 210 in some embodiments first encodes a sequence of images and stores this encoded sequence in storage 255. For example, encoder 210 obtains a first encoding of a sequence of images using multi-pass encoding process 100. It then simulates the decoder input buffer 215 and re-encodes the images that would have caused the buffer underflow. After all buffer underflow conditions are removed, the re-encoded image is via a connection 255, which may be a network connection (Internet, cable, PSTN line, etc.), a non-network direct connection, media (DVD, etc.), or the like. Provided to the decoder 205.

3 illustrates an encoding process 300 of an encoder of some embodiments. This process attempts to find an optimal encoding solution that does not cause the decoder buffer to underflow. As shown in FIG. 3, process 300 may determine the sequence of images that satisfy a desired target bitrate (eg, the average bitrate for each image in the sequence satisfies the desired average target bitrate). Identifies the first encoding (302). For example, process 300 may obtain 302 a first encoding of a sequence of images using multi-pass encoding process 100.

After reference numeral 302, encoding process 300 may include various factors such as connection speed (i.e., speed at which the decoder receives encoded data), size of decoder input buffer, size of encoded image, speed of decoding process, and the like. By simulating the decoder input buffer 215 (305). At reference numeral 310, process 300 determines whether any segment of the encoded image will cause the decoder input buffer to underflow. The technique that the encoder uses to determine (and then remove) the underflow condition is further described below.

If process 300 determines (310) that the encoded image does not cause an underflow condition, the process ends. On the other hand, if process 300 determines that there is a buffer underflow condition in any segment of the encoded image (310), the encoding parameters are refined (315) based on the values of these parameters from the previous encoding pass. The process then re-encodes the segment with underflow to reduce the segment bit size (320). After re-encoding the segment, process 300 examines the segment to determine if the underflow condition has been removed (325).

If the process determines that the segment still causes underflow (325), the process 300 further refines the encoding parameters to go to reference 315 to eliminate underflow. Alternatively, if the process determines that the segment will not cause any underflow (325), the process will review and reconsider the video sequence as the frame after the end of the re-encoded segment at the last iteration at 320. Specify a starting point for encoding (330). Then, at reference numeral 335, the process continues the portion of the video sequence specified at reference numeral 330 up to the first IDR frame following the underflow segment specified at reference numerals 315 and 320 (this frame is not included). Re-encode. After reference number 335, the process moves back to reference number 305 to simulate the decoder buffer to determine if the remainder of the video sequence still causes buffer underflow after re-encoding. The flow of process 300 from reference numeral 305 has been described above.

A. Determining Underflow Segments in a Sequence of Encoded Images

As mentioned above, the encoder simulates the decoder buffer state to determine if any segment in the segment of the encoded or re-encoded image causes an underflow in the decoder buffer. In some embodiments, the encoder may be configured such as the size of the encoded image, network conditions such as bandwidth, decoder elements (e.g., input buffer size, initial and nominal time to remove the image, decoding process time, display of each image). A simulation model that takes into account time, etc.).

In some embodiments, an MPEG-4 AVC Coded Picture Buffer (CPB) model is used to simulate the decoder input buffer state. CPB is a term used in the MPEG-4 H. 264 standard to refer to the simulated input buffer of the HRD (Hypothetical Reference Decoder). HRD is a hypothetical decoder model that describes constraints on variability in the stream that an encoding process can produce. CPB models are well known and described in section 1 below for convenience. A more detailed description of CPB and HRD is described in "Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec. H. 264 / ISO / IEC 14496-10 AVC)".

1. Using the CPB Model to Simulate a Decoder Buffer

The following paragraphs describe how in some embodiments the decoder input buffer is simulated using a CPB model. The time at which the first bit of image n begins to enter the CPB is called the initial arrival time point t _ai (n), which is derived as follows.

If the image is a first image (ie image 0), t _ai (0) = 0

If the image is not the first image in the sequence being encoded or re-encoded (ie n> 0), t _ai (n) = MAX (t _af (n-1), t _ai , earliest (n)) .

Where

T _ai, earliest (n) = t _r , n (n) -initial_cpb_removal_delay,

Where t _r , n (n) is the nominal removal point of image n from the CPB as described below and initial_cpb_removal_delay is the initial buffering period.

The final arrival time for the image is

t _af (n) = t _ai (n) + b (n) / BitRate,

Induced by

Where b (n) is the bit size of image n.

In some embodiments, the encoder calculates the nominal removal time calculation directly, as described below, instead of reading from an optional portion of the bit stream as in the H.264 specification. For image 0, the nominal removal point of the image from the CPB is

Is described.

For image n (n> 0), the nominal removal point of the image from the CPB is

t _r , n (n) = t _r , n (0) + sum _{i = 0 to n-1} (t _i )

It is described as

Where t _r , n (n) is the nominal time of removal of image n, and t _i is the display period for image i.

The removal point of image n is described as follows.

If t _r , n (n)> = t _af (n), then t _r (n) = t _r , n (n),

If t _r , n (n) <t _af (n), then t _r (n) = t _af , n (n)

The latter case indicates that the size of image n, b (n), is too large to remove at nominal removal point.

2. Detection of Underflow Segments

As described in the previous section, the encoder can simulate the decoder input buffer state and obtain the number of bits in the buffer at a given instant. Alternatively, the encoder determines how each individual image enters the decoder input through the difference between its nominal elimination point and its final arrival point (ie t _b (n) = t _{r, n} (n) -t _af (n)). You can track if the state of the buffer changes. If t _b (n) is less than 0, the buffer underflows instantaneously between t _{r, n} (n) and t _af (n), and possibly before t _{r, n} (n) and after t _af (n) Are going through.

Images directly related to underflow can be easily found by checking if t _b (n) is less than zero. However, an image with t _b (n) less than zero does not necessarily cause underflow, and conversely, an image causing underflow may not have t _b (n) less than zero. Some embodiments define underflow segments (in decoding order) as a contiguous image range that causes underflow by continuously emptying the decoder input buffer until the underflow reaches its worst point.

4 is a diagram of the difference t _b (n) versus image number between the last arrival time of the image and the nominal removal time in some embodiments. This figure is shown for a sequence of 1500 encoded images. 4A shows the underflow segment with arrows indicating its start and end. Note that in FIG. 4A, although not explicitly indicated by arrows for brevity, there are other underflow segments that occur after the first underflow segment.

5 shows a process 500 that an encoder uses to perform an underflow detection operation at 305. Process 500 first determines 505 the last arrival time point t _af , and the nominal removal time point t _r , n of each image by simulating the decoder input buffer state as described above. Note that since this process can be called many times during the iterative process of buffer underflow management, it receives the image number as a starting point and checks the sequence of images from this predetermined starting image. Clearly, in the first iteration, the starting point is the first image of the sequence.

At 510, process 500 compares the last arrival time of each image in the decoder input buffer with the nominal removal time of that image by the decoder. If the process determines that there is no image with the latest arrival time after the nominal removal time point (ie no underflow state exists), the process ends. On the other hand, if an image is found whose last arrival time is after the nominal removal point, the process determines that there is an underflow and moves to reference 515 to identify the underflow segment.

At reference numeral 515, process 500 continues to decode the decoder buffer to the next global minimum value at which the underflow condition begins to increase (i.e., t _b (n) is not negative across the image range). Identifies the underflow segment as the segment of the image to begin emptying with. Process 500 then ends. In some embodiments, the start of the underflow segment is further adjusted to begin with an I-frame, which is an image in the encoding that indicates the start of the set of images between the associated encodings. Once one or more segments that are causing the underflow are identified, the encoder proceeds to eliminate the underflow. The following section B describes underflow elimination in the case of a single-segment (ie, when the entire sequence of encoded images contains only a single underflow segment). Section C then describes underflow elimination for the multiple-segment underflow case.

B. Remove single-segment underflow

Referring to a of FIG. 4, if the t _b (n) versus n curve crosses the n-axis only once with decreasing slope, there is only one underflow segment in the entire sequence. The underflow segment starts at the nearest local maximum preceding the point where it meets zero and ends at the next global minimum between the point where it meets zero and the end of the sequence. The end point of the segment can be followed by another zero that takes a curve that increases in slope if the buffer overcomes underflow.

6 illustrates a process 600 that the encoder uses in some embodiments (at 315, 320, and 325) to remove an underflow condition in a single image segment. At reference numeral 605, process 600 reduces the underflow segment by calculating the product of the longest delay found at the end of the segment (e.g., the minimum value t _b (n)) and the input bitrate into the buffer. Estimate the total number of bits (#B) you want to make.

Then, at reference numeral 610, process 600 uses the total number of bits of the current segment from the last encoding pass (or passes) and the predetermined number of bits for the segment using the average masked frame QP (AMQP), Estimate a given AMQP to achieve B _T = B− ㅿ B _P (where p is the current number of iterations of process 600 for the segment). If this iteration is the first iteration of the process 600 for a particular segment, then the AMQP and total number of bits are the AMQP and total number of bits for this segment derived from the initial encoding solution identified at 302. On the other hand, if this iteration is not the first iteration of process 600, these parameters may be derived from the encoding solution or solutions obtained in the last pass or the last few passes of process 600.

Then, at reference numeral 615, the process 600 is based on the masking intensity Φ _F (n) such that an image that is more permissible to mask using a predetermined AMQP can obtain more bit reductions. Modify the average masked frame QP, MQP (n). The process then re-encodes the video segment 610 based on the parameter defined at 315. The process then examines the segment to determine if the underflow condition has been removed (625). 4B illustrates the removal of the underflow state of FIG. 4A after process 600 has been applied to the underflow segment to re-encode this segment. When the underflow condition is removed, the process terminates. Otherwise, go to reference 605 and further adjust the encoding parameter to reduce the total bit size.

C. Eliminate Underflow in Multiple Underflow Segments

If there are a plurality of underflow segments in the sequence, the re-encoding of the segments changes the time t _b (n) the buffer is full for every subsequent frame. To account for the modified buffer state, the encoder searches for one underflow segment at a time, starting from the point where it first meets zero (ie, the lowest n) with decreasing slope.

The underflow segment starts at the nearest local maximal point preceding this zero, and then the next global minimum between the point where zero meets and the next zero (or the end of the sequence if no more zero). Terminate at. After finding one segment, the encoder virtually eliminates underflow in this segment and updates the buffer saturation by setting t _b (n) to 0 at the end of that segment and recovering the buffer simulation for all subsequent frames. estimate the fullness.

The encoder then continues to search for the next segment using the modified buffer saturation. As mentioned above, once all underflow segments have been identified, the encoder derives the AMQP and modifies the masked frame QP for each segment independently of the other segments, just as with the single-segment case.

Those skilled in the art will recognize that other embodiments may be implemented differently. For example, some embodiments will not identify multiple segments that cause an underflow of the decoder's input buffer. Instead, some embodiments will perform a buffer simulation as described above to identify the first segment causing the underflow. After identifying the segment in this way, these embodiments recover from modifying the segment to correct the underflow condition in that segment and then encoding the subsequent modified region. After encoding the rest of the sequence, these embodiments will repeat this process for the next underflow segment.

D. Application of Buffer Underflow Management

The decoder buffer underflow technique described above is applied to various encoding and decoding systems. Some examples of such systems are described below.

7 shows a network 705 that connects the video streaming server 710 with some client decoders 715-725. Clients are connected to the network 705 via links having different bandwidths, such as 300 Kb / sec and 3 Mb / sec. The video streaming server 710 controls the streaming of the encoded video image from the encoder 730 to the client decoders 715-725.

The streaming video server may decide to stream the encoded video image using the slowest bandwidth in the network (ie, 300 Kb / sec) and the smallest client buffer size. In this case, the streaming server 710 needs only one encoded image set optimized for the target bitrate of 300 Kb / sec. On the other hand, the server can generate and store different encodings optimized for different bandwidths and different client buffer states.

8 shows another example of an application for decoder underflow management. In this example, HD-DVD player 805 is receiving an encoded video image from HD-DVD 840 that stores encoded video data from video encoder 810. The HD-DVD player 805 has an input buffer 815, a set of decoding modules 820 shown as one block 820 for brevity, and an output buffer 825.

The output of player 805 is sent to a display device, such as TV 830 or computer display terminal 835. HD-DVD players can have very high bandwidth, for example 29.4 Mb / sec. In order to maintain a high quality image on the display device, the encoder ensures that no segment in the image sequence is encoded in such a way that it is not large enough to be delivered to the decoder input buffer in time.

Ⅵ. Computer systems

9 illustrates a computer system on which embodiments of the present invention may be implemented. Computer system 900 includes bus 905, processor 910, system memory 915, read-only memory 920, permanent storage 925, input device 930, and output device 935. . Bus 905 collectively represents all system buses, peripheral buses, and chipset buses that communicatively connect various internal devices of computer system 900. For example, bus 905 communicatively couples processor 910 to read-only memory 920, system memory 915, and persistent storage 925.

From these various memory units, the processor 910 retrieves instructions to execute and data to be processed in order to execute the process of the present invention. Read-only memory (ROM) 920 stores instructions and static data required by the processor 910 and other modules of the computer system.

On the other hand, persistent storage 925 is a read-and-write memory device. This device is a non-volatile memory device that stores instructions and data even when the computer system 900 is powered off. Some embodiments of the present invention utilize mass-storage devices (such as magnetic disks or optical disks and their corresponding disk drives) as persistent storage 925.

Another embodiment uses removable storage devices (such as floppy disks or zip® disks, and their corresponding disk drives) as permanent storage. As with persistent storage 925, system memory 915 is a read-and-write memory device. However, unlike storage 925, system memory is volatile read-and-write memory, such as RAM. System memory stores some of the instructions and data that a process needs at run time. In some embodiments, the processes of the present invention are stored in system memory 915, persistent storage 925, and / or ROM 920.

Bus 905 is also connected to input device 930 and output device 935. The input device allows the user to communicate information and select commands to the computer system. The input device 930 includes an alphanumeric keyboard and a cursor-controller. Output device 935 displays an image generated by the computer system. Output devices include printers and display devices such as CRTs or LCDs.

Finally, as shown in FIG. 9, bus 905 also connects computer 900 to network 965 via a network adapter (not shown). In this way, the computer may be part of a network of computers (such as a LAN, WAN, or the Internet) or a network of networks (such as the Internet). Any or all components of computer system 900 may be used in connection with the present invention. However, one of ordinary skill in the art will recognize that any other system configuration may also be used in connection with the present invention.

While the present invention has been described with reference to various specific details, those skilled in the art will recognize that the present invention may be practiced in other specific forms without departing from the spirit of the invention. For example, instead of using the H264 method to simulate the decoder input buffer, other simulation methods may be used that take into account the buffer size, the time of arrival and removal of the image in the buffer, and the decoding and display time of the image.

Some embodiments described above calculate the average deleted SAD to obtain an indication of image change in the macroblock. However, other embodiments may identify image changes differently. For example, some embodiments may predict the image values expected for the pixels of the macroblock. These embodiments then generate a macroblock SAD by subtracting this predicted value from the luminance value of the pixel of the macroblock and summing the absolute value of this subtraction value. In some embodiments, the predicted value is based on the pixel value of one or more neighboring macroblocks as well as the value of the pixel of the macroblock.

In addition, the embodiment described above directly uses the derived spatial and pre-time masking values. Another embodiment will apply smoothing filtering to successive spatial masking values and / or to successive temporal masking values before using them to find general tendencies of these values throughout the video image. Therefore, those of ordinary skill in the art will understand that the present invention is not limited by the above-described exemplary details.

1 illustrates a process conceptually illustrating the encoding method of some embodiments of the present invention.

2 conceptually illustrates the codec system of some embodiments.

3 is a flow chart illustrating an encoding process of some embodiments.

FIG. 4A illustrates an image number that indicates the difference between the nominal removal time and the last arrival time of the image versus the underflow state in some embodiments.

4b shows the image number for the same image as shown in a of FIG. 4 after the difference between the nominal removal time and the final arrival time of the image versus the underflow condition has been removed.

FIG. 5 illustrates a process that an encoder uses to perform underflow detection in some embodiments. FIG.

FIG. 6 illustrates a process the encoder uses in some embodiments to remove underflow conditions in a single segment of an image. FIG.

7 illustrates application of buffer underflow management in a video streaming application.

8 illustrates the application of buffer underflow management in an HD-DVD system.

9 illustrates a computer system in which one embodiment of the present invention is implemented.

Claims (13)

A method of encoding a plurality of images, a) defining a nominal quantization parameter for encoding the image; b) deriving at least one image-specific quantization parameter for at least one image based on the nominal quantization parameter; c) encoding the image based on the image-specific quantization parameter; And d) iteratively repeating the definition, derivation, and encoding operations to optimize the encoding Image encoding method comprising a. The method of claim 1, a) deriving a plurality of image-specific quantization parameters for a plurality of images based on the nominal quantization parameters; b) encoding the image based on the image-specific quantization parameter; And c) repeating the definition, derivation and encoding operations to optimize the encoding Image encoding method further comprising. The method of claim 1, Stopping the iteration when an encoding operation meets a set of termination criteria Image encoding method further comprising. The method of claim 3, And the set of termination criteria comprises an identification of the image encoding that is acceptable. The method of claim 4, wherein And the allowable image encoding is encoding of the image within a specific range of a target bitrate. A method of encoding a plurality of images, a) identifying a plurality of image attributes, each particular image attribute quantifying the complexity of at least a particular portion of a particular image; b) identifying reference attributes that quantify the complexity of the plurality of images; c) identifying a quantization parameter for encoding the plurality of images based on the identified image attribute, the reference attribute and the nominal quantization parameter; d) encoding the plurality of images based on the identified quantization parameters; And e) optimizing the encoding by repeatedly performing the identification and encoding operations, where a plurality of different iterations use a plurality of different reference attributes Image encoding method comprising a. The method of claim 6, The plurality of attributes are visual masking intensities of at least a portion of each image, the visual masking intensities being encoding artifacts that are not recognizable to the viewer of the video sequence after the video sequence has been encoded according to the method and then decoded. ) To estimate the amount. The method of claim 6, A plurality of said attributes are visual masking intensities of at least a portion of each image, and visual masking intensities for a portion of an image in quantifying the complexity of a portion of said image and in quantifying the complexity of a portion of an image, wherein said visual masking intensity Provides an indicator of the amount of compressed artifacts that may result from the encoding without visual distortion of the encoded image after the image is decoded. A computer readable medium storing a computer program for encoding a plurality of images, comprising: The computer program a) instructions for defining a nominal quantization parameter for encoding the image; b) instructions for deriving at least one image-specific quantization parameter for at least one image based on the nominal quantization parameter; c) instructions for encoding the image based on the image-specific quantization parameter; And d) instructions for optimizing the encoding by iteratively repeating the definition, derivation, and encoding operations. A computer readable medium comprising an assembly. The method of claim 9, The computer program a) instructions for deriving a plurality of image-specific quantization parameters for a plurality of images based on the nominal quantization parameters; b) instructions for encoding the image based on the image-specific quantization parameter; And c) instructions for optimizing the encoding by repeating the definition, derivation, and encoding operations And further comprising an assembly. The method of claim 9, And a set of instructions for stopping the repetition when an encoding operation satisfies a set of termination criteria. The method of claim 11, And the set of termination criteria includes an identification of the image encoding that is acceptable. The method of claim 12, And the acceptable image encoding is the encoding of the image within a specific range of a target bitrate.

Images