JP4709189B2 - Scalable video encoding method, scalable video encoding device, program thereof, and computer-readable recording medium recording the program - Google Patents

Description

The present invention relates to a scalable video encoding technique, particularly to a scalable video encoding technique to achieve effective prefilter you can reduce the code amount without impairing the subjective image quality.

  Conventionally, studies have been made to improve coding efficiency by using human visual characteristics in moving picture coding. In this research, an attempt is made to reduce the amount of information that is difficult to degrade, taking into account the visual sensitivity to the spatio-temporal frequency of the human visual system.

  As a typical example, there is a pre-filter process that is placed before the encoding process. In the image, there are an area where visual deterioration is difficult to understand and an area where it is easy to understand. Therefore, the amount of code can be reduced without lowering the subjective image quality by deleting information on spatial high-frequency components in a region where deterioration is difficult to visually recognize using a low-pass filter in advance. The key here is how to calculate the visibility of visual degradation in each region and control the bandwidth limit of the low-pass filter to be applied.

  In the moving picture coding method described in Non-Patent Document 1, a low-pass filter given as an initial value is applied, and a PSNR (Peak Signal to Noise Ratio) compared with the case of non-application is calculated for each macroblock. The strength of the low-pass filter is controlled by referring to a table that stores the correspondence relationship between the PSNR and the band limit width of the filter created based on empirical rules.

  In the high-efficiency video coding method described in Patent Document 1, the amount of motion is calculated from the norm of the motion vector of the input video signal, and a two-dimensional spatial filter is applied according to the value of the amount of motion. Information is reduced while minimizing unnecessary degradation.

  On the other hand, H. Standardization of JSVC (Joint Scalable Video Coding), which is a scalable extension method of H.264 / AVC, is being tackled by JVT, a joint organization of ISO and ITU-T. In JSVC, the following layer structure is adopted in order to support spatial scalability. First, an image with a resolution to be supported is created in advance by reduction processing from the original image. Next, one resolution is regarded as one layer, and a non-scalable H.264 format is used for each layer. H.264 / AVC is applied. Finally, the obtained encoded streams of each layer are integrated. Here, the layer that processes the image with the lowest resolution is called the basic layer, and the layer above it is called the extension layer.

  In JSVM of JSVC encoding reference software shown in Non-Patent Document 2, encoding is performed in order from the lower layer. At this time, the enhancement layer performs inter-layer prediction in order to remove redundancy between layers. More specifically, motion information prediction for predicting information related to motion prediction such as motion vectors and residual signals from the immediately lower layer, and texture prediction using an upsampled internal decoded signal in the immediately lower layer as prediction reference are performed.

  In the JSVC pre-filter processing proposed in Non-Patent Document 3, the conspicuousness of visual degradation is calculated as a visual distortion amount, and the texture prediction generated by the immediately lower layer is calculated according to the visual distortion amount value. Is applied to the original signal to eliminate redundancy between layers.

Non-Patent Document 4 describes an example of a spatial contrast sensitivity function indicating the relationship between the spatio-temporal frequency component and the contrast sensitivity that can be used in the embodiment of the present invention. Non-Patent Document 5 describes a technique of a non-scalable encoding method for moving images, which is well known as JM.
Sankeizaki, K., Ono, N., Uekura, Y., Yashima, Y .: "Examination of adaptive pre-filter considering image quality variation within and between frames," IEICE Tech. Reports, CS2006-5, IE2006-118, Dec. 2006. J. Reichel, H. Schwarz and M. Wien: “Joint Scalable Video Model JSVM-8.0,” ISO / IEC JTC1 / SC29 / WG11 and ITU-T SG16 Q.6, JVT-U202, 2006. Kazuya Hayase, Yukihiro Bando, Noriyuki Takamura, Hitoshi Uekura, Yoshiyuki Yashima: “JSVC Spatial Extension Layer Prefilter Using Inter-layer Prediction,” IEICE, Mar. 2007. NBNill: “A visual model weighted cosine transform for image compression and quality assessment,” IEEE Trans. Commun., Vol. COM-33, No. 12, pp.551-557, June 1985. KPLim, G. Sullivan and T. Wiegand: “Text Description of Joint model Reference Encoding Methods and Decoding Concealment Methods,” Joint Video Team (JVT) of ISO / IEC MPEG and ITU-T VCEG, JVT-R95, Jan. 2006. JP-A-6-62392

  The value of the simple square error of the image signal often deviates from the subjective image quality. When the frequency of the difference signal in the block is converted, the simple square error value of the signal in the block is regarded as equivalent to the contribution to visual degradation of each spatial frequency. However, it is generally known that human contrast sensitivity with respect to spatial high-frequency components in an image is small. In other words, the spatial high-frequency component of the differential signal has a small contribution to visual degradation compared to the low frequency. If it is assumed that the PSNR value of an image having a high frequency component is the same as that of two images having no high frequency component, the image having a high frequency component is less noticeable in visual deterioration.

  Non-Patent Document 1 described above calculates the conspicuousness of deterioration based on the average of the square error values of the signals in the block, and derives the band pass width. However, an appropriate bandpass width cannot be applied to a region where the square error value deviates from the subjective image quality. Further, in Patent Document 1, since the ease of conspicuous deterioration in space is not used for controlling the bandwidth limit width, there remains room for reduction of spatial high-frequency components.

  In addition, the pre-filter for scalable coding proposed in Non-Patent Document 3 has only two degrees of freedom whether to replace the original signal with the texture prediction signal from the lower layer. If it is determined not to be replaced, the original signal is directly input to the subsequent encoding process, but there is a high possibility that a spatial high-frequency component that can be removed remains in the original signal. Non-patent document 1 can be applied to remove the remaining spatial high-frequency components, but similarly, the aforementioned problem of the difference between PSNR and subjective image quality is encountered.

  The present invention has been made in view of such circumstances, and it is possible to increase the internal decoded signal in the immediately lower layer according to the amount of visual distortion indicating the conspicuous visual degradation between the original image and the filtered image. When the filter processing to replace with the sample signal and the normal low-pass filter processing are adaptively performed, and the low-pass filter is used, the band-limiting width of the low-pass filter according to the value of the visual distortion amount The purpose is to establish a design method of a moving image encoder that performs filtering by determining the above.

  The prefiltering process of the present invention in scalable coding with a layer structure is composed of the following two steps. The following processing is applied to the original signal of the enhancement layer in units of small areas of any size.

[Step 1: Replacement determination using the immediately below layer texture prediction signal]
Only a region where visual deterioration in the original signal of the enhancement layer is not conspicuous is replaced with a texture prediction signal from the immediately lower layer (an image signal obtained by up-sampling the decoded image signal of the immediately lower layer). The procedure is the same as the prefilter procedure in Non-Patent Document 3.

A visual distortion amount D ₀ between the original signal of the enhancement layer and the texture prediction signal of the immediately lower layer is calculated. The amount of visual distortion is a measure of the degree of conspicuous deterioration calculated using the contrast sensitivity of the visual system, and the smaller the value, the less noticeable it is. The calculation procedure of the visual distortion amount will be described later.

If the calculated visual distortion amount D ₀ is equal to or less than the threshold value D _th , the original signal is replaced with a texture prediction signal that is an interpolation signal of the immediately lower layer. If D ₀ > D _th , the process proceeds to Step 2. The threshold value D _th is an allowable value of the visual distortion amount in which deterioration is not perceived or is not worrisome, and is given from the outside. As an example of how to give the threshold value _Dth , a method of empirically determining from the contrast and brightness of the display can be considered.

[Step 2: Replacement determination by low-pass filter signal]
In Step 1, when the visual distortion amount D ₀ between the enhancement layer original signal and the texture prediction signal of the immediately lower layer exceeds the threshold D _th , a low-pass filter is applied to the original signal of the block.

Previously, different low-pass filter f _n of the band pass width and N patterns prepared. Here, it is assumed that n = 1,... Although the method of giving the N pattern is arbitrary, it is desirable to set the fluctuation amount of energy that is reduced by the band limitation to be equal.

Next, filtering is performed on the original signal of the block in order from the filter with the strong band limitation, and the visual distortion amount D _n is calculated each time. When D _n ≦ D _th , the original signal is replaced with the filtering signal at that time. When D _N > D _th , the original signal is output as it is. The threshold value D _th used here is the same value as the threshold value used in Step 1.

Incidentally, as a reference , in the case of non-scalable coding that does not have a layer structure, if only the replacement determination by the low-pass filter signal described in Step 2 is performed without performing Step 1, it corresponds to the value of the visual distortion amount. Pre-filter processing. In the case of scalable coding having the layer structure of the present invention, both Step 1 and Step 2 are executed for the enhancement layer original signal, and only Step 2 is not performed for the base layer original signal. Just do it.

  In addition, it is determined whether to apply a filter according to prefilter application frame designation information given from the outside for each frame image (including field images) or a predetermined filter application determination condition, and according to the determination result. The application of the filter may be controlled. As an example of the determination condition for applying a filter, there may be a condition of non-application when the frame is a referenced frame for motion prediction, and application when the frame is not a referenced frame. For example, when the frame is a reference frame for motion prediction, as pre-filter application frame designation information, designation information indicating that the filter is not applied is externally given to the filter processing unit, and the frame is referred to for motion prediction. When it is not a frame, designation information for applying a filter is given from the outside as pre-filter application frame designation information. Only in the latter case, pre-filter processing using the present invention is executed.

  Also, the filter is applied in units of image partial areas including several small areas such as slices, rather than externally specifying information on whether to apply the filter in units of frames or setting application judgment conditions. It may be possible to specify whether or not to perform determination and to set determination conditions.

[Visual distortion calculation method]
As described above, it is known that human contrast sensitivity with respect to spatial high-frequency components in an image is small. Non-Patent Document 4 shows the relationship between the time-space frequency component and the contrast sensitivity as a spatial contrast sensitivity function. This function gives the contrast sensitivity for each frequency component, and the relative value for the maximum contrast sensitivity is derived. In the present invention, this contrast sensitivity is assumed to be a contribution rate to visual degradation of each spatial frequency. The difference signal between the original signal and the comparison signal is developed in the frequency domain, the contrast sensitivity is weighted to the energy of each spatial frequency, and the sum of the weighted energy is regarded as the visual distortion amount. The calculation procedure will be described in detail below.

Visual distortion amount D _n between the visual distortion amount D _0, and the original signal and the low-pass filter signal by low-pass filter f _n of the enhancement layer between the original signal and the texture prediction signal enhancement layers , As follows.

Here, F ₀ (u, v) is a Fourier coefficient of the spatial frequency (u, v) for the difference signal between the enhancement layer original signal and the texture prediction signal, and F _n (u, v) is the enhancement layer original. It is a Fourier coefficient of the spatial frequency (u, v) for the difference signal between the signal and the low-pass filter signal by the low-pass filter f _n . w (u, v) is a weighting coefficient for the Fourier coefficient, and indicates the contrast sensitivity with respect to the spatial frequency. This value is calculated as follows based on the spatial contrast sensitivity function. An example is described in Non-Patent Document 4, but the usable contrast sensitivity function is not limited to this.

w (η) = (0.2 + 0.45η) exp (−0.18η) (3)
Here, η and (u, v) have the following relationship.

  θ is a viewing angle of an M × M block when an image having a vertical width h is observed at a viewing distance ah, and is given as follows.

  The viewing distance ah is given from the outside. As an example of how to provide the viewing distance ah, a method of determining according to a precondition for subjective image quality evaluation, a method of determining according to an assumed viewing environment, and the like can be considered.

The present invention, without degrading the subjective image quality, the optimum band-limited width of the low-pass filter to reduce the high spatial frequency components as possible, can be determined for each local region in an image.

Also, a redundancy reduction of information in the reduction and space information redundancy between Layer, can be achieved simultaneously.

[Process flow]
An embodiment of the scalable encoding process of the present invention will be described with reference to FIG.

  Step S11: A process of determining whether or not the current encoding target layer is a base layer is performed. If true, the process of step S12 is performed. If false, the process of step S13 is performed.

  Step S12: Read the original signal in the base layer, perform prefilter processing for the base layer, and output the generated prefilter signal. This process will be described in detail later with reference to FIG.

  Step S13: Read the original signal in the enhancement layer, perform prefilter processing for the enhancement layer, and output the generated prefilter signal. This process will be described in detail later with reference to FIG.

  Step S14: The prefilter signal generated by the process of step S12 or the process of step S13 is read, the encoding process in the encoding target layer is performed, and the encoding information of the encoding target layer is output. An example of this encoding process is JSVM shown in Non-Patent Document 2.

  Step S15: A process for determining whether or not all the encoding target layers have been encoded is performed. If true, the process of step S17 is performed. If false, the process of step S16 is performed.

  Step S16: Move to the next encoding target layer, and perform the process of step S11 on the new encoding target layer.

  Step S17: The encoded information of each layer is multiplexed and output as an encoded stream.

An example of non-scalable encoding processing will be described with reference to FIG. 2 as a related technique of the present invention.

  Step S21: Read the original signal, perform prefilter processing, and output the generated prefilter signal. This process will be described in detail later with reference to FIG.

  Step S22: The prefilter signal generated in step S21 is read, encoding processing is performed, and encoding information is output. Examples of this encoding process include JSVM shown in Non-Patent Document 2 and JM shown in Non-Patent Document 5.

  Details of the processing in step S12 shown in FIG. 1 (prefilter processing for the base layer in the scalable encoding processing) and the processing in step S21 shown in FIG. 2 (prefiltering processing in the non-scalable encoding processing) are shown in FIG. It explains using.

  Step S31: Read prefilter application frame designation information given from the outside, and determine whether or not the frame is a prefilter application frame. If true, the process proceeds to step S32; The process proceeds to S33. The pre-filter application frame designation information is information that designates whether or not to apply the above filter.

  Step S32: Read the original signal of the frame, perform a replacement process for each small area using a low-pass filter, and output a prefilter signal of the frame. This process will be described in detail later with reference to FIG.

  Step S33: A determination process is performed to determine whether or not pre-filter processing has been performed on all frames. If true, a pre-filter signal is output. If false, the process proceeds to step S34. In the prefiltering process for the base layer, the output prefilter signal is a base layer prefilter signal.

  Step S34: Move to the next frame and perform the process of step S31 on the new frame.

  Details of the processing in step S13 shown in FIG. 1 (prefiltering processing for the enhancement layer in scalable coding processing) will be described with reference to FIG.

  Step S41: Read prefilter application frame designation information given from the outside, and determine whether or not the frame is a prefilter application frame. If true, the process proceeds to step S42; The process proceeds to S43.

  Step S42: Read the original signal of the frame, perform an adaptive replacement process for each small region with the filter signal from the immediately lower layer texture prediction signal or the low-pass filter, and output the pre-filter signal of the frame. This process will be described in detail with reference to FIG.

  Step S43: A determination process is performed to determine whether or not prefiltering has been performed on all frames in the enhancement layer. If true, the prefilter signal of the enhancement layer is output. If false, step S44. Move on to processing.

  Step S44: Move to the next frame and perform the process of step S41 on the new frame.

  Details of the processing in step S32 shown in FIG. 3 (replacement processing using a low-pass filter signal) will be described with reference to FIG.

Step S51: The visual distortion amount threshold value _Dth given from the outside is read and the value is written in the register.

  Step S52: The prefilter target frame is divided into arbitrary small areas, and the original signal of each small area is written in the buffer.

Step S53: The filter coefficient of the low-pass filter f _n applied to the small area is read, and the coefficient value is written in the register.

Step S54: The filter coefficient of the low-pass filter f _n is read from the register, the low-pass filter constituted by the filter coefficient is applied to the original signal of the small area, and the low band of the small area obtained is obtained. Write the pass filter signal to the buffer.

Step S55: Read the low-pass filter signal by the original signal of the small region and the low-pass filter f _n from the buffer, generate a difference signal thereof, apply Fourier transform to the difference signal, and perform the Fourier transform performs weighting using the contrast sensitivity function on each frequency component of, calculating the visual distortion amount D _n between the two signals by taking the sum of the frequency components thereof weighting, the value in the register Write.

Step S56: The visual distortion amount D _n calculated using the low-pass filter f _n in the small region and the visual distortion amount threshold value D _th are read from the register, and when D _n ≦ D _th Moves to the process of step S57, and if D _n > D _th , moves to the process of step S58.

Step S57: reads the low-pass filter signal by the low-pass filter f _n of the small area, replacing the original signal of the small region, write to the buffer as a pre-filter signal.

  Step S58: It is determined whether or not all the low-pass filters have already been applied to the small area. If true, the process proceeds to step S60. If false, the process proceeds to step S59. Move.

  Step S59: The process proceeds to the next low-pass filter, and the process of step S53 is performed using the new low-pass filter.

  Step S60: A determination process is performed as to whether or not the replacement determination process has been executed for all the small regions in the frame, and if true, a prefilter signal is output. If false, the process proceeds to step S61.

  Step S61: Move to the next small area and perform the process of step S53 on the new small area.

  Details of the process (adaptive replacement process) in step S42 shown in FIG. 4 will be described with reference to FIG.

Step S71: The visual distortion amount threshold value _Dth given from the outside is read and the value is written in the register.

  Step S72: The prefilter target frame is divided into arbitrary small areas, and the original signal of each small area is written in the buffer.

  Step S73: The layer texture prediction signal directly under the small area is read from the buffer.

Step S74: Read the original signal of the small region from the buffer, generate a difference signal between the original signal of the small region and the immediately below layer texture prediction signal, apply Fourier transform to the difference signal, Each frequency component is weighted using a contrast sensitivity function, and the visual distortion amount D ₀ between the two signals is calculated by taking the sum of the weighted frequency components, and the value is written to the register. .

Step S75: The visual distortion amount D ₀ and the visual distortion amount threshold value D _th calculated by using the immediately lower layer texture prediction signal in the small region are read from the register. If D ₀ ≦ D _th , step S75 is performed. The process proceeds to S83, and if D ₀ > D _th , the process proceeds to Step S76.

Step S76: The filter coefficient of the low-pass filter f _n applied to the small area is read, and the coefficient value is written in the register.

Step S77: The filter coefficient of the low-pass filter f _n is read from the register, the low-pass filter constituted by the filter coefficient is applied to the original signal of the small area, and the low band of the small area obtained is obtained. Write the pass filter signal to the buffer.

Step S78: The original low-frequency signal and the low-pass filter signal by the low-pass filter f _n are read from the buffer, a difference signal between them is generated, a Fourier transform is applied to the difference signal, and the Fourier transform performs weighting using the contrast sensitivity function on each frequency component of, calculating the visual distortion amount D _n between the two signals by taking the sum of the frequency components thereof weighting, the value in the register Write.

Step S79: The visual distortion amount D _n and the visual distortion amount threshold value D _th calculated using the low-pass filter f _n in the small region are read from the register, and when D _n ≦ D _th Moves to step S80, and if D _n > D _th , moves to step S81.

Step S80: The low-pass filter signal by the low-pass filter f _{n in the} small region is read, replaced with the original signal in the small region, and written in the buffer as a pre-filter signal.

  Step S81: It is determined whether or not all the low-pass filters have already been applied to the small area. If true, the process proceeds to step S84. If false, the process proceeds to step S82. Move.

  Step S82: The process proceeds to the next low-pass filter, and the process of step S76 is performed using a new low-pass filter.

  Step S83: The layer texture prediction signal immediately under the small area is read, replaced with the original signal of the small area, and written into the buffer as a prefilter signal.

  Step S84: It is determined whether or not replacement determination processing has been executed for all the small regions in the frame, and if true, an adaptively replaced prefilter signal is output, and if false, Moves to the process of step S85.

  Step S85: Move to the next small area and perform the process of step S73 on the new small area.

[Processing equipment]
An example of the configuration of the scalable coding apparatus according to this embodiment is shown in FIG. FIG. 7 is a diagram of an encoding apparatus in the case of having spatial scalability of n layers (where n ≧ 1).

  Original signal reduction processing unit 101: Reads an original signal which is an encoding target of scalable encoding, prepares a plurality of desired resolutions for providing spatial scalability by reduction processing, and generates an original signal storage unit (basic layer original signal storage) for each layer. Unit 102, first enhancement layer original signal storage unit 110,..., Nth enhancement layer original signal storage unit 118). Arbitrary reduction methods can be applied, an example of which is Lanczos 3-lobed sinc filter.

  Base layer prefilter processing unit 103: Reads the base layer original signal from the base layer original signal storage unit 102, performs prefilter processing, and outputs the prefilter signal to the base layer prefilter signal storage unit 104. This processing apparatus will be described in detail later with reference to FIG.

  Base layer encoding unit 105: reads a base layer prefilter signal from base layer prefilter signal storage unit 104, performs base layer encoding processing, and outputs encoded information to base layer encoded information storage unit 106 . Examples of this encoding process include JSVM shown in Non-Patent Document 2 and JM shown in Non-Patent Document 5.

  Base layer inner decoding unit 107: Reads base layer coding information from the base layer coding information storage unit 106, performs decoding processing within the base layer coding processing, and writes the generated decoded signal to a buffer.

  Base layer texture prediction signal generation unit 108: reads a base layer decoded signal from the buffer, generates a texture prediction signal to be used on the first enhancement layer using an interpolation filter, and uses the generated texture prediction signal as a base layer texture prediction The signal is output to the signal storage unit 109.

  First enhancement layer prefilter processing unit 111: reads the first enhancement layer original signal from the first enhancement layer original signal storage unit 110, performs prefilter processing, and converts the prefilter signal into the first enhancement layer prefilter signal storage unit 112. Output to. This processing apparatus will be described in detail later with reference to FIG.

  First enhancement layer encoding unit 113: Reads the first enhancement layer prefilter signal from the first enhancement layer prefilter signal storage unit 112, and performs enhancement layer encoding processing. At this time, a reference signal used for texture prediction is read from the base layer texture prediction signal storage unit 109. The obtained encoded information is output to first enhancement layer encoded information storage section 114. An example of this encoding process is JSVM shown in Non-Patent Document 2.

  First enhancement layer inner decoding unit 115: reads the first enhancement layer coding information from the first enhancement layer coding information storage unit 114, performs the decoding process in the first enhancement layer coding process, and generates the generated decoding Write signal to buffer.

  First enhancement layer texture prediction signal generation unit 116: reads a decoded signal of the first enhancement layer from the buffer, generates a texture prediction signal to be used on the second enhancement layer using an interpolation filter, and generates the generated texture prediction signal The first enhancement layer texture prediction signal storage unit 117 outputs the result.

  N-th enhancement layer pre-filter processing unit 119: reads the n-th enhancement layer original signal from the n-th enhancement layer original signal storage unit 118, performs pre-filter processing, and converts the pre-filter signal into the n-th enhancement layer pre-filter signal storage unit 120. Output to. This processing apparatus will be described in detail later with reference to FIG.

  N-th enhancement layer encoding unit 121: reads an n-th enhancement layer pre-filter signal from the n-th enhancement layer pre-filter signal storage unit 120, and performs enhancement layer encoding processing. At this time, a reference signal used for texture prediction is read from the (n-1) th enhancement layer texture prediction signal storage unit (not shown). The obtained encoded information is output to the nth enhancement layer encoded information storage unit 122. An example of this encoding process is JSVM shown in Non-Patent Document 2.

  Coding information multiplexing unit 123: Coding information storage unit of each layer (base layer coding information storage unit 106, first enhancement layer coding information storage unit 114,..., Nth enhancement layer coding information storage unit The encoded information is read from 122), multiplexed, and output as an encoded stream.

As a related technique of the present invention, a configuration example of a non-scalable encoding apparatus is shown in FIG.

  Original signal storage unit 201: Reads an original signal and writes it to a buffer.

  Pre-filter processing unit 202: Reads the original signal from the original signal storage unit 201, performs pre-filter processing, and outputs the pre-filter signal to the pre-filter signal storage unit 203. This processing apparatus will be described in detail later with reference to FIG.

  Encoding unit 204: reads a prefilter signal from the prefilter signal storage unit 203, performs an encoding process using the prefilter signal as an input, and outputs the obtained encoded information to the encoded information storage unit 205. An example of this encoding process is JM shown in Non-Patent Document 5.

  Details of the base layer prefilter processing unit 103 in the scalable encoding device shown in FIG. 7 and the prefilter processing unit 202 in the non-scalable encoding device shown in FIG. 8 will be described with reference to FIG.

  Prefilter application frame designation information storage unit 301: Reads prefilter application frame designation information given from the outside and writes it to a register. The pre-filter application frame designation information is information that designates whether or not to apply the above filter.

  Pre-filter application frame determination unit 302: Reads pre-filter application frame designation information from the pre-filter application frame designation information storage unit 301, determines whether or not the frame is a pre-filter application frame, The process proceeds to the pass filter signal replacement processing unit 304. If false, the process proceeds to the prefilter processing target frame update unit 306.

  Low-pass filter set storage unit 303: Reads a low-pass filter set given from the outside and writes it to a register. The low pass filter set is a set of a plurality of low pass filters having different band pass widths.

  Low-pass filter signal replacement processing unit 304: Reads the original signal of the frame and uses a low-pass filter composed of filter coefficients of the low-pass filter read from the low-pass filter set storage unit 303 for each small region. The pre-filter signal of the obtained frame is written in the buffer, and the pre-filter processing end determination unit 305 moves. Note that in the base layer prefilter processing unit 103, the read original signal is the base layer original signal. This processing apparatus will be described in detail later with reference to FIG.

  Pre-filter processing end determination unit 305: Determines whether or not pre-filter processing has been performed on all frames, outputs a pre-filter signal if true, and pre-filter processing target frame if false Move to update unit 306. In the base layer prefilter processing unit 103, the output prefilter signal is a base layer prefilter signal.

  Prefilter processing target frame update unit 306: Updates the processing target to the next frame, and moves to the prefilter application frame determination unit 302.

  Details of the first enhancement layer prefilter processing unit 111, the nth enhancement layer prefilter processing unit 119, and other enhancement layer prefilter processing units in the scalable coding apparatus shown in FIG. 7 will be described with reference to FIG.

  Prefilter application frame designation information storage unit 401: Reads prefilter application frame designation information given from the outside and writes it to the register.

  Pre-filter application frame determination unit 402: Reads pre-filter application frame designation information from the pre-filter application frame designation information storage unit 401, determines whether or not the frame is a pre-filter application frame, and is adaptive if true. The process proceeds to the replacement processing unit 404. If false, the process proceeds to the prefilter processing target frame updating unit 406.

  Low-pass filter set storage unit 403: Reads a low-pass filter set given from the outside and writes it to a register.

  Adaptive replacement processing unit 404: Reads the original signal of the frame, and receives the filter signal from the low pass filter composed of the filter coefficients of the low pass filter read from the low pass filter set storage unit 403, The texture prediction signal is subjected to adaptive replacement processing for each small region, the obtained prefilter signal of the frame is written in the buffer, and the prefilter processing end determination unit 405 moves. This processing apparatus will be described in detail later with reference to FIG.

  Pre-filter processing end determination unit 405: Performs determination processing on whether or not pre-filter processing has been performed on all frames in the enhancement layer. If true, the pre-filter signal of the enhancement layer is output, and false In this case, the process proceeds to the prefilter processing target frame update unit 406.

  Prefilter processing target frame update unit 406: Updates the processing target to the next frame, and moves to the prefilter application frame determination unit 402.

  Details of the low-pass filter signal replacement processing unit 304 shown in FIG. 9 will be described with reference to FIG.

  Small region dividing unit 501: Reads the original signal of the frame, divides it into small regions of a specified size, and outputs the original signal of each small region to the small region signal storage unit 502.

Applied low-pass filter coefficient reading unit 503: Reads the filter coefficient of the low-pass filter f _n to be applied to the small area, and writes the coefficient value to the register.

Low-pass filter processing unit 504: Reads the original signal of the small region from the small region signal storage unit 502, reads the filter coefficient of the low-pass filter f _n from the register, and configures the low band constituted by the filter coefficient The pass filter is applied to the original signal of the small region, and the obtained low pass filter signal of the small region is output to the low pass filter signal storage unit 505.

  Viewing distance parameter storage unit 506: Reads a viewing distance parameter given from the outside and writes it to a register.

Visual distortion amount calculation unit 507: Reads the original signal of the small region from the small region signal storage unit 502, reads the low-pass filter signal from the low-pass filter signal storage unit 505, and generates a difference signal thereof. The Fourier transform is applied to the difference signal, the frequency components of the Fourier transform are weighted using the contrast sensitivity function, and the sum of the weighted frequency components is taken to obtain a difference between the two signals. The visual distortion amount D _n is calculated and the value is output to the visual distortion amount storage unit 508.

Visual distortion amount threshold storage unit 509: Reads a visual distortion amount threshold value _Dth given from the outside, and writes it in a register.

Visual distortion amount comparison unit 510: The visual distortion amount D _n calculated by using the low-pass filter f _n in the small region is read from the visual distortion amount storage unit 508, and the visual distortion amount threshold storage unit The visual distortion amount threshold value D _th is read from 509. If D _n ≦ D _th , the process proceeds to the low-pass filter signal replacement processing unit 511, and if D _n > D _th , the end of the applied low-pass filter set is determined. Move to section 512.

Low-pass filter signal replacement processing unit 511: Reads a low-pass filter signal by the low-pass filter f _n of the small region, replaces it with the original signal of the small region, writes it to the buffer as a pre-filter signal, and completes the replacement process Move to determination unit 514.

  Applicable low-pass filter set end determination unit 512: Determines whether all low-pass filters have already been applied to the small region, and if true, moves to a replacement processing end determination unit 514. In the case of false, the process proceeds to the applicable low-pass filter coefficient update unit 513.

  Applied low-pass filter coefficient updating unit 513: The application target is transferred to the next low-pass filter, and the applied low-pass filter coefficient reading unit 503 is moved.

  Replacement processing end determination unit 514: Performs determination processing on whether or not replacement determination processing has been executed for all the small regions in the frame, outputs a prefilter signal if true, and processing if false The process proceeds to the target small area update unit 515.

  Processing target small area update unit 515: Moves the processing target to the next small area and moves to the low-pass filter processing unit 504.

  Details of the adaptive replacement processing unit 404 shown in FIG. 10 will be described with reference to FIG.

  Small region dividing unit 601: Reads the original signal of the frame, divides it into small regions of a specified size, and outputs the original signal of each small region to the small region signal storage unit 602.

  Texture prediction signal reading unit 603: Reads a texture prediction signal from the immediately lower layer and writes it to the buffer.

Texture prediction signal visual distortion amount calculation unit 604: Reads the original signal of the small region from the small region signal storage unit 602, reads the texture prediction signal of the immediately lower layer from the buffer, generates a difference signal thereof, Applying a Fourier transform to the signal, weighting each frequency component of the Fourier transform using the contrast sensitivity function, and taking the sum of the weighted frequency components gives a visual The distortion amount D ₀ is calculated, and the value is output to the texture prediction signal visual distortion amount storage unit 606.

  Viewing distance parameter storage unit 605: Reads a viewing distance parameter given from the outside and writes it to a register.

Visual distortion amount threshold storage unit 607: Reads a visual distortion amount threshold value _Dth given from the outside, and writes it into a register.

Texture prediction signal visual distortion quantity comparing unit 608: reads the visual distortion amount D ₀ calculated using a texture prediction signal immediately-lower layer from the texture prediction signal visual distortion amount storage unit 606, visual distortion amount threshold storage The visual distortion amount threshold value D _th is read from the unit 607. If D _n ≦ D _th , the process proceeds to the texture prediction signal replacement processing unit 618. If D _n > D _th , the process proceeds to the low-pass filter processing unit 610. .

Applied low-pass filter coefficient reading unit 609: Reads the filter coefficient of the low-pass filter f _n to be applied to the small area, and writes the coefficient value to the register.

Low-pass filter processing unit 610: Reads a filter coefficient of the low-pass filter f _n from the register, applies a low-pass filter constituted by the filter coefficient to the original signal of the small area, and obtains the obtained The small-pass low-pass filter signal is output to the low-pass filter signal storage unit 611.

Low-pass filter signal visual distortion amount calculation unit 612: Reads the original signal of the small region from the small-region signal storage unit 602, reads the low-pass filter signal from the low-pass filter signal storage unit 611, Generate a difference signal, apply a Fourier transform to the difference signal, weight each frequency component of the Fourier transform using a contrast sensitivity function, and take the sum of the weighted frequency components To calculate the visual distortion amount D _n between the two signals, and output the value to the low-pass filter signal visual distortion storage unit 613.

Low-pass filter signal visual distortion amount comparison unit 614: Visual distortion amount D _n calculated from the low-pass filter signal visual distortion amount storage unit 613 using the low-pass filter f _n in the small region. And the visual distortion amount threshold value D _th is read from the visual distortion amount threshold value storage unit 607. If D _n ≦ D _th , the process proceeds to the low-pass filter signal replacement processing unit 615, and if D _n > D _th Then, the process proceeds to the applied low-pass filter set end determination unit 616.

Low-pass filter signal replacement processing unit 615: Reads a low-pass filter signal by the low-pass filter f _n of the small region, replaces it with the original signal of the small region, writes it to the buffer as a pre-filter signal, and completes the replacement process Move to determination unit 619.

  Applicable low-pass filter set end determination unit 616: Determines whether all the low-pass filters have already been applied to the small region, and if true, moves to a replacement processing end determination unit 619, In the case of false, the process proceeds to the applicable low-pass filter coefficient update unit 617.

  Applied low-pass filter coefficient updating unit 617: The application target is transferred to the next low-pass filter, and the applied low-pass filter coefficient reading unit 609 is moved.

  Texture prediction signal replacement processing unit 618: Reads the texture prediction signal of the immediately lower layer in the small region, replaces it with the original signal of the small region, writes it into the buffer as a prefilter signal, and moves to the replacement processing end determination unit 619.

  Replacement processing end determination unit 619: Performs determination processing as to whether or not replacement determination processing has been executed for all the small regions in the frame, and outputs an adaptively replaced prefilter signal if true, If false, the process proceeds to the processing target small area update unit 620.

  Processing target small region update unit 620: Moves the processing target to the next small region, and moves to the texture prediction signal visual distortion amount calculation unit 604.

[Experiment]
An experiment for verifying the performance of the prefilter according to the present invention was performed using the JSVC reference encoder JSVM8.0 described in Non-Patent Document 2. For the video, City, Crew, Harbor, and Soccer of JSVC standard video were used. The lower layer is QCIF, the upper layer is CIF, and the same quantization parameter QP is used in both layers. The picture structure is IPPP and 300 pictures are encoded. The viewing distance used for the pre-filter according to the present invention was 3h, the threshold was 1, and five low-pass filters were prepared.

  Table 1 shows the result of code amount reduction when the prefilter is on / off. A high code amount reduction of 4.5% to 24% (average of 11.9%) has been realized, and the effect tends to increase as the rate increases. It has been confirmed that there is no difference in the subjective image quality of the decoded signal between on / off of the prefilter. The higher the rate, the more likely a non-zero quantization level that causes an increase in the amount of code occurs in the high frequency, but since the level is made zero by the prefilter, the effect is relatively higher than the low rate.

  In addition, the smaller the high frequency components such as Crew and Soccer, the greater the reduction effect. When the high-frequency component included is small, the amount of visual distortion with the original signal is small. Therefore, it is considered that the signal is replaced with a signal by a filter having a strong band limitation or an interpolation signal, and the reduction effect is increased. We also measured the macroblock replacement rate by prefilter for each video, but it was confirmed that a video with a larger high-frequency component used more filters with higher bandwidth restrictions.

  The above moving image encoding processing can be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or provided through a network. .

It is a figure which shows the flow of the scalable encoding process by this Embodiment. It is a figure which shows the flow of the non-scalable encoding process by related technology . It is a figure which shows the flow of the pre filter process by this Embodiment. It is a figure which shows the flow of the pre filter process for the enhancement layers by this Embodiment. It is a figure which shows the flow of the replacement process by the low-pass filter signal by this Embodiment. It is a figure which shows the flow of the adaptive replacement process by this Embodiment. It is a figure which shows the structural example of the scalable coding apparatus by this Embodiment. It is a figure which shows the structural example of the non-scalable encoding apparatus by related technology . It is a figure which shows the apparatus structural example of a base layer pre filter process part and a pre filter process part. It is a figure which shows the apparatus structural example of an extended layer prefilter process part. It is a figure which shows the apparatus structural example of a low-pass filter signal substitution process part. It is a figure which shows the apparatus structural example of an adaptive replacement process part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Original signal reduction process part 102 Base layer original signal storage part 103 Base layer pre-filter process part 104 Base layer pre-filter signal storage part 105 Base layer encoding part 106 Base layer encoding information storage part 107 Base layer internal decoding part 108 Basic Layer texture prediction signal generation unit 109 Base layer texture prediction signal storage unit 110 First enhancement layer original signal storage unit 111 First enhancement layer prefilter processing unit 112 First enhancement layer prefilter signal storage unit 113 First enhancement layer encoding unit 114 1st enhancement layer coding information storage part 115 1st enhancement layer inner decoding part 116 1st enhancement layer texture prediction signal generation part 117 1st enhancement layer texture prediction signal storage part 118 nth enhancement layer original signal storage part 119 nth Enhanced layer prefilter Processing unit 120 nth enhancement layer prefilter signal storage unit 121 nth enhancement layer encoding unit 122 nth enhancement layer encoded information storage unit 123 encoded information multiplexing unit 201 original signal storage unit 202 prefilter processing unit 203 prefilter Signal storage unit 204 Coding unit 205 Coding information storage unit

Claims (5)

In a scalable video coding method that forms a layer structure,
A small area dividing process for dividing the original image signal in the enhancement layer to be encoded into small areas;
A process for calculating the visual distortion amount of the immediately lower layer decoded upsample signal for calculating the visual distortion amount of the original image signal and the image signal obtained by upsampling the decoded image signal of the immediately lower layer for each divided small area;
It is determined whether or not the calculated visual distortion amount is equal to or less than a predetermined threshold value. A sub-region image signal direct layer decoding up-sample signal replacement process for performing a low-pass filter process described later;
A low-pass filter processing process in which the filter processing for the small region is performed from a low-pass filter having a small band pass width among low-pass filter sets having different band pass widths prepared in advance;
A low-pass filter signal visual distortion amount calculation process for calculating a visual distortion amount between the derived filter signal and the original image signal;
It is determined whether or not the calculated visual distortion amount is equal to or less than a predetermined threshold value. If the calculated visual distortion amount is equal to or less than the threshold value, the original image signal is replaced with the filter signal. Determines whether all low-pass filter sets have been applied. If all are applied, the original image signal is output as it is. If not, the band-pass width of the low-pass filter applied to the small region is applied. Is updated to the one with the smallest bandpass width next to the bandpass width, and the low-pass filter process for re-executing the low-pass filter process,
An image signal that is finally calculated by performing processing from the immediately lower layer decoded upsample signal visual distortion amount calculation process to the small area image signal low-pass filter signal replacement process for all the small areas And a process of inputting the signal as an encoded input signal to the enhancement layer to the encoder and performing the encoding. The scalable video encoding method according to claim 1 ,
Only for a specific frame image or its image partial area designated from the outside, or for a specific frame image or its image partial area defined by a predetermined criterion, for all small areas in the area The processing from the immediately lower layer decoded upsample signal visual distortion amount calculation process to the small area image signal low-pass filter signal replacement process is executed, and for the small area other than the specific frame image or its image partial area, The scalable moving picture coding method, wherein the process from the immediately lower layer decoded upsample signal visual distortion amount calculation process to the small area image signal low-pass filter signal replacement process is not executed. In a scalable video encoding device that constitutes a layer structure,
A small area division processing means for dividing the original image signal in the enhancement layer to be encoded into small areas;
A direct layer decoded upsample signal visual distortion amount calculation processing means for calculating a visual distortion amount between an original image signal and an image signal obtained by upsampling a decoded image signal of a direct layer for each of the divided small regions;
It is determined whether or not the calculated visual distortion amount is equal to or less than a predetermined threshold value. A small region image signal direct layer decoding up-sample signal replacement processing means for executing low pass filter processing means described later;
Low-pass filter processing means for performing filter processing on the small region from a low-pass filter having a small bandpass width among low-pass filter sets having different bandpass widths prepared in advance;
A low-pass filter signal visual distortion amount calculation processing means for calculating a visual distortion amount between the derived filter signal and the original image signal;
It is determined whether or not the calculated visual distortion amount is equal to or less than a predetermined threshold value. If the calculated visual distortion amount is equal to or less than the threshold value, the original image signal is replaced with the filter signal. Determines whether all low-pass filter sets have been applied. If all are applied, the original image signal is output as it is. If not, the band-pass width of the low-pass filter applied to the small region is applied. A small region image signal low-pass filter signal replacement processing means for renewing the band-pass width next to the band-pass width and re-executing the filter processing by the low-pass filter processing means;
An image signal finally calculated by executing the processing from the immediately lower layer decoded upsample signal visual distortion amount calculation processing means to the small area image signal low-pass filter signal replacement processing means for all the small areas. And a means for performing encoding by inputting the signal as an encoded input signal to the enhancement layer to the encoder. Scalable video encoding program for executing a scalable video encoding method according to claim 1 or claim 2 wherein the computer. A computer-readable recording medium on which a scalable video encoding program for causing a computer to execute the scalable video encoding method according to claim 1 or 2 is recorded.

Images