JP2008199252A - Motion picture decoding device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce an overhead of concealment processing by decreasing the operational amount when the type of an area adjacent to an area to be interpolated is classified. <P>SOLUTION: When the type of a reference area adjacent to a missing macroblock is classified, first classification processing S421 is performed first to find a feature vector 1 representing a distribution state of luminance values of pixels constituting the reference area and then to decide "Smoothness" or "Texture" based upon the feature vector 1. Then only when the type of the reference area can not be classified through the first classification processing, second classification processing S422 is carried out to calculate a feature quantity vector 2 representing a gradient of pixels present in the reference area and to decide to which the type of the reference area belongs among "Smoothness", "Edge", or "Texture" based upon the calculated feature quantity vector 2 and feature quantity vector 1. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image decoding apparatus having an error concealment function that repairs an error in an image frame when decoding moving image data.

  In recent years, mobile terminals such as mobile phones and PDAs (Personal Digital Assistants) are increasingly equipped with terminals having a function of receiving one-segment broadcasting of terrestrial digital broadcasting. In the one-segment broadcasting of terrestrial digital broadcasting, the ITUT H.264 method is adopted as a moving image encoding method. For this reason, the mobile terminal is provided with a moving picture decoding apparatus corresponding to the ITUT H.264 system.

  By the way, in a moving picture decoding apparatus of a mobile terminal, for example, if an error is mixed in a bit stream in a wireless transmission path, it becomes impossible to decode an image until the synchronization of variable-length decoding is restored, resulting in a large visual deterioration. . Therefore, an error concealment process is performed in which an area lost due to an error is interpolated using an already decoded image signal stored in the frame memory.

  The error concealment is roughly classified into a temporal concealment that uses a correlation between frames, a spatial concealment that uses a correlation within a frame, and a spatiotemporal concealment that combines the above two. In general, interpolation processing is performed by specifying the presence or absence of an error for each macroblock (MB), which is a reference unit of encoding, but error concealment with a higher image quality improvement effect by adaptively switching the interpolation processing method. Is possible.

  Among these, for spatial concealment, the reference area adjacent to the missing macroblock is classified into three types called “Smoothness”, “Edge”, and “Texture”, and the characteristics of the missing macroblock are identified from the classification results. There is a method of estimating and switching the content of interpolation processing. Here, “Smoothness” is an area where pixel values are almost uniform, such as the sky, and examples thereof are shown in FIGS. “Edge” is an area including an edge pattern, and an example thereof is shown in FIGS. “Texture” is a region having a large variation in pixel values, such as a geometric pattern or a flower bed, and examples thereof are shown in FIGS.

By the way, as a method of classifying the reference region adjacent to the missing macroblock into the above three types, conventionally, for example, a method using gradient information representing an edge pattern as one of indices (for example, non-patent) is known. See reference 1.)
Zhang Rongfu, et. Al, “Content-Adaptive Spatial Error Concealment for Video Communication”, IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, pp.335-340, Feb. 2004. and classification.

  However, in the method described in Non-Patent Document 1, gradient calculation for detecting edge patterns is performed for all reference regions used for classification of missing macroblocks. The calculation of the gradient is generally performed by applying a filter such as Sobel Filter or Prewitt Filter to each pixel in the gradient calculation region. For this reason, gradient calculation is performed for all the pixels in the region, and the amount of calculation becomes enormous. The gradient information is not wasted when the type of the reference area is “Edge” because the gradient information is used for the interpolation process. However, when the type of the reference area is “Smoothness” or “Texture”, the gradient information is not used for the interpolation process, which is a wasteful calculation. This arithmetic processing is a non-negligible overhead when performing concealment processing, and is a very big problem especially for devices with limited image processing capability such as mobile phones and PDAs (Prsonal Digital Assistant). It becomes.

  The present invention has been made paying attention to the above circumstances, and its object is to reduce the amount of calculation when classifying the reference area type adjacent to the interpolation target block, thereby reducing the overhead of concealment processing. The object is to provide an image decoding apparatus.

  In order to achieve the above object, according to a first aspect of the present invention, there is provided an image decoding apparatus that decodes image data encoded for each block and outputs decoded image data. The restoration target block including the missing pixel that has not been correctly decoded by the above is specified, and a reference area that is adjacent to the specified restoration target block and that has been correctly decoded by the decoding process is selected. Then, for each of the selected reference regions, a feature amount that represents the dispersion state of the pixel values in the reference region is detected using a statistical method, and based on the feature amount that represents the dispersion state of the detected pixel values. The image type of the reference area is a first type in which pixel values are uniformly distributed, a second type with a large variation in pixel values, or a third type including an edge pattern. Whether or not there is is determined by the first determination means. And based on the determination result of the type of the reference region by the first determination means, it is estimated whether the type of the block to be repaired is the first type, the second type, or the third type, Based on the estimation result, an interpolation processing unit is selected from a plurality of interpolation processing units prepared in advance in association with the first, second, and third types, and the restoration target block including the missing pixel is selected. An interpolation process is performed.

  According to a second aspect of the present invention, when the image type of the reference area is not determined by the first determination unit, a feature amount representing a gradient of pixel values in the reference area is calculated. Then, based on the calculated feature value representing the gradient of the pixel value, the image type of the reference region is the first type, the second type, or the third type. Is determined by the second determination means.

  Therefore, according to the first aspect of the present invention, when determining the type of the image of the reference area, a feature value representing the dispersion state of the pixel values in the reference area is detected using a statistical method. The type of the image is determined based on the feature value of the dispersion state of the detected pixel value. For this reason, compared to the case where the feature amount representing the gradient of the pixel value in the reference region is always calculated and the image type is determined based on the feature amount representing the gradient, the image type of the reference region is determined. The amount of computation required is reduced, and the overhead of concealment processing is reduced.

  Further, according to the second aspect of the present invention, only when the type of the image of the reference area is not determined by the first determination unit, the feature amount representing the gradient of the pixel value in the reference area is calculated, Re-determination of the image type of the reference area is performed based on the calculated feature value representing the gradient of the pixel value. Therefore, it is possible to perform accurate type determination based on the feature value representing the gradient of the pixel value even when the type value cannot be accurately determined from the feature value representing the dispersion state of the pixel value while suppressing the calculation amount to a small amount. Is possible.

  That is, according to the present invention, it is possible to provide an image decoding apparatus that can reduce the amount of calculation when classifying the type of the region adjacent to the interpolation target block and reduce the overhead of concealment processing. .

(First embodiment)
In the first embodiment of the present invention, a moving picture decoding apparatus according to the present invention is provided in a portable terminal such as a cellular phone, and a code received by one-segment broadcasting of terrestrial digital broadcasting (hereinafter referred to as one-segment broadcasting). When interpolating missing macroblocks in a converted video data stream based on a reference area adjacent to the macroblock, the type of the reference area is first determined by calculating the pixel distribution using a statistical method. The feature amount representing the gradient of the pixel value is calculated only when there is a possibility that the type of determination is “Edge”, and the type is re-determined based on the calculation result.

FIG. 1 is a block diagram showing a configuration of a portable terminal provided with an image decoding apparatus according to the first embodiment of the present invention.
The portable terminal according to this embodiment includes a wireless unit 1, a baseband unit 2, a user interface unit 3, a storage unit 4, a digital broadcast receiving unit 5, and a power supply unit 6.

  In the figure, first, in a state where the call mode is set, the user's transmitted voice signal output from the microphone 32 of the user interface unit 3 is input to the encoder 23 of the baseband unit 2. A video signal output from the camera (CAM) 31 is also input to the encoder 23. The encoder 23 encodes the transmitted voice signal by a predetermined coding scheme of CELP (Code Excited Linear Predictive coding) and encodes the video signal in accordance with, for example, MPEG-4 (Moving Picture Coding Experts Group-4) scheme. The encoded audio data and video data are respectively packetized and multiplexed to output transmission multimedia data. The transmission multimedia data is supplied to the transmission circuit (TX) 15 of the wireless unit 1 after the control module 21 further multiplexes various control information such as destination information.

  The transmission circuit 15 includes a modulator, a frequency converter, and a transmission power amplifier. The transmission data is digitally modulated by a modulator, and then mixed with a transmission local oscillation signal generated from the frequency synthesizer 14 by a frequency converter and frequency-converted to a radio frequency signal. As a modulation method, a digital modulation method such as a QPSK (Quadriphase Phase Shift Keying) method or a QAM (Quadrature Amplitude Modulation) method and a spread spectrum method using a spread code are used. The modulated transmission radio frequency signal is amplified to a predetermined transmission level by a transmission power amplifier, and then supplied to the antenna 11 via the antenna duplexer (DUP) 12, from which a base station (not shown) is supplied. Sent to.

  In contrast, a radio frequency signal arriving from a base station via a radio channel is received by an antenna 11 and then input to a receiving circuit (RX) 13 via an antenna duplexer 12. The reception circuit 13 includes a high frequency amplifier, a frequency converter, and a demodulator. The radio frequency signal is amplified with a low noise amplifier and then mixed with a reception local oscillation signal generated from a frequency synthesizer (SYN) 14 in a frequency converter to generate a reception intermediate frequency signal or a reception baseband signal. The output signal is digitally demodulated by a demodulator. As a demodulation method, for example, an orthogonal demodulation method and a spectrum despreading method using a spread code are used. The reception local oscillation signal frequency generated from the frequency synthesizer 14 is instructed from the control module 21 provided in the baseband unit 2.

  The received packet data output from the demodulator is input to the baseband unit 2. In the baseband unit 2, the signal is input to the decoder 22 via the control module 21. The decoder 22 separates the received packet data into audio packets and video packets, then depackets the audio packets and then decodes them into audio data. The decoded audio frame is converted into an analog signal and then output as a received voice from the speaker 35 of the user interface unit 3. The decoder 22 depackets the input video packet and then decodes it into a video frame in accordance with, for example, the MPEG4 system. The decoded video frame is displayed on the display device 34 of the user interface unit 3. The display device 34 includes, for example, an LCD (Liquid Crystal Display) 33.

  On the other hand, in a state where the digital broadcast viewing mode is set, a broadcast signal transmitted from a broadcast station (not shown) is received and demodulated by the digital broadcast receiving unit 5 via the antenna 51. The received and demodulated received broadcast data is analyzed by the control module 21 of the baseband unit 2 and then input to the decoder 22. The decoder 22 separates the input received broadcast data into audio packets and video packets. The audio packet is depacked and then decoded into an audio frame, and the video packet is depacketed and then decoded into an image frame.

  For example, in one-segment broadcasting of terrestrial digital broadcasting, audio data is encoded by the MPEG2 AAC format, and video data is encoded by the ITUT H.264 format. For this reason, the decoder 22 decodes the audio packet and the video packet by a decoding method corresponding to these encoding methods. The decoded audio signal is loudly output from the speaker 35 of the user interface unit 3, and the video signal is supplied to the display device 34 and displayed.

  The power supply unit 6 includes a battery 61 such as a lithium ion battery, a charging circuit 62 for charging the battery 61 based on a commercial power output (AC 100 V), and a voltage generation circuit (PS) 63. It has been. The voltage generation circuit 63 is composed of, for example, a DC / DC converter, and generates a predetermined power supply voltage Vcc based on the output voltage of the battery 61.

  By the way, the decoder 22 is configured using, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and has the following functions as functions according to the present invention. FIG. 2 is a block diagram showing the functional configuration. That is, the decoder 22 includes a decoding unit 100, a missing area information memory 200, a concealment unit 300, and a decoded picture buffer 400.

  Among these, the decoding unit 100 includes a decoder, an error detection unit 110, and a switching control unit 120. The decoder performs variable-length decoding on the input video data bit stream in units of macroblocks according to the variable-length code table, and outputs the decoded picture data to the encoded picture buffer 400.

The error detection unit 110 detects a syntax error mixed in the bit stream of the video data decoded by the decoder. Then, in accordance with the detection result, flag information StatusFrag indicating whether or not each macroblock has been normally decoded is generated, and the flag information StatusFrag is stored in the missing area information memory 200 for each image frame.
When the decoding process for one frame is completed by the decoder and the next picture boundary is detected, the switching control unit 120 performs a concealment process on the restoration target frame for which the decoding process has been completed. Notification to instruct

  The concealment unit 300 includes a concealment control unit 310, a peripheral region information acquisition unit 320, a peripheral region buffer 330, a gradient information calculation unit 240, a gradient selection unit 350, a threshold setting unit 360, boundary coordinates. A calculation unit 370 and an interpolation pixel value calculation unit 380 are provided.

  The concealment control unit 310 comprehensively controls the operation of each part in the concealment unit 300. Upon receiving a notification for instructing the concealment process from the switching control unit 120, the concealment control unit 310 Each part is caused to perform a concealment process on a missing region in the repair target frame. At the same time, when the interpolation process for all missing macroblocks in the repair target frame is completed, the decoding unit 100 is notified of the end of the concealment process via the switching control unit 120, and the decoding unit 100 is notified of the next image frame. The decryption process is started. At this time, the switching control unit 120 clears the flag information corresponding to the restoration target frame stored in the missing area information memory 200.

  The peripheral area information acquisition unit 320 refers to the flag information StatusFlag stored in the missing area information memory 200 and uses a macroblock that has been correctly decoded from among the macroblocks adjacent to the macroblock to be repaired. Select as possible reference area. For example, assuming that the restoration target macroblock is X as shown in FIG. 9 (a), whether or not four macroblocks A to D adjacent to the top, bottom, left and right are usable for restoration is determined. The determination is based on the D flag information StatusFlag. In the example of FIG. 9B, the flag information StatusFlag of the adjacent macroblocks A to D is 1, 1, 0, and 1, respectively, so that the reference areas that can be used by the adjacent macroblocks A, B, and D for restoration are used. It is determined.

  The peripheral area buffer 330 acquires pixel information of adjacent macroblocks determined to be usable for restoration by the peripheral area information acquisition unit 320 from the decoded picture buffer 400 and stores them. Then, the coordinate value of the boundary pixel is received from the boundary coordinate detection unit 370, and based on the received coordinate value of the boundary pixel, the pixel information corresponding to the coordinate value is stored in the pixel information of the stored adjacent macroblock. Read out and output to the interpolation pixel value calculation unit 380.

  The missing area classification unit 340 reads the pixel information of the adjacent macroblock from the peripheral area buffer 330, classifies the type of the read adjacent macroblock, and the type of the macroblock to be interpolated is “ Estimate whether it is “Smoothness”, “Edge”, or “Texture”, and is configured as follows. FIG. 3 is a block diagram showing the configuration.

In other words, the missing region classification unit 340 includes a statistical information calculation unit 341, a gradient information calculation unit 342, a peripheral region analysis unit 343, and a missing region classification determination unit 344, and further includes a switching unit 345.
The statistical information calculation unit 341 reads pixel information corresponding to a preset analysis region size from the peripheral region buffer 330, and based on the read image information, calculates a feature amount representing pixel value distribution information by a statistical method. The statistical information representing the calculation result is notified to the peripheral region analysis unit 343 via the switching unit 345. The gradient information calculation unit 342 reads pixel information corresponding to a preset analysis region size from the peripheral region buffer 330, calculates a feature value representing the gradient of the pixel value based on the read image information, and the calculation result Is transmitted to the peripheral region analysis unit 343 via the switching unit 345. Here, when analyzing image information in a certain reference area, a luminance signal is used as pixel information of the reference area used for calculating statistical information and gradient information. However, since there is a correlation between the luminance information and the color difference information, a color difference signal can be used instead of the activation information.

  The peripheral region analysis unit 343 compares the information calculated by the statistical information calculation unit 341 and the gradient information calculation unit 342 with a threshold value for statistical information and a threshold value for gradient information, which are set in advance, respectively. It is determined whether the type is “Smoothness”, “Edge”, or “Texture”. At the same time, the calculation result of the statistical information and the calculation result of the gradient information are notified to the missing region classification determination unit 344.

  The missing region classification unit 344 determines whether the type of the macro block to be interpolated is “Smoothness”, “Edge”, or “Texture” based on the determination result of the reference region type by the peripheral region analysis unit 343. presume. As the estimation method, for example, a method is used in which a majority decision process is performed on the reference region type determination result.

  When classifying the types of adjacent macroblocks, the switching unit 345 first selects the statistical information calculation unit 341 and causes the calculation of statistical information. Then, only when the peripheral region analysis unit 343 cannot determine the type of each reference region based on the statistical information, the gradient information calculation unit 342 is selected to perform the calculation processing of the gradient information.

  The missing area interpolation processing unit 350 interpolates the interpolation target macroblock using the pixel information of the boundary pixels with the interpolation target macroblock in the reference area read from the peripheral area buffer 330. In this interpolation processing, the type estimation result of the macro block to be interpolated by the missing area classification unit 340 is referred to.

  FIG. 4 is a block diagram showing a configuration of the missing area interpolation processing unit 350. That is, the missing region interpolation processing unit 350 includes a Smoothness interpolation processing unit 351, an Edge interpolation processing unit 352, and a Texture interpolation processing unit 353, and further, an input side and an output side of these interpolation processing units 351, 352, and 353 Are provided with a pair of switching sections 354 and 355. The switching units 354 and 355 switch the Smoothness interpolation processing unit 351, the Edge interpolation processing unit 352, and the Texture interpolation processing unit 353 according to the estimation result of the type of the interpolation target macroblock estimated by the missing region classification unit 340. Switch alternatively. Pixel information (luminance value or color difference value) corrected by the Smoothness interpolation processing unit 351, Edge interpolation processing unit 352, and Texture interpolation processing unit 353 is supplied to the decoded picture buffer 400 and stored in the position of the corresponding interpolation target region. Is done.

  The decoded picture buffer 400 obtains, for each image frame, the pixel value of the region normally decoded by the decoding unit 100 and the pixel value of the missing pixel calculated by the missing region interpolation processing unit 350 of the concealment unit 300. Remember. The concealment unit 300 reads out an image frame for which all missing pixel values have been interpolated, and supplies the read image frame to the display device 34.

Next, the procedure and contents of concealment processing by the decoder 22 configured as described above will be described. FIG. 5 is a flowchart showing an outline of the procedure and contents of the concealment process, and FIG. 6 is a flowchart showing a specific example of the process contents.
When a bit stream of video data is input from the control module 21, the decoding unit 100 decodes the input video data bit stream for each macroblock, and the decoded picture data is stored in the decoded picture buffer 400. It is sent and temporarily stored.

In parallel with this decoding process, the error detection unit 110 determines whether or not an error in the syntax is mixed in the bit stream of the video data to be decoded (step S41). Then, flag information representing the determination result of the presence or absence of this error is stored in the missing area information memory 200. For example, StatusFlag = 1 is set for a macroblock that is normally decoded, and StatusFrag = 0 is set for a macroblock that cannot be decoded, and these are stored in the missing area information memory 200 in units of frames. FIG. 8 shows an example of the storage result of the flag information StatusFrag in the missing area information memory 200.
When the decoding process of video data for one frame is completed, the switching control unit 120 sends a notification instructing the concealment process for the frame for which the decoding process has been completed to the concealment control unit 310.

When the concealment processing instruction is notified, the concealment unit 300 performs the concealment processing as follows.
That is, for the macroblock X including the missing pixel, the neighboring macroblock is first selected by the peripheral area information acquisition unit 320. For example, there are eight macroblocks A to H around the missing macroblock X as shown in FIG. 7, and from among these, first, for example, as shown in FIG. Four adjacent macroblocks A to D are selected. Next, referring to the flag information StatusFlag stored in the missing area information memory 200, for example, as shown in FIG. 9B, a macroblock A, which does not contain a missing pixel from the above four adjacent macroblocks A to D, B and D are selected as reference macroblocks. Then, pixel values of the respective pixels in the selected reference macroblocks A, B, and D are selectively read out from the decoded picture buffer 400, and the read reference macroblocks A, B, and D are read out. Each pixel value is stored in the peripheral area buffer 330.

  Note that the area size of the reference macroblock for obtaining the pixel value from the decoded picture buffer 400 is not necessarily equal to the area size of the interpolation processing unit, and is a pixel necessary for detecting feature vectors 1 and 2 described later. You only need to get a few minutes. FIG. 9C shows an example in which the size of the reference area to be acquired and the number of pixels included in the area size (16 × 16 pixels) of the interpolation processing unit are not equal. In this example, a case where an area of 16 × 8 pixels or 8 × 16 pixels is used as a reference area is shown. In short, the size of the number of pixels of the reference area to be acquired can be arbitrarily determined according to the detection accuracy of the feature vector of the reference area and the buffer size that can be secured.

  Next, the concealment unit 300 proceeds to step S42, and the missing region separation unit 340 performs a process of classifying the type of the selected reference macroblock (reference region). This reference area classification process includes a first classification process in step S421, a second classification process in step S422, and an end determination process in step S423. First, the first classification process is performed. Is called.

  That is, in the first classification process S421, the missing region separation unit 340 first selects one of the reference regions from the peripheral region buffer 330. Then, the feature quantity vector 1 representing the distribution state of the luminance values of the pixels constituting the selected reference area is calculated, and the selected reference area is classified based on the feature quantity vector 1.

  For example, as shown in FIG. 6, first, in step S51, the frequency distribution of the luminance values of the pixels included in the selected reference area is examined to detect the maximum value deg_max. Then, deg_max_rate representing the ratio of the detected maximum value deg_max to the number N of pixels in the reference region is calculated. At the same time, the maximum value and the minimum value of the luminance values of the pixels included in the selected reference area are respectively detected, and a range representing the difference between the detected maximum value and minimum value is calculated.

  Next, the missing region separation unit 340 compares the difference range between the calculated maximum value and the minimum value of the calculated brightness value with a preset threshold value Th_r in step S52. If the result of this comparison is range <Th_r, it is determined in step S4213 that the selected reference area is “Smoothness”, and the process proceeds to classification processing for the next reference area that has not yet been selected. On the other hand, if the result of the comparison is range> = Th_r, deg_max_rate representing the ratio of the calculated maximum value deg_max is compared with a preset threshold value Th_d in step S53. If the result of this comparison is deg_max_rate <Th_d, it is determined in step S4214 that the reference area is “Texture”, and the process proceeds to the classification process for the next unselected reference area. The threshold values Th_r and Th_d are set to values obtained experimentally because optimum values differ for each image.

  In the above classification process, “range” is used as an index for classifying a region that is clearly Texture. However, if even a single pixel is taken away, it is not an intended index, and classification can be performed with higher accuracy by using variance. However, the amount of processing increases when dispersion is used. Further, in the first classification process, after clearly classifying the area that is clearly “Smoothness”, the area that is clearly “Texture” is classified. However, the present invention is not limited to this, and a configuration in which an area that is clearly “Texture” is classified first, and then an area that is clearly “Smoothness” is classified.

  On the other hand, it is assumed that deg_max_rate> = Th_d as a result of the comparison in step S53. In this case, the missing area separation unit 340 proceeds to the second classification process S422. In the second separation process S422, a feature quantity vector 2 representing a gradient is calculated for the pixels existing in the selected reference region, and the calculated feature quantity vector 2 and the feature previously calculated in step S4211 are calculated. Based on the quantity vector 1, a process for classifying the type of the selected reference area is performed.

  For example, as shown in FIG. 6, first, in step S54, a gradient vector of pixels included in the selected reference region is calculated as the feature vector 2. The calculation of the gradient vector is performed by applying a filter operation such as Sobel Filter or Prewitt Filter to each pixel in the reference region. Next, in step S55, classification conditions are set based on the calculated gradient vector and the range previously calculated in step S51. Based on this classification condition, the type of the selected reference area is determined in steps S4223, S4224, and S4225.

  As described above, when the second classification process in step S422 is completed, the missing area separation unit 340 determines in step S423 whether the classification process for all reference areas has been completed. If there remains a reference area that has not yet been classified, the process returns to the first classification process S421 to execute the classification process according to the procedure described above.

  On the other hand, when the classification process for all the reference regions is completed, the missing region separation unit 340 proceeds from step S423 to step S43, where the type of the missing macroblock to be interpolated is based on the classification result for each reference region. Perform estimation processing. For example, when the reference areas A, B, and D in FIG. 9A are determined to be “Smoothness”, “Texture”, and “Smoothness”, respectively, the interpolation target macroblock is estimated as “Smoothness”.

  When the type of the missing macroblock is estimated, the concealment unit 300 moves from steps S44, S46, and S48 to steps S45, S47, and S49, respectively. In these steps S45, S47, and S49, “Smoothness” is obtained. , “Edge” and “Texture” are executed.

For example, when the type of the missing macroblock is estimated as “Smoothness”, an interpolation process using Linear Interpolation is performed. In this interpolation process using Linear Interpolation, for example, as shown in FIG. 10, perpendicular lines d1, d2, d3, and d4 are drawn from the missing pixel p (i, j) to be interpolated toward the adjacent vertical and horizontal boundaries. The pixel values p1 (i1, j1), p2 (i2, j2), p3 (i3, j3), and p4 (i4, j4) at the four boundaries of the intersection are weighted by distances to determine the pixel values to be captured. As for interpolation processing using this Linear Interpolation, Viktor Varsa et al., “Non-normative error concealment algorithms”, ITU-Telecommunications Standardization Section, 14 ^th Meeting: Santa Barbara, CA, USA, 21-24 September, It is described in detail in VCEG-N62 (Proposal), 2001.

  On the other hand, when the type of the missing macroblock is estimated to be “Edge”, for example, interpolation processing using Directional Interpolation is performed. In the interpolation processing using Directional Interpolation, a gradient vector representing an edge pattern that will be included in the missing macroblock is calculated from a reference area adjacent to the missing macroblock. Then, in consideration of the direction of the calculated gradient vector, for example, as shown in FIG. 11, the pixel values p1 (i1, j1) and p2 (i2, j2) at the boundary are weighted by the distances d1 and d2, The pixel value p (i, j) to be captured is determined.

  In this embodiment, since the gradient vector is already calculated in the second classification process S422 described above, the interpolation pixel value can be calculated using the calculated gradient vector. is there. For interpolation processing using Directional Interpolation, see O. Nemethova, A. Al Moghrabi, M. Rupp, “Flexible Error Concealment for H.264 Based on Directional Interpolation”, Proc. Of the 2005 International Conference on Wireless Networks Communications and It is described in detail in Mobile Computing, Maui, Hawaii, USA, June, 2005.

  Further, when the type of the missing macroblock is estimated as “Texture”, for example, an interpolation process using Searching the best match block is performed. In this interpolation process, a macroblock that is considered to be highly similar is identified by searching for a macroblock that minimizes the value of the cost function prepared from the periphery of the missing macroblock. Then, the interpolation value is obtained by copying the pixel value of the specified macroblock. For interpolation processing using this Searching the best match block, see Zhang Rongfu, et. Al, “Content-Adaptive Spatial Error Concealment for Video Communication”, IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, pp. .335-340, Feb. 2004.

  The interpolated pixel value of the missing pixel P (i, j) calculated by the missing region interpolation processing unit 350 is written at the corresponding pixel position in the picture frame held in the decoded picture buffer 400. Thereafter, similarly, the concealment processing is performed by the concealment unit 300 for all the macroblocks including the missing pixel P (i, j), whereby the interpolation pixel value of the missing pixel P (i, j) is calculated. , And written in the corresponding pixel position in the picture frame.

  As described above in detail, in the first embodiment, when the type of the reference area adjacent to the missing macroblock is classified in the concealment unit 300, first, the pixels constituting the reference area by the first classification processing S421. A feature vector 1 representing the distribution state of the luminance value of is obtained, and it is determined based on the feature vector 1 whether it is “Smoothness” or “Texture”. Then, only when the type of the reference area cannot be classified by the first classification process, the process proceeds to the second classification process S422, and the gradient of the pixels existing in the reference area is determined by the second classification process S422. A feature quantity vector 2 to be expressed is calculated, and based on the calculated feature quantity vector 2 and the feature quantity vector 1, whether the type of the reference region is “Smoothness”, “Edge”, or “Texture” Judgment is made.

  Specifically, in the first classification process, deg_max_rate representing the ratio of the maximum frequency deg_max to the number N of pixels in the reference region is calculated, and the maximum value of the luminance value of the pixels included in the reference region is calculated. The range representing the difference between the value and the minimum value is calculated. Then, it is determined whether the reference area is “Smoothness” by comparing the range with a threshold value Th_r, and the reference area is “Texture” by comparing the deg_max_rate with a threshold value Th_d. Judgment is made whether there is.

  Therefore, when it is clearly determined as “Smoothness” or “Texture” in the first classification processing S421, the calculation processing of the gradient vector with a large amount of calculation becomes unnecessary. For this reason, compared to the case where the feature amount representing the gradient of the pixel value in the reference region is always calculated and the image type is determined based on the feature amount representing the gradient, the image type of the reference region is determined. It is possible to reduce the necessary calculation amount and reduce the overhead of concealment processing. This effect is extremely effective in smoothly performing the decoding process of the image data stream in a portable terminal whose processing capability of the decoder is lower than that of a personal computer or a television receiver.

  If the type of the image of the reference area cannot be determined in the first classification process S421, the type determination based on the gradient vector of the pixel value is performed in the second classification process S422. For this reason, even when accurate type determination cannot be performed from the feature value representing the distribution of pixel values, accurate classification can be performed based on the feature value representing the gradient of the pixel value.

(Second Embodiment)
In the second embodiment of the present invention, when the type of the reference area adjacent to the missing macroblock is classified, the feature vector 1 representing the distribution state of the luminance values of the pixels constituting the reference area is obtained by the first classification process. Based on this feature vector 1, it is determined whether it is “Smoothness” or “Texture”. If it is neither “Smoothness” nor “Texture”, it is regarded as “Edge”. Is.

  FIG. 12 is a flowchart showing the reference area classification processing procedure and its processing contents according to the second embodiment of the present invention. Since the configuration of the decoder is the same as that described with reference to FIGS. 2 to 4 in the first embodiment, the description of FIGS. 2 to 4 is also extended in the second embodiment. I do.

  The missing area separation unit 340 first proceeds to the first classification process S111, and in step S1111, the frequency distribution of the luminance value of the pixel is examined for the reference area selected from the peripheral area buffer 330, and the maximum value deg_max is detected. . Then, deg_max_rate representing the ratio of the detected maximum value deg_max to the number N of pixels in the reference region is calculated. At the same time, the missing region separation unit 340 detects the maximum value and the minimum value of the luminance values of the pixels included in the selected reference region, and calculates a range representing the difference between the detected maximum value and minimum value. .

  Next, the missing region separation unit 340 compares the difference range between the calculated maximum value and the minimum value of the calculated brightness value with a preset threshold value Th_r in step S1112. If the result of this comparison is range <Th_r, it is determined in step S1114 that the selected reference area is “Smoothness”. If there is a reference area that has not yet been selected, the process proceeds to the next reference area classification process. On the other hand, if the result of the comparison is range> = Th_r, deg_max_rate representing the ratio of the calculated maximum value deg_max is compared with a preset threshold Th_d in step S1113. If the result of this comparison is deg_max_rate <Th_d, it is determined in step S1115 that the reference area is “Texture”, and the process proceeds to classification processing for the next reference area that has not yet been selected.

On the other hand, it is assumed that deg_max_rate> = Th_d as a result of the comparison in step S53. In this case, the missing area separation unit 340 proceeds to the second classification process S112, and regards the reference area as “Edge” in step S1121. If there is a reference area that has not yet been selected, the process proceeds to the next reference area classification process.
It should be noted that the procedure and content of the classification process of the missing macroblock to be interpolated using the classification result of the reference region and the interpolation process of the missing pixel in the missing block based on the classification result are the same as those in the first embodiment. The same.

Below, the specific example of the evaluation result by the classification | category process procedure and content shown in the said FIG. 12 is shown.
Now, the threshold values Th_r and Th_d are set to values Th_r = 32 and Th_d = 0.3 obtained experimentally as illustrated in FIG. 12, and the size of the reference area is set to 16 × 8 (8 × 16) pixels. The scale of the horizontal axis (luminance value) when obtaining the frequency distribution is set to 8. Under this classification condition, as shown in FIG. 13, there are missing macroblocks having an image size of QCIF (Quarter Common Intermediate Format) and a bit rate of 128 kbps. Assume that classification is performed for adjacent reference regions.

  The classification result at this time is shown in FIG. In the figure, each item has a macroblock address (upper left corner address is MbX = 0, MbY = 0), a reference area position (upper: upper, lower: lower, left: left, right: right), The conditions used for the classification (conditions illustrated in FIG. 12), the classification results, and the classification of the target area by the subjectivity (the circles indicate that the classification result and the classification by the subjectivity match).

  In this evaluation result, the same result as the subjective classification is obtained by performing the first classification processing S111 on eight of the 35 reference areas to be classified, and the gradient is about 23%. The amount of calculation related to calculation is reduced. There are some areas that differ from the subjective classification results, but this is an area that is originally difficult to classify only with range and deg_max_rate, such as the presence of multiple edge patterns. As shown in FIG. 6 of the first embodiment, the reference region in which the plurality of edge patterns exist is classified by using the gradient information together in the second classification process S422, so that high-accuracy classification can be performed. It becomes possible. Further, when the reference region sizes to be classified are the same, it is considered that the correlation between pixel values included in one region increases as the image size increases. This makes it easy to classify “Smoothness”, “Edge”, and “Texture”.

  As described above, in the second embodiment, the process of calculating the gradient vector of the pixel in the reference area is performed only when the reference area is regarded as “Edge” in the first classification process S111. Therefore, the opportunity for calculating the gradient vector of the reference region is further reduced, thereby further reducing the calculation processing burden of the decoder and performing the decoding process more smoothly.

(Other embodiments)
In the above embodiment, the case where the concealment unit 300 is provided in the decoder 22 has been described as an example. However, the application program for realizing the function of the concealment unit 300 may be executed by the CPU of the control module 21. Good. Further, the functions in the concealment unit 300 may be divided so that a part is executed by the decoder 22 and the other part is executed by the control module 21.

  Furthermore, in the above-described embodiment, the case where the ITUT H.264 method is adopted as an image coding method has been described as an example. However, any coding method may be used as long as the image data is coded in block units. The present invention can also be applied to image data. Further, the present invention is applicable not only to moving images but also to still images as long as the image data is encoded using this type of encoding method.

  Further, in the above embodiment, the case where image data received by one-segment broadcasting of terrestrial digital broadcasting is decoded has been described as an example. However, the present invention is not limited thereto, and the present invention can be similarly applied to the case of decoding image content data downloaded from a content server on the web or image content data recorded on a recording medium such as a memory card or a high-definition DVD. It is.

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, you may combine suitably the component covering different embodiment.

It is a block diagram which shows the structure of the portable terminal provided with the moving image decoding apparatus concerning 1st Embodiment of this invention. It is a block diagram which shows the function structure of the decoder of the portable terminal shown in FIG. FIG. 3 is a block diagram showing a functional configuration of a missing region separation unit in the concealment unit of the decoder shown in FIG. 2. It is a block diagram which shows the function structure of the missing area interpolation process part in the concealment part of the decoder shown in FIG. It is a flowchart which shows the procedure and content of the reference area classification | category process by the concealment part provided in the decoder shown in FIG. 6 is a flowchart showing more specifically the content of the reference area classification process shown in FIG. 5. It is a figure which shows the relationship between the missing macroblock and its reference area. FIG. 3 is a diagram showing an example of flag information stored in a missing area information buffer provided in the decoder shown in FIG. 2. It is a figure which shows an example of the positional relationship of the missing macroblock and the macroblock selected from the reference area used in order to repair the said macroblock. It is a figure which shows an example of the interpolation process based on the adjacent macroblock determined as "Smoothness". It is a figure which shows an example of the interpolation process based on the adjacent macroblock determined as "Edge". It is a flowchart which shows the procedure and content of the reference area classification | category process by the moving image decoding apparatus concerning 2nd Embodiment of this invention. It is a figure which shows an example of the image for demonstrating the reference area classification | category process shown in FIG. It is a figure which shows an example of the information showing the classification result of the image shown in FIG. It is a figure which shows an example of the macroblock determined as "Smoothness". It is a figure which shows an example of the macroblock determined as "Edge". It is a figure which shows an example of the macroblock determined as "Texture".

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Wireless unit, 2 ... Baseband unit, 3 ... User interface unit, 4 ... Memory | storage unit, 5 ... Digital broadcast receiving unit, 11 ... Mobile communication antenna, 12 ... Antenna duplexer (DUP), 13 ... Reception circuit ( RX), 14 ... synthesizer (SYN), 15 ... transmission circuit (TX), 21 ... control module, 22 ... decoder, 23 ... encoder, 31 ... camera (CAM), 32 ... microphone, 33 ... input device, 34 ... display Device 35 ... Speaker 100 ... Decoding unit 110 Error detection unit 120 Switch control unit 200 Missing region information memory 300 Concealment unit 310 Concealment control unit 320 Peripheral region information acquisition unit , 330 ... Peripheral area buffer, 340 ... Missing area classification part, 341 ... Statistical information calculation part 342 ... Gradient information calculation unit, 343 ... Peripheral region analysis unit, 344 ... Missing region classification determination unit, 345 ... Switching unit, 350 ... Missing region interpolation processing unit, 351 ... Smoothness region interpolation processing unit, 352 ... Edge region interpolation processing unit 353 ... Texture region interpolation processing unit, 354, 355 ... switching unit, 400 ... decoded picture buffer.

Claims (3)

In an image decoding apparatus that decodes image data encoded for each block and outputs decoded image data,
Means for identifying a block to be repaired that includes a missing pixel that has not been correctly decoded by the decoding process from the decoded image data;
Means for selecting a reference area adjacent to the identified block to be repaired and correctly decoded by the decoding process;
Detecting means for detecting, for each selected reference area, a feature value of a dispersion state of pixel values in the reference area using a statistical method;
Based on the feature value of the dispersion state of the detected pixel values, the type of the image of the reference area is the first type in which the pixel values are uniformly distributed, or the second type having a large variation in pixel values. First determination means for determining whether the pattern is a type or a third type including an edge pattern;
Means for estimating whether the type of the block to be repaired is the first type, the second type, or the third type based on the determination result of the type of the reference region;
Interpolation corresponding to the estimated type from among a plurality of interpolation processing means prepared in advance in association with the first, second and third types based on the type estimation result for the block to be repaired Means for selecting a processing means;
An image decoding apparatus, comprising: means for performing interpolation processing on the restoration target block including the missing pixel based on the selected interpolation processing means. The detection means includes
Means for calculating a frequency distribution of pixel values included in the reference area, calculating a maximum value of the frequency, and calculating a ratio of the calculated maximum value of the frequency to the number of pixels in the reference area;
Means for calculating a difference between a maximum value and a minimum value of pixel values included in the reference region;
The first determination means includes
The difference between the maximum value and the minimum value of the calculated pixel values is compared with a preset first threshold value, and when the difference is less than the first threshold value, the image type of the reference region First comparing means for determining the first type as
As a result of the comparison by the first comparison means, when the difference is equal to or greater than a first threshold value, the ratio of the calculated maximum value of the frequency is compared with a preset second threshold value, When the ratio of the maximum value of frequency is less than a preset second threshold, the type of the image in the reference area is determined as the second type, and in other cases, the type of the image in the reference area is The image decoding apparatus according to claim 1, further comprising: a second comparison unit that determines the third type. A gradient calculating unit that calculates a feature amount representing a gradient of a pixel value in the reference region when the image type of the reference region is not determined by the first determining unit;
Based on the calculated feature value representing the gradient of the pixel value, the image type of the reference region is the first type, the second type, or the third type. The image decoding apparatus according to claim 1, further comprising second determination means for determining whether or not there is any.

Images