JP2006196959A - Moving picture data receiver - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving picture data receiver capable of realizing interpolation of an error image caused in a transmission path in a way of not causing unnaturalness without the need for preparing a special contrivance to a transmission side. <P>SOLUTION: The moving picture data receiver is configured to include: a reception means for receiving coded moving picture data; an error detection means for detecting errors of the received data caused on a communication path; an image interpolation means for interpolating image data at an error location detected by the error detection means; a decoding means for decoding the received data; an area division means for applying area division to the decoded image; a feature quantity calculation means for calculating a feature quantity by each divided area; an area detection means for detecting areas such as character areas, bar code and QR code areas not including a natural picture by using the calculated feature quantity; and an interpolation processing switching means for switching an interpolation method by the image interpolation means in response to a result of the detection by the area detection means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention provides a moving image receiving / decoding device that receives and decodes encoded moving image information in real time, and has a function of improving a decrease in image quality caused by data error or omission in a transmission system. The present invention relates to a moving image data receiving apparatus.

  As an application protocol for transmitting moving images in real time using a network system such as the Internet, there is RTP (Real Time Protocol). In order to transmit data with low delay in real time using RTP, it is common to use UDP (User Datagram Protocol). In UDP, since it does not know about packet data loss, discard due to delay, order reversal, etc. that occur on the network, the processing is left to the application. When data loss or discard occurs during moving image transmission, interpolation processing is performed on the receiving device side using temporal and spatial similarity of images (see, for example, Patent Document 1).

  Since the image quality deteriorates by simply interpolating packets discarded on the UDP by image processing on the receiving device side, FEC (Forward Error Correction) that can correct some errors by increasing redundancy, and a data retransmission request A system that dynamically switches and controls ARQ (Auto Repeat reQuest), which is a process, has also been proposed (for example, see Patent Document 2).

  There has also been proposed a system that reduces image degradation due to packet discarding by dividing an image into regions and transmitting the resolution by changing the hierarchical encoding resolution for each region (see, for example, Patent Document 3).

Japanese Patent Laid-Open No. 07-111654 (pages 5-6, FIG. 1) Japanese Patent Laying-Open No. 2003-179580 (pages 19 to 21, FIG. 14) Japanese Unexamined Patent Publication No. 2004-096309 (pages 5-7, FIG. 2)

  In the above conventional example, images that are not natural images such as characters, barcodes, and QR codes are processed with the same algorithm as natural images such as landscapes. For example, interpolation from adjacent blocks in the same frame is performed. When is selected, there is a case where interpolation that is clearly recognized as an error by human eyes is performed. Further, in the example of Patent Document 3, a method of changing the encoding method for each area of the image is adopted, but this method requires special processing for switching the encoding process for each area on the transmitting device side. There is a problem that it cannot be applied when receiving image data from a general content transmission server.

  The present invention has been made in view of the above technical problem, and an object of the present invention is to transmit an error image generated on a transmission path to a transmission side in a moving image data receiving apparatus that displays moving image data transmitted in real time. An object of the present invention is to provide a moving image data receiving apparatus capable of realizing interpolation that does not become unnatural without preparing a special mechanism.

  In order to achieve the above object, in the first aspect of the present invention, a receiving means for receiving encoded moving image data, an error detecting means for detecting an error of the received data on the communication path, and an error detecting means An image interpolating means for interpolating the image data at the error position, a decoding means for decoding the received data, an area dividing means for dividing the decoded image into areas, and a feature amount for each divided area is calculated. Feature amount calculating means, region detecting means for detecting a region that is not a natural image such as a character, barcode or QR code using the calculated feature amount, and the image interpolating means according to the detection result of the region detecting means The moving image data receiving apparatus is configured to include an interpolation processing switching means for switching the interpolation method in (1).

  The invention according to claim 2 has frame storage means for storing one frame of the image decoded by the decoding means in the invention according to claim 1, and the area dividing means is stored by the frame storage means. The image data is used, and the switching by the interpolation processing switching unit is performed using the detection result of the region detecting unit detected based on the image one frame before.

  According to a third aspect of the present invention, in the second aspect of the present invention, the interpolation processing switching means searches the image data of the previous frame for a region that matches the periphery of the region to be interpolated as an interpolation method for a region that is not a natural image. Then, an interpolation method for interpolation is selected.

  According to a fourth aspect of the present invention, in the second aspect of the invention, there is provided a retransmission request means for sending a retransmission request to the transmission side for the error area, and the interpolation processing switching means is for the area that is not a natural image. In this case, a retransmission request by the retransmission request unit is selected, and image interpolation processing by the interpolation processing unit is selected for other regions.

  According to the present invention, in a moving image data receiving apparatus that displays moving image data that is transmitted in real time, an error image generated on a transmission path is interpolated without preparing a special mechanism on the transmission side. It becomes possible to do.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

<Embodiment 1>
FIG. 1 is a block diagram showing a schematic configuration of a moving image data receiving apparatus according to Embodiment 1 of the present invention.

  In FIG. 1, reference numeral 100 denotes a wirelessly connected communication interface which receives encoded moving image data transmitted in real time. The moving image data here is compressed according to the MPEG4 (Moving Picture Experts Group phase 4) standard, and is transmitted as an RTP packet using UDP. In addition, FEC (Forward Error Correction) is used for communication data as an error correction method corresponding to packet errors, and some packet errors can be corrected. The FEC decode processing unit 110 performs error correction by this FEC and deletion of the added redundant data. The output of the FEC decoding processing unit 110 is sent to the image decoding processing unit 130 and also to the error detection unit 120. The error detection unit 120 detects an error that the FEC decoding processing unit 110 could not complete error correction. The errors here include not only data destruction on the communication path but also those whose delay time is longer than specified.

  The image decoding processing unit 130 decodes MPEG4-compressed image data. The decoded result is sequentially stored in the current frame buffer 140. At this time, motion vector data is simultaneously written as the attached information. The contents of the current frame buffer 140 are copied to the previous frame buffer 150 before decoding of the data of the next frame is started.

  The area determination unit 160 determines an interpolation method when an error occurs in the next frame for each area divided into blocks based on the contents of the previous frame buffer 150 and writes the interpolation method in the interpolation method table 170. Detailed processing of the area determination unit 160 will be described later with reference to the drawings.

  The image interpolation processing unit 180 receives the error position information from the error detection unit 120, performs interpolation processing using the contents of the current frame 140 and the previous frame 150 according to the information on the corresponding position in the interpolation method table, and the corresponding position of the current frame 140. Write to. Detailed processing of the image interpolation processing unit 180 will be described later with reference to the drawings.

  After the decoding by the image decoding processing unit 130 and the interpolation processing by the image interpolation processing unit 180 are finished, the contents of the current frame buffer 140 are displayed on the display panel 191 by the display control unit 190.

  FIG. 2 is a diagram showing the internal configuration of the area determination unit 160. FIG. 3 is a diagram showing actual images and processing results.

  The block cutout unit 200 reads out the image in the previous frame buffer 150 while cutting it out into blocks. The unit of this divided block is set larger than the macro block which is a processing unit in MPEG4. A state of being cut out by the block cutout unit 200 is shown in FIG. In FIG. 3A, the data are read in the order of blocks 30, 31, 32, and 33.

  The data cut out by the block cutout unit 200 is sent to the contrast calculation unit 210 and the run length measurement unit 220, and the feature amount is calculated.

  The contrast calculation unit 210 performs processing for further dividing the block into smaller areas and accumulating the maximum density difference in the smaller areas.

  The run length measurement unit 220 measures continuous run lengths of the same color in the vertical, horizontal, and diagonal directions.

  The block classification unit 230 determines that a region with a high contrast and a large number of short runs is a character region based on information from the contrast calculation unit 210 and the run length measurement unit 220, outputs “1”, and the contrast is low. The area is determined as an inconspicuous area with low importance and “3” is output, “2” is output for the other areas and written in the corresponding position of the interpolation table 170. An example of the determination result is shown in FIG.

  FIG. 4 is a diagram showing the internal configuration of the image interpolation unit 180.

  The motion vector calculation unit 400 receives the error position information from the error detection unit 120, reads the motion vector information of the non-errored macroblock around the error position from the current frame buffer 140 based on the position information, and moves the error position. Calculate the vector. The motion vector is calculated by taking a weighted average of surrounding motion vectors.

  Based on the motion vector calculated by the motion vector calculation unit 400, the matching search unit 410 uses the image data around the error position in the current frame buffer 140 and the image around the position indicated by the motion vector in the previous frame buffer 150. Are searched for a position having the highest similarity.

  According to the value indicating the interpolation method selection information of the block including the error position in the interpolation method table 170, the selector 420 is error position information from the error detection unit 120 in the case of “3”, and in the case of “2”. The position information from the motion vector calculation unit 400 is selected. In the case of “1”, the position information from the matching search unit is selected. Also, according to the value indicating the interpolation method selection information, when “3”, the motion vector calculation unit 400 and the matching search unit 410 are disabled, and when “2”, the matching search unit 410 is disabled. .

  In the block transfer unit 430, image data is read from the previous frame buffer 150 in accordance with the position information output from the selector 420 and written in the corresponding error position of the current frame data 140.

  As a result, in the inconspicuous area “3”, which is less important, the same position as the error position, in the normal area “2”, the position indicated by the calculated motion vector, and in the character area “1”, the calculated motion Interpolation processing is performed on the image of the previous frame buffer 150 at the position detected by the matching processing around the vector.

<Embodiment 2>
In the present embodiment, a retransmission request processing unit 500 is added as compared with the first embodiment. In response to the value indicating the interpolation method selection information of the block including the error position in the interpolation method table 170, the retransmission request processing unit 500 resends error information to the transmitting device in the case of the character region “1”. Make a request. At this time, as in the first embodiment, interpolation by the image interpolation unit 180 is also performed at the same time, and when the retransmitted data is in time for the display timing of the display control unit 190, the retransmitted data is replaced.

  In the present embodiment, by limiting the retransmission processing to a case where a retransmission request for error information is included in the character region “1”, errors in regions that are likely to cause discomfort due to interpolation can be reduced.

<Other embodiments>
In the moving image data receiving apparatus described in the above embodiment, the moving image is compressed by MPEG4. The same effect can be obtained when other compression formats such as H.264 and Motion JPEG are used.

  Further, although the communication means for moving image data is wireless communication, it may be communication by wired connection or optical connection, and the communication protocol is not limited to RTP or UDP.

  Regarding the image interpolation process, various methods such as a method of interpolating from frequency components of surrounding data are known, and other interpolation methods can be replaced without impairing the effects of the present invention.

It is a block diagram which shows schematic structure of the moving image data receiver which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the area | region determination means of the moving image data receiver which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the area | region determination processing result of the moving image data receiver which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the image interpolation means of the moving image data receiver which concerns on Embodiment 1 of this invention. It is a block diagram which shows schematic structure of the moving image data receiver which concerns on Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Communication part 110 FEC decoding process part 120 Error detection part 130 Image decoding part 150 Previous frame buffer 160 Area determination part 170 Interpolation method table 180 Image interpolation process part 200 Block cut-out part 210,220 Feature amount calculation part 230 Area | region detection part 410 Matching Search unit 420 Interpolation method switching unit 500 Retransmission request unit

Claims (4)

  Receiving means for receiving the encoded moving image data; error detecting means for detecting an error on the communication path of the received data; image interpolating means for interpolating the image data at the error position detected by the error detecting means; Decoding means for decoding received data, area dividing means for dividing the decoded image into areas, feature quantity calculating means for calculating feature quantities for each of the divided areas, and characters using the calculated feature quantities A region detection unit that detects a region that is not a natural image, such as a barcode or a QR code, and an interpolation processing switching unit that switches an interpolation method in the image interpolation unit according to a detection result of the region detection unit. A moving image data receiving device.   A frame storing unit that stores one frame of the image decoded by the decoding unit; the region dividing unit uses image data stored by the frame storing unit; The moving image data receiving apparatus according to claim 1, wherein the moving image data receiving apparatus performs the detection using a detection result of the area detecting unit detected based on a previous image.   The interpolation processing switching means selects an interpolation method for searching and interpolating a region matching around the region to be interpolated from image data of the previous frame as an interpolation method for a region that is not a natural image. 2. The moving image data receiving device according to 2.   Retransmission request means for sending a retransmission request to the transmitting side for the error area, the interpolation processing switching means selects a retransmission request by the retransmission request means for an area that is not a natural image, and others 3. The moving image data receiving apparatus according to claim 2, wherein image interpolation processing by said interpolation processing means is selected for said area.

Images