JP3976975B2 - Image processing apparatus and method, and storage medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method for encoding and / or decoding image data, and a storage medium.
[0002]
[Prior art]
Conventionally, as a moving image encoding method, an encoding method such as Motion JPEG or Digital Video, which is an intra-frame (screen) encoding method, or H.264 using inter-frame (screen) predictive encoding is used. 261, H.H. Coding schemes such as H.263, MPEG-1 and MPEG-2 are known. The former intra-frame coding method is an independent coding method in units of frames, and is easy to manage frames, so that it is most suitable for an apparatus that requires editing of moving images and special reproduction. The latter interframe coding scheme has a feature that coding efficiency is high because interframe prediction is used.
[0003]
[Problems to be solved by the invention]
In transmission of moving images over a general telephone line or wirelessly, errors due to noise or radio wave interference occur. In the method using only intra-frame coding, even if an error occurs, the influence is closed for each frame and does not propagate to other frames. However, since only the intra-frame coding has low coding efficiency, it is difficult to transmit a high-resolution moving image over a telephone line or the like. On the other hand, in encoding using inter-frame prediction, a prediction frame is decoded using erroneous image data due to noise or the like, so that the error is propagated to other frames, resulting in degradation of image quality over a long period of time. There is a disadvantage of letting it.
[0004]
The present invention has been made in view of the above conventional example, and provides an image processing apparatus and method and storage medium capable of suppressing deterioration in image quality that occurs even if an error occurs and maintaining high coding efficiency. With the goal.
[0005]
  It is another object of the present invention to ensure that each block of a plurality of frames of image data is always intra-coded at regular intervals, so that intra-frame coding is performed at a cycle shorter than the frame interval for performing intra-frame coding. It is an object of the present invention to provide an image processing apparatus, a method thereof, and a storage medium that can suppress the propagation of image quality degradation due to transmission errors in inter-frame coding.
[0006]
[Means for Solving the Problems]
  In order to achieve the above object, the image processing apparatus of the present invention comprises the following arrangement. That is,An input means for inputting image information of a plurality of frames;
  Dividing means for dividing each of the plurality of frames input by the input means into blocks;
  Encoding means for encoding each block obtained by the dividing means in an intra-picture coding mode or an inter-picture coding mode;
  The encoding mode of frame I located at a certain number of frame intervals in the plurality of frames is set to the intra-screen encoding mode, and the encoding mode of other frames P is set to the intra-screen encoding mode or the inter-screen encoding mode. Encoding mode setting means for selectively setting the encoding mode;
  Storage means for storing a plurality of types of mode patterns periodically assigned to a plurality of successive frames P;
  Output means for multiplexing and outputting information indicating the mode pattern stored in the storage means to the plurality of encoded frames;
  The encoding mode setting means sets the encoding mode of each frame P based on the mode pattern, and each of the plurality of types of mode patterns includes a plurality of blocks that constitute one frame P in the screen. Information including whether to perform the inner coding mode or the inter-screen coding mode is included.
[0007]
  In order to achieve the above object, the image processing apparatus of the present invention comprises the following arrangement. That is,
  In the image information of a plurality of frames, the frame I positioned at a fixed number of frame intervals is encoded in the intra-frame encoding mode, and each block of the other frame P is set in the intra-screen encoding mode or the inter-screen encoding mode. Intra-screen encoding mode or inter-screen based on a plurality of types of mode patterns that are selectively used and encoded, and each block of the encoded frame P is periodically assigned to a plurality of consecutive frames P Input means for inputting a plurality of frames encoded in the encoding mode and a mode pattern corresponding to the frame P;
  Determining means for determining an error in the encoding mode by comparing the encoding mode of each block recorded in the header of the plurality of encoded frames and the corresponding mode pattern;
  Decoding means for correcting an error in the encoding mode based on the determination result notified from the determination means and decoding the plurality of encoded frames.
[0008]
  In order to achieve the above object, the image processing method of the present invention includes the following steps. That is,
  An input process for inputting image information of a plurality of frames;
  A dividing step of dividing each of the plurality of inputted frames into blocks;
  An encoding step of encoding each block obtained in the dividing step in an intra-screen encoding mode or an inter-screen encoding mode;
  The encoding mode of frame I positioned at a certain number of frame intervals in the plurality of frames is set to the intra-screen encoding mode, and the encoding mode of other frames P is set to the intra-screen encoding mode or the inter-screen encoding An encoding mode setting step for selectively setting the encoding mode;
  A storage step of storing a plurality of types of mode patterns periodically assigned to a plurality of successive frames P;
  An output step of multiplexing and outputting the information indicating the mode pattern stored in the storage step to the plurality of encoded frames;
  in frontThe encoding mode setting step sets the encoding mode of each frame P based on the mode pattern, and each of the plurality of types of mode patterns includes a plurality of blocks constituting one frame P in the screen. Information including whether to perform the inner coding mode or the inter-screen coding mode is included.
[0009]
  In order to achieve the above object, the image processing method of the present invention includes the following steps. That is,
  In the image information of a plurality of frames, the frame I positioned at a fixed number of frame intervals is encoded in the intra-frame encoding mode, and each block of the other frame P is set in the intra-screen encoding mode or the inter-screen encoding mode. Intra-screen encoding mode or inter-screen based on a plurality of types of mode patterns that are selectively used and encoded, and each block of the encoded frame P is periodically assigned to a plurality of consecutive frames P An input step of inputting a plurality of frames encoded in the encoding mode and a mode pattern corresponding to the frame P;
  A determination step of determining an error in an encoding mode by comparing an encoding mode of each block recorded in a header of the plurality of encoded frames and the mode pattern;
  And a decoding step of correcting an encoding mode error based on the determination result notified from the determination step and decoding the plurality of encoded frames.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings.
[0011]
[Embodiment 1]
FIG. 1 is a block diagram showing a configuration of an image coding apparatus according to Embodiment 1 of the present invention. In the present embodiment, H.264 is used as a moving image encoding method. A case where the H.263 encoding method is used will be described. For ease of explanation, the image size is described as QCIF (176 × 144), but the present invention is not limited to these.
[0012]
In FIG. 1, reference numeral 100 denotes an input terminal, which is a terminal for inputting moving image data to be encoded. Reference numeral 101 denotes a frame memory, which stores the input moving image data in units of frames, and outputs a signal 201 for notifying that data is stored in units of frames. A block extractor 102 reads out moving image data from the frame memory 101 in units of blocks that are encoding units, and outputs them to the subsequent subtractor 105 and motion compensator 104. A signal 202 indicating that the block has been cut out is output to the encoding mode setting unit 114. In the first embodiment, H.264 is used. Since the H.263 coding method is described as an example, a block that is a coding unit corresponds to a macro block. A frame memory 103 stores the decoded image data in units of frames. A motion compensator 104 determines a mode for encoding by performing motion compensation based on the contents of the frame memory 103 and the moving image data in units of blocks output from the block cutout unit 102, and outputs the predicted image data. To do. When this motion compensation is performed, the motion vector is also output.
[0013]
A subtractor 105 calculates a difference in units of pixels based on block-unit moving image data output from the block cutout unit 102 and prediction image data from the motion compensator 104 to obtain a prediction error. A DCT unit 106 DCT-transforms the prediction error or pixel value obtained by the difference unit 105. Reference numeral 107 denotes a quantizer that quantizes the DCT coefficient obtained by the DCT unit 106. A Huffman encoder 108 assigns a Huffman code to the quantization result obtained by the quantizer 107. Reference numeral 109 denotes an inverse quantizer, which dequantizes the output of the quantizer 107 to obtain a DCT coefficient. Reference numeral 110 denotes an inverse DCT, which inversely transforms the DCT coefficient inversely quantized by the inverse quantizer 109 and outputs data corresponding to a prediction error. An adder 111 adds the predicted image data from the motion compensator 104 and the prediction error from the inverse DCT unit 110 in units of pixels.
[0014]
Reference numeral 112 denotes a motion vector encoder which inputs and encodes the motion vector 203 from the motion compensator 104. Reference numeral 113 denotes a header encoder that generates and encodes a frame-based header or a macroblock-based header. This header contains H.264. According to the H.263 coding method, encoded data such as a frame start code, a frame encoding mode, and a macroblock encoding mode are included. An encoding mode setting unit 114 determines a frame encoding mode, a macroblock encoding mode, and the like. The encoding mode setting unit 114 will be described in detail with reference to FIG. Reference numeral 115 denotes a mode pattern memory which stores information on the encoding mode of the macroblock in the frame. Details of this will be described later. Reference numeral 116 denotes a multiplexer, which outputs the outputs of the Huffman encoder 108, the motion vector encoder 112, and the header encoder 113 to H.264. Multiplexing is performed according to the code format of the H.263 coding method. Reference numeral 117 denotes a terminal for outputting multiplexed encoded data.
[0015]
In such a configuration, each unit is initialized prior to the operation. The encoding mode setting unit 114 sets the leading frame in the input moving image data as a frame to be subjected to intraframe encoding, and sets the frame to be subjected to intraframe encoding every 132 frames. For other frames, the encoding mode is set so as to perform interframe encoding using the immediately preceding frame as a reference frame.
[0016]
FIG. 3 is a block diagram showing a detailed configuration of the encoding mode setting unit 114.
[0017]
In FIG. 3, an input terminal 150 receives a signal 201 for notifying that data is stored in units of frames from the frame memory 101. Reference numeral 151 denotes a frame counter, which counts the number of frames to be processed in accordance with the signal 201. This counter 151 is set to “0” at initialization. A frame mode setting unit 152 sets the frame mode in which the first frame and every 132th frame are frames to be intra-coded as described above. The output terminal 153 is a terminal that outputs a signal 204 indicating the frame mode to the header encoder 113. The input terminal 154 is a terminal for inputting a signal 202 for notifying that a block has been cut out from the block cutout unit 102. A block counter 155 counts the number of blocks from the head of the frame. The block counter 155 is reset to “0” at the head of the frame. An address generator 156 calculates an address on the mode pattern memory 115 and outputs it to the mode pattern memory 115 via the output terminal 157. The input terminal 158 is a terminal for inputting the mode signal 205 from the mode pattern memory 115. The latch 159 latches the mode signal 205 from the input terminal 158, and outputs it based on the signals 204 and 202 from the frame mode setting unit 152 and the input terminal 154.
[0018]
In such a configuration, the signal 201 input from the terminal 150 is input to the frame counter 151 and counted. The value counted by the frame counter 151 is supplied to the frame mode setting unit 152. When the remainder of the value obtained by dividing the count value by “132” is “1”, the frame mode setting unit 152 sets the encoding mode of the frame from the output terminal 153 as the mode for performing the intra-frame encoding. A mode signal 204 is output. Otherwise, a frame coding mode signal 204 indicating a mode for performing interframe coding is output from the output terminal 153.
[0019]
Here, in the case of a frame for which intraframe encoding is performed, the operation is as follows.
[0020]
When instructed to perform intraframe coding from the frame mode setting unit 152, the latch 159 always writes a value so that the macroblock performs intraframe coding. Then, in accordance with the block cutout signal 202 from the block cutout device 102, a signal 206 for performing intraframe coding is output from the terminal 160 to the motion compensator 104. At this time, the block counter 155 does not count the number of blocks. Also, the address generator 156 does not operate.
[0021]
In the case of a frame for performing interframe coding, the operation is as follows.
[0022]
The block counter 155 is reset to “0” before processing the frame. Each time the block cutout signal 202 is input from the block cutout unit 102, the block counter 155 counts. The count value by the block counter 155 is input to the address generator 156. The address generator 156 also receives the frame count value from the frame counter 151. In this way, the address generator 156 generates an address for reading the mode value from the mode pattern memory 115.
[0023]
2B to 2I are diagrams showing an example of patterns stored in the mode pattern memory 115. FIG.
[0024]
In FIG. 2, a QCIF image is taken as an example, and each square represents a macroblock. Black squares in the figure represent the positions of macroblocks that instruct the corresponding macroblocks to always perform intraframe coding regardless of the determination by the motion compensator 104. Here, if the count value of the frame counter 151 in a certain frame is divisible by “8”, the pattern of FIG. 2B is obtained. If the remainder is “1”, the pattern of (c) is obtained, and the remainder is “2”. If there is a pattern of (d), if the remainder is “3”, the pattern of (e), if the remainder is “4”, the pattern of (f), if the remainder is “5” (g) If the remainder is “6”, the pattern (h) is selected, and if the remainder is “7”, the pattern (i) is selected.
[0025]
This indicates that each block is always intra-frame encoded in units of 8 frames in the thick frame shown in FIG. That is, according to the count value of the block counter 155, if the cell corresponding to the corresponding macroblock is white, the mode determined by the motion compensator 104 is used. A mode signal 206 is output so as to be in the mode.
[0026]
The mode signal of the macro block designated by the address generator 156 is read from the mode pattern memory 115, inputted via the terminal 158, and outputted from the terminal 160 to the motion compensator 104 via the latch 159.
[0027]
Returning to FIG. 1 again, the operation of FIG. 1 will be described.
[0028]
When moving image data is sequentially input from the terminal 100, the input moving image data is stored in the frame memory 101. When image data for one frame is accumulated in the frame memory 101, the encoding mode setting unit 114 is notified by the signal 201 that the image data of one frame has been stored. Thus, the encoding mode setting unit 114 determines whether to perform intraframe encoding or interframe encoding for one frame of image data stored in the frame memory 101.
[0029]
First, a case where intra-frame coding is performed will be described.
[0030]
The block cutout unit 102 extracts image data in units of macroblocks from the frame memory 101 and inputs them to the motion compensator 104 and the difference unit 105. At this time, the motion compensator 104 does not perform motion compensation on the macroblock, but outputs the intraframe coding mode as the macroblock coding mode, and the difference unit 105 and the adder 111 set all pixel values. In contrast, “0” is output.
[0031]
The data of the macroblock cut out by the block cutout unit 102 is subtracted by “0” by the difference unit 105 and is input to the DCT unit 106 as it is and DCT transformed, and the DCT coefficient is quantized by the quantizer 107. The result is input to the Huffman encoder 108 and the inverse quantizer 109. The Huffman encoder 108 is an H.264 encoder. A Huffman code determined by the H.263 coding method is assigned to the quantization result and output to the multiplexer 116.
[0032]
Also, the quantization result input to the inverse quantizer 109 is inversely quantized to become a DCT transform coefficient, which is converted to a prediction error by the inverse DCT device 110. “0” is added to the prediction error (pixel value) by the adder 111 and stored in a predetermined position in the frame memory 103.
[0033]
At this time, since the motion compensator 104 operates in the intra-frame encoding mode, the motion vector encoder 112 does not operate. The header encoder 113 encodes the frame start code, the frame encoding mode indicating that intra-frame encoding has been performed, the encoding mode of each macroblock, and the like, and outputs the result to the multiplexer 116. Multiplexer 116 is connected to H.264. The encoded data are multiplexed and output from the output terminal 117 in the order determined by the H.263 encoding method.
[0034]
Next, a case where interframe coding is performed will be described.
[0035]
The block extractor 102 extracts macroblock unit data from the frame memory 101 and inputs the data to the motion compensator 104 and the differencer 105. The motion compensator 104 determines whether or not the corresponding macroblock is a macroblock for which intraframe coding is performed, based on the signal 206 from the coding mode setting unit 114, and if it is a macroblock that performs intraframe coding. Without performing motion compensation, the intra-frame coding mode is output to the header encoder 113 as the macroblock encoding mode, and “0” is output to the difference unit 105 and the adder 111 for all pixel values. .
[0036]
On the other hand, when the intra-frame coding is not necessarily performed, the motion compensator 104 performs motion compensation on the macro block to obtain a prediction error, compares the block data, and performs intra-frame coding on a macro block basis. Or whether to perform interframe coding. In the case of performing intraframe encoding, the intraframe encoding mode is output to the header encoder 113 as the macroblock encoding mode, and the subtractor 105 and the adder 111 receive “0” for all pixel values. Is output.
[0037]
On the other hand, when performing inter-frame encoding, the inter-frame encoding mode is output to the header encoder 113 as the macroblock encoding mode, and each prediction image data is output to the differentiator 105 and the adder 111. Further, the motion vector at that time is output to the motion vector encoder 112. The motion vector encoder 112 encodes this motion vector and outputs it to the multiplexer 116.
[0038]
The macro block data cut out by the block cutout unit 102 is subtracted from “0” or a predicted value by the difference unit 105, input to the DCT unit 106 and DCT transformed, and the DCT coefficient is quantized by the quantizer 107. And input to the Huffman encoder 108 and the inverse quantizer 109. The Huffman encoder 108 is an H.264 encoder. A Huffman code determined by the H.263 coding method is assigned to the quantization result and output to the multiplexer 116.
[0039]
Further, the quantization result input to the inverse quantizer 109 is inversely quantized to become a DCT transform coefficient, and is converted into a pixel value by the inverse DCT device 110. This pixel value is added with “0” or a predicted value by the adder 111 and stored in a predetermined position in the frame memory 103.
[0040]
The header encoder 113 encodes the frame start code, the frame encoding mode indicating that inter-frame encoding has been performed, the encoding mode of each macroblock, and the like, and outputs the result to the multiplexer 116. Multiplexer 116 is connected to H.264. Each encoded data is multiplexed in the order determined by the H.263 encoding method and output from the terminal 117.
[0041]
By such a series of selection operations, the encoding mode of each block can be easily determined so that each macroblock always performs intra-frame encoding at regular intervals (every 8 frames in the example of FIG. 2). Can do. In addition, it is possible to suppress the propagation of deterioration due to transmission errors in inter-frame coding at a cycle shorter than the frame interval for performing actual intra-frame coding and without significantly increasing the code amount. Further, the obtained encoded data is completely compliant with the standard system, and no special configuration is required on the decoding side.
[0042]
FIG. 4 is a flowchart for explaining a coding mode setting method by the coding mode setting unit 114 of the image coding apparatus according to Embodiment 1 of the present invention.
[0043]
First, in step S1, moving image data is input from the terminal 100 in units of frames, and the number of input frames is counted. In step S2, the input frame is divided into a plurality of blocks (macro blocks), and each macro block is cut out and input. In step S3, it is determined whether the input frame is the first frame or a frame that appears every 132nd frame. If so, the process proceeds to step S4, and the encoding mode of the image data of the frame is set to the intra-frame encoding mode.
[0044]
On the other hand, when it is not the first frame or every 132nd frame at step S3, that is, when it is not a frame that is unconditionally encoded in the intraframe encoding mode, the process proceeds to step S5. See if it is a frame. Specifically, the count value of the number of frames is divided by “8”, the mode pattern memory 115 is accessed based on the remainder, and the mode pattern data (see FIG. 2) stored in the memory 115 is read (see FIG. 2). Step S6).
[0045]
In step S7, it is checked whether the macroblock to be processed is black (1) or white (0) based on the read pattern data. If it is black, the process proceeds to step S9, and the coding mode of the block is determined. Is set to the intra-frame coding mode. On the other hand, if it is white, the process proceeds to step S10, and the coding mode of the block is set to the interframe coding mode.
[0046]
Here, in the case of the intraframe coding mode, the motion compensator 104 outputs “0” to the differentiator 105 and the adder 111, stops the operation of the motion vector encoder 112, and causes the header encoder 113 to Informs that it is intra-frame coding.
[0047]
In the case of the interframe coding mode, the motion compensator 104 outputs a prediction error to the difference unit 105 and the adder 111, outputs a motion vector to the motion vector encoder 112, and outputs to the header encoder 113. Informs that it is interframe coding. Then, the decoded image data output from the adder 111 is stored in the frame memory 103, the corresponding macroblock of the next frame is predicted based on the contents of the frame memory 103, and the difference becomes smaller. As described above, the position (motion vector) is obtained and the motion compensation interframe prediction is performed.
[0048]
In the first embodiment, the video encoding method is H.264. Of course, other encoding methods may be used.
[0049]
The configuration of the frame memory can be changed as appropriate according to the processing speed.
[0050]
As a block group, GOB may be used as a unit, and of course, a configuration straddling GOB may be used.
[0051]
In the above-described embodiment, the DCT transform is used for the orthogonal transform. However, the present invention is not limited to this. For example, a wavelet transform or the like may be used.
[0052]
[Embodiment 2]
FIG. 5 is a block diagram showing the configuration of the image coding apparatus according to Embodiment 2 of the present invention. In addition, the same number is attached | subjected about the component similar to above-mentioned Embodiment 1, and those detailed description is abbreviate | omitted.
[0053]
Reference numeral 201 denotes a mode pattern memory, which is different from the mode pattern memory 115 of the first embodiment in the storage contents and the output destination. Reference numeral 202 denotes a header encoder, to which the function of encoding and outputting the contents of the pattern memory 201 is added to the header encoder 113 of the first embodiment. A multiplexer 203 is added with a function of multiplexing encoded data representing the contents of the pattern memory 201.
[0054]
In the second embodiment, as in the first embodiment described above, H.264 is used. A description will be given taking the H.263 coding method as an example.
[0055]
Prior to this encoding, the contents of the mode pattern memory 201 are encoded by the header encoder 202.
[0056]
The contents of the mode pattern memory 201 according to the second embodiment are shown in FIG.
[0057]
In FIG. 6 (a), the numbers from (1) to (13) are written in each square obtained by dividing the frame, and each of them is a group of macroblocks that are periodically intra-coded. Yes. For example, a macroblock marked with (1) is always coded in an intraframe coding mode with 8 frames in a frame to be subjected to interframe coding. The same applies to the blocks (2) to (12) below. Since there are only three blocks marked with (13), they are intra-frame encoded with a period of 3 frames. The situation is shown in FIGS. 6B to 6I as in FIG. 2 of the first embodiment.
[0058]
First, the header encoder 202 sets a macroblock (black) for which intra-frame encoding is always performed to “1”, and sets a macroblock (white) other than that to “0”, and runs each frame in the order in which the macroblocks are arranged. Perform length coding. These encoded data are stored in the H.264 of the first frame. It is inserted as preliminary insertion information (PEI) and spare information (PSPARE) of the H.263 coding system, and multiplexed by the multiplexer 203 according to a predetermined format.
[0059]
The encoding of the moving image data according to the second embodiment is performed in the same manner as in the first embodiment. That is, normal intra-frame coding is performed for a frame to be subjected to intra-frame coding, and in a frame to be subjected to inter-frame coding, the coding mode setting unit 114 determines the mode pattern memory from the frame count value and the block count value. 201, it is determined whether or not the macroblock to be encoded should be subjected to intraframe encoding. When the intraframe encoding is not necessarily performed, the encoding target is determined based on the determination of the motion compensator 104. The encoding mode of the macroblock is determined. The data encoded in each encoding mode in this manner is multiplexed by the multiplexer 203 and output from the terminal 117.
[0060]
As described above, according to the second embodiment, it is possible to notify the decoding side of the position of the macroblock subjected to intraframe coding. This makes it possible to know the position of the macroblock coded by the intraframe coding mode that can cut off the propagation of errors even when degradation due to transmission path errors occurs. There is an effect that correction can be made easily.
[0061]
Further, by adopting the pattern data configuration as shown in FIG. 6, it is possible to randomly arrange a block group of macroblocks for which intraframe coding is always performed, and to make the period of the block in which intraframe coding is performed conspicuous. There is an effect that can be eliminated.
[0062]
In addition, the macroblock to be encoded in the intraframe encoding mode is appropriately present in one frame, thereby widening the interval between the intraframe encoded frames even in the encoding schemes such as MPEG-1, 2, 4 and the like. Therefore, the encoding efficiency is not deteriorated.
[0063]
[Embodiment 3]
FIG. 7 is a block diagram showing the configuration of the image decoding apparatus according to Embodiment 3 of the present invention.
[0064]
In FIG. 7, 300 is a terminal for inputting encoded data, 301 is a separator, and each encoded data is separated for passing to each subsequent decoder. Here, the encoded data input from the terminal 300 will be described with reference to an example in which the encoded data is generated by the encoding apparatus according to the second embodiment and input via a transmission path that may cause an error on the transmission path. . The operation of separator 301 is the reverse operation of multiplexer 203 in the second embodiment.
[0065]
Reference numeral 302 denotes a header decoder that decodes a header in frame units or a header in macroblock units. The operation of the header decoder 302 is the reverse operation of the header encoder 202 in the second embodiment. Reference numeral 303 denotes a mode pattern memory, and 304 denotes an encoding mode determination unit. These operate in the same manner as the mode pattern memory 201 and the encoding mode setting unit 114 in the second embodiment.
[0066]
Reference numeral 305 denotes an error determiner that detects an error in the encoded data. Reference numeral 306 denotes a motion vector decoder that decodes a motion vector, and reference numeral 309 denotes a Huffman decoder that decodes a Huffman code and obtains a quantized DCT coefficient. These operate in reverse to the motion vector encoder 112 and the Huffman encoder 108 in the second embodiment.
[0067]
Reference numerals 307 and 314 denote frame memories for storing decoded images. A motion compensator 308 generates a predicted image in units of macroblocks from the frame memory 307 using the decoded motion vector. Reference numeral 310 denotes an inverse quantizer that performs inverse quantization on the Huffman-decoded code to obtain DCT coefficients, and reference numeral 311 denotes an inverse DCT apparatus that obtains a pixel value or a prediction error value from the DCT coefficients. Reference numeral 312 denotes an adder that adds the predicted image and the decoded prediction error. Reference numeral 313 denotes an error corrector that corrects image data in accordance with the error detected by the error determiner 305. Reference numeral 315 denotes an output terminal for outputting the reproduced decoded image.
[0068]
In such a configuration, first, each part is initialized prior to the operation. The encoded data input via the terminal 300 is input to the separator 301. This separator 301 outputs the encoded data related to the header of the frame and macroblock and the encoded data obtained by encoding the contents of the mode pattern memory 201 of FIG. 5 to the header decoder 302, and the encoded data related to the motion vector is output. The data is output to the motion vector decoder 306, and the Huffman encoded data is separated and output to the Huffman decoder 309, respectively.
[0069]
The header decoder 302 decodes the encoded data input from the separator 301, and decodes each frame, each header of the macroblock, and the frame period and position of the intrablock encoded macroblock. Based on the decoding result, these patterns are stored in the mode pattern memory 303. This pattern is, for example, a pattern as shown in FIG.
[0070]
At the time of decoding each frame, the header decoder 302 decodes the header of the frame and knows by decoding whether the frame is encoded in the intra-frame encoding mode or the inter-frame encoding mode.
[0071]
Here, when the frame is intra-frame encoded, since each macroblock of the frame is intra-frame encoded, the motion vector decoder 306 does not operate, and the motion compensator 308 also has each pixel value. In contrast, “0” is always output to the adder 312. The Huffman decoder 309 decodes input encoded data and obtains quantized DCT coefficients. If an error is detected during the Huffman decoding, the information is output to the error corrector 313. For example, the case where 64 or more DCT transform coefficients appear as a result of decoding, or a case where an attempt is made to decode a nonexistent code corresponds to this error detection. In this case, decoding is interrupted to the beginning of the next GOB, and the error corrector 313 is notified that the encoded data has been skipped.
[0072]
On the other hand, when there is no error, the quantized DCT coefficient is inversely quantized by the inverse quantizer 310 and then converted into a pixel value by the inverse DCT 311. The pixel value thus obtained is added with “0” by the adder 312 and input to the error corrector 313. The error corrector 313 reads and corrects the skipped macroblock data from the frame memory 307 to substitute the macroblock value of the previous frame if there is an error. On the other hand, if there is no error, the input is output as it is. The pixel data output from the error corrector 313 is stored in predetermined positions in the frame memory 314 and the frame memory 307, and the contents of the frame memory 314 are output via the terminal 315.
[0073]
Here, when the frame is encoded in the inter-frame encoding mode, the encoding mode determination unit 304 first determines how many frames from the beginning of the frame to be decoded are the inter-frame encoding frame. Detecting based on the count value of the number of frames. Furthermore, the position of the intra-coded macroblock of the frame to be decoded is read from the mode pattern memory 303 by comparison with the period in which the intra-coded macroblock appears first in the frame header information. The encoding mode determination unit 304 detects the position of the macroblock to be decoded based on the count value of the macroblock, and inputs to the error determination unit 305 whether or not the macroblock is an intraframe encoded macroblock.
[0074]
The header decoder 302 decodes the header of each macroblock, and the decoding mode of the decoded macroblock is input to the error determiner 305. If the macroblock is a macroblock to be intraframe encoded, the error determiner 305 compares whether or not the macroblock is intraframe encoded. If the macroblock is not intraframe encoded, Is notified to the error corrector 313, the Huffman decoder 309, and the header decoder 302.
[0075]
The header decoder 302 and the Huffman decoder 309 continue decoding, but the error corrector 313 stores the decoded image data until the head of the next GOB is decoded. The number of macroblocks to be intra-coded and the number of macroblocks up to GOB are counted, and it is determined whether the previous macroblock returned by the counted number is intra-coded. If these macroblocks are in the intraframe encoding mode, the reproduced image data is stored in the corresponding positions in the frame memories 307 and 314, assuming that the macroblock is to be intraframe encoded. This is because it can be determined that an error has occurred in the count value of the macroblock.
[0076]
If the error determiner 305 does not detect an error, the motion vector decoder 306 is not operated if the macroblock is encoded in the intraframe encoding mode, and the motion compensator 308 is always applied to each pixel value. “0” is output to the adder 312. On the other hand, if the macro block is encoded by performing motion compensation, the motion vector decoder 306 is operated, the decoded motion vector is input to the motion compensator 308, and the predicted pixel value by motion compensation is received from the frame memory 307. And output to the adder 312.
[0077]
The Huffman decoder 309 decodes input encoded data and obtains quantized DCT coefficients. If an error is detected during the Huffman decoding, the information is output to the error corrector 313 and corrected in the same manner as the error correction at the time of intraframe coding. On the other hand, when there is no error, the quantized DCT coefficient is inversely quantized by the inverse quantizer 310 and converted to a pixel value by the inverse DCT 311. The pixel value thus obtained is added with “0” by the adder 312 if it is in the intra-frame coding mode, and is added by the adder 312 if not. The pixel data output from the error corrector 313 is stored in predetermined positions in the frame memory 314 and the frame memory 307, and the contents of the frame memory 314 are output as decoded image data via the terminal 315. .
[0078]
FIG. 8 is a flowchart for explaining the decoding process in the image decoding apparatus according to Embodiment 3 of the present invention.
[0079]
First, in step S21, code data is input from a terminal 300, and the input code is separated into a header, a motion vector, and a Huffman code portion by a separator 301. In step S22, the decoded frame and block headers are analyzed and notified to the encoding mode determination unit 304. In the case of inter-frame encoding, the operations of the motion vector decoder 306 and the motion compensator 308 are enabled. To do. In step S 23, the encoded mode pattern data is decoded and stored in the mode pattern memory 303.
[0080]
In step S24, it is checked whether the frame is an intra-frame encoded frame. If so, the process proceeds to step S25, and the input frame is decoded by an intra-frame decoding process. In this case, the operations of the motion vector decoder 306 and the motion compensator 308 are disabled.
[0081]
When it is not intra-frame coding in step S24, the process proceeds to step S26, and it is checked what number the frame is. In step S27, it is checked whether it is the first frame or every 132nd frame, and if so, the process proceeds to step S33, and the frame that should be intra-frame encoded has been inter-frame encoded. The error determiner 305 is notified of the error, and error correction by the error corrector 313 is performed in step S34.
[0082]
If it is determined in step S27 that the frame is not to be intraframe-encoded, the process proceeds to step S28 to determine whether the block to be decoded is an intraframe-encoded block or not (the number of received blocks in the frame). ) And the pattern data stored in the mode pattern memory 303 in step S23. If the block is to be intra-frame encoded, the process proceeds to step S31 to check whether or not the block is intra-frame encoded. If not, the process proceeds to error processing of step S33. If YES in step S32, the flow advances to step S32 to execute the decoding process of the intra-frame encoded code in the same manner as in step S25.
[0083]
If it is determined in step S28 that the block is not to be intraframe-encoded, the process proceeds to step S29, where the motion vector is decoded and motion compensation (step S30) is performed to decode the block.
[0084]
As described above, according to the third embodiment, on the decoding side, by detecting the position of a macroblock to be intra-coded, an error in the transmission path is detected depending on whether the data is intra-coded. I can do it. Based on this, it is possible to easily correct image degradation due to an error and to suppress degradation of image quality.
[0085]
In the third embodiment, the description has been made using the code generated in the second embodiment, but the same contents as the mode pattern memory 115 in the first embodiment are stored in the mode pattern memory 303 in advance. Needless to say, similar operations can be performed.
[0086]
<Other embodiments>
In the above description, the configuration of the macroblock group and the mode for transmitting the mode setting method (configuration of the mode pattern memory 201) in the encoding mode setting unit 114 of Embodiment 2 together with the encoded data have been described. It is not limited to this, and it is also possible to transmit in advance using an error-free transmission line or a method with strong error tolerance. Of course, in a limited terminal, these may be stored in advance in individual devices. Is possible.
[0087]
The configuration of the block group is not limited to these, and the macroblock to be intra-coded may not have a fixed number of block groups, and may have a distribution as shown in FIG. 9, for example. In the example of FIG. 9, in consideration of the fact that a person often comes to the center in a TV phone or the like, in order to shorten the period of the macroblock to be intra-frame encoded in the substantially central portion, a small number of A block group is formed by macroblocks. Conversely, by constructing a block group with a larger number of macroblocks at the periphery of the screen, it is possible to actively suppress error propagation in important parts of the image while reducing the amount of encoded data as a whole. Is possible.
[0088]
In the present embodiment, H.264. The H.263 encoding method has been described as an example, but other encoding methods for motion compensation, for example, MPEG-1, 2, 4 may be used, and it is of course possible to target an object having an arbitrary shape. .
[0089]
Furthermore, in the present embodiment, the inter-frame prediction encoding using the motion vector has been described, but the present invention is not limited to this. For example, you may apply to the method of using a fractal between frames.
[0090]
Further, as shown in the image encoding apparatus of another embodiment shown in FIG. The same can be applied to the bi-directional prediction inter-frame encoded frame in the H.263 option.
[0091]
In FIG. 10, a motion compensator 402 is obtained by adding a bi-directional prediction coding mode to determination in addition to intra-frame coding / inter-frame coding. The motion vector encoder 403 encodes the bidirectional motion vector, and the header encoder 404 encodes the bidirectional predictive encoding mode into the macroblock encoding mode.
[0092]
Further, as in the mode pattern memory 406, the contents may be appropriately updated from the outside via the terminal 405. For example, the pattern shown in FIG. 2 may be changed to the pattern shown in FIG. In that case, in Embodiment 2 described above, it is also possible to encode the contents of the mode pattern memory at the beginning of the frame at the time of updating in the same manner as the contents of the mode pattern memory at the beginning frame are encoded. is there.
[0093]
In the present embodiment, the term “frame” is used, which indicates one screen, and when a frame composed of several fields is used as a frame, it corresponds to a field.
[0094]
Of course, each part or all of the functions of this embodiment may be described by software and processed by an arithmetic device such as a CPU.
[0095]
In this embodiment, the frame is described as a unit. However, it is of course possible to make each correspond to a field as in the MPEG-2 system.
[0096]
Note that the present invention can be applied to a system (for example, a copier, a facsimile machine, etc.) composed of a single device even if it is applied to a system composed of a plurality of devices (for example, a host computer, interface device, reader, printer). May be.
[0097]
Another object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU) of the system or apparatus. Or MPU) can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. A case where part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.
[0098]
Further, after the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. This includes a case where the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.
[0099]
As described above, according to the present embodiment, by inserting macroblocks for intraframe encoding at intervals shorter than the period of intraframe encoded frames, the transmission path can be reduced without reducing encoding efficiency. It is possible to suppress degradation of image quality due to errors in
[0100]
In addition, according to the present embodiment, error propagation is minimized by inserting a block to be intra-coded even in a frame for which inter-frame predictive coding is performed without significantly reducing the coding efficiency. Can do.
[0101]
Further, according to the present embodiment, the configuration of the macroblock group and the mode setting method in the encoding mode setting device are also transmitted to the decoding side together with the encoded data, so that the position of the intra-frame encoding block is determined on the decoding side. This can be known in advance, and can also be used for detecting transmission errors.
[0102]
Further, according to the present embodiment, on the decoding side, a transmission error is detected based on the collation between the position information of the intra-frame coded block and the decoded data, and the deterioration of the transmission image is suppressed based on the detection result. Can do.
[0103]
【The invention's effect】
  As described above, according to the present invention,Each block of a plurality of frames of image data can be surely subjected to intra-picture encoding at a constant interval. Since there are blocks that perform intra-frame coding with a period shorter than the frame interval for performing intra-frame coding, the propagation of image quality degradation due to transmission errors in inter-frame coding is suppressed without extremely increasing the code amount. It becomes possible.
[0104]
  Also according to the invention,By transmitting the encoding mode of each block of the frame P to the decoding side together with the encoded data of a plurality of frames, the position of the block that has been intra-encoded on the decoding side can be known in advance. This can also be useful for detecting transmission errors.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a main configuration of an image coding apparatus according to Embodiment 1 of the present invention.
FIG. 2 is a diagram showing an example of an arrangement of macroblocks to be intra-frame encoded according to Embodiment 1 of the present invention.
FIG. 3 is a block diagram showing a configuration of a coding mode setting unit according to Embodiment 1 of the present invention.
FIG. 4 is a flowchart showing mode setting processing by an encoding mode setting unit according to Embodiment 1 of the present invention.
FIG. 5 is a block diagram showing a main configuration of an image coding apparatus according to a second embodiment of the present invention.
[Fig. 6] Fig. 6 is a diagram illustrating an example of an arrangement of macroblocks to be subjected to intraframe coding according to the second embodiment.
FIG. 7 is a block diagram showing the main configuration of an image decoding apparatus according to Embodiment 3 of the present invention.
FIG. 8 is a flowchart showing decoding processing in the image decoding apparatus according to Embodiment 3 of the present invention.
FIG. 9 is a diagram illustrating an arrangement example of macroblocks to be intra-coded according to another embodiment of the present invention.
FIG. 10 is a block diagram showing a main configuration of an image encoding device according to another embodiment of the present invention.

Claims (8)

An input means for inputting image information of a plurality of frames;
Dividing means for dividing each of the plurality of frames input by the input means into blocks;
Encoding means for encoding each block obtained by the dividing means in an intra-picture coding mode or an inter-picture coding mode;
The encoding mode of frame I positioned at a fixed number of frame intervals in the plurality of frames is set to the intra-screen encoding mode, and the encoding mode of other frames P is set to the intra-screen encoding mode or between the screens Encoding mode setting means for selectively setting the encoding mode ;
Storage means for storing a plurality of types of mode patterns periodically assigned to a plurality of successive frames P;
Output means for multiplexing and outputting information indicating the mode pattern stored in the storage means to the plurality of encoded frames ;
The encoding mode setting means, on the basis of the mode pattern sets an encoding mode of each frame P, wherein each of the plurality of types of mode pattern, one of a plurality of blocks constituting the frame P each said screen An image processing apparatus comprising: information indicating whether to perform an inner coding mode or the inter-screen coding mode. The image processing apparatus according to claim 1, wherein the encoding unit performs an encoding process compliant with the MPEG standard. In the image information of a plurality of frames, the frame I positioned at a fixed number of frame intervals is encoded in the intra-frame encoding mode, and each block of the other frame P is set in the intra-screen encoding mode or the inter-screen encoding mode. Intra-screen encoding mode or inter-screen based on a plurality of types of mode patterns that are selectively used and encoded, and each block of the encoded frame P is periodically assigned to a plurality of consecutive frames P Input means for inputting a plurality of frames encoded in the encoding mode and a mode pattern corresponding to the frame P;
Determining means for determining an error in the encoding mode by comparing the encoding mode of each block recorded in the header of the plurality of encoded frames and the corresponding mode pattern;
Decoding means for correcting an error in the encoding mode based on the determination result notified from the determination means and decoding the plurality of encoded frames;
An image processing apparatus comprising: The image processing apparatus according to claim 3 , wherein the mode pattern is obtained by decoding information included in the plurality of encoded frames. An input process for inputting image information of a plurality of frames;
A dividing step of dividing each of the plurality of inputted frames into blocks;
An encoding step of encoding each block obtained in the dividing step in an intra-screen encoding mode or an inter-screen encoding mode;
The encoding mode of frame I positioned at a certain number of frame intervals in the plurality of frames is set to the intra-screen encoding mode, and the encoding mode of other frames P is set to the intra-screen encoding mode or inter-screen encoding An encoding mode setting step for selectively setting the encoding mode ;
A storage step of storing a plurality of types of mode patterns periodically assigned to a plurality of successive frames P;
An output step of multiplexing and outputting the information indicating the mode pattern stored in the storage step to the plurality of encoded frames ;
The encoding mode setting step, on the basis of the mode pattern sets an encoding mode of each frame P, wherein each of the plurality of types of mode pattern, one of a plurality of blocks constituting the frame P each said screen An image processing method comprising: information indicating whether to perform an inner coding mode or the inter-screen coding mode. The image processing method according to claim 5, readable storage medium by a computer, characterized in that storing a control program Ru cause the computer to execute. In the image information of a plurality of frames, the frame I positioned at a fixed number of frame intervals is encoded in the intra-frame encoding mode, and each block of the other frame P is set in the intra-screen encoding mode or the inter-screen encoding mode. Intra-screen encoding mode or inter-screen based on a plurality of types of mode patterns that are selectively used and encoded, and each block of the encoded frame P is periodically assigned to a plurality of consecutive frames P An input step of inputting a plurality of frames encoded in the encoding mode and a mode pattern corresponding to the frame P;
A determination step of determining an error in an encoding mode by comparing an encoding mode of each block recorded in a header of the plurality of encoded frames and the mode pattern;
A decoding step of correcting an error in the encoding mode based on the determination result notified from the determination step, and decoding the plurality of encoded frames;
An image processing method comprising: Image processing method, a storage medium readable by a computer, characterized in that storing a control program Ru cause the computer to execute according to claim 7.

Images