JP5236088B2 - Image encoding / decoding device, encoding / decoding program, and encoding / decoding method - Google Patents

Description

  The present invention relates to a video encoding device for encoding video, a video encoding program, a video encoding method, a video decoding device for decoding encoded video, a decoding program, a decoding method, The present invention relates to a medium on which an encoded video encoded stream is recorded and a method for transmitting encoded video data.

  As a method for recording and transmitting large volume video and audio information as compressed digital data, an encoding method such as the MPEG (Moving Picture Experts Group) method has been established, and the MPEG-1 standard and the MPEG-2 standard. It is an international standard encoding method such as MPEG-4 standard. As a method for further improving the compression ratio, H. The H.264 / AVC (Advanced Video Coding) standard is defined (see Non-Patent Document 1). These systems are adopted as encoding systems in digital satellite broadcasting, DVDs, mobile phones, digital cameras, and the like, and the range of use is now expanding and becoming familiar.

  Of these encoding schemes, H.264. In the H.264 / AVC format, in addition to the techniques of motion compensation prediction and arithmetic transform coding using Discrete Cosine Transform (DCT), which have been used in MPEG and the like, Intra-coded prediction coding and The so-called technology is adopted. This is a technique for predicting a signal level from a coded adjacent block in an intra frame without using information of other pictures.

  FIG. 3 is an explanatory diagram of intra prediction encoding. Here, as an example, a case will be described in which prediction is performed in block units of 4 × 4 pixel size in a raster scan method in which scanning is sequentially performed from the upper left to the lower right of the screen. The signal level of the pixel 304 of the prediction block 303 (block to be predicted) is the boundary pixel 302 of the adjacent block 301 that has already been encoded (pixels adjacent to the prediction block among the pixels of the adjacent block 301. However, the prediction block For the adjacent block on the upper right side of (1), it is predicted from the signal level of the pixel in the lowermost row), and the difference (residual component) between the predicted value and the actual value is subjected to arithmetic transform coding by DCT or the like. At this time, the pixel 305 of the adjacent block which is not yet encoded cannot be used for prediction. There are a plurality of prediction directions (that is, encoding modes) 306, prediction is performed for all directions, the residual components are compared, the direction with the smallest residual component is selected, and the residual component is the arithmetic transform code. It becomes. Here, the direction “0: DC” is a mode in which an average value of signal levels of all boundary pixels is a predicted value.

  Even when prediction is performed in units of blocks each having a size of 16 × 16 pixels, the prediction direction is limited, but intra prediction encoding can be performed in the same manner.

  The inter prediction encoding is the same as the conventional MPEG system, but the motion vector of each block is encoded as a difference between the already encoded upper and left motion vectors.

Joint Video Team (JVT) of ISO / IEC MPEG & ITU-T VCEG: “Text of International Standard of Joint Video Specification”, ITU-T Rec. H.264 | ISO / IEC 14496-10 Advanced Video Coding, (December, 2003)

  As described above, in intra prediction encoding, pixels of adjacent blocks that are not yet encoded cannot be used. For this reason, a difference occurs in prediction accuracy depending on the prediction direction 306. For example, when a block is scanned by a raster scan method that sequentially scans from the upper left to the lower right of the screen, the prediction accuracy from the upper left to the lower right direction increases, but the prediction accuracy from the upper right to the lower left direction decreases. In other words, it is easy to predict an image in which pixels with the same signal level continue from the upper left to the lower right (lower-right image), so the compression rate can be increased, but the signal level is equal from the upper right to the lower left. Since it is difficult to predict an image with a continuous pixel (upwardly rising image), the compression rate is low.

  Similarly in inter prediction coding, there is a direction dependency regarding image features by raster scanning. Since the motion vector of each block encodes the length of the difference between the motion vector of the left side and the adjacent block on the upper side, the sequence with the closer motion correlation in the lower right direction has higher prediction accuracy and better encoding efficiency. Moreover, since the selection of the context model in entropy coding uses information on the left and upper adjacent blocks, the same tendency occurs in prediction accuracy.

  Further, in the conventional intra prediction coding, since prediction is performed using only boundary pixels, there is a problem that continuous changes in the signal level inside the block cannot be predicted with sufficient accuracy. Usually, the signal level of the pixel changes gently. However, in the conventional intra prediction coding, prediction is performed by extending the boundary pixel at the block boundary (that is, the pixel in the predicted block is predicted to be the same as the boundary pixel). For this reason, as the pixel is farther from the boundary pixel, the residual component becomes larger and the amount of code after arithmetic conversion increases.

  The present invention has been made in view of the above problems, and before performing intra prediction, the input image is inverted in the vertical direction or the horizontal direction, or rotated to an arbitrary angle to thereby change the scan direction. It is an object of the present invention to provide a video encoding device capable of always performing intra prediction in a direction with high prediction accuracy without change, and a video decoding device corresponding to the video encoding device. In addition, the present invention reverses the input image in the vertical direction or the horizontal direction before performing the inter prediction, or rotates it to an arbitrary angle, so that the interpolation is always performed in the direction with high prediction accuracy without changing the scan direction. It is an object of the present invention to provide a video encoding device capable of performing the above and a video decoding device corresponding thereto.

  In addition, the present invention provides a video encoding device that realizes intra prediction with high accuracy by a predetermined interpolation formula using not only the boundary pixels but also the pixels inside the block at the time of intra prediction, and corresponding to this. An object is to provide a video decoding device.

  According to the present invention, in an encoding apparatus that encodes an image signal, an image conversion unit that converts the direction of the image, an encoding unit that encodes the image, and the image whose direction has been converted; The code amount of the encoded image is compared with the code amount of the image obtained by converting the encoded direction, the code with the small code amount is selected, the selected code, and the selection And a mode selection unit for outputting a flag for displaying the generated code.

  According to the present invention, in the encoding device that encodes an image signal, information on the first pixel in the prediction region of the image and one second pixel in the prediction direction from the first pixel. And calculating the first difference of the prediction region, encoding the first difference of the prediction region, and a plurality of third pixels in the prediction direction from the first pixel. A prediction value is calculated by applying a forward interpolation formula to the information, a second difference from the prediction value is calculated, and the second difference in the prediction region is encoded, and a plurality of the encoding units The code amounts of the first and second differences in the prediction region calculated for the prediction direction are compared, the code with the smallest code amount is selected, and the selected code and the selected code are either Whether the difference is the first or second difference calculated for the prediction direction of Those comprising a mode selection unit for outputting the A flag, is provided.

  Therefore, according to the present invention, there is provided a video encoding device in which the compression rate is increased by always performing intra prediction and inter prediction in a direction with high prediction accuracy without changing the scan direction, and a video decoding device corresponding thereto. Can be provided.

  Further, according to the present invention, it is possible to provide a video encoding device that increases the compression rate by performing intra prediction with high accuracy using pixels in the block, and a video decoding device corresponding to this.

It is a block diagram explaining the hardware constitutions of the video coding apparatus of the 1st Embodiment of this invention. It is a block diagram explaining the function of the video coding apparatus of the 1st Embodiment of this invention. It is explanatory drawing of intra prediction encoding. It is explanatory drawing of the procedure of the video encoding of the 1st Embodiment of this invention. It is a block diagram explaining the hardware constitutions of the video decoding apparatus of the 2nd Embodiment of this invention. It is a block diagram explaining the function of the video decoding apparatus of the 2nd Embodiment of this invention. It is explanatory drawing of the procedure of the video decoding of the 2nd Embodiment of this invention. It is explanatory drawing of the intra prediction method of the 3rd Embodiment of this invention. It is a block diagram explaining the function of the intra estimation part to which the intra prediction method of the 3rd Embodiment of this invention is applied. It is explanatory drawing of the procedure of the intra prediction of the 3rd Embodiment of this invention. It is explanatory drawing of the data recording medium of the 4th Embodiment of this invention. It is explanatory drawing of the packet of the 5th Embodiment of this invention. It is a block diagram explaining the function of the video coding apparatus of the 6th Embodiment of this invention. It is explanatory drawing of the procedure of the video encoding of the 6th Embodiment of this invention. It is a block diagram explaining the function of the video decoding apparatus of the 7th Embodiment of this invention. It is explanatory drawing of the procedure of the video decoding of the 7th Embodiment of this invention.

  Embodiments of the present invention and advantages of the present invention will be described in detail below with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a hardware configuration of a video encoding apparatus according to the first embodiment of this invention.

  The video encoding apparatus 101 according to the first embodiment includes a processor 102, a memory 103, an input interface (input I / F) 104, and an output interface (output I / F) 106 that are connected so as to communicate with each other. The input I / F 104 is connected to the input device 105. The output I / F 106 is connected to the output device 107.

  The processor 102 is a processor that performs video encoding processing according to the present invention, executes a program stored in the memory 103, encodes data received from the input I / F 104, and sends the data to the output I / F 106.

  The memory 103 stores a program executed by the processor 102. In addition, data to be processed by the processor 102 is temporarily stored.

  The video encoding device 101 may be provided with a plurality of processors 102 and memories 103. For example, a dedicated processor that executes only a part of the program for performing the video encoding process of the present invention may be provided. A plurality of dedicated processors that perform the same processing may be provided.

  The processor 102 and memory 103 may be implemented on a single chip.

  The input I / F 104 is an interface that receives video data processed by the processor 102 from the input device 105.

  The input device 105 is a device that inputs a video signal processed by the video encoding device 101 to the input I / F 104, and is, for example, a video camera or a TV tuner. In this case, the input I / F 104 is, for example, a video capture card. The input device 105 may be a storage device that stores unencoded video data. In this case, the input I / F 104 is, for example, a SCSI interface.

  The output I / F 106 is an interface that transmits data encoded by the processor 102 to the output device 107.

  The output device 107 is an output destination of the data encoded by the video encoding device 101, and is, for example, a storage device that stores the encoded data. In this case, the output I / F 106 is, for example, a SCSI interface. The output device 107 may be a computer device connected to the output I / F 106 via a LAN, an IP network, or the like (not shown). In this case, the output I / F 106 is a network interface. The output device 107 may be an image receiving device connected to the output I / F 106 via a telephone communication network (not shown). In this case, the output I / F 106 is a telephone signal transmitter. The output device 107 may be a digital TV receiver. In this case, the output I / F 106 is a digital TV signal transmitter.

  The video encoding device 101 may include a plurality of input I / Fs 104 and output I / Fs 106, and different types of input devices and output devices may be connected to the respective input I / Fs 104 and output I / Fs 106. For example, the video encoding device 101 may include two output I / Fs 106, one of which is connected to a hard disk device and the other is connected to a magneto-optical disk device. Further, a hard disk device may be connected to one side, and a computer device may be connected to the other side via a LAN or the like.

  FIG. 2 is a block diagram illustrating functions of the video encoding device 101 according to the first embodiment of this invention.

  The video encoding apparatus 101 includes an original image memory 201, an image conversion unit 202, an encoding unit 203, a mode control unit 213, and a mode selection unit 214. Here, the original image memory 201 is a partial area of the memory 103, and the image conversion unit 202, the mode control unit 213, and the mode selection unit 214 are programs stored in the memory 103 and executed by the processor 102. . In addition, the encoding unit 203 is a program executed by the processor 102, such as a motion prediction unit 204, an intra prediction unit 205, an arithmetic conversion unit 206, a quantization unit 207, a coefficient encoding unit 208, an inverse quantization unit 209, and an inverse quantization unit 209. It comprises an arithmetic conversion unit 210 and a decoded image memory 211 and a predicted image memory 212 which are partial areas of the memory 103.

  Next, functions of each unit of the video encoding device 101 will be described along a procedure for encoding an image.

  The original image memory 201 temporarily buffers an original image to be encoded. Next, the image conversion unit 202 performs conversion processing on all or part of the frame of the image acquired from the original image memory 201. Here, the part of the frame may be, for example, a macro block or a block obtained by dividing the frame into a predetermined size, or may be a predetermined rectangular area. In general, encoding is performed in units of macroblocks.

  The image conversion unit 202 may divide the data obtained by performing the conversion process on the entire frame into units of macro blocks and transmit the data to the encoding unit 203. Alternatively, the image conversion unit 202 may perform the conversion process after dividing the frame into units of macro blocks. You may go. Information on whether or not conversion processing has been performed is transmitted to the mode control unit 213.

  The conversion process performed by the image conversion unit 202 is a process of converting the direction of the image. The process for converting the direction of the image is, for example, line symmetry conversion or point symmetry conversion. The line symmetry conversion is, for example, left-right symmetry conversion that inverts the frame left and right or up-down symmetry conversion that inverts the frame up and down. The point symmetry conversion is, for example, rotation conversion that rotates a frame. The image conversion unit 202 may perform a combination of these conversions. Hereinafter, as an example, a case will be described in which the case where the entire image is subjected to left-right symmetric conversion is compared with the case where no conversion is performed, and the smaller code amount is selected.

  The encoding unit 203 acquires from the image conversion unit 202 an image that has undergone left-right symmetry conversion and an image that has not been converted, and sequentially encodes each image. Also, a plurality of encoding units 203 may be provided, and two encoding units 203 may encode each image in parallel. When a plurality of encoding units 203 are provided, the video encoding apparatus 101 is provided with a plurality of dedicated processors 102 that execute only the program of the encoding unit 203.

  The motion prediction unit 204 performs inter-frame prediction on the image acquired from the image conversion unit 202 using the images in the predicted image memory 212 and the decoded image memory 211, and transmits information such as a motion vector to the mode control unit 213. Then, the residual component of the encoded block obtained by the prediction is transmitted to the arithmetic conversion unit 206.

  The intra prediction unit 205 performs intra prediction on the image acquired from the image conversion unit 202 using the image in the decoded image memory 211, transmits mode information and the like to the mode control unit 213, and is obtained by prediction. The residual component of the encoded block is transmitted to the arithmetic conversion unit 206.

  The arithmetic conversion unit 206, the quantization unit 207, and the coefficient encoding unit 208 are the same as those of the conventional encoding device, and perform DCT operation, quantization of conversion coefficients, conversion of coefficients to codes, and the like, respectively. . Also, the inverse quantization unit 209 and the inverse arithmetic conversion unit 210 are the same as those of the conventional encoding device, and respectively return the encoded data to the image information by inversely transforming the encoded data, and the decoded image memory 211 and the prediction Store in the image memory 212.

  The mode control unit 213 manages the encoding mode of the entire image (frame) and the macroblock being encoded. With respect to the entire image, information on an encoding process for an image that has undergone conversion processing and an encoding process for an image that has not been subjected to conversion processing are held. That is, information (motion vector, reference frame information, etc.), intra coding mode (intra prediction direction) information, etc., which is a reference for motion prediction, for each of the cases where conversion processing is performed and not performed. Hold. Regarding the macro block, information about whether the current macro block is intra-encoded or inter-frame predictively encoded, and intra-encoding mode, motion vector, reference frame information and the like related thereto are held. The mode control unit 213 transmits these pieces of information to the mode selection unit 214.

  The mode selection unit 214 configures and outputs the encoded data of the image from the encoded data regarding the entire image and the macroblock being encoded and the information of the encoding mode. The entire code amount when encoded by performing the conversion process by the image conversion unit 202 is compared with the code amount when encoded without performing the conversion process, and the smaller code is output as a stream. In addition, a flag indicating whether or not conversion processing has been performed and, if a partial area of the image is converted, position information of the area is added to the stream as data.

  The flag indicating whether or not the conversion process has been performed may be one bit each for the left-right symmetric conversion and the vertical symmetric conversion. In the case of rotation conversion, if the number is incremented by 1 every 90 degrees clockwise, 360 degrees can be expressed with 2 bits.

  FIG. 4 is an explanatory diagram of a video encoding procedure according to the first embodiment of this invention.

  First, the image conversion unit 202 determines the encoding method (step 401). That is, it is determined whether conversion processing is performed on the entire frame or conversion is performed in units of macroblocks. Hereinafter, as an example, a case where the entire frame is converted will be described. The process that performs the conversion process proceeds to step 402, and the process that does not perform the conversion process proceeds to step 404.

  In step 402, the image conversion unit 202 converts the input image. That is, for each frame, left-right symmetry conversion, up-down symmetry conversion, rotation conversion, and the like are performed. Then, it progresses to step 403.

  In step 403 and step 404, the encoding unit 203 encodes the image. This encoding is as described in FIG.

  Next, the mode selection unit 214 performs code amount comparison and mode determination (step 405). That is, as described with reference to FIG. 2, the code amount obtained by encoding is compared between the process that performs the conversion process and the process that does not perform the conversion process, and determines that data with a small code amount is output.

  Next, the mode selection unit 214 outputs a code and a flag (step 406). As described in FIG. 2, according to the determination in step 405, a flag indicating whether or not conversion processing has been performed, the position information of the region when a partial region of the image is converted, and the selected code Output the digitized data as a stream. Thus, the encoding process ends.

  FIG. 5 is a block diagram illustrating the hardware configuration of the video decoding apparatus according to the second embodiment of the present invention.

  The video decoding apparatus 501 according to the second embodiment includes a processor 502, a memory 503, an input interface (input I / F) 504, and an output interface (output I / F) 506 that are connected so as to communicate with each other. The input I / F 504 is connected to the input device 505. The output I / F 506 is connected to the output device 507.

  The processor 502 is a processor that performs video decoding processing according to the present invention, executes a program stored in the memory 503, decodes data received from the input I / F 504, and sends the data to the output I / F 506.

  The memory 503 stores a program executed by the processor 502. In addition, data to be processed by the processor 502 is temporarily stored.

  The video decoding apparatus 501 may be provided with a plurality of processors 502 and memories 503. For example, a dedicated processor that executes only a part of the program that performs the video decoding process of the present invention may be provided.

  The processor 502 and memory 503 may be implemented on a single chip.

  The input I / F 504 is an interface that receives encoded data to be processed by the processor 102 from the input device 505.

  The input device 505 is a device that inputs encoded data to be processed by the video decoding device 501 to the input I / F 504. For example, the input device 505 is a storage device that stores the encoded data. In this case, the input I / F 504 is, for example, a SCSI interface. The input device 505 may be a computer device connected to the input I / F 504 via a LAN, an IP network, or the like (not shown). In this case, the input I / F 504 is a network interface. The input device 505 may be a video data transmission device connected to the input I / F 504 via a telephone communication network (not shown). In this case, the input I / F 504 is a telephone signal receiver. The input device 505 may be a digital TV broadcast station. In this case, the input I / F 504 is a digital TV tuner.

  The output I / F 506 is an interface that transmits data decoded by the processor 502 to the output device 507.

  The output device 507 is an output destination of the data decoded by the video decoding device 501 and is, for example, a display that outputs video. In this case, the output I / F 506 is, for example, a video card. The output device 507 is a storage device that records the decoded data as a stream. In this case, the output I / F 506 is, for example, a SCSI interface.

  The video decoding device 501 may include a plurality of input I / Fs 504 and output I / Fs 506, and different types of input devices and output devices may be connected to the respective input I / Fs 504 and output I / Fs 506. For example, the video encoding device 101 may include two input I / Fs 504, one of which is connected to a hard disk device and the other is connected to an optical disk device. Further, a hard disk device may be connected to one side, and a computer device may be connected to the other side via a LAN or the like.

  FIG. 6 is a block diagram illustrating functions of the video decoding apparatus 501 according to the second embodiment of this invention.

  The video decoding apparatus 501 includes a stream analysis unit 601, a mode determination unit 602, a decoding unit 603, an image conversion unit 610, and a decoded image memory 611. Here, the stream analysis unit 601, the mode determination unit 602, the decoding unit 603, and the image conversion unit 610 are programs stored in the memory 503 and executed by the processor 502, and the decoded image memory 611 is stored in the memory 503. Part area. Also, the decoding unit 603 is a program executed by the processor 502, a motion prediction unit 604, an intra prediction unit 605, a coefficient analysis unit 606, an inverse quantization unit 607, an inverse arithmetic operation unit 608, and one of the memories 503. A prediction image memory 609 which is a region of a part.

  The decoding apparatus 501 according to the second embodiment can decode the stream encoded by the video encoding apparatus 101 according to the first embodiment. Next, the function of each unit of the video decoding apparatus 501 will be described along a procedure for decoding an encoded stream.

  The stream analysis unit 601 analyzes input encoded stream data, and transmits flag and data information to the mode determination unit 602. The stream analysis unit 601 analyzes the stream data and flag created by the encoding apparatus 101.

  Next, the mode determination unit 602 controls modes related to motion prediction, intra prediction, and image conversion based on the information analyzed by the stream analysis unit 601. When a stream is flagged indicating that conversion processing such as inversion and rotation has been performed on the image at the time of encoding, information on the type of conversion processing performed (for example, symmetrical conversion) And the like) is transmitted to the image conversion unit 610.

  The motion prediction unit 604 performs inter-frame prediction using information such as motion vectors transmitted from the mode determination unit 602 and images in the prediction image memory 609 and the decoded image memory 611, and transmits the prediction information to the coefficient analysis unit 606. To do.

  The intra prediction unit 605 performs intra prediction using information such as the intra coding mode transmitted from the mode determination unit 602 and the image in the decoded image memory 611, and transmits the prediction information to the coefficient analysis unit 606.

  The coefficient analysis unit 606, the inverse quantization unit 607, and the inverse arithmetic conversion unit 608 are the same as those of the conventional decoding device, and each of them combines prediction information and coefficient information, inverse quantization of transform coefficients, DCT calculation, etc. I do.

  The image conversion unit 610 performs conversion processing on the whole or part of the decoded image frame in accordance with the information transmitted from the mode determination unit 602. That is, a process for returning the conversion process performed by the image conversion unit 202 of the encoding apparatus 101 in FIGS. 1 and 2 is performed.

  The decoded image memory 611 stores the decoded image after the conversion processing by the image conversion unit 610, transmits the decoded image to the output device 504, and displays the decoded image on a screen or outputs it to a stream.

  FIG. 7 is an explanatory diagram of a video decoding procedure according to the second embodiment of this invention.

  First, the stream analysis unit 601 and the mode determination unit 602 perform stream and flag analysis (step 701). Next, the decoding unit 603 decodes the image (step 702). Next, the image conversion unit 610 converts the decoded image and stores it in the decoded image memory 611 (step 703). Finally, the image stored in the decoded image memory 611 is output for display or stream recording (step 704). This completes the decoding process. Since the contents of the processing of each of the above parts are as described in FIG. 6, detailed description thereof will be omitted.

  The video encoding apparatus 101 according to the first embodiment of the present invention and the video decoding apparatus 501 according to the second embodiment described above may be implemented as the same hardware. In this case, each program described in FIGS. 2 and 6 is stored in the memory 103 (or the memory 503), and each area described in FIGS. 2 and 6 is secured.

  According to the first and second embodiments of the present invention described above, the amount of code after encoding is reduced by always performing intra prediction in a direction with high prediction accuracy without changing the scan direction (that is, It is possible to provide a video encoding device (with a high compression rate) and a video decoding device corresponding thereto. As a result, compared with the conventional intra prediction, the code amount is reduced by about 10% at the maximum. However, since the code amount changes according to the contents of the original image, it is not always a constant reduction amount.

  FIG. 8 is an explanatory diagram of the intra prediction method according to the third embodiment of this invention.

  The present embodiment is a method of using not only the signal level of the boundary pixel but also the signal level of the entire pixel of the adjacent block 801 that has already been encoded when predicting the signal level of the pixel of the prediction block 802. It is used in the intra prediction units 205 and 605 in FIGS. Here, “prediction block” refers to a block that is to be encoded and has not been encoded yet, and “adjacent block” refers to an encoded block that is adjacent to the prediction block.

  The direction of prediction is as indicated by 306 in FIG. In this embodiment, for the sake of simplicity, only predictions for some directions are described, but in actual prediction, prediction is performed for all prediction directions 306, and the residual component is the smallest direction. Is selected. Here, prediction refers to a difference (residual) between an actual signal level of each pixel of the prediction block 802 and a prediction value calculated from a signal level of a pixel of an adjacent block used for prediction in encoding. This is a procedure for encoding (component). On the other hand, in decoding, it means a procedure for obtaining a decoded image by adding a decoded residual component to a predicted value calculated in the same manner. The encoding procedure will be described below.

  The conventional intra prediction method is a prediction method using only boundary pixels. For example, when predicting in the vertical direction, the prediction is performed in the downward direction using the boundary pixel (the pixel in the lowest row) of the adjacent block 801 directly above the prediction block 802. That is, the prediction block 802 is divided into four columns, and the signal level value of the boundary pixel in contact with the column including the pixel is subtracted from the signal level value of each pixel. Similarly, when predicting in the horizontal direction, the prediction is performed in the right direction using the boundary pixel (the pixel in the rightmost column) of the adjacent block on the left side of the prediction block. That is, the prediction block 802 is divided into four rows, and the signal level value of the boundary pixel in contact with the left of the row including the pixel is subtracted from the signal level value of each pixel. That is, in any of the above predictions, the pixel level of the prediction block is predicted to be the same as the signal level of the boundary pixel, and the difference (residual component) between the predicted value and the actual value is calculated. calculate.

  On the other hand, the intra prediction method according to the third embodiment of the present invention is a prediction method based on composite pixels that uses not only boundary pixels but also pixels inside adjacent blocks. Here, the composite pixel refers to a plurality of blocks used for predicting the pixel of the prediction block. For example, when predicting in the horizontal direction, it is one row (803) of an adjacent block adjacent to the left side of the prediction target row. In this case, a prediction value is calculated from the value of the composite pixel 803 by an interpolation formula, and the value of the pixel in one row of the adjacent prediction block is predicted. For this prediction, for example, a Newton forward interpolation formula is used. For example, the number n of the composite pixel of the adjacent block and the prediction target pixel of the prediction block is set to 0 to 7, the signal level of the pixel n is set to yn, and n = 0, 1, 2, 3 (encoded) to n = The case of obtaining 4, 5, 6, 7 (unencoded) will be described. At this time, the signal level predicted value yn of the pixel n is calculated by Expression (1).

  The residual component is obtained by subtracting the predicted value calculated by Equation (1) from the pixel value of the predicted block. For example, when n = 7, the residual component of the pixel 7 is a value obtained by subtracting y7 from the actual value of the pixel 7. Similarly, when predicting in an oblique direction, the number n (n = 0 to 7) is assigned to the pixel of the prediction block to be predicted and the pixel of the adjacent block that is the basis of prediction, and the equation ( A predicted value is calculated according to 1) (805). When the number of pixels to be predicted is less than 4, calculation is performed with the range of n limited. For example, when the number of pixels to be predicted is 3, n = 0 to 6 is set (806).

  Equation (1) is an example of a mathematical formula that is applied when the block size is 4 × 4 pixels, but the prediction method according to the present embodiment is for other block sizes (for example, 16 × 16 pixels). Can also be applied. In general, when predicting the signal level values of the pixels m + 1 to n from the known signal level values of the pixels 0 to m for the pixels numbered from 0 to n, the predicted value yn is expressed by the equation (2). ). Here, nCj is a binomial coefficient.

  Next, the residual by the conventional prediction method is compared with the residual by the prediction method of the present embodiment. The signal level 808 according to the conventional prediction method is the same as the signal level of the pixel 3 (boundary pixel). On the other hand, the signal level 809 according to the prediction method of the present embodiment is a value calculated by Expression (1).

  For example, when the signal levels of the pixels 0 to 3 change at a constant rate, the actual signal levels 810 of the pixels 4 to 7 often change at a rate close thereto. That is, the actual signal level 810 is often closer to the signal level 809 according to the prediction method of the present embodiment than the signal level 808 according to the conventional prediction method. As a result, the residual 812 according to the prediction method of the present embodiment is smaller than the residual 811 according to the conventional prediction method, and the code amount is reduced. In Expressions (1) and (2), the signal levels of all known pixels are used for prediction, but prediction can also be performed using the signal levels of some pixels.

  Note that a least square method may be applied to this embodiment. In this case, a linear expression that is estimated by a least square method from a known signal level is used. For example, similarly to the above, based on the known signal level yn of the pixel n when n = 0, 1, 2, 3, the signal level yn of the pixel n when n = 4, 5, 6, 7 is obtained. If so, equation (3) is used. Here, m is the number of pixels serving as a basis for prediction, and in this example, m = 4.

  FIG. 9 is a block diagram illustrating functions of the intra prediction unit 205 to which the intra prediction method according to the third embodiment of this invention is applied.

  The intra prediction unit 205 according to the present embodiment uses the intra block coding mode of the prediction block (that is, the prediction direction 306 in FIG. 3 above) based on the signal level information 907 of the adjacent block that has already been encoded and the current image. ) Is performed, prediction processing is performed, and information 906 on the encoding mode and residual component is transmitted to the mode control unit 213 and the arithmetic conversion unit 206.

  The intra prediction unit 205 includes an intra prediction control unit 901 and a plurality of prediction units corresponding to the respective encoding modes. The plurality of prediction units are classified into those that perform the prediction using the conventional boundary pixels and those that perform the prediction using the composite pixel of the present embodiment, each of which is shown in the prediction direction 306 of FIG. Includes those that make predictions in each direction.

  For the sake of simplicity, FIG. 9 illustrates a vertical boundary pixel mode prediction unit 902 that predicts in the vertical direction (“0: Vertical” in FIG. 2) using boundary pixels, and a horizontal direction (FIG. 2, “1: Horizontal”), a horizontal boundary pixel mode prediction unit 903 that predicts in a vertical direction using a composite pixel, a horizontal composite pixel mode prediction unit 904 that predicts in a vertical direction using a composite pixel, and a horizontal prediction using a composite pixel. Only the directional composite pixel mode prediction unit 905 is shown, but actually, a prediction unit that uses boundary pixels and a prediction unit that uses composite pixels are provided for each of the other prediction directions 306 in FIG.

  The intra prediction control unit 901 manages an intra prediction method. That is, it checks whether a block adjacent to the prediction block is usable, and controls information for intra prediction.

  Each mode prediction unit 902 to 905 predicts the signal level of the pixel of the prediction block from the signal level of the pixel of the adjacent block, and calculates a residual component from the signal level of the actual pixel. Next, the mode information with the smallest calculated residual component is transmitted to the mode control unit 213, and the residual component calculated in that mode is transmitted to the arithmetic conversion unit 206.

  The vertical boundary pixel mode prediction unit 902 predicts downward using the boundary pixel of the adjacent block above the prediction block. The horizontal boundary pixel mode prediction unit 903 performs prediction in the right direction using the boundary pixel of the adjacent block on the left side of the prediction block.

  The vertical direction composite pixel mode prediction unit 904 and the horizontal direction composite pixel mode prediction unit 905 perform intra prediction using the composite pixel of the present embodiment. The vertical composite pixel mode prediction unit 904 uses a composite pixel (that is, a boundary pixel and an internal pixel) of an adjacent block on the upper side of the prediction block, and predicts a signal level prediction value using Equation (1) or Equation (2). And a residual component with the actual signal level is obtained. The horizontal composite pixel mode prediction unit 905 calculates a prediction value of the signal level according to the formula (1) or the formula (2) using the composite pixel of the adjacent block on the left side of the prediction block, and the residual of the actual signal level. Find the difference component.

  FIG. 10 is an explanatory diagram of an intra prediction procedure according to the third embodiment of this invention.

  First, the intra prediction control unit 901 checks the intra prediction mode (step 1001). That is, it is determined whether or not the pixels of the adjacent block can be used, and the pixels to be used for prediction are specified in the applicable prediction direction 306 and each prediction direction 306 based on the determination result.

  Next, the mode prediction units 902 to 905 perform intra prediction for the applicable prediction direction 306. In FIG. 10, as an example, prediction using a conventional boundary pixel and prediction using a composite pixel of the present embodiment are performed for each of the vertical direction and the horizontal direction (steps 1002 to 1005).

  Next, an optimal intra prediction mode is selected from the prediction results of steps 1002 to 1005 (step 1006). As a result of the prediction in steps 1002 to 1005, a total value for each mode of the residual component for each pixel is calculated. Since the code amount after encoding can be reduced as the mode having a smaller total value, the mode having the smallest total value of residual components is selected as the optimum mode. Next, the selected residual component is transmitted to the arithmetic conversion unit 206, and information on the selected mode is transmitted to the mode control unit 213. In order to make a more accurate determination, the mode selection unit 214 may select a mode having the smallest total value of residual components after the arithmetic conversion by the arithmetic conversion unit 206 as the optimum mode. This completes the intra prediction process.

  The mode selection unit 214 adds a flag indicating the selected optimum mode to the output code stream.

  In FIG. 10, only the prediction in the vertical direction and the horizontal direction is described for the sake of simplicity, but actually, prediction is performed for all prediction directions determined by the intra prediction control unit 901 to be applicable, The optimum mode can be selected from the result.

  9 and 10 describe the video encoding apparatus 101, the intra prediction method of the present embodiment can be similarly applied to the video decoding apparatus 501. In this case, the intra prediction unit 605 includes a composite pixel mode prediction unit for each prediction direction 306 in addition to the conventional boundary pixel mode prediction unit, and performs intra prediction according to the prediction mode of the input encoded data. Thus, the video decoding apparatus 501 corresponding to the intra prediction method of the present embodiment can be realized.

  That is, the mode determination unit 602 refers to the flag attached to the stream of the image to be decoded, and determines the intra prediction mode selected when the image is encoded. Next, the intra prediction unit 605 performs prediction according to the determined intra prediction mode, and decodes the image. For example, if the image to be decoded is encoded in the boundary pixel mode in the right direction, the signal level of the pixel of the prediction block is predicted to be the same as the boundary pixel of the adjacent block on the left side of the prediction block. And decrypt. Further, for example, when the image to be decoded is encoded in the composite pixel mode in the right direction, the signal level of the pixel of the prediction block is expressed in the composite pixel of the adjacent block on the left side of the prediction block ( 1) is applied to calculate a prediction value and decode it.

  According to the third embodiment of the present invention described above, the amount of code after encoding is reduced by performing highly accurate intra prediction using composite pixels (pixels in a block) (that is, the compression rate). (High) video encoding device and video decoding device corresponding to this can be provided.

  FIG. 11 is an explanatory diagram of a data recording medium according to the fourth embodiment of this invention.

  The data recording medium 1101 is a recording medium when the output device 107 or the input device 505 is a storage device, and is, for example, a magnetic disk. The encoded data created by the video encoding apparatus 101 according to the first embodiment of the present invention is recorded as a data string 1102 on the data recording medium 1101. The data string 1102 is recorded as an encoded stream according to a certain grammar. Hereinafter, H.C. An example of the H.264 / AVC standard will be described.

  H. In H.264 / AVC, a stream includes a sequence parameter set 1103, a picture parameter set 1104, and slices 1105, 1106, and 1107. Hereinafter, a case where one image (frame) is stored in one slice will be described as an example.

  In a stream in which conversion processing such as left-right symmetric conversion, up-down symmetric conversion, or rotation conversion is performed on the entire frame, a slice header 1108 is recorded at the head of the slice 1105, and the slice header 1108 includes a conversion process. Information 1109 such as a flag to be stored is stored. As contents of this information, a flag indicating whether or not conversion processing has been performed and, when a partial area of the image is converted, position information of the partial area and the like are stored. A flag indicating whether or not conversion processing has been performed may be one bit each for left-right symmetric conversion and up-down symmetric conversion. In the case of rotation conversion, if the number is incremented by 1 every 90 degrees clockwise, 360 degrees can be expressed with 2 bits. The position information of the area subjected to the conversion process is information such as an abscissa, ordinate, width, and height in the image. Further, when the conversion process is performed for each macroblock, these pieces of information can be stored in a portion where a macroblock flag is recorded instead of the slice header.

  With regard to the flag indicating whether or not the above-described conversion processing has been performed, when the conversion processing is performed only on the intra prediction frame, the conversion processing is performed on the inter prediction frame, or in units of GOP (group of pictures). A flag indicating information such as when conversion processing may be stored. An example in which conversion is performed for an inter prediction frame will be described later.

  Further, when the intra prediction method according to the third embodiment of the present invention is applied, A new mode using composite pixels is added to the H.264 / AVC standard encoded stream. In this case, a numerical value indicating a new mode is added to information 1109 such as a conventional flag and stored. As a new mode expression format, a bit indicating that prediction is performed using composite pixels may be added, or a numerical value obtained when mode prediction is performed using composite pixels may be assigned to each prediction direction. Good.

  FIG. 12 is an explanatory diagram of a packet according to the fifth embodiment of this invention.

  FIG. 12 is a diagram illustrating, as an example, a procedure in which the output I / F 106 of the video encoding device 101 generates an IP packet to be transmitted to the IP network from the data string 1102 in FIG.

  First, the data string 1102 is divided into a predetermined size, TCP headers 1204 to 1206 are attached, and TCP segments 1201 to 1203 are generated. FIG. 12 shows an example in which one segment is generated from one slice. At this time, the slice header 1108 included in each slice is also included in the segment. As described in FIG. 11, information 1109 such as a flag related to the conversion process is stored in the slice header 1108. Although not described in FIG. 12, segments are similarly generated for slices 1107 and later.

  Next, each segment is divided into a predetermined size, an IP header is added, and an IP packet is generated. For example, the segment 1202 is divided into a predetermined size, IP headers 1209 and 1210 are added, and IP packets 1207 and 1208 are generated. Here, only IP packets 1207 and 1208 generated from a part of the segment 1202 are shown, but an IP packet is generated in the same manner for the entire segment 1202. Although not described in FIG. 12, the other segments 1201 and the like are divided in the same manner to generate an IP packet.

  The IP packet 1209 and the like are generated by, for example, the output I / F 106 of the video encoding device 101 and transmitted to the output device 107 via an IP network (not shown).

  Further, the IP packet 1209 and the like include a storage device in which the data string 1102 is stored and an output interface that generates and transmits the IP packet 1209 and the like from the data string 1102 and has no video encoding function ( (Not shown) may be transmitted.

  Similarly to FIG. 12, the video encoding device 101 or the packet transmission device may generate a wireless packet from the data string 1102 and transmit the wireless packet to the output device 107 via the wireless packet communication network.

  FIG. 13 is a block diagram illustrating functions of the video encoding device 101 according to the sixth embodiment of the present invention. This relates to the video encoding apparatus according to the first embodiment of the present invention shown in FIG. 2 and describes the case where the conversion process is performed also in inter prediction.

  The video encoding apparatus 101 includes an original image memory 201, an image conversion unit 202, an encoding unit 1302, a mode control unit 213, and a mode selection unit 214. Here, the original image memory 201, the image conversion unit 202, the mode control unit 213, and the mode selection unit 214 are the same as those in the video encoding apparatus according to the first embodiment. The video encoding apparatus according to the sixth embodiment is characterized in that the encoding unit 1302 includes an image conversion unit 1301 for performing inverse conversion on the predicted image controlled by the mode control unit 213.

  The conversion process performed by the image conversion unit 1301 is similar to the process performed by the image conversion unit 202 described above, but the target is a reference frame. That is, the image conversion unit 1301 performs, for example, line-symmetric conversion or point-symmetric conversion of the reference frame. The line symmetry conversion is, for example, left-right symmetry conversion that inverts the reference frame left and right or up-down symmetry conversion that inverts the reference frame up and down. Point symmetry conversion is, for example, rotation conversion that rotates a reference frame. The image conversion unit 1301 may perform a combination of these conversions.

  Note that the video encoding apparatus 101 according to the present embodiment includes two image conversion units 202 and 1301 as shown in FIG. However, these may be physically one image conversion unit. That is, the video encoding apparatus 101 may include one image conversion unit, and the image conversion unit may be used as the image conversion unit 202 and the image conversion unit 1301 by switching input / output thereof.

  FIG. 14 is an explanatory diagram of a video encoding procedure according to the sixth embodiment of this invention. This relates to the video encoding procedure of the first embodiment of the present invention shown in FIG. 4 and describes the case where the conversion process is performed also in the inter prediction.

  Among these, the procedure differs from the video encoding of the first embodiment in that the image conversion unit 1301 performs mode control in step 1401 before encoding an image that has already undergone left-right symmetry conversion in step 403. This is a part for converting the reference image in accordance with the information indicating whether or not the conversion process transmitted to the unit 213 has been performed. The image conversion method is the same as the conversion method in step 402, and the processing content is the same as in FIG.

  As a result, the image conversion units 202 and 1301 encode the encoded frame stored in the original image memory 201 and the reference frame stored in the predicted image memory 212 by performing conversion processing in the same direction. Processing such as motion prediction in inter prediction may be the same as that in the existing method. Also in this embodiment, if the method in which the code amount is reduced with respect to the same level of image quality is compared between the case where the entire image is symmetrically converted and the case where it is not converted, the encoding efficiency can be improved. The better data can be adopted.

  FIG. 15 is a block diagram illustrating functions of the video decoding apparatus 501 according to the seventh embodiment of this invention. This relates to the video decoding apparatus according to the second embodiment of the present invention shown in FIG. 6 and describes the case where the conversion process is also performed in inter prediction.

  The video decoding apparatus 501 includes a stream analysis unit 601, a mode determination unit 602, a decoding unit 1502, an image conversion unit 610, and a decoded image memory 611. Here, the stream analysis unit 601, the mode determination unit 602, the image conversion unit 610, and the decoded image memory 611 are the same as those of the video decoding apparatus according to the second embodiment. In the video decoding apparatus according to the seventh embodiment, the decoding unit 1502 includes an image conversion unit 1501 for performing inverse conversion on a predicted image (reference frame) controlled by the mode determination unit 602. And The conversion process performed by the image conversion unit 1501 is the same as the process performed by the image conversion unit 610 described above.

  In the decoding of inter prediction according to the present invention, when the mode determination unit 602 detects an inversion flag, the image conversion unit 610 and the image conversion unit 1501 in the decoding unit are instructed to perform conversion processing, and the reference frame is processed. After performing the inversion process, the current frame is inter-predicted and decoded, and the resulting image is inverted and output.

  Note that the video decoding apparatus 501 according to the present embodiment includes two image conversion units 610 and 1501 as shown in FIG. However, these may be physically one image conversion unit. That is, the video decoding apparatus 501 may include one image conversion unit, and the image conversion unit may be used as the image conversion unit 610 and the image conversion unit 1501 by switching the input / output.

  FIG. 16 is an explanatory diagram of a video decoding procedure according to the seventh embodiment of this invention. This relates to the video decoding procedure of the second embodiment of the present invention shown in FIG. 7 and describes the case where conversion processing is also performed in inter prediction.

  Among these, the procedure differs from the video decoding of the second embodiment in the part of converting the reference frame in step 1601 before performing the image decoding in step 702. The image conversion method is the same as the conversion method in step 703, and the processing content is the same as in FIG.

  According to the sixth and seventh embodiments of the present invention described above, the amount of code after encoding is reduced by performing inter prediction in the direction with high prediction accuracy without changing the scan direction (that is, It is possible to provide a video encoding device (with a high compression rate) and a video decoding device corresponding thereto. As a result, the amount of code can be reduced by about 10% compared to the conventional coding method.

  In addition, this invention is H.264. The present invention is not limited to H.264 / AVC and can be applied to video encoding devices and video decoding devices based on various standards.

  The present invention can be used for recording and transmission of image data, and contributes to a reduction in recording capacity and an increase in transmission speed by improving the compression rate and reducing the amount of data. For example, the present invention can be used for a video recorder and a video player using a hard disk or a DVD. In addition, the present invention can be used for an image distribution service using a wired or wireless communication network including a mobile phone and a television broadcast. Further, the present invention can be used for a video phone, a video conference system, and the like.

101 Video encoding device 102, 502 Processor 103, 503 Memory 104, 504 Input interface (I / F)
105, 505 Input device 106, 506 Output interface (I / F)
107, 507 Output device 201 Original image memory 202, 610, 1301, 1501 Image conversion unit 203, 1302 Encoding unit 204, 604 Motion prediction unit 205, 605 Intra prediction unit 206 Arithmetic conversion unit 207 Quantization unit 208 Coefficient encoding unit 209, 607 Inverse quantization unit 210, 608 Inverse arithmetic conversion unit 211, 611 Decoded image memory 212, 609 Predicted image memory 213 Mode control unit 214 Mode selection unit 501 Video decoding device 601 Stream analysis unit 602 Mode determination unit 603, 1502 Decoding unit 606 Coefficient analysis unit 901 Intra prediction control unit 902 Vertical direction boundary pixel mode prediction unit 903 Horizontal direction boundary pixel mode prediction unit 904 Vertical direction composite pixel mode prediction unit 905 Horizontal direction composite pixel mode prediction unit 1101 Data recording medium 1102 Data Column

Claims (2)

In a decoding device for decoding a signal of an encoded image,
Wherein the portion of the image, whether conversion by symmetric mode for inverted transform or the lateral direction by the vertically symmetric mode for inverting the vertical direction is such, the determination unit according to the information sent come together with the signal of the image When,
And inter prediction processing using the reference image of the image and the image, performs a conversion process corresponding to the conversion that is determined by the determination unit, Bei example a decoded image generating unit for generating a decoded image,
The decoding apparatus, wherein the information includes information indicating a direction of inversion for a part of the image and information on a position of the inverted area . About a decoding method for decoding a signal of an encoded image,
A determination step of determining, based on information sent together with the signal of the image, whether a part of the image has been converted in a vertically symmetric mode that is inverted in the up-down direction or converted in a left-right symmetric mode that is inverted in the left-right direction; ,
An inter prediction process using the image and a reference image of the image, and a decoded image generation step of performing a conversion process corresponding to the conversion determined in the determination step and generating a decoded image,
  The decoding method, wherein the information includes information indicating a direction of inversion for a part of the image and information on a position of the inverted region.

Images