JP2010041161A - Image decoder, image decoding method, image encoder, and image encoding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce an amount of operation processing upon performing high definition processing on an image. <P>SOLUTION: An image decoder is provided with: a decoding part for decoding an encoded image included in an encoding stream to be inputted to generate a decoded image; and an image high definition part for performing high definition processing on the decoded image generated by the decoding part. The image high definition part has: an object to be processed determination part for determining a region of high definition processing object on which high definition processing is performed in the decoded image on the basis of information included in the encoding stream; and an object region high definition part which mixes corresponding high frequency component data with image data in the region of high definition processing object in the decoded image to perform high definition processing. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image decoding device, an image decoding method, an image encoding device, and an image encoding method.

  A technique called Image Hallucination has been proposed as a method for obtaining a high-frequency component in a region targeted for high definition processing. For example, in Non-Patent Document 1, a pair group of normalized low-frequency components and high-frequency components is held, and a high frequency corresponding to a low-frequency component in a region targeted for high definition processing is based on the pair group. A technique for obtaining a high-frequency component in the region by integrating the components is disclosed.

Jian Sun, et al. Image Hallucination with Primal Sketch Priors, IEEE Conf.Computer Vision and Pattern Recognition (CVPR), 2003, P5, Figure5

  When the technique described in Non-Patent Document 1 is simply applied to the entire frame, there is a problem that the amount of calculation processing at the time of high definition is large.

  The present invention has been made in view of the above problems, and an object of the present invention is to reduce the amount of calculation processing when making an image high-definition.

  In order to solve the above-described problems, an embodiment of the present invention may be configured as described in the claims, for example.

  According to the present invention, it is possible to reduce the amount of calculation processing when making an image high-definition.

  Hereinafter, embodiments of the present invention will be described.

  The first embodiment of the present invention will be described below.

  FIG. 1 is an explanatory diagram of image transmission from an image encoding device to an image decoding device according to Embodiment 1 of the present invention. An image encoding device (200) that encodes input image data and transmits the encoded image data via a transmission path as an encoded stream, and an image decoding device (100) that decodes image data from the transmitted encoded stream Is shown.

  The image decoding device (100) includes a decoding unit (101) that decodes image data from an encoded stream, and an image enhancement unit (102) that enhances the decoded image data. Prepare.

  Each unit described above may be configured by hardware or software. Moreover, the module which combined hardware and software may be sufficient. The transmission path may be a network or a broadcast wave.

  The image encoding device (200) in FIG. 1 is configured as shown in FIG. 2, for example. The original image memory (201) stores an input image. The subtracter (202) calculates the difference between the input image and the predicted image. The frequency conversion unit (203) performs frequency conversion on the difference image between the original image and the predicted image calculated by the subtracter (202). The quantization unit (204) quantizes the image data frequency-converted by the frequency conversion unit (203). The variable length coding unit (205) performs variable length coding on the image data quantized by the quantization unit (204).

  The inverse quantization unit (206) inversely quantizes the image data quantized by the quantization unit (204). The inverse frequency transform unit (207) performs inverse frequency transform on the image data inversely quantized by the variable length coding unit (206). The adder (208) adds the predicted image generated by the predicted image generation unit (210) to the image data subjected to the reverse frequency conversion by the reverse frequency conversion unit (207), and generates a decoded image. The frame memory (209) stores the decoded image generated by the adder (208). The predicted image generation unit (210) performs inter-screen prediction or intra-screen prediction using the image data stored in the frame memory (209), and generates a predicted image.

  According to the image encoding device of FIG. 2, the input image data can be encoded, and the encoded image data can be transmitted as an encoded stream to the image decoding device via a transmission path.

  The image encoding device (200) according to the present embodiment is not limited to the above configuration, and may be configured by an existing encoding scheme such as MPEG-2 / MPEG-4, H.264, or the like.

  The image decoding apparatus (100) in FIG. 1 is configured as shown in FIG. 3, for example.

  The variable length decoding unit (301) decodes the encoded stream sent from the encoding side. The syntax analysis unit (302) analyzes the syntax of the image data decoded by the variable length decoding unit (301), and generates encoding parameters and quantized image data. The inverse quantization unit (303) inversely quantizes the quantized image data sent from the syntax analysis unit (302). The inverse frequency transform unit (304) performs inverse frequency transform on the image data inversely quantized by the inverse quantization unit (303). The adder (305) adds the image data subjected to the inverse frequency conversion by the inverse frequency conversion unit (304) and the prediction image generated by the prediction image generation unit (308) to generate a decoded image. The frame memory (306) stores the decoded image generated by the adder (305). The predicted image generation unit (307) generates a predicted image using the image data stored in the frame memory (306).

  The image refinement unit (102) refines the decoded image generated by the adder (305) and outputs the highly refined image data.

Next, details of the image enhancement unit (102) will be described with reference to FIG. As shown in FIG. 4, the image refinement unit (102) receives the encoding parameters from the syntax analysis unit (302) and the decoded image from the adder (305). 401) determines a region for high definition using the encoding parameter and the decoded image input to the image high definition unit (102). The high definition processing will be described later with reference to FIG. Here, the processing target region determination unit (401) excludes a region in which a large image quality deterioration has not occurred during the encoding process from the high definition processing target using the encoding parameter. By the exclusion process, it is possible to reduce the amount of calculation processing at the time of high definition process described later. In the processing of the processing target area determination unit (401), the area may be a rectangular area such as a macro block or a sub macro block.

  Next, the target area refinement unit (402) refines the area selected by the processing target area determination unit (401) using dictionary data stored in the dictionary data storage unit (403) in advance.

  The dictionary data in the present invention will be described with reference to FIG. The dictionary data in the present invention is a database showing a pair group of a low frequency component (552) and a high frequency component (551) in a partial region in an image. The dictionary data is stored in advance in the dictionary data storage unit (403). Alternatively, when the URL of dictionary data is encoded in the encoded stream, the decoding apparatus may acquire the dictionary data from this URL in advance. Accordingly, even when the dictionary data is updated, the high definition processing can be performed using the latest dictionary data.

  Here, the partial area in the image for generating the dictionary data is a lot of images such as lines, edges, corners, T-shaped pixel value boundaries, that is, pixel value boundaries that have a predetermined shape. You may restrict to the area | region containing a common element.

  Next, a processing flow of the image enhancement unit (102) of the first embodiment will be described with reference to FIG. In the processing flow of the image enhancement unit (102) shown in FIG. 6, for example, a region to be enhanced is determined using a quantization parameter. If the quantization parameter is smaller than a predetermined threshold, it is determined that the image is less distorted, and the block is shifted to the next block without performing the high definition processing. If the quantization parameter is larger than a predetermined threshold value, it is determined that the image has a large amount of distortion, and high definition is performed using dictionary data. The above threshold value may be encoded in the encoded stream and updated as needed. In this case, this threshold value is input as an encoding parameter.

  The decoded image and the encoding parameter are input to the image refinement unit (102) (601, 602).

  Next, of the input encoding parameters, for example, a region to be refined is determined using a quantization parameter (603). If the quantization parameter is smaller than the predetermined threshold value, the process proceeds to step (606). If the quantization parameter is larger than the predetermined threshold, the process proceeds to step (604). Note that the high-definition processing target region determination processing in step (603) is performed by the processing target region determination unit (401) in FIG.

  In step (604), the region determined as the high definition processing target region in step (603) is used as a reference image, and the low frequency component corresponding to the region is specified from the low frequency component in the dictionary data. That is, the residual between each low-frequency component stored in the dictionary data storage unit and the high-definition processing target area is obtained, and the low-frequency component that minimizes the residual is specified. Using the ID, which is identification information indicating the identified low frequency component, as a key, the high frequency component in the dictionary data corresponding to the low frequency component is extracted. The extraction processing in step (604) is performed by the processing target region determination unit (401) or the target region high definition unit (402) in FIG. Then, it moves to step (605).

  In step (605), the high-frequency component extracted in step (604) is mixed with the area determined in step (603) to achieve high definition. Here, in the description of the present specification and each drawing, mixing means high-definition such as weighted addition in which a pixel value of an addition or high-definition processing target region and a pixel value of an extracted high-frequency component are multiplied and added. A new pixel value is generated based on a function in which each of the pixel value of the region to be processed and the extracted high-frequency component pixel value is a variable. The high definition processing in step (605) is performed by the target region high definition section (402) in FIG. Then, it moves to step (606).

  In step (606), it is determined whether or not all the regions in the frame have been processed. When all the areas in the frame have been processed, the high definition processing in the frame is finished. When the processing for all the high definition processing target areas in the frame has not been completed, the process proceeds to the next block and the processing is repeated from step (603). The intra-frame determination process in step (606) is performed by a control unit (not shown).

  That is, in the processing flow of the image enhancement unit (102) illustrated in FIG. 6, for example, a region to be enhanced is determined using a quantization parameter. If the quantization parameter is smaller than a predetermined threshold, it is determined that the image is less distorted, and the block is shifted to the next block without performing the high definition processing. If the quantization parameter is larger than a predetermined threshold value, it is determined that the image has a large amount of distortion, and high definition is performed using dictionary data.

  As described above, according to the processing flow of FIG. 6, the target area to be refined is determined using the value of the encoding parameter, and the refinement is performed only on the target area. The amount of calculation processing is reduced. The above threshold value may be encoded in the encoded stream and updated as needed.

  In the image encoding device, the image decoding device, and the method thereof in Embodiment 1 described above, the image decoding device performs high definition using the encoding parameters included in the normal encoded stream. Since the area is determined, the image encoding apparatus does not need to send extra additional information during transmission. In addition, since the target region to be refined is determined using the value of the encoding parameter and the refinement is performed only on the target region, the amount of calculation processing at the time of high definition is reduced.

  Thus, according to the image encoding device, the image decoding device, and the method thereof in the present embodiment, it is possible to suppress the amount of generated code and the amount of calculation processing when the decoded image is highly refined.

  In the above example, it has been described that the high-definition processing is performed for any region determined to be the high-definition processing determination region by the encoding parameter. Of the partial areas in the image, there are many areas such as lines, edges, corners, T-shaped pixel value boundaries, that is, pixel value boundaries that have a predetermined shape, among partial areas in the image. High definition processing can be performed only in a region including elements common to images. In this case, the high definition processing target region may be limited to a region including the boundary portion of the pixel value having the predetermined shape described above.

  That is, when the high definition processing is performed only in the region including the element, an element determination step is provided after step (603) which is the high definition processing target region determination step in FIG. In the determination step, an edge extraction filter such as a Canny filter is used to determine whether the block is a characteristic part such as an edge or a corner. The filter to be used may be a Laplacian filter or a Gabor filter. When it is determined by the determination unit that the target block is not a characteristic part such as an edge or a corner, the process proceeds to step (606). When the determination unit determines that the block is a characteristic part such as an edge or a corner, the process proceeds to step (605), and high definition is performed only for the characteristic part. Note that the determination processing in the element determination step is performed by the processing target region determination unit (401) in FIG.

  In this way, when the region to be subjected to high definition processing is limited to a region including a boundary portion of a pixel value having a predetermined shape such as a boundary portion of a line, edge, corner, or T-shaped pixel value, For example, it is possible to prevent generation of an unnatural image by performing high definition for noise or the like in the image.

  Further, if the data to be stored in the dictionary data is limited to the area, the storage capacity necessary for holding the dictionary data and the calculation amount when specifying the dictionary data can be reduced, and matching errors can be reduced.

  Next, the configuration and operation of the present invention according to Embodiment 2 will be described.

  As shown in FIG. 4, the image encoding apparatus, the image decoding apparatus, and the method thereof according to the first embodiment are subject to a high-definition processing according to the encoding parameter included in the encoded stream in the image high-definition unit (102). Determine the area. On the other hand, in the image encoding device, the image decoding device, and the method thereof according to the second embodiment, the image encoding device (200) generates and transmits additional information, and the high definition processing indicated by the additional information is performed. The image decoding apparatus (900) performs high definition processing on the target area.

  A configuration example of the image encoding device (700) in the present embodiment is shown in FIG. The image coding apparatus (700) in the present embodiment has the same configuration as the image coding apparatus (200) in the first embodiment, except for the configuration of the processing target region determination unit (701) and the variable length coding unit (702). Since this is an operation, the description of other configurations and operations is omitted. Here, in the image encoding device (700) according to the present embodiment, a configuration excluding the processing target region determination unit (701) is referred to as an encoding unit (703).

  In FIG. 7, the processing target region determination unit (701) inputs the encoding parameter output from the quantization unit (204) or the predicted image generation unit (210) and the decoded image stored in the frame memory (209). As described above, an area for high definition is determined, and an ID (address) that is identification information indicating the position of the area is generated.

  Here, as a coding parameter, SAD (Sum of Absolute Difference), SSD (Sum of Square Difference), SATD (Sum of Absolute Transformed Difference), a quantization parameter, or the like can be used. Alternatively, the high-definition processing target area determination may be performed using a uniquely designed encoding parameter. The SAD, SSD, and SATD are generated by the predicted image generation unit (210), and the quantization parameter is generated by the quantization unit (204).

  The variable length encoding unit (702) encodes the ID (address) generated by the processing target region determination unit (701), and encodes encoded image data as additional information that is identification information data indicating the high definition processing target region. Are transmitted to the image decoding apparatus as an encoded stream. Here, the transmission method may be a simple sequence of coordinate values, or a difference value from an adjacent block may be encoded. Alternatively, other compression encoding methods may be used.

A processing flow of the image encoding device (700) according to the second embodiment will be described with reference to FIG. In the processing flow of the image encoding device (700) shown in FIG. 8, the image encoding device uses the encoding parameter to determine the encoding cost, determines the area to be refined, and determines the position of the area. An ID (address) indicating is generated and output. As a result, if SAD, SSD, SATD and quantization parameters are smaller than a predetermined threshold, it is determined that the coding cost is low, that is, a region with little image distortion, and high definition processing is performed on the block. Absent. When SAD, SSD, SATD, and the quantization parameter are larger than a predetermined threshold, it is determined that the coding cost is high, that is, the image is highly distorted, and high definition is performed using dictionary data. The above threshold value may be encoded in the encoded stream and updated as needed.

  The decoded image and the encoding parameter are input to the processing target area determination unit (701) (801, 802).

  Next, the coding cost is judged using the inputted coding parameter (803). Here, the coding cost is a numerical value calculated by a function having a coding parameter as a variable. For example, if the encoding parameter is SAD, SSD, SATD, or a quantization parameter, if the SAD, SSD, SATD, or quantization parameter is greater than a threshold value, the encoding cost is large, and the process proceeds to step (804). If SAD, SSD, SATD, and the quantization parameter are smaller than the threshold value, the encoding cost value is small, and the process proceeds to step (805). The encoding cost determination process in step (803) is performed in the processing target area determination unit (701).

  In step (804), the region determined in step (803) is determined as a high-definition processing target region, and an ID (address) that is identification information indicating the high-definition processing target region is generated and output. The determination and the ID (address) generation processing are performed by the processing target region determination unit (701), but may be performed by the variable length coding unit (702). Then, it moves to step (805).

  In step (805), it is determined whether or not the region for high definition processing in the frame has been completely processed for the region determined in step (804). When the processing for all the high definition processing target areas in the frame has been completed, the high definition processing for the frame ends. If all of the high definition processing target areas in the frame have not been processed, the process proceeds to the next block and the process is repeated from step (803). The determination process in step (805) is performed in the process target area determination unit (701).

  That is, in the processing flow of the image encoding device (700) shown in FIG. 8, the image encoding device uses the encoding parameter to determine the encoding cost, determines the region to be refined, and the region Generate and output an ID (address) indicating the location of As a result, if SAD, SSD, SATD and quantization parameters are smaller than a predetermined threshold, it is determined that the coding cost is low, that is, a region with little image distortion, and high definition processing is performed on the block. Absent. When SAD, SSD, SATD, and the quantization parameter are larger than a predetermined threshold, it is determined that the coding cost is high, that is, the image is highly distorted, and high definition is performed using dictionary data.

  In addition, parameters such as SAD, SSD, and SATD that cannot be referred to by the normal image decoding apparatus can also be used for determination as coding parameters. As a result, it is possible to determine the region for higher definition processing with higher accuracy. When SAD, SSD, and SATD are smaller than a predetermined threshold value, it is determined that the coding cost is low, that is, the image distortion is small, and the high-definition processing is not performed in the block, and the process proceeds to the next block. If SAD, SSD, and SATD are larger than a predetermined threshold, it is determined that the coding cost is high, that is, the image has a large amount of distortion, and high definition is performed using dictionary data.

  As described above, according to the flow of the processing in FIG. 8, in order to determine a region for high definition in the image encoding device, it is possible to simplify the determination processing of the region for high definition processing on the image decoding device side. This can reduce the amount of calculation processing in the image decoding apparatus. In addition, in the determination process for the high-definition processing target area, since it is possible to refer to the encoding parameters that cannot be referred to on the decoding side, it is possible to determine the high-definition processing target area with higher accuracy.

  Here, a configuration example of the image enhancement unit (900) in the present embodiment is shown in FIG.

  The configuration of the image decoding apparatus in the present embodiment will be described. The image decoding apparatus according to the present embodiment has the same configuration and operation as the image decoding apparatus (100) according to the first embodiment except for the configuration of the image enhancement unit (900). Description is omitted.

  The image enhancement unit (900) in the present embodiment has the same configuration and operation as the image enhancement unit (102) in Example 1 except for the configuration of the processing area determination unit (901). The description of the configuration and operation is omitted.

  As shown in FIG. 9, the image refinement unit (900) has an ID that is identification information indicating the decoded image from the adder (305) and the region to be refined from the image encoding device (700). Additional information such as (address) is input.

  The processing target area determination unit (901) determines an area to be refined using the additional information and the decoded image input to the image refinement unit (900). The high definition processing will be described later with reference to FIG. Here, the processing target region determination unit (901) identifies a region where no significant image quality degradation has occurred during the encoding process from the processing target by referring to the additional information transmitted from the image encoding device (700). exclude. When the high-definition processing is performed in the image decoding apparatus (900) by the exclusion process, unlike the first embodiment, the identification information indicating the high-definition processing target area generated in the image encoding apparatus (700) The area where the high definition processing is performed is determined by the ID (address). Thereby, it is possible to simplify the determination processing of the high-definition processing target area on the image decoding apparatus side, and it is possible to reduce the calculation processing amount of the processing target area determination unit (901) in the image decoding apparatus.

  A processing flow of the image enhancement unit (900) of the second embodiment will be described with reference to FIG. In the processing flow of the image refinement unit (900) shown in FIG. 10, with respect to the region for high definition processing determined by the image encoding device, the dictionary data corresponding to the region for high definition processing is included. The high-frequency components are mixed to achieve high definition.

  The decoded image and additional information, which is identification information indicating the high definition processing target area, are input to the image enhancement unit (900) (1001, 1002).

  Next, in step (1003), the high-frequency processing target area indicated by the additional information is used as a reference image, and the low-frequency component corresponding to the area is identified from the low-frequency components in the dictionary data storage unit. That is, the residual between each low-frequency component stored in the dictionary data storage unit and the high-definition processing target area is obtained, and the low-frequency component that minimizes the residual is specified. Using the ID, which is identification information indicating the identified low frequency component, as a key, the high frequency component in the dictionary data corresponding to the low frequency component is extracted. The specifying process in step (1003) is performed by the processing target area determination unit (901) in FIG. Then, it moves to step (1004).

  In step (1004), the high-frequency component extracted in step (1003) is mixed with the region indicated by the ID (address), that is, the region determined as the high-definition processing target region by the image encoding device (700). Realize high definition. The high definition processing in step (1004) is performed by the target region high definition section (402) in FIG. Then, it moves to step (1005).

  In step (1005), it is determined whether or not all the areas in the frame have been processed. When all the areas in the frame have been processed, the high definition processing in the frame is finished. If the processing for all the high definition processing target areas in the frame has not been completed, the process proceeds to the next block and the process is repeated from step (1003). The intra-frame determination process in step (1005) is performed by a control unit (not shown).

  That is, in the processing flow of the image refinement unit (900) shown in FIG. 10, for the high definition processing target area determined by the image encoding device, a dictionary corresponding to the high definition processing target area. The high-frequency components in the data are mixed to achieve high definition.

  As described above, according to the processing flow of FIG. 10, unlike the processing flow of the image high-definition unit (102) of the first embodiment, the high-definition processing target is used using the identification information transmitted from the image encoding device. Since the region is determined, the determination processing of the high definition processing target region on the image decoding device side can be further simplified, and the amount of calculation processing of the image decoding device can be reduced. Further, a higher definition image can be obtained.

  In the image encoding device, the image decoding device, and these methods in the above-described second embodiment described above, the image encoding device determines the region for high definition processing, and the image decoding device uses the high definition processing target. The identification information indicating the area is transmitted.

  Thereby, according to the image coding apparatus, the image decoding apparatus, and the method thereof in the present embodiment, it is possible to reduce the calculation processing amount of the image decoding apparatus. In addition, since the types of encoding parameters that can be used for the determination processing of the high-definition processing target area increase, a higher-definition image can be obtained.

  In the above example, it has been described that the high definition processing is performed for any region determined by the encoding parameter. However, the present embodiment is not limited to this, and a portion in the image is not limited thereto. Among the typical areas, the high-definition processing target area includes elements common to many images such as lines, edges, corners, T-shaped pixel value boundaries, that is, pixel value boundaries having a predetermined shape. Only the area can be determined as the high definition processing target area.

  That is, in the case where only the region including the element is determined as a region for high definition processing, an element determination step is provided after the encoding cost determination processing step of step (803) in FIG. In the determination step, an edge extraction filter such as a Canny filter is used to determine whether the block is a characteristic part such as an edge or a corner. The filter to be used may be a Laplacian filter or a Gabor filter. When it is determined by the determination unit that the target block is not a characteristic part such as an edge or a corner, the process proceeds to step (805). When the determination unit determines that the block is a characteristic part such as an edge or a corner, the process proceeds to step (804), where the block is determined as a high-definition processing target region, and an ID ( Address) is generated and output. Note that the determination processing in the element determination step is performed by the processing target region determination unit (701) in FIG.

  As described above, when the region for high definition processing is limited to a region including a boundary portion of a pixel value having a predetermined shape such as a boundary portion of a line, edge, corner, or T-shaped pixel value, for example, decoding In the high-resolution processing on the side, it is possible to prevent generation of an unnatural image by performing high-definition for all regions including, for example, noise in the image. Further, if the data to be stored in the dictionary data is limited to the area, the storage capacity necessary for holding the dictionary data and the calculation amount when specifying the dictionary data can be reduced, and matching errors can be reduced.

  Next, the configuration and operation of the present invention related to Example 3 will be described.

  FIG. 11 shows a configuration example of an image encoding device (1100) according to the third embodiment. The image encoding device (1100) in the present embodiment has the same configuration and operation as the image encoding device (700) in Embodiment 2 except for the configuration of the processing target region determination unit (1101). The description of the operation is omitted. Here, in the image encoding device (1100) in the present embodiment, a configuration excluding the processing target region determination unit (1101) is defined as an encoding unit (1103).

  In FIG. 11, the processing target region determination unit (1101) includes an original image stored in the original image memory (201), and the encoding parameter generated by the quantization unit (204) or the predicted image generation unit (210). Then, an area to be refined is determined using the decoded image stored in the frame memory (209) as an input, and an ID (address) that is identification information indicating the position of the area is generated.

  Here, as the encoding parameter, for example, SAD (Sum of Absolute Difference), SSD (Sum of Square Difference), SATD (Sum of Absolute Transformed Difference), quantization parameter, and the like can be used. SAD, SSD, and SATD are generated by the predicted image generation unit (210), and the quantization parameter is generated by the quantization unit (204).

  In the present embodiment, the variable length encoding unit (1102) encodes ID (address) that is identification information indicating the high definition processing target region generated by the processing target region determination unit (1101), and sends the encoded information to the image decoding apparatus. And transmit. The image coding apparatus according to the present embodiment uses the original image and the decoded image, so that the photographer intentionally determines whether the original image is blurred due to loss of high-frequency components during the encoding process. It can be distinguished whether it is blurred because it is narrowed. As a result, it is possible to determine such as excluding the blurred area in the original image from the high definition processing target area.

  A processing flow of the image encoding device (1100) of the third embodiment will be described with reference to FIG. In the processing flow of the image encoding device (1100) shown in FIG. 12, the size of the encoding cost is determined using the input encoding parameter, and if the value of the encoding cost is large, frequency conversion is performed. The high-definition processing target region is determined based on the distribution of the frequency conversion coefficient generated by the processing in the frequency region, or based on the variance of the original image and the calculation result of the first and second derivatives.

  The decoded image, the encoding parameter, and the original image are input to the processing area determination unit (1101) (1201, 1202, 1203).

  Next, the size of the encoding cost is determined using the input encoding parameter (1204). Here, the coding cost is a numerical value calculated by a function having a coding parameter as a variable. For example, if the encoding parameter is SAD, SSD, SATD, or a quantization parameter, if SAD, SSD, SATD, or the quantization parameter is large, the value of the encoding cost is large, and the process proceeds to step (1205). If SAD, SSD, SATD and the quantization parameter are small, the value of the coding cost is small, and the process proceeds to step (1207).

  In step (1205), it is determined whether the degree of blur of the original image is large or small. It is determined whether the original image is blurred due to loss of high-frequency components in the encoding process or whether the photographer is blurred because the depth of field is intentionally reduced. Here, the degree of blur of the original image is determined from the frequency distribution of the frequency conversion coefficient. For example, if the frequency conversion coefficient is constantly generated up to the high frequency component, it is highly possible that the original image has a clear texture. Therefore, the degree of blur of the original image is determined to be small, and the low frequency component is distributed mainly. If so, the degree of blur of the original image is determined to be large.

  Alternatively, the degree of blurring of the original image may be determined by calculating the variance of the original image and the first and second derivatives. If it is not necessary to increase the definition of the original image, the process proceeds to step (1207). If high definition of the original image is required, the process proceeds to step (1206).

  In step (1206), the region determined in step (1204) and step (1205) is determined as a region for high definition processing, and an ID (address) that is identification information indicating the position of the region is generated and output. . Then, it moves to step (1207).

  In step (1207), it is determined whether or not all the regions in the frame have been processed for the region determined in step (1206). When all the areas in the frame have been processed, the high definition processing in the frame is finished. When the processing for all the high-definition processing target areas in the frame has not been completed, the process proceeds to the next block and the process is repeated from step (1204).

  In other words, if the frequency conversion coefficient is constantly generated up to the high-frequency component, the original image is likely to have a clear texture. Therefore, the degree of blur of the original image is determined to be small, and the low-frequency component is mainly distributed. If the original image is determined to have a high degree of blur, the original image is blurred due to loss of high-frequency components during the encoding process, or the photographer intentionally reduces the depth of field. Determine if you are blurred. Only when the original image is blurred due to loss of high-frequency components in the encoding process, it is determined as the high-definition processing determination region.

  As described above, according to the flow of the processing in FIG. 12, it is possible to determine that a region blurred in the original image is not highly defined, and it is possible to avoid high definition contrary to the photographer's intention.

  Note that the configuration and processing flow of the image decoding apparatus according to the present embodiment is the same as the configuration and processing flow of the image decoding apparatus according to the second embodiment. Then, an ID (address) that is identification information indicating the position of the area is generated and transmitted. The processing in the image decoding apparatus according to the present embodiment is the same as that according to the second embodiment because high definition processing is performed on the area indicated by the ID (address). Therefore, the configuration and operation of the image decoding apparatus according to the present embodiment are the same as those of the second embodiment, and thus description thereof is omitted.

  According to the image coding apparatus, the image decoding apparatus, and the methods thereof in the third embodiment described above, the high-definition processing target region determination can be performed by the image coding apparatus. It is possible to simplify the determination processing of the high definition processing target area in the image processing apparatus, and to reduce the calculation processing amount of the image decoding apparatus.

  In addition, since the original image can be referred to, it is possible to determine whether it is blurred because the depth of field is narrow or is lost due to loss of high-frequency components in the encoding process. High definition can be avoided.

  In the above example, it has been described that the high definition processing is performed for any region determined by the encoding parameter. However, the present embodiment is not limited to this, and a portion in the image is not limited thereto. Among the typical areas, the high-definition processing target area includes elements common to many images such as lines, edges, corners, T-shaped pixel value boundaries, that is, pixel value boundaries having a predetermined shape. Only the area can be determined as the high definition processing target area.

  That is, in the case where only the region including the element is determined as a region for high definition processing, an element determination step is provided after the encoding cost determination processing step of step (1204) in FIG. In the determination step, an edge extraction filter such as a Canny filter is used to determine whether the block is a characteristic part such as an edge or a corner. The filter to be used may be a Laplacian filter or a Gabor filter. When it is determined by the determination unit that the target block is not a characteristic part such as an edge or a corner, the process proceeds to step (1207). When the determination unit determines that the block is a characteristic part such as an edge or a corner, the process proceeds to step (1205), the original image is determined, and the original image loses high frequency components during the encoding process. It is determined whether the subject is blurred or the subject is blurred because the photographer intentionally reduces the depth of field. Note that the determination processing in the element determination step is performed by the processing target region determination unit (1101) in FIG.

  As described above, when the region for high definition processing is limited to a region including a boundary portion of a pixel value having a predetermined shape such as a boundary portion of a line, edge, corner, or T-shaped pixel value, for example, decoding In the resolution-enhancement processing on the image forming side, it is possible to prevent generation of an unnatural image by performing high-definition for the entire region including, for example, noise in the image. Further, if the data to be stored in the dictionary data is limited to the area, the storage capacity necessary for holding the dictionary data and the calculation amount when specifying the dictionary data can be reduced, and matching errors can be reduced.

It is explanatory drawing which shows an example of a structure of the image coding apparatus which concerns on Example 1 of this invention, and an image decoding apparatus. It is explanatory drawing which shows an example of a structure of the image coding apparatus which concerns on Example 1 of this invention. It is explanatory drawing which shows an example of a structure of the image decoding apparatus which concerns on Example 1 of this invention. It is explanatory drawing which shows an example of a structure of the image high definition part which concerns on Example 1 of this invention. It is explanatory drawing which shows the dictionary data based on Example 1 of this invention. It is a flowchart which shows an example of a process of the image high definition part which concerns on Example 1 of this invention. It is explanatory drawing which shows an example of a structure of the image coding apparatus which concerns on Example 2 of this invention. It is a flowchart which shows an example of a process of the image coding apparatus which concerns on Example 2 of this invention. It is explanatory drawing which shows an example of a structure of the image high definition part which concerns on Example 2 of this invention. It is a flowchart which shows an example of a structure of the image high definition part which concerns on Example 2 of this invention. It is explanatory drawing which shows an example of a structure of the image high definition part which concerns on Example 3 of this invention. It is a flowchart which shows an example of a structure of the image high definition part which concerns on Example 3 of this invention.

Explanation of symbols

100: Image decoding device
101: Decoding unit
102, 900 ... Image enhancement section
200, 700, 1100 ... Image encoding device
201 ... Original image memory
202 ... Subtractor
203 ... Frequency converter
204 ... Quantization part
205, 702, 1102 ... Variable length coding unit
206, 303 ... Inverse quantization part
207, 304 ... Inverse frequency converter
208, 305 ... adder
209, 306 ... Frame memory
210, 307 ... predicted image generation unit
301: Variable length decoding unit
302 ... Syntactic analyzer
401, 701, 901, 1101 ... processing target area determination unit
402… Target area high-definition department
403 ... Dictionary data storage
551… Elements of high frequency components
552… Elements of low frequency components
703, 1103 ... Coding section

Claims (20)

A decoding unit that decodes an encoded image included in an input encoded stream and generates a decoded image;
An image high-definition unit that performs high-definition processing on the decoded image generated by the decoding unit;
The image high-definition unit includes a processing target region determination unit that determines a high-definition processing target region for performing high-definition processing in the decoded image based on information included in the encoded stream; An image decoding apparatus, comprising: a target region high-definition unit that performs high-definition by mixing corresponding high-frequency component data with image data of the high-definition processing target region. 2. The processing target region determination unit according to claim 1, wherein the high definition processing target region is determined by comparing a value of an encoding parameter included in the encoded stream with a predetermined threshold. Image decoding device. The image refinement unit has a data storage unit that stores a plurality of pairs of low frequency component data and high frequency component data in a partial region in an image,
The target region high-definition unit identifies low-frequency component data corresponding to the high-definition processing target region from the low-frequency component data stored in the data storage unit, and among the high-frequency component data stored 3. The image decoding according to claim 1, wherein high definition is performed by mixing high frequency component data corresponding to the specified low frequency component data and image data of the high definition processing target area. apparatus. The processing target area determination unit determines, as the high definition processing target area, an area indicated by additional information that is identification information data indicating the high definition processing target area included in the encoded stream. The image decoding apparatus according to claim 1. 4. The processing area determination unit according to claim 1, 2 or 3, wherein the processing area determination section determines the high-definition processing area based on whether or not a boundary portion of pixel values having a predetermined shape is included. Image decoding device. The decoding unit
A variable length decoding unit that performs a variable length decoding process on the encoded stream transmitted from the encoding side;
A syntax analysis unit for analyzing the syntax of the data decoded by the variable length decoding unit;
An inverse quantization unit that inversely quantizes the image data analyzed by the syntax analysis unit;
An inverse frequency transform unit for inverse frequency transforming the image data inversely quantized by the inverse quantization unit;
An adder that generates the decoded image by adding the image data and the predicted image obtained by the inverse frequency conversion by the inverse frequency conversion unit;
A frame memory for storing the decoded image generated by the adder;
A predicted image generation unit that generates the predicted image using the decoded image stored in the frame memory and the image data analyzed by the syntax analysis unit;
6. The image decoding device according to claim 1, further comprising: A decoding step of decoding the encoded image included in the input encoded stream and generating a decoded image;
A determination step of determining a high-definition processing target region for performing high-definition processing in the decoded image, based on information included in the encoded stream;
A high-definition step of performing high-definition by mixing corresponding high-frequency component data with respect to the image data of the high-definition processing target region,
An image decoding method comprising: 8. The image decoding according to claim 7, wherein in the determination step, the high-definition processing target region is determined by comparing a value of an encoding parameter included in the encoded stream with a predetermined threshold. Method. The determination step includes determining, as the high-definition processing target area, an area indicated by additional information that is identification information data indicating the high-definition processing target area included in the encoded stream. Item 8. The image decoding method according to Item 7. The image decoding method according to claim 7 to 9, comprising:
A data holding step for holding high-frequency component data for a partial region in the image before the decoding step;
In the high-definition step, low-frequency component data corresponding to the high-definition processing target region is specified from low-frequency component data in a partial region in the image held in the data holding step, and the data holding step The high-frequency component data held in the step is mixed with the high-frequency component data corresponding to the specified low-frequency component data and the image data of the region for high-definition processing to perform high-definition processing. The image decoding method according to any one of claims 7 to 9. 11. The image decoding according to claim 7, 8 or 10, wherein, in the determination step, the high-definition processing target region is determined based on whether or not a boundary portion of pixel values having a predetermined shape is included. Method. An encoding unit that encodes an input original image to generate encoded image data;
A processing target region determination unit that determines a high-definition processing target region for performing the high-definition processing on the decoding side using the encoding parameters used when the encoding unit generates the encoded image data. ,
The encoding unit generates additional information that is identification information data indicating identification information of the high definition processing target region determined by the processing target region determination unit, and includes the additional information and the encoded image data. An image encoding device that generates an encoded stream. The processing target area determination unit calculates an encoding cost by a function having the encoding parameter as a variable, compares the encoding cost value with a predetermined threshold value, and the encoding cost value is greater than the threshold value. 13. The image encoding device according to claim 12, wherein if it is larger, it is determined as the high-definition processing determination region, and if the value of the encoding cost is smaller than the threshold value, it is determined as a region in which high-definition processing is not performed. . The encoding unit generates the encoded image data by frequency-converting a difference image between the input original image and the predicted image,
13. The processing target region determination unit determines the high-definition processing target region based on a distribution in a frequency region of a frequency transform coefficient generated by a frequency conversion process of the encoding unit. Image coding apparatus. 13. The processing area determination unit determines the high-definition processing area based on a result of calculating variance or primary / secondary differentiation for the input original image. Image coding apparatus. The encoding unit includes:
An original image memory for holding the input original image;
A differentiator for calculating a difference between the original image and the predicted image;
A frequency converter that converts the frequency of the difference image generated by the differentiator;
A quantization unit for quantizing the image data subjected to frequency conversion processing by the frequency conversion unit;
A variable length coding unit for variable length coding the image data quantized by the quantization unit;
An inverse quantization unit that inversely quantizes the image data quantized by the quantization unit;
An inverse frequency transform unit for inverse frequency transforming the image data inversely quantized by the inverse quantizer;
An adder that adds the image data subjected to inverse frequency conversion by the inverse frequency conversion unit and the predicted image;
A frame memory for storing the image data added by the adder;
A predicted image generation unit that generates the predicted image using image data stored in the frame memory;
The variable length encoding unit generates an encoded stream including the image data quantized by the quantization unit and the additional information input from the processing target region determination unit. The image encoding device according to 12 to 15. An encoding step of encoding the input original image to generate encoded image data;
A processing target region determination step for determining a high definition processing target region for performing a high definition processing on the decoding side, using the encoding parameters used when generating the encoded image data in the encoding step;
An encoded stream generation step for generating an encoded stream including additional information that is identification information data indicating the high-definition processing target area generated in the processing target area determination step and encoded image data. A characteristic image encoding method. In the processing target region determination step, an encoding cost is calculated by a function using the encoding parameter as a variable, the value of the encoding cost is compared with a predetermined threshold value, and the value of the encoding cost is greater than the threshold value. 18. The image encoding method according to claim 17, wherein if it is larger, the region is determined as the region for high-definition processing, and if the value of the encoding cost is smaller than the threshold, it is determined as a region for which high-definition processing is not performed. . In the encoding step, the encoded image data is generated by frequency-converting a difference image between the input original image and the predicted image,
18. The high-definition processing target region is determined based on a distribution in the frequency domain of frequency transform coefficients generated by the frequency conversion process of the encoding step in the processing target region determining step. Image coding method. 18. The high-definition processing target region is determined based on a result of calculating a variance or a first-order / second-order derivative for the input original image in the processing target region determining step. Image coding method.

Images