WO2010021039A1

WO2010021039A1 - Image processing device, image processing method, and image processing program

Info

Publication number: WO2010021039A1
Application number: PCT/JP2008/064866
Authority: WO
Inventors: 淑貴田中
Original assignee: パイオニア株式会社
Priority date: 2008-08-21
Filing date: 2008-08-21
Publication date: 2010-02-25
Also published as: JPWO2010021039A1

Abstract

An image processing device is conveniently applied to for example, one segment receiver of terrestrial digital broadcasting. Specifically, image quality judgment means judges an image quality for each frame and image quality adjustment means controls image quality adjustment strength according to the judged image quality. Accordingly, because a degradation state of the image can be properly grasped, by controlling the image quality adjustment strength according to the judged image quality, promotion of image quality degradation caused by accentuation of coding degradation or the like can be properly suppressed. Therefore, image quality can be optimally improved while making a degradation factor inconspicuous.

Description

Image processing apparatus, image processing method, and image processing program

The present invention relates to a technical field in which image quality is determined and image processing is performed based on the image quality.

This type of technology is proposed in Patent Document 1, for example. Specifically, Patent Document 1 proposes a technique for controlling the degree of enhancement in sharpness correction according to the data rate (bit rate) of digitally compressed video data.

JP 2004-180043 A

However, since the technique described in Patent Document 1 uses the bit rate information described in the sequence header, fixed bit rate information is used in the content of the same sequence (in other words, sharpness). The same emphasis was used in degree correction). For this reason, there is a case where the image quality for each frame cannot be correctly determined even when information relating to other image quality is taken into consideration. Also, in digital broadcasting such as terrestrial digital broadcasting, the bit rate per unit frame tends to decrease in scenes with large movements and images where scene changes frequently occur. In this case, the same enhancement is used in sharpness correction. In some cases, image quality deterioration is promoted. On the other hand, in an encoded image, even if the bit rate is the same, there is a tendency that an image containing more edges tends to deteriorate the image quality. Similarly, when an object is deformed, the image quality is also deteriorated. Due to the tendency, it is considered inappropriate to control the enhancement degree of sharpness correction based only on the bit rate as described above.

The above is one example of problems to be solved by the present invention. An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of appropriately controlling image quality adjustment intensity by determining image quality for each frame.

According to the first aspect of the present invention, the image processing apparatus includes: an image quality determination unit that determines an image quality for each frame of the input image; and an image quality adjustment strength based on the image quality determined by the image quality determination unit. Image quality adjusting means for controlling the image quality.

According to a fifteenth aspect of the present invention, an image processing method includes: an image quality determination step for determining image quality for each frame of an input image; and an image quality adjustment strength based on the image quality determined by the image quality determination step. An image quality adjustment process for controlling the image quality.

According to the sixteenth aspect of the present invention, an image processing program executed by a computer is determined by the image quality determination unit that determines image quality for each frame of the input image, and the image quality determination unit. Based on the image quality, it functions as image quality adjustment means for controlling the image quality adjustment intensity.

1 is a schematic configuration diagram of a system to which an image processing apparatus according to an embodiment is applied. It is a schematic block diagram of an adaptive image sharpening process part. It is a figure for demonstrating the block size in a macroblock. It is a figure for demonstrating the image processing method in another example. It is a flowchart which shows the whole process in a present Example. It is a flowchart which shows the process for I pictures. It is a flowchart which shows the bit rate determination status confirmation process at the time of I picture. It is a flowchart which shows the process for P pictures. It is a flowchart which shows the process performed when the number of intra blocks is not more than predetermined in the process for P pictures. It is a flowchart which shows the bit rate determination status confirmation process at the time of a P picture.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Tuner part 12 Demodulator part 22 Video decoder 24 Adaptive image sharpening process part 24a Bit amount determination part 24b Edge detection part 24c Edge condition determination part 24d Syntax element acquisition part 24e Sharpening level determination part 24f Image quality sharpening process part 24x Image quality Determination unit 25 Signal processing unit

In one aspect of the present invention, an image processing apparatus determines, for an input image, an image quality determination unit that determines image quality for each frame, and an image quality adjustment strength based on the image quality determined by the image quality determination unit. Image quality adjusting means for controlling.

The above-described image processing apparatus is suitably applied to, for example, a 1 seg (1 seg) receiver for terrestrial digital broadcasting. Specifically, the image quality determination unit determines the image quality for each frame, and the image quality adjustment unit controls the image quality adjustment intensity based on the determined image quality. As a result, it is possible to appropriately grasp the deterioration state of the image and the like, and by controlling the image quality adjustment intensity based on the determined image quality, the deterioration of the image quality is promoted by enhancing the coding deterioration and the like. This can be suppressed appropriately. Therefore, it is possible to optimally improve the image quality while making the deterioration factor inconspicuous.

In one aspect of the image processing apparatus, the image quality determination unit changes the image quality determination method according to a picture type. Specifically, the image quality determination means changes the image quality determination method according to which picture type the compressed image data is. For example, picture types include an I (Intra) picture, a P (Predictive) picture, and a B (Bidirectionally predictive) picture. By doing so, it becomes possible to accurately grasp the degradation state of the image.

In another aspect of the image processing apparatus, the image quality determination unit determines the image quality based on a bit amount when the picture type is an I picture, and the picture type is other than the I picture. If it is, the image quality is determined based on the bit amount and the number of intra blocks. In this case, the image quality determination means determines that the image quality is lower when the bit amount is small than when it is large when the bit amount is small, and when it is other than the I picture, When the bit amount is small, it can be determined that the image quality is lower than when it is large, and when the number of intra blocks is large, it can be determined that the image quality is lower than when it is small.

In another aspect of the above-described image processing apparatus, the image quality determination unit further determines the image quality based on the number of blocks whose size in a macroblock is less than a predetermined size when the image is other than the I picture. Do. In this case, the image quality determination means can determine that the image quality is degraded when the number of blocks whose macroblock size is less than the predetermined size is large compared to when the number of blocks is small.

In another aspect of the above-described image processing apparatus, the image quality determination unit further determines the image quality based on an edge situation for both the case of the I picture and the case other than the I picture. Thereby, the image quality can be determined in consideration of the complexity of the image appropriately.

Preferably, in the above-described image processing apparatus, the image quality adjusting means adaptively controls the image quality adjustment intensity for each area in the image. Preferably, the image quality adjustment unit does not control the image quality adjustment intensity based on the image quality determined by the image quality determination unit for a specific area in the image. As a result, it is possible to reduce the processing burden caused by the control of the image quality adjustment strength.

Preferably, in the above-described image processing apparatus, the image quality adjustment unit controls the image quality adjustment intensity based on the image quality determined by the image quality determination unit when a scene change occurs. This can also reduce the processing burden.

In a preferred example, the image quality adjustment means controls a sharpening level as the image quality adjustment intensity. In this case, the image quality adjusting means reduces the sharpening level according to the image quality deterioration level when the image quality determining means determines that the image quality has deteriorated. As a result, the deterioration factor can be appropriately made inconspicuous, and the image quality can be improved optimally.

Also preferably, the image quality adjustment means controls a noise removal level as the image quality adjustment intensity. In this case, when the image quality determination unit determines that the image quality is degraded, the image quality adjustment unit increases the noise removal level according to the degradation level of the image quality. As a result, the deterioration factor can be appropriately made inconspicuous, and the image quality can be improved optimally.

In another aspect of the present invention, an image quality determination step for determining image quality for each frame of an input image, and an image quality adjustment step for controlling image quality adjustment strength based on the image quality determined by the image quality determination step. And comprising.

In another aspect of the present invention, an image processing method includes: an image quality determination step for determining image quality for each frame of an input image; and an image quality adjustment based on the image quality determined by the image quality determination step. And an image quality adjustment step for controlling the intensity.

The above-described image processing method and image processing program can also appropriately grasp the degradation state of the image, and therefore, by controlling the image quality adjustment intensity based on the determined image quality, the coding degradation is emphasized. Thus, it is possible to appropriately suppress the deterioration of the image quality. Note that the image processing program can be suitably handled in a state where it is recorded on a recording medium.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Device configuration]
FIG. 1 is a schematic configuration diagram of a system to which an image processing apparatus according to the present embodiment is applied. The system mainly includes a pre-processing unit 1 and a post-processing unit 2, and is applied to, for example, a 1 seg (1seg) receiver for terrestrial digital broadcasting. As an example, the present invention is applied to a mobile phone or a navigation device for a vehicle. The image data compression format used for one-segment broadcasting is H.264. H.264 encoding system, and in the following, H.264 encoding is used. A case of handling data of the H.264 encoding method will be described as an example.

The pre-processing unit 1 mainly includes a tuner unit 11 and a demodulation unit 12. The tuner unit 11 receives the modulated radio wave via the antenna 11a and supplies this to the demodulation unit 12 as a signal S11. The demodulation unit 12 demodulates the signal S11 supplied from the tuner unit 11, and supplies the demodulated signal S12 to the post-processing unit 2. Specifically, the demodulation unit 12 performs conversion processing into MPEG-TS (for example, MPEG2-TS).

The post-processing unit 2 mainly includes a Demux unit 21, a video decoder 22, a frame buffer memory 23, an adaptive image sharpening processing unit 24, and a signal processing unit 25. The Demux unit 21 obtains a signal S12 (corresponding to MPEG-TS) from the demodulator 12 of the pre-processing unit 1, and performs processing for separating the signal S12 for each ES data such as audio, video, and data broadcasting. More specifically, the Demux unit 21 performs MPEG2-TS header analysis, PES header analysis, ES data extraction, and the like. Then, the Demux unit 21 supplies the video decoder 22 with a signal S21 corresponding to video in the separated data.

The video decoder 22 acquires the signal S21 from the Demux unit 21 and performs a process of decoding the encoded information and converting it into image data. Specifically, the video decoder 22 supplies the image data obtained by decoding to the frame buffer memory 23 as a signal S22a. Further, the video decoder 22 supplies information corresponding to the bit amount for each frame (for example, the number of bytes of the NAL unit in the case of the decoder in the H.264 encoding method) to the adaptive image sharpening processing unit 24 as a signal S22b. H. obtained in the middle of decoding. Information (hereinafter referred to as “syntax element”) such as a macroblock type, a picture type, and a vector in the H.264 encoding method is supplied to the adaptive image sharpening processing unit 24 as a signal S22c.

The frame buffer memory 23 acquires the signal S22a corresponding to the image data from the video decoder 22 and temporarily stores it. The data stored in this way is used as a reference frame or the like.

The adaptive image sharpening processing unit 24 acquires the signal S23 corresponding to the image data from the frame buffer memory 23, and the signal S22b corresponding to the bit amount for each frame and the signal S22c corresponding to the syntax element from the video decoder 22. get. Then, the adaptive image sharpening processing unit 24 determines the quality of the image quality for each frame based on these acquired signals, and controls the image quality adjustment intensity for the image based on the determination result. The emphasis level (hereinafter simply referred to as “sharpening level”) is controlled. In this case, if the adaptive image sharpening processing unit 24 determines that the image quality is relatively low, the adaptive image sharpening processing unit 24 performs the sharpening process by setting the sharpening level low, and determines that the image quality is relatively high. Performs a sharpening process with a high sharpening level. Alternatively, if the adaptive image sharpening processing unit 24 determines that the image quality is relatively low, the adaptive image sharpening processing unit 24 performs a sharpening process with a high noise removal level, and determines that the image quality is relatively high. Set the noise removal level low and perform sharpening. Then, the adaptive image sharpening processing unit 24 supplies the image data after the sharpening processing as described above to the signal processing unit 25 as a signal S24. As described above, the adaptive image sharpening processing unit 24 corresponds to the image processing apparatus according to the present invention, and functions as an image quality determination unit and an image quality adjustment unit. Details of processing performed by the adaptive image sharpening processing unit 24 will be described later.

The signal processing unit 25 acquires the signal S24 from the adaptive image sharpening processing unit 24, converts it into data suitable for the output destination such as a video composite signal, and outputs this as the signal S25.

Here, H. A specific example of syntax elements (encoding information) in the H.264 encoding method will be described. Examples of the syntax element include a picture type, a macroblock type, “mv (motion vector)”, “pmv (motion vector prediction value)”, “total_coeff”, “nC”, and the like. The picture type is H.264. In the H.264 encoding method, it corresponds to either an I picture or a P picture, and “total_coeff” is a value indicating the number of residual components (difference values) in a 4 × 4 size pixel block, and “nC” is The value increases when the peripheral block gives a large amount of the residual component. In this case, a complex image tends to exist in the peripheral block. Note that “total_coeff” and “nC” can be determined from “coeff_token” or “CBP (Code_Block_Pattern)”. “CBP” is an H.264 standard. Is a value existing in the macroblock layer in the H.264 encoding system, and when the difference value exists in the 8 × 8 block in the corresponding macroblock, the bit is set, and the lower 4 bits indicate the block of “luma”, The upper 2 bits indicate the status of a block having “chroma” (“CBP” is, for example, a total of 6 bits).

Next, the adaptive image sharpening processing unit 24 will be described in detail with reference to FIG. FIG. 2 is a schematic configuration diagram of the adaptive image sharpening processing unit 24. As illustrated, the adaptive image sharpening processing unit 24 includes an image quality determination unit 24x and an image quality sharpening processing unit 24f. Basically, the image quality determination unit 24x evaluates the image quality for each frame, and determines the enhancement level (sharpening level) in the sharpening based on the evaluation. Then, the image quality sharpening processing unit 24f performs a process of sharpening the image according to the sharpening level determined by the image quality determination unit 24x. Thus, in the adaptive image sharpening processing unit 24, the image quality determination unit 24x functions as the image quality determination unit in the present invention, and the image quality sharpening processing unit 24f functions as the image quality adjustment unit in the present invention.

Specifically, the image quality determination unit 24x includes a bit amount determination unit 24a, an edge detection unit 24b, an edge condition determination unit 24c, a syntax element acquisition unit 24d, and a sharpening level determination unit 24e. The bit amount determination unit 24a acquires the signal S22b corresponding to the bit amount per frame (in other words, the bit rate) from the video decoder 22, and makes a rough determination on the sharpening level based on the bit amount. Then, the bit amount determination unit 24a supplies the determination result to the sharpening level determination unit 24e.

The edge detection unit 24b acquires the signal S23 corresponding to the image data from the frame buffer memory 23, and performs edge detection by using a general filter used for edge extraction such as “sobel filter” on the signal S23 ( Specifically, edge detection is performed when the threshold value is exceeded. Further, the edge detection unit 24b holds information on the location where the edge is detected, and counts how many pieces of this information are present. The edge status determination unit 24c acquires edge detection information from the edge detection unit 24b, calculates the ratio of edge regions and the like based on the edge detection information, and determines the complexity of the image. Then, the edge state determination unit 24c performs a rough determination on the sharpening level from the edge image based on the complexity of the image, and supplies the determination result to the sharpening level determination unit 24e.

The syntax element acquisition unit 24d acquires the signal S22c corresponding to the syntax element from the video decoder 22, and extracts information below the macroblock layer necessary for the sharpening level determination in the sharpening level determination unit 24e. Specifically, the syntax element acquisition unit 24d extracts the picture type, macroblock type, “mv”, “pmv”, “total_coeff”, “nC”, and the like as described above. Is supplied to the sharpening level determination unit 24e. By using such a syntax element, for example, even if the bit amount per frame is substantially the same, it is possible to appropriately determine whether the image quality is good or bad.

The sharpening level determination unit 24e determines the sharpening level by comprehensively considering the information supplied from the bit amount determination unit 24a, the edge state determination unit 24c, and the syntax element acquisition unit 24d, The determined sharpening level is supplied to the image quality sharpening processing unit 24f. Specifically, the sharpening level determination unit 24e sets the sharpening level high when it is determined that the image quality is relatively high, and sharpens when it is determined that the image quality is relatively low. Set the level low.

The image quality sharpening processing unit 24f acquires the signal S23 corresponding to the image data from the frame buffer memory 23, and sharpens the image according to the sharpening level acquired from the sharpening level determination unit 24e for the image data. Perform processing. Specifically, the image quality sharpening processing unit 24f performs processing with a high sharpening strength when the sharpening level is high, and performs processing with a low sharpening strength when the sharpening level is low. For example, the image quality sharpening processing unit 24f performs sharpening using an edge enhancement filter such as a Laplacian filter. In this case, the sharpening process is performed by changing the operator coefficient of the filter or the like according to the sharpening level determined by the sharpening level determination unit 24e.

Here, with reference to FIG. 3, the block size in the macro block which is one of the syntax elements will be specifically described. H. In the H.264 encoding method, a mechanism called variable block size is used, and seven types of block sizes are prepared for a macro block of 16 × 16 pixels. The macro block type indicates which block size it is. Specifically, as shown in FIG. 3, seven types of block sizes of 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, and 4 × 4 are used.

[Image quality judgment method]
Next, an image quality determination method according to the present embodiment (in other words, a method for determining a sharpening level) will be described.

In the present embodiment, as described above, the image quality determination unit 24x mainly considers the three viewpoints of bit amount information for each frame, syntax elements, and edge conditions after edge detection, Determine the image quality. Specifically, the image quality determination unit 24x changes the image quality determination method according to the picture type. Specifically, when the picture type is an I picture, the image quality determination unit 24x mainly determines the image quality based on the bit amount and the edge status (such as the number of pixels having an edge). On the other hand, when the picture type is other than I picture, specifically, when the picture type is P picture, the image quality determination unit 24x determines not only the bit amount but also the number of intra blocks and the size of the macro block. The image quality is determined based on the number of blocks that are less than the size. Note that an intra block does not have a vector (motion vector) and corresponds only to a difference value (residual component) or a block that creates an image by prediction from surrounding blocks, and is a macro that is one of the syntax elements described above. It can be grasped from the block type. In addition, the size of the macroblock can be grasped from the macroblock type.

The reason for determining the image quality as described above is as follows. First, confirmation of syntax elements in the case of a P picture will be described. In the case of an image with a low bit rate such as one-segment broadcasting, there is a case where the image quality is inevitably lowered than the average image quality due to the limitation of the bit rate when the image is encoded. In this case, it can be said that the quality of image quality cannot be determined appropriately only by observing only the bit amount per frame. This is because the I-picture has less inter-frame prediction than the P-picture, so that the data rate required for producing the same image quality tends to be larger. Also, in the P picture, even if the bit amount per frame is small, the image quality may not be lowered depending on the movement of the image. For example, when an image can be created using only motion vector information such as a pan scene, a small bit amount may be sufficient. Conversely, if there are many “total_coeff” (or many blocks with a large value of “nC”) and the macroblock type is also a small size (for example, 8 × 8), there is a high possibility that a deformed object exists, It can be said that there is a high possibility that a bit amount is necessary. As described above, in the present embodiment, for the P picture, the image quality is determined by comprehensively determining the macroblock type information, the “total_coeff” and “nC” information indicating the residual component, and the like.

On the other hand, when picture quality is likely to deteriorate, there is a tendency of appearance of P pictures with many intra blocks. This is because, in the case of a scene change, if the amount of bits per frame is sufficient, the image quality is less deteriorated when all of the I-pictures are composed of intra blocks. However, in a scene where scene changes continue frequently, the amount of bits per frame becomes insufficient. In that case, although the correlation with the previous frame is not originally high, an image is generated using a vector for a block that has the same correlation by chance. For blocks that have no correlation at all, intra blocks are used. Even for blocks made using vectors, the correlation is not high, so the image quality often decreases. As described above, in this embodiment, the image quality is determined based on the situation of both the picture type and the macroblock type.

Next, the confirmation of edge status will be described. The confirmation of the edge state is basically performed in order to see the complexity of the image, and is used for determining the image quality of the I picture. Here, in the case of an I picture, even if the bit amount per frame is low to some extent, if the image is flat, it is considered that the image quality does not deteriorate so much, but if the image is complicated, the image quality decreases. Therefore, the edge condition is confirmed as a reference for the standard. On the other hand, in the case of a P picture, an image can be generated with a motion vector. Therefore, it can be said that it is necessary to consider the proportion of intra blocks in proportion to the confirmation of the edge situation. In a P picture, if a deformed object such as a person moves greatly, it is difficult to predict with a vector, so a residual component is required and a relatively large amount of bits is required. In a low bit rate stream, there is a possibility that a sufficient amount of bits may not be given, resulting in degradation of image quality. If “total_coeff” is large and the macroblock type is also a small size (for example, 8 × 8), there is a high possibility that a deformed object exists, and a large amount of bits is inherently necessary. Therefore, it can be said that such a block should also be handled according to the intra block (hereinafter also referred to as “quasi-intra block”). As described above, in the present embodiment, the image quality of a P picture is determined from the elements of both the number of intra blocks and the number of quasi-intra blocks and the bit amount per frame.

According to the image quality determination method according to the present embodiment as described above, it is possible to appropriately grasp the degradation state of the image. Therefore, by performing the sharpening process based on the determined image quality, it is possible to appropriately prevent the deterioration of the image quality from being promoted by emphasizing the deterioration of the encoding and the like, while making the deterioration factor inconspicuous. The image quality can be optimally improved. In particular, in low-bit-rate images such as one-segment broadcasting, when scene changes occur frequently, images with severe deterioration (such as coding degradation noise) may appear. By applying this embodiment, it can be said that encoding degradation noise and the like can be effectively suppressed.

In addition, it is not limited to changing the sharpening level according to the image quality determined as described above for the entire screen. In another example, the area that adaptively changes the sharpening level can be limited within the screen. For example, the fixed sharpening level is used for a specific area where characters such as telop are displayed, and for other areas, the sharpening level can be changed according to the image quality by the method according to the present embodiment. it can.

FIG. 4 is a diagram for explaining an image processing method in another example. Specifically, FIG. 4 shows an example of an image, and broken line areas T1 and T2 show areas including telops. It can be said that the image quality of such a fixed image area such as a telop hardly changes even if the bit amount per frame changes. This is because the image of the previous frame can be referred to as it is for the telop portion, so that it can be said that the telop portion is hardly affected by the change in image quality even if the required bit amount of the telop portion is small. Therefore, in another example, the sharpening level is not changed according to the image quality for a specific area including a telop or the like. That is, the sharpening level is fixed. On the other hand, since there is a possibility that image degradation has occurred in the image area other than the telop, the sharpening level is changed in accordance with the image quality by the procedure described above.

In still another example, it is possible to divide the area in the image into a plurality of areas and adaptively change the sharpening level for each area depending on the image condition. Specifically, the sharpening level can be changed according to the object in the image. As an example, the sharpening level can be set low for the face portion in the image, and the sharpening level can be set high for the collar portion or the like in the image. When the sharpening level is changed in this way, for example, the area can be determined by using the edge density (determined by the edge condition determining unit 24c), and the object in the image is It can be recognized by image analysis.

[Processing according to embodiment]
Next, processing performed by the adaptive image sharpening processing unit 24 (more specifically, the image quality determination unit 24x) in the present embodiment will be described with reference to FIGS. In this processing, as described above, the image quality determination unit 24x determines the image quality by comprehensively considering the information on the bit amount for each frame, the syntax element, the edge condition after edge detection, and the like. The sharpening level is determined. Here, in the case of setting the sharpening level in four steps, in detail, any one of the four steps of “no sharpening”, “sharpening level 1”, “sharpening level 2”, and “sharpening level 3” Take the case of setting as an example. It is assumed that the sharpening strength increases in the order of “not sharpening” → “sharpening level 1” → “sharpening level 2” → “sharpening level 3”.

FIG. 5 is a flowchart showing the overall processing in this embodiment. This process is repeatedly executed by the image quality determination unit 24x.

First, in step S101, the image quality determination unit 24x determines whether the processing target is an area other than the telop. This determination is performed in order to change the handling of the sharpening level by dividing the telop area and the non-telop area on the screen. Specifically, the image quality determination unit 24x determines whether the area is other than the telop by performing image analysis or the like. If it is not a telop (step S101; Yes), the process proceeds to step S102. In this case, the sharpening level is changed according to the image quality in the processing after step S102. On the other hand, when it is not an area other than the telop (step S101; No), the process proceeds to step S106. In this case, for the reason described above, the sharpening level is fixed to a predetermined level (step S106). That is, the sharpening level is not changed according to the image quality. Then, the process ends.

In step S102, the image quality determination unit 24x acquires picture type information. Specifically, the image quality determination unit 24 x receives the H.264 from the video decoder 22. The picture type (included in the syntax element) in the H.264 encoding method is acquired. Then, the process proceeds to step S103. In step S103, the image quality determination unit 24x determines whether the picture type is an I picture or a P picture. If it is an I picture, the process proceeds to step S104, and the image quality determination unit 24x determines a sharpening level by executing an I picture process described later. Thereafter, the process ends. On the other hand, if it is a P picture, the process proceeds to step S105, and the image quality determination unit 24x determines a sharpening level by executing a P picture process described later. Thereafter, the process ends.

After such processing, the adaptive image sharpening processing unit 24 in the adaptive image sharpening processing unit 24 uses a general edge enhancement filter such as a Laplacian filter to filter according to the determined sharpening level. The sharpening process is performed by changing the operator coefficient.

FIG. 6 is a flowchart showing the I picture processing performed in step S104 of FIG. In this processing, first, the image quality is roughly determined based on the bit amount, and then the sharpness level is determined by finely determining the image quality based on the edge state (the number of pixels having edges). The I picture processing is also executed by the image quality determination unit 24x.

In step S201, the image quality determination unit 24x performs processing for classifying frames based on the bit rate per frame, specifically, processing for classifying frames based on the number of bytes of Nal units (hereinafter referred to as “bit rate determination for I picture”). This is referred to as “situation confirmation processing”. Then, the process proceeds to step S202.

Here, with reference to FIG. 7, the bit rate determination status confirmation processing at the time of I picture performed in step S201 of FIG. 6 will be described. First, in step S301, the image quality determination unit 24x acquires the number of bytes of Nal unit from the video decoder 22. Then, the process proceeds to step S302. In step S302, the image quality determination unit 24x performs determination based on the number of bytes of the Nal unit. Specifically, the image quality determination unit 24x classifies the frames to be processed into four stages (X1 <X2 <X3 <X4) by using four byte numbers X1, X2, X3, and X4. .

When the number of bytes of the Nal unit is less than X1 bytes, the process proceeds to step S303, and the image quality determination unit 24x determines that the frame is an ultra-low bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes of the Nal unit is greater than or equal to X1 bytes and less than X2 bytes, the process proceeds to step S304, and the image quality determination unit 24x determines that the frame is a low bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes of the Nal unit is greater than or equal to X2 bytes and less than X3 bytes, the process proceeds to step S305, and the image quality determination unit 24x determines that the frame is a medium bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes of the Nal unit is greater than or equal to X3 bytes and less than X4 bytes, the process proceeds to step S306, and the image quality determination unit 24x determines that the frame is a high bit rate allocation frame. Then, the process ends.

Referring back to FIG. 6, the processing after step S202 will be described. In step S202, the image quality determination unit 24x determines the bit rate determination status. Specifically, the image quality determination unit 24x performs determination based on the results classified by the processing illustrated in FIG.

In the processing of subsequent steps S203 to S210, the processing for determining the sharpening level is performed according to the bit rate determination status (specifically, according to the frames classified by the number of bytes of the Nal unit). Is called. In this case, basically, the sharpening level is determined according to the number of bytes of the Nal unit. That is, basically, the sharpening level is set higher in the order of ultra-low bit rate allocation frame → low bit rate allocation frame → medium bit rate allocation frame → high bit rate allocation frame. In this case, the sharpening level is adjusted in consideration of not only the number of bytes of the Nal unit but also the edge state. That is, in this embodiment, the sharpening level is determined taking into account both the number of bytes of Nal unit and the edge state. For example, even if the number of bytes of the Nal unit is large to some extent, when the number of pixels having edges is large, the sharpening level is set to be relatively low.

First, if it is an ultra-low bit rate allocation frame, the process proceeds to step S203. In step S203, the image quality determination unit 24x determines not to perform sharpening. In this case, it is considered that the image quality is considerably lowered because the bit amount is considerably small. Then, the process ends.

On the other hand, if it is a low bit rate allocation frame, the process proceeds to step S204. In step S204, the image quality determination unit 24x performs processing for confirming the edge state. Specifically, the image quality determination unit 24x (specifically, the edge detection unit 24b in the image quality determination unit 24x) counts pixels having edges in one frame. That is, the information of the location where the edge is detected is held and the number of pieces of this information is counted. Then, the process proceeds to step S205.

In step S205, the image quality determination unit 24x determines whether or not the number of pixels having the counted edges is equal to or less than a predetermined value α1. When the number of pixels having an edge is equal to or less than the predetermined value α1 (step S205; Yes), the sharpening level 2 is set (step S209). On the other hand, when the number of pixels having edges is larger than the predetermined value α1 (step S205; No), the sharpening level 1 is set (step S206). That is, the sharpening level is lowered by one as compared with the case where the number of pixels having edges is equal to or less than the predetermined value α1. This is because when the number of pixels having an edge is larger than the predetermined value α1, the image is more complex and the image quality may be lower than when the number is less than the predetermined value α1. After the processes in steps S206 and S209, the process ends.

On the other hand, if it is a medium bit rate allocation frame, the process proceeds to step S207. In step S207, the image quality determination unit 24x performs processing for confirming the edge state. Specifically, the image quality determination unit 24x (specifically, the edge detection unit 24b in the image quality determination unit 24x) counts pixels having edges in one frame. Then, the process proceeds to step S208.

In step S208, the image quality determination unit 24x determines whether or not the number of pixels having the counted edges is equal to or less than a predetermined value α2. When the number of pixels having an edge is equal to or less than the predetermined value α2 (step S208; Yes), the sharpening level 3 is set (step S210). On the other hand, when the number of pixels having edges is larger than the predetermined value α2 (step S208; No), the sharpening level 2 is set (step S209). That is, the sharpening level is lowered by one as compared with the case where the number of pixels having edges is equal to or less than the predetermined value α2. This is because when the number of pixels having an edge is larger than the predetermined value α2, the image is more complicated and the image quality may be lower than when the number is less than the predetermined value α2. After the processes in steps S209 and S210, the process ends. Note that the predetermined value α2 used in the determination in step S208 may be larger than the predetermined value α1 used in the determination in step S205, or the same value may be used.

On the other hand, if it is a high bit rate allocation frame, the process proceeds to step S210. In step S210, the image quality determination unit 24x sets the sharpening level 3. That is, the maximum sharpening level is set. In this case, since the bit amount is large, there is a high possibility that the image quality exceeds the standard. Then, the process ends.

Next, FIG. 8 is a flowchart showing the P picture processing performed in step S105 of FIG. In this process, in addition to the bit amount, the image quality is determined based on the number of intra blocks, the size of the macro block, and the like, and the sharpening level is determined. The P picture processing is also executed by the image quality determination unit 24x.

First, in step S401, the image quality determination unit 24x performs a process of confirming the intra block status. Specifically, the image quality determination unit 24x obtains macroblock type information (included in the syntax element) from the video decoder 22, thereby counting the number of blocks of the intra block. Then, the process proceeds to step S402.

In step S402, the image quality determination unit 24x determines whether or not the number of intra blocks is greater than or equal to a predetermined value. If the number of intra blocks is greater than or equal to the predetermined number (step S402; Yes), the process proceeds to step S403. In this case, since there is a possibility that the image quality has deteriorated due to a scene change or the like, processing for determining the image quality is further performed in step S403 and subsequent steps. On the other hand, if the number of intra blocks is not greater than or equal to the predetermined number (step S402; No), the process shown in FIG. 9 described later is executed.

In step S403, the image quality determination unit 24x performs processing for confirming the edge state in the intra block. Specifically, the image quality determination unit 24x performs edge detection for a region that is an intra block (intra block region), and counts pixels having edges. Then, the process proceeds to step S404.

In step S404, the image quality determination unit 24x determines whether the ratio of pixels having edges in the intra block is equal to or greater than a predetermined value in the intra block area. When the ratio of pixels having an edge is equal to or greater than a predetermined value of the intra block area (step S404; Yes), the process proceeds to step S405. In this case, since there are many edges in the intra block region, it is highly possible that the image quality is degraded. On the other hand, when the ratio of pixels having edges is not equal to or greater than the predetermined value of the intra block area (step S404; No), the process shown in FIG. 9 described later is executed.

In step S405, the image quality determination unit 24x performs a process of confirming the number of blocks whose macroblock size is less than a predetermined size. Specifically, the image quality determination unit 24x obtains the number of blocks whose macroblock size is less than a predetermined size by obtaining macroblock type information (included in the syntax element) from the video decoder 22. For example, the predetermined size used in the determination can be set to various sizes such as “16 × 16” and “8 × 8”. Then, the process proceeds to step S406. In step S406, the image quality determination unit 24x determines whether the number of blocks whose macroblock size is less than a predetermined size is greater than or equal to a predetermined value.

If the number of blocks whose macroblock size is less than the predetermined size is greater than or equal to the predetermined number (step S406; Yes), the process proceeds to step S407. In step S407, the image quality determination unit 24x determines not to perform sharpening. In this case, since there are many edges in the intra block area, it can be said that the image is complicated, and it can be said that there are many macro block sizes, so there is a possibility that the object has moved or deformed. Conceivable. Therefore, since it is highly likely that the amount of bits per frame is not sufficient, sharpening is not performed. Then, the process ends.

On the other hand, when the number of blocks whose macroblock size is less than the predetermined size is not equal to or larger than the predetermined number (step S406; No), the process proceeds to step S408. In step S408, the image quality determination unit 24x sets the sharpening level 1. In this case, it can be said that there are few macroblocks having a small size compared to the situation in which the process proceeds to step S407 described above, so the sharpening level is increased by one and set to the sharpening level 1. Then, the process ends.

Next, referring to FIG. 9, when the number of intra blocks is not greater than or equal to a predetermined value (step S402; No), and when the ratio of pixels having edges is not greater than or equal to a predetermined value of the intra block area (step S404; No). A process to be executed will be described. In this process, basically, as in the process (I picture process) shown in FIG. 6 described above, the image quality is determined based on the bit amount and the edge status, and the sharpening level is determined. This process is also executed by the image quality determination unit 24x.

In step S501, the image quality determination unit 24x performs processing for classifying frames based on the bit rate per frame, specifically, processing for classifying frames based on the number of bytes of Nal units (hereinafter referred to as “bit rate determination for P picture”). This is referred to as “situation confirmation processing”. Then, the process proceeds to step S502.

Here, with reference to FIG. 10, a description will be given of the bit rate determination status confirmation processing for a P picture performed in step S501 of FIG. First, in step S <b> 601, the image quality determination unit 24 x acquires the number of bytes of Nal unit from the video decoder 22. Then, the process proceeds to step S602. In step S602, the image quality determination unit 24x performs determination based on the number of bytes of the Nal unit. Specifically, the image quality determination unit 24x classifies the frame to be processed into four stages (Y1 <Y2 <Y3 <Y4) by using four byte numbers Y1, Y2, Y3, and Y4. .

Since the P picture can have a motion vector, the threshold used for the number of bytes of the Nal unit in the bit rate determination status confirmation process shown in FIG. 10 is the bit rate determination status for the I picture shown in FIG. It is set to a small value compared to the threshold used in the confirmation process. Specifically, “Y1 <X1”, “Y2 <X2”, “Y3 <X3”, “Y4 <X4” are set.

If the number of bytes in the Nal unit is less than Y1 bytes, the process proceeds to step S603, and the image quality determination unit 24x determines that the frame is an ultra-low bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes of the Nal unit is greater than or equal to Y1 bytes and less than Y2 bytes, the process proceeds to step S604, and the image quality determination unit 24x determines that the frame is a low bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes of the Nal unit is Y2 bytes or more and less than Y3 bytes, the process proceeds to step S605, and the image quality determination unit 24x determines that the frame is a medium bit rate allocation frame. Then, the process ends. On the other hand, if the number of bytes in the Nal unit is Y3 bytes or more and less than Y4 bytes, the process proceeds to step S606, and the image quality determination unit 24x determines that the frame is a high bit rate allocation frame. Then, the process ends.

Referring back to FIG. 9, the processing after step S502 will be described. In step S502, the image quality determination unit 24x determines the bit rate determination status. Specifically, the image quality determination unit 24x performs determination based on the results classified by the processing illustrated in FIG.

In the processing of subsequent steps S503 to S510, processing for determining the sharpening level is performed according to the bit rate determination status (specifically, according to the frames classified by the number of bytes of the Nal unit). Is called. In this case, basically, the sharpening level is determined according to the number of bytes of the Nal unit. That is, basically, the sharpening level is set higher in the order of ultra-low bit rate allocation frame → low bit rate allocation frame → medium bit rate allocation frame → high bit rate allocation frame. In this case, the sharpening level is adjusted in consideration of not only the number of bytes of the Nal unit but also the edge state. That is, in this embodiment, the sharpening level is determined taking into account both the number of bytes of Nal unit and the edge state. For example, even if the number of bytes of the Nal unit is large to some extent, when the number of pixels having edges is large, the sharpening level is set to be relatively low.

First, if it is an ultra-low bit rate allocation frame, the process proceeds to step S503. In step S503, the image quality determination unit 24x determines not to perform sharpening. In this case, it is considered that the image quality is considerably lowered because the bit amount is considerably small. Then, the process ends.

On the other hand, if it is a low bit rate allocation frame, the process proceeds to step S504. In step S504, the image quality determination unit 24x performs processing for confirming the edge state. Specifically, the image quality determination unit 24x (specifically, the edge detection unit 24b in the image quality determination unit 24x) counts pixels having edges in one frame. That is, the information of the location where the edge is detected is held and the number of pieces of this information is counted. Then, the process proceeds to step S505.

In step S505, the image quality determination unit 24x determines whether or not the number of pixels having the counted edges is equal to or less than a predetermined value β1. When the number of pixels having an edge is equal to or less than the predetermined value β1 (step S505; Yes), the sharpening level 2 is set (step S509). On the other hand, when the number of pixels having an edge is larger than the predetermined value β1 (step S505; No), the sharpening level 1 is set (step S506). That is, the sharpening level is lowered by one as compared with the case where the number of pixels having edges is equal to or less than the predetermined value β1. This is because when the number of pixels having an edge is larger than the predetermined value β1, the image is more complex and the image quality may be lower than when the number is less than the predetermined value β1. After the processes in steps S506 and S509 described above, the process ends. For example, the predetermined value β1 used in the determination in step S505 can be the same value as the predetermined value α1 used in the determination in step S205 of FIG.

On the other hand, if it is a medium bit rate allocation frame, the process proceeds to step S507. In step S507, the image quality determination unit 24x performs processing for confirming the edge state. Specifically, the image quality determination unit 24x (specifically, the edge detection unit 24b in the image quality determination unit 24x) counts pixels having edges in one frame. Then, the process proceeds to step S508.

In step S508, the image quality determination unit 24x determines whether or not the number of pixels having the counted edges is equal to or less than a predetermined value β2. When the number of pixels having an edge is equal to or less than the predetermined value β2 (step S508; Yes), the sharpening level 3 is set (step S510). On the other hand, when the number of pixels having an edge is larger than the predetermined value β2 (step S508; No), the sharpening level 2 is set (step S509). That is, the sharpening level is lowered by one as compared with the case where the number of pixels having edges is equal to or less than the predetermined value β2. This is because when the number of pixels having edges is larger than the predetermined value β2, the image is more complex and the image quality may be lower than when the number is less than the predetermined value β2. After the processes in steps S509 and S510, the process ends. Note that the predetermined value β2 used in the determination in step S508 may be larger than the predetermined value β1 used in the determination in step S505, or the same value may be used. For example, the same value as the predetermined value α2 used in the determination in step S208 of FIG. 6 can be used as the predetermined value β2.

On the other hand, if it is a high bit rate allocation frame, the process proceeds to step S510. In step S510, the image quality determination unit 24x sets the sharpening level 3. That is, the maximum sharpening level is set. In this case, since the bit amount is large, there is a high possibility that the image quality exceeds the standard. Then, the process ends.

According to the processing described above, it is possible to appropriately grasp the degradation state of the image. Therefore, by performing the sharpening process based on the determined image quality, it is possible to appropriately prevent the deterioration of the image quality from being promoted by emphasizing the deterioration of the encoding and the like, while making the deterioration factor inconspicuous. The image quality can be optimally improved.

In the above, an example in which the sharpening level is divided into four stages (in other words, an example in which the frame is classified into four according to the number of bytes of the Nal unit) is shown. However, the present invention is not limited to this. It can be divided into various stages.

[Modification]
In the above, an example in which an I picture and a P picture are used as the picture type and the image quality determination method is changed depending on whether the picture is an I picture or a P picture has been described. However, the present invention is not limited to this. That is, it is not limited to using P pictures as picture types other than I pictures. In another example, when there is a B picture in addition to an I picture and a P picture as a picture type, the image quality determination method is changed according to whether the picture is an I picture, a P picture, or a B picture. be able to. For the B picture, the image quality can be determined using a method similar to the image quality determination method used for the P picture.

Further, the present invention is not limited to executing the sharpening process corresponding to the determined sharpening level for each frame. That is, the image quality is determined for each frame and the sharpening level is determined, but the sharpening process may not be performed for each frame. For example, the sharpening process may be executed only when a scene change or the like occurs. Furthermore, the determination of image quality is not limited to being performed for each frame. For example, the sharpness level may be determined by determining the image quality only when a scene change or the like occurs.

Furthermore, in the above, the example in which the sharpening level is controlled as the image quality adjustment strength is shown, but the present invention is not limited to this. In another example, the noise removal level can be controlled as the image quality adjustment strength. That is, it is possible to control the noise removal level or the like according to the determined image quality.

The present invention can be used for a digital broadcast receiver or an Internet broadcast receiver using an image encoding method.

Claims

Image quality determination means for determining image quality for each frame of the input image;
An image processing apparatus comprising: an image quality adjustment unit that controls an image quality adjustment intensity based on the image quality determined by the image quality determination unit.
The image processing apparatus according to claim 1, wherein the image quality determination unit changes the image quality determination method according to a picture type.
The image quality judging means is
When the picture type is an I picture, the image quality is determined based on the bit amount,
3. The image processing apparatus according to claim 2, wherein when the picture type is other than the I picture, the image quality is determined based on a bit amount and the number of intra blocks.
The image quality judging means is
In the case of the I picture, when the amount of bits is small, it is determined that the image quality is lower than when it is large,
If it is other than the I picture, it is determined that the image quality is lower than when it is large when the bit amount is small, and the image quality is lower than when it is small when the number of intra blocks is large. The image processing apparatus according to claim 3, wherein the image processing apparatus determines that the image processing is performed.
5. The image quality determination unit according to claim 3, wherein the image quality determination unit further determines the image quality based on the number of blocks whose size in a macroblock is less than a predetermined size when the image is other than the I picture. The image processing apparatus described.
6. The image according to claim 5, wherein the image quality determination means determines that the image quality is lower when the number of blocks whose macroblock size is less than the predetermined size is larger than when the number of blocks is small. Processing equipment.
7. The image quality determination unit according to claim 3, wherein the image quality determination unit further determines the image quality based on an edge situation for both the case of the I picture and the case other than the I picture. The image processing apparatus according to one item.
The image processing apparatus according to any one of claims 1 to 7, wherein the image quality adjustment unit adaptively controls the image quality adjustment intensity for each area in the image.
9. The image processing apparatus according to claim 8, wherein the image quality adjustment unit does not control the image quality adjustment intensity based on the image quality determined by the image quality determination unit for a specific area in the image.
10. The image quality adjustment unit according to claim 1, wherein when a scene change occurs, the image quality adjustment unit controls the image quality adjustment intensity based on the image quality determined by the image quality determination unit. The image processing apparatus described.
The image processing apparatus according to claim 1, wherein the image quality adjusting unit controls a sharpening level as the image quality adjustment intensity.
12. The image quality adjusting unit, when it is determined by the image quality determining unit that the image quality is degraded, the sharpening level is reduced according to the image quality degradation level. The image processing apparatus described.
The image processing apparatus according to claim 1, wherein the image quality adjustment unit controls a noise removal level as the image quality adjustment intensity.
14. The image quality adjusting unit, according to claim 13, wherein when the image quality determining unit determines that the image quality is degraded, the image quality adjusting unit increases the noise removal level according to the image quality degradation level. The image processing apparatus described.
An image quality determination step for determining image quality for each frame of the input image;
An image processing method comprising: an image quality adjustment step of controlling an image quality adjustment intensity based on the image quality determined by the image quality determination step.
An image processing program executed by a computer,
The computer,
Image quality determination means for determining image quality for each frame of the input image;
An image processing program that functions as image quality adjustment means for controlling image quality adjustment intensity based on the image quality determined by the image quality determination means.
A recording medium on which the image processing program according to claim 16 is recorded.