JP2010087754A - Image quality evaluation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform image quality evaluation of an interpolation frame to be generated by frame rate conversion with successful precision while excluding an influence by image quality deterioration by a factor other than the frame rate conversion. <P>SOLUTION: A block evaluation value calculation means 10 calculates distortion of an image for evaluation to an original image as block evaluation values in blocks by in-frame division. A non-interpolation frame distortion totaling means 20 determines a distortion threshold on the basis of a block evaluation value in a non-interpolation frame. A distortion block evaluation means 30 determines a distortion block in an interpolation frame to output a distortion block number on the basis of the block evaluation value and the distortion threshold in the interpolation frame among the block evaluation values. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image quality evaluation apparatus that performs image quality evaluation of digital moving images, and more particularly to an image quality evaluation apparatus that can accurately perform image quality evaluation of an interpolated frame image generated by frame rate conversion.

  On the transmission side of one-segment broadcasting, for example, an original image of 30 frames / second is reduced to a low frame rate such as 15 frames / second, and then encoded and transmitted according to H.264. On the other hand, on the receiving side, motion features in the previous and next frames are extracted, and based on the extracted motion features, an interpolated frame is artificially generated using a frame rate conversion technique to increase the frame rate. The smoothness of the movement, which is impaired by the low frame rate, can be compensated.

  FIG. 10 is a block diagram showing the configuration of the transmission side and the reception side. The transmission side includes a frame decimation unit 60 and an H.264 encoder 70, and the reception side includes an H.264 decoder and a frame rate converter 90. The original image signal is reduced in frame rate by the frame reducer 60 on the transmission side, encoded by the H.264 encoder 70, and then transmitted as a broadcast wave. The encoded image received on the receiving side is decoded by the H.264 decoder, further increased in frame rate by the frame rate converter 90, and output as a converted image signal.

  Here, in the frame rate conversion on the receiving side, motion features may be erroneously detected due to the complexity of motion in the screen, the complexity of texture, compression distortion, etc., and an unnatural interpolation frame image may be generated There is. Therefore, in order to evaluate the performance of the frame rate conversion, an image quality evaluation apparatus capable of evaluating the image quality of the interpolation frame is required.

  Interpolated frame image quality is basically evaluated by comparing the interpolated frame image generated by frame rate conversion with the frame image at the same playback time as the interpolated frame image in the original image, and measuring the distortion. Can do.

  Patent Document 1 describes a digital image quality evaluation apparatus that can obtain an evaluation result close to the subjective evaluation result. In this digital image quality evaluation device, the SN ratio based on the difference value between the value of the evaluation target image (reproduced image) data from the image quality evaluation target system such as a video codec and a digital image transmission path and the value of the original image data is calculated. The SN ratio is weighted based on the AC component power of the original image data of the entire frame.

Patent Document 2 describes an image quality evaluation method that reflects human visual characteristics and can improve evaluation accuracy. In this case, a spatiotemporal region is extracted from the original image, the evaluation target image is divided into each spatiotemporal region, and image quality deterioration determination is performed for each divided spatiotemporal region.
Japanese Patent Laid-Open No. 9-307930 JP 2008-28707 A

  According to the digital image quality evaluation apparatus disclosed in Patent Document 1 and the image quality evaluation method disclosed in Patent Document 2, the image quality of the evaluation target image can be evaluated based on the distortion between the original image and the evaluation target image. When the original image is not compressed and encoded, the frame rate is simply reduced, and an interpolated frame image is generated from the image with the reduced frame rate by frame rate conversion, the quality of the interpolated frame image is the interpolated frame image. Can be evaluated by comparing the interpolated frame image and the frame image at the same reproduction time in the original image.

  On the other hand, when the frame rate of the original image is reduced and compression encoded, and an interpolated frame image is generated from the image with the reduced frame rate by frame rate conversion, the compression code Both the distortion caused by the conversion and the distortion caused by the unnatural frame interpolation are mixed.

  In order to determine the performance of frame rate conversion, it is necessary to be able to evaluate image quality degradation caused only by distortion due to frame interpolation, without being influenced by distortion due to compression coding.

  However, when the frame rate of the original image is reduced and compressed and encoded, and an interpolated frame image is generated from the image with the reduced frame rate by frame rate conversion, the interpolated frame can be used regardless of which of the conventional techniques is applied. There is a problem that it is impossible to determine whether image quality deterioration of an image is caused by distortion due to frame rate interpolation or distortion caused by compression coding.

  An object of the present invention is to provide an image quality evaluation apparatus capable of accurately evaluating the image quality of an interpolated frame generated by frame rate conversion while eliminating the influence of image quality degradation caused by factors other than frame rate conversion, such as compression encoding. Is to provide.

  In order to solve the above-described problem, the present invention inputs an original image and an evaluation target image including an interpolation frame in which the original image is once reduced in frame rate and then increased again by frame rate conversion, An image quality evaluation apparatus for evaluating the image quality of an interpolation frame by rate conversion, a block evaluation value calculation means for calculating distortion of an evaluation target image with respect to an original image as a block evaluation value for each block by intra-frame division, and a non-interpolated frame A non-interpolated frame distortion totaling unit that determines a distortion threshold based on the block evaluation value in the block, a block evaluation value in an interpolated frame among the block evaluation values calculated by the block evaluation value calculating unit, and the non-interpolated frame distortion totaling The distortion in the interpolated frame based on the distortion threshold calculated by the means There is first characterized in having a distortion block evaluation means for outputting a distortion block number to determine the lock.

  The second aspect of the present invention is that the block evaluation value calculation means calculates a difference absolute value sum or a sum of squared differences of pixels at the same position of corresponding blocks of the original image and the evaluation target image as a block evaluation value. There are features.

  Further, the present invention provides an evaluation value calculation unit in which the block evaluation value calculation unit calculates distortion of an evaluation target image with respect to an original image in units of blocks by intra-frame division, and a block evaluation calculated by the evaluation value calculation unit. A non-interpolated frame evaluation value storage memory for storing a block evaluation value in a non-interpolated frame of values, and an interpolation frame evaluation for storing a block evaluation value in an interpolated frame among block evaluation values calculated by the evaluation value calculation unit A third feature is that it has a value storage memory.

  Further, according to the present invention, the non-interpolated frame distortion counting unit determines a maximum value of block evaluation values of all blocks in a non-interpolated frame among the block evaluation values calculated by the block evaluation value calculating unit as a distortion threshold value. The point has the fourth feature. The fifth aspect of the present invention is that the distortion block evaluation means outputs the block number of the block as the distortion block number when the block evaluation value of each block in the interpolation frame exceeds the distortion threshold. There are features.

  Further, the present invention has a sixth feature in that it further includes a distortion block totaling unit that calculates and outputs a distortion block appearance rate based on a distortion block number output from the distortion block evaluation unit.

  Further, the present invention is characterized in that the distorted block totaling unit totals the number of appearances of the distorted block number and outputs a ratio of the number of appearances of distorted blocks to the total number of blocks of the non-interpolated frame as a distorted block appearance rate. There are 7 features.

  According to the present invention, it is possible to accurately evaluate the image quality of an interpolated frame generated by frame rate conversion while eliminating the influence of image quality degradation caused by factors other than frame rate conversion, such as compression encoding.

  The present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a first embodiment of the present invention. The image quality evaluation apparatus of the first embodiment includes a block evaluation value calculation means 10, a non-interpolated frame distortion totaling means 20, and a distortion block evaluation means 30. The block evaluation value calculation means 10 includes an evaluation value calculation unit 11, a switching unit 12, a non-interpolated frame evaluation value storage memory 13, and an interpolation frame evaluation value storage memory 14. Each of these means and units can be realized by hardware or software.

  The original image signal and the converted image signal are input to the evaluation value calculation unit 11 of the block evaluation value calculation means 10. The post-conversion image signal is an image signal including an interpolated frame in which the original image signal is thinned out, once reduced to a low frame rate, and again increased in frame rate by frame rate conversion. Hereinafter, the frame image of the original image signal is referred to as an original frame image, the image of the converted image signal is referred to as an evaluation target image, and the frame image is referred to as an evaluation target frame image.

  FIG. 2 is a conceptual diagram illustrating an example of a correspondence relationship between frames of an original image and an evaluation target image. When the frame rate of the original image is F, the original image is an original frame image (1) with a time interval 1 / F whose reproduction times are t, t + 1 / F, t + 2 / F,. It consists of (2), (3), ... Among the original images, even-numbered original frame images (2),... Are thinned out, and only odd-numbered original frame images (1), (3),. It is an image. That is, the H.264 encoded image is composed of H.264 encoded frame images (1 ′), (3 ′),... And is reduced to a low frame rate.

  The evaluation target image is generated from the decoded images of the H.264 encoded frame images (1 ′), (3 ′),. In this example, the frame rate of the evaluation target image is the same as the frame rate of the original image, and the odd-numbered evaluation target frame images (1 ″), () whose reproduction times are t, t + 2 / F,. 3 ″),... Are generated by decoding H.264 encoded frame images (1 ′), (3 ′),..., Respectively, and playback times are t + 1 / F, t + 3 The images to be evaluated (2 ″), (4 ″),... Of / F,... Decode the H.264 encoded frame images (1 ′), (3 ′),. The frame rate is converted by the frame rate converter 50. For example, the evaluation target image (2 ″) whose reproduction time is t + 1 / F is generated by decoding the H.264 encoded frame images (1 ′) and (3 ′) and interpolating them.

  The original image is, for example, 30 frames / second. An H.264 encoded image is an original image that has been frame-rate converted to 15 frames / second and compression encoded, and each H.264 encoded frame image is distorted by compression encoding. The evaluation target image is obtained by decoding an H.264 encoded image and further converting the frame rate to 30 frames / second.

  Returning to FIG. 1, the evaluation value calculation unit 11 includes the original frame images (1), (2), (3),... And the evaluation target frame images (1 ″), (2 ″), (3 ″),... Are divided into unit blocks, and a block evaluation value for each unit block between the evaluation target frame image and the original frame image is calculated. The block evaluation value is a unit block of the evaluation target frame image. It shows how much the image in the image is different from the image in the corresponding unit block of the original frame image at the same playback time, where the size of the unit block is arbitrary but general Each unit block has a size of 16 × 16 pixels.

  The block evaluation value is calculated for all frames having the same reproduction time of the original frame image and the evaluation target frame image. For example, in the case of FIG. 2, between the original frame image (1) and the evaluation target frame image (1 ″), between the original frame image (2) and the evaluation target frame image (2 ″), the original frame image (3). And the evaluation frame image (3 ″), block evaluation values are calculated for.

  FIG. 3 is a conceptual diagram showing the operation of the evaluation value calculation unit 11. The evaluation value calculation unit 11 divides the original frame image and the evaluation target frame image into unit blocks, calculates a difference norm for each unit block at the same position, and sets this as a block evaluation value. The difference norm is a value representing the difference between the original frame image and the evaluation target frame image, and a difference absolute value sum or a difference square sum of pixels at the same position of the original frame image and the evaluation target frame image can be used. For example, the sum of absolute differences or the sum of squared differences of each pixel at the same position in the unit block shown in the bold frame in the original frame image and the evaluation target frame image is obtained, and this is used as the block evaluation value of the unit block. In this case, a unit block having a large difference norm indicates that the distortion of the evaluation target frame image is larger than that of the original frame image.

  The switching unit 12 inputs the block evaluation value P for the non-interpolated frame calculated by the evaluation value calculating unit 11 to the non-interpolated frame evaluation value storage memory 13, and stores the block evaluation value R for the interpolation frame in the interpolated frame evaluation value It operates to input to the memory 14.

  The non-interpolated frame evaluation value storage memory 13 stores the block evaluation value P for the non-interpolated frame calculated by the evaluation value calculation unit 11, and the interpolation frame evaluation value storage memory 14 stores the block evaluation value R for the interpolation frame. Remember.

  The non-interpolated frame distortion totaling means 20 totals the block evaluation values P for all blocks of the original frame image and the evaluation target frame image at the reproduction time of all non-interpolated frames, and a threshold value for determining a distorted block in the interpolated frame (Distortion threshold) Thr is determined.

  FIG. 4 is a conceptual diagram showing the operation of the non-interpolated frame distortion totaling means 20. Frames for evaluation images (1 ″), (3 ″), (5 ″),... With playback times t, t + 2 / F, t + 4 / F,. And generated by the frame rate converter 50. These evaluation target frame images (1 ″), (3 ″), (5 ″),... And the original frame images (1), (3), (5 ),... Are calculated by the evaluation value calculation means 11 and stored in the non-interpolated frame evaluation value storage memory 13. The non-interpolated frame distortion totaling means 20 totals this block evaluation value P and, for example, gives the block evaluation value P that is the maximum value over all frames to the distortion block evaluation means 30 as a distortion threshold Thr.

  The distortion block evaluation means 30 compares the block evaluation value R in the interpolation frame stored in the interpolation frame evaluation value storage memory 14 with the distortion threshold Thr given from the non-interpolation frame distortion aggregation means 20, and the block evaluation value R is The block number exceeding the distortion threshold Thr is output as the distortion block number.

  FIG. 5 is a conceptual diagram showing the operation of the distortion block evaluation means 30. Frames of evaluation object frames (2 ″), (4 ″),... With reproduction times t + 1 / F, t + 3 / F,... Are interpolation frames, and these evaluation object frames. The block evaluation value R between the image (2 ″), (4 ″),... And the original frame image (2), (4),... Is calculated by the evaluation value calculation unit 11 and is not interpolated. It is stored in the frame evaluation value storage memory 13. The distorted block evaluation means 30 compares this block evaluation value R with the distortion threshold Thr given from the non-interpolated frame distortion totaling means 20, and outputs the block number where the block evaluation value R exceeds the distortion threshold Thr as the distortion block number. . The distorted block evaluation means 30 outputs the distorted block numbers for all blocks in all interpolated frames. In the block in which the block evaluation value in the non-interpolated frame exceeds the distortion threshold value Thr, it indicates that distortion has occurred due to the influence of elements other than compression coding.

  FIG. 6 is a diagram illustrating an example of an image having distortion generated due to the influence of elements other than compression coding. Here, an example is shown in which lines and regions not included in the original image are generated in the evaluation target image in the blocks indicated by the thick frames. These lines and regions are distortions generated by the influence of elements other than compression coding, and the distortion block evaluation means 30 can determine such blocks as unnatural distortion blocks generated by frame rate conversion. If the determined distortion block is displayed, for example, by enclosing it in a red frame or changing the display of that portion, it is possible to easily recognize the occurrence of distortion other than distortion due to compression coding.

  FIG. 7 is a flowchart showing an image quality evaluation procedure in the first embodiment. Here, the block evaluation value P for the non-interpolated frame is already stored in the non-interpolated frame evaluation value storage memory 13, and the block evaluation value R for the interpolation frame is already stored in the interpolation frame evaluation value storage memory 14. To do. The following steps S2 to S5 are processes performed by the non-interpolated frame distortion totaling means 20, and steps S6 to S9 are processes performed by the distortion block evaluating means 30.

  First, a distortion threshold Thr for determining a distortion block in an interpolation frame is set to an initial value 0 (S1). Steps S2 to S5 are performed for each of those blocks through all the non-interpolated frames. In step S2, a block evaluation value P for each block in each non-interpolated frame is acquired. Initially, the block evaluation value P for the first block in the first non-interpolated frame is acquired. Next, it is determined whether or not the block evaluation value P acquired in S2 exceeds the distortion threshold Thr (S3). In this case, the distortion threshold value Thr is set as the block evaluation value P (S4). That is, the distortion threshold Thr is rewritten to the block evaluation value P.

  Next, it is determined whether or not the block is the final block (S5). If it is determined that the block is the final block, the process proceeds to S6. If it is determined that the block is not the final block, the process returns to S2.

  In S2, the block evaluation value P for the next block is acquired, and thereafter the steps of S3 to S5 are repeated. By repeating the steps S2 to S5, the maximum value of the block evaluation value P in the entire non-interpolated frame is obtained as the distortion threshold Thr.

  The following steps S6 to S9 are executed for each of those blocks through all the interpolation frames. In S6, a block evaluation value R for each block in each interpolation frame is acquired. Initially, the block evaluation value R for the first block in the first interpolation frame is acquired. Next, it is determined whether or not the block evaluation value R acquired in S6 exceeds the distortion threshold Thr (S7), and if it does not exceed, the process proceeds to S9, but it is determined that it exceeds In this case, the block number is output as a distortion block number.

  Next, it is determined whether or not the block is the final block (S9). If it is determined that the block is the final block, the process ends. If it is determined that the block is not the final block, the process returns to S6. .

  In S6, the block evaluation value R for the next block is acquired, and thereafter the steps of S7 to S9 are repeated. By repeating the steps S6 to S9, the block number whose block evaluation value R exceeds the distortion threshold Thr throughout the entire interpolation frame is output as the distortion block number.

  FIG. 8 is a block diagram showing a second embodiment of the present invention. In FIG. 8, the same or equivalent parts as in FIG. The image quality evaluation apparatus according to the second embodiment is different from the first embodiment in that a distortion block totaling means 40 is added.

  The configurations and operations of the block evaluation value calculation unit 10, the non-interpolated frame distortion totaling unit 20, and the distortion block evaluation unit 30 are the same as those in the first embodiment, and thus description thereof is omitted.

  The distorted block totaling means 40 generates a distorted block for each block in the entire interpolated frame based on the determination result of whether or not the block evaluation value R exceeds the distortion threshold Thr in the distorted block evaluating means 30. Count the number. Then, when the number of blocks in the entire interpolation frame is N and the number of occurrences of distortion blocks is D, D / N is output as the distortion block appearance rate.

  FIG. 9 is a flowchart showing an image quality evaluation procedure in the second embodiment. In FIG. 9, the same or equivalent parts as in FIG. Here, steps S1 to S9 are the same as those in FIG. 7 is different from FIG. 7 in that S10 and S11 are added, and these are processes performed by the distortion block totaling means 40.

  In S10, the number of occurrences of blocks for which it is determined in S7 that the block evaluation value R exceeds the distortion threshold Thr, that is, the number of occurrences of distortion blocks is counted. In S11, when the number of blocks of the entire interpolation frame is N, D / N is calculated as a distortion block appearance rate from the count value D and the number of blocks N in S10, and is output. When the distortion block appearance rate D / N is high, it indicates that unnatural image distortion due to frame rate conversion frequently occurs.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment, and can be variously modified. For example, if an original image of 120 frames / second is reduced to 60 frames / second, compression encoded, and then frame rate converted to 120 frames / second again, the original image of 60 frames / second is reduced to 15 frames / second. The present invention can be applied to the case where both frames of the original image and the image to be evaluated can be acquired at a plurality of the same reproduction times, such as when dropped and compressed and then converted again to 60 frames / second.

  The present invention is also effective as an image quality evaluation for frames in a predetermined section of the converted image. This may be done by processing the frame image of a predetermined section of the converted image and the frame image of the original image corresponding to the playback time.

1 is a block diagram showing a first embodiment of the present invention. It is a conceptual diagram which shows the example of the correspondence of the flame | frame of an original image and an evaluation object image. It is a conceptual diagram which shows operation | movement of an evaluation value calculation part. It is a conceptual diagram which shows operation | movement of a non-interpolation frame distortion total means. It is a conceptual diagram which shows the operation | movement in a distortion block evaluation means. It is a figure which shows the example of the image with the distortion which generate | occur | produced by the influence by elements other than compression encoding. It is a flowchart which shows the image quality evaluation procedure in 1st Embodiment. It is a block diagram which shows 2nd Embodiment of this invention. It is a flowchart which shows the image quality evaluation procedure in 2nd Embodiment. It is a block diagram which shows the structure of a transmission side and a receiving side.

Explanation of symbols

10 ... block evaluation value calculation means, 11 ... evaluation value calculation unit, 12 ... switching unit, 13 ... non-interpolated frame evaluation value storage memory, 14 ... interpolation frame evaluation value storage memory, 20 ... Non-interpolated frame distortion totaling means, 30 ... Distorted block evaluation means, 40 ... Distorted block totaling means, 50, 90 ... Frame rate converter, 60 ... Frame decimation unit, 70 ..H.264 encoder, 80 ... H.264 decoder

Claims (7)

Image quality for evaluating the quality of an interpolated frame by frame rate conversion using the original image and the evaluation target image including the interpolated frame, the original image having been once reduced in frame rate and increased in frame rate conversion again. An evaluation device,
A block evaluation value calculating means for calculating distortion of the evaluation target image with respect to the original image as a block evaluation value in block units by intra-frame division;
Non-interpolated frame distortion aggregation means for determining a distortion threshold based on a block evaluation value in a non-interpolated frame;
Based on the block evaluation value in the interpolation frame among the block evaluation values calculated by the block evaluation value calculation means and the distortion threshold value calculated by the non-interpolated frame distortion aggregation means, the distortion block in the interpolation frame is determined. An image quality evaluation apparatus comprising distortion block evaluation means for outputting a distortion block number.   The block evaluation value calculation means calculates a difference absolute value sum or a sum of squared differences of pixels at the same position of corresponding blocks of the original image and the evaluation target image as a block evaluation value. Image quality evaluation device.   The block evaluation value calculation means includes an evaluation value calculation unit that calculates distortion of the evaluation target image with respect to the original image in units of blocks by intra-frame division, and non-interpolation among the block evaluation values calculated by the evaluation value calculation unit A non-interpolated frame evaluation value storage memory for storing a block evaluation value in a frame; and an interpolation frame evaluation value storage memory for storing a block evaluation value in an interpolation frame among the block evaluation values calculated by the evaluation value calculation unit. The image quality evaluation apparatus according to claim 1, wherein:   The non-interpolated frame distortion totaling unit determines a maximum value of block evaluation values of all blocks in a non-interpolated frame among the block evaluation values calculated by the block evaluation value calculating unit as a distortion threshold value. Item 6. The image quality evaluation apparatus according to any one of Items 1 to 3.   5. The distorted block evaluation means outputs the block number of the block as a distorted block number when the block evaluation value of each block in the interpolation frame exceeds the distortion threshold value. The image quality evaluation apparatus according to any one of the above.   6. The apparatus according to claim 1, further comprising a distortion block totaling unit that calculates and outputs a distortion block appearance rate based on a distortion block number output from the distortion block evaluation unit. Image quality evaluation device.   The distortion block totaling unit totalizes the number of appearances of the distortion block number and outputs a ratio of the number of appearances of the distortion block with respect to the total number of blocks of the non-interpolated frame as a distortion block appearance rate. The image quality evaluation apparatus described.

Images