JP2010109640A - Video display device, video display method, and image high resolution method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain high image quality by attaining the high resolution of a compressed animation. <P>SOLUTION: A video display device includes: a decoding part for decoding the encoded animation and encoding parameter information encoded together with the encoded animation; a generating part of image quality determination feature for generating a featured value to be used for determining the image quality, based on the plurality of images included in the decoded animation; an image quality determining part for determining whether the image quality is low or high through the use of the featured value which is generated in the generating part of image quality determination feature; a high resolution part for performing the high resolution processing of the animation which is decoded by using the image quality determination result obtained by the determination of the image quality determining part; and a display part for displaying the animation where the high resolution processing is performed by the high resolution part. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for increasing the resolution of moving images.

  In digital cameras and digital video devices, there is a strong demand for high-definition video and images, and various high-resolution technologies have been devised.

  For example, in the technique for improving the image quality of a high-resolution image disclosed in Patent Document 1, the reference image is first removed from an image to be increased in resolution (hereinafter referred to as a reference image) among a plurality of images. A plurality of image data are weighted according to the following procedure. Image data with a shorter distance from the reference image is given a higher weight, and an image having a larger amount of misalignment correction between the reference image and each image is given a smaller weight, and the position difference between the pixels constituting the image after high resolution The smaller the inter-pixel distance from the pixels in the vicinity of the pixels constituting the high-resolution image among the pixels constituting the image after correction, the smaller the weight.

  Next, a plurality of image data is synthesized using these weights to generate high resolution image data. After decoding the moving image encoded by the encoding technique shown in Non-Patent Document 1, the resolution enhancement process shown in Patent Document 1 is performed using a plurality of images included in the moving image.

  Further, Patent Document 2 describes a technique that can accurately detect block noise from an input video signal in which block noise is generated even if there is no block boundary information from the video compression data decoding device. .

  Patent Document 3 describes a signal processing technique in which high-frequency data of an original signal having a Nyquist frequency or higher is restored from a plurality of sets of digital data having different sampling positions to obtain high-resolution data with less blur. ing.

  Further, in Patent Document 4, the distortion of aliasing is reduced by canceling the aliasing component for a plurality of two-dimensional digital data having a two-dimensional sampling position shift and an arbitrary number of aliasing components, A two-dimensional digital signal processing technique for restoring the high-frequency component of the original signal is described.

JP 2006-33062 A JP 2007-28460 A JP-A-8-336046 JP-A-9-069755 Sullivan, G.J .; Wiegand, T .; Video Compression- from concepts to the H.264 / AVC standard. IEEE proc vol.93, No.1, p.18-31 (January2005) Richard O. Duda, Peter E. Hart, David G. Stork, "Pattern Classification (2nd. Ed.)", John Wiley & Sons, Inc. (2001)

  When the high resolution processing described in Patent Document 1 is applied to the high resolution of video data, the weight of the B picture with low image quality is high in the processing when the picture immediately before the reference image is a B picture. Therefore, there is a problem in that the image quality of the high resolution image synthesized using the B picture is deteriorated.

  The present invention has been made to solve the above problems, and an object thereof is to generate a high-resolution image with higher image quality.

  In order to solve the above problems, the video display device of the present invention includes a decoding unit that decodes an encoded moving image and encoding parameter information encoded together with the encoded moving image, and the decoded An image quality determination feature generation unit that generates a feature amount used to determine image quality from a plurality of images included in the moving image, and a low image quality or a high image quality using the feature amount generated by the image quality determination feature generation unit An image quality determination unit for determining whether the image quality determination result is determined by the image quality determination unit, a resolution enhancement unit that performs a resolution enhancement process on the decoded moving image, and a resolution at the resolution enhancement unit. And a display unit for displaying the processed moving image.

  Furthermore, a reference image selection unit that selects an image to be used for the resolution enhancement processing among a plurality of images included in the moving image using the image quality determination result determined by the image quality determination unit is provided.

  In addition, the reference image selection unit includes an I / P picture selection unit that preferentially selects an image of an I picture or a P picture over a picture determined to have low image quality by the image quality determination unit.

  According to the present invention, it is possible to generate a high-resolution moving image with higher image quality.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments for implementing the invention will be described with reference to the drawings. In addition, in each drawing, the component to which the same code | symbol is attached | subjected shall have the same function.

First, an example of a video display apparatus according to Embodiment 1 of the present invention will be described with reference to FIG.
2 includes an input unit 201, a decoding unit 202, a resolution enhancement unit 203, a display unit 204, an output unit 205, and a storage medium 206. Details will be described below.

  The input unit 201 acquires an encoded stream including a moving image by receiving an input from another video device or receiving a digital television broadcast. The acquired encoded stream is recorded on the recording medium 203 or sent to the decoding unit 202.

  The decoding unit 202 decodes the encoded stream acquired from the input unit 201 or the recording medium 203, and generates a moving image signal in, for example, RGB or component (Y / Cb / Cr) format. The decoding unit 202 transmits the generated video signal and encoding parameter information decoded from the encoded stream to the high resolution unit 203.

  The high resolution unit 203 generates a single high resolution image using a plurality of images included in the moving image signal, and generates a high resolution moving image by repeating this. The number of images used for generating one high-resolution image is two or more. Details of the resolution enhancement processing in the resolution enhancement unit 203 will be described later. The high resolution unit 203 outputs the generated high resolution moving image to the display unit 204 or the output unit 205.

  The display unit 204 is a unit having a display element, and is realized by a liquid crystal display device, a plasma display device, a projector device, or the like, and displays a high-resolution moving image acquired from the high-resolution unit 203. In this case, the video display device constitutes a liquid crystal television device, a plasma television device, or the like.

  Further, the output unit 205 outputs the high-resolution moving image acquired from the high-resolution unit 203 to another device or the like.

  As another configuration example, for example, if the video display device according to the first embodiment does not include the display unit 204 but includes the recording medium 203 and the output unit 205, a video storage device such as a recorder device is configured. Is also possible.

  Next, FIG. 3 illustrates a detailed configuration and an operation flow of the decoding unit 202 according to the first embodiment. The decoding unit 202 includes an inverse quantization unit 301, an inverse transformation unit 302, a deblocking filter 303, a variable length decoder 304, an intra prediction 305, a motion compensation 306, and a frame memory 307. The encoded stream input to the decoding unit 202 stores encoded parameter information used for encoding together with the encoded moving image signal. The encoding parameter information is information such as picture type and quantization accuracy. Such information is described in detail in Non-Patent Document 1, for example.

  In the decoding unit 202, first, the variable length decoder 304 performs variable length decoding processing on the input signal to the decoding unit 202. At this time, the quantized video signal and the encoding parameter information are decoded. The variable length decoder 304 outputs the quantized moving image signal and the encoding parameter information to the inverse quantizer 302. At this time, the encoding parameter information is also output to the resolution enhancement unit 203.

Next, the inverse quantizer 301 performs an inverse quantization process on the output of the variable length decoder 304. At this time, the inverse quantizer 301 uses a quantization accuracy parameter in the encoding parameter information.
Next, the inverse converter 302 performs conversion processing such as inverse DCT conversion.
Next, the signal processed by the inverse transformer 302 is added to the output 305 from the intra prediction unit or the output of the motion compensation result 306 and output to the deblocking filter unit 303.

Next, the output image from the deblocking filter unit 303 is stored in the frame memory 307, and is simultaneously output to the high resolution unit 203 as a decoded image output from the decoder 202.
Note that in FIG. H.264 decoder is displayed. However, the configuration is not limited to this, and a configuration without a deblocking filter is also possible.

  FIG. 4 illustrates a detailed configuration and operation of the resolution enhancement unit 203. The resolution enhancement unit 203 includes a reference image selection unit 100 and a high resolution image generation unit 401.

  In the resolution enhancement unit 203, first, the reference image selection unit 100 inputs coding parameter information and a moving image signal that are inputs to the resolution enhancement unit 203, selects a reference image, and selects the reference image as a high resolution image. The data is output to the generation unit 401. Next, the high-resolution image generation unit 401 inputs the reference image selected from the moving image signal that is an input to the high-resolution unit 203 and the reference image output from the reference image selection unit 100, and generates a high-resolution image. To do.

  Returning to FIG. 1, a detailed configuration of the reference image selection unit 100 according to the first embodiment will be described. The reference image selection unit 100 includes an input unit 101, a frame memory 102, an image quality determination feature generation unit 103, an image quality determination unit 104, a previous image acquisition unit 105, a neighborhood I / P picture search unit 106, and a neighborhood I / P picture acquisition unit 107. Consists of

The input unit 101 stores, in the frame memory 102, encoding parameter information and a moving image signal that are inputs to the resolution enhancement unit 203.
The image quality determination feature generation unit 103 extracts the image immediately before the reference image and the encoding parameter information from the frame memory 102, calculates the feature amount used for the image quality determination, and collects it as a feature vector to collect the image quality determination unit 104. Output to.

Next, details of the image quality determination feature generation unit 103 will be described with reference to FIG.
The image quality determination feature generation unit 103 includes a feature amount generation unit 500 and a feature amount integration unit 505. The feature amount generation unit 500 includes a picture information acquisition unit 501, a quantization information acquisition unit 502, a picture coding size acquisition unit 503, a block noise amount calculation unit 504, and an execution control unit 507.

  The picture information acquisition unit 501 acquires the picture parameter of the immediately preceding image from the encoding parameter information. In general, in moving picture coding, a B picture has a property that image quality is likely to be lower than an I picture and a P picture. Therefore, whether or not an image is a B picture is effective information for determining image quality.

  The quantization information acquisition unit 502 acquires the quantization accuracy of the immediately preceding image from the encoding parameter information. Since the quantization accuracy is a parameter that greatly affects the image quality, it is effective information for determining the image quality.

  The picture coding size acquisition unit 503 acquires the picture coding size from the coding parameter information. The size of the encoded picture corresponds to the compression rate of the picture, and is information that is effective for determining the image quality when used in combination with the picture parameter.

The block noise amount calculation unit 504 calculates the amount of block noise of the image. Here, the method described in Patent Document 2 can be used to calculate the amount of block noise.
Here, as an example of the feature quantity related to the image quality by encoding, the feature quantity generated by the function blocks 501 to 504 has been described, but the feature quantity representing the image quality deterioration by encoding is not dependent on this.

  The feature amount integration unit 505 combines the feature amounts generated by the feature amount generation unit 500 into a feature vector and outputs it to the image quality determination unit 104.

  The execution control unit 507 determines whether or not to execute the processing of the picture information acquisition unit 501, the quantization information acquisition unit 502, the picture coding size acquisition unit 503, and the block noise amount calculation unit 504 using the execution control information 506. Switch. The execution control information 506f determines whether to acquire a predetermined value stored in the frame memory 102 or to execute for each value of the processing load of the resolution enhancement unit 203, and execute control Whether or not to execute the processing may be switched by using the processing load as the information 506.

  Returning to the description of FIG. 1, the image quality determination unit 104 receives the feature vector output from the image quality determination feature generation unit 103, and determines the class to which the feature vector belongs, in this case, two classes of low image quality and high image quality. The discrimination function determines whether the image quality is low or high. The discriminant function prepares teacher data for each of the encoded N images, in which a plurality of people determine low image quality or high image quality, and the determination result of the majority occupies the final result. The N images and teacher data can be input to design using the method described in Non-Patent Document 2.

  The immediately preceding image acquisition unit 105 acquires the image immediately before the reference image from the frame memory 102 and outputs the image to the high resolution image generation unit 401.

The neighborhood I / P picture search unit 106 searches the frame memory 102 for an I picture or P picture in the neighborhood of the reference image, and if there is an I picture or P picture in the neighborhood, the neighborhood I / P picture On the other hand, the picture acquisition unit 107 is turned on. On the other hand, if there is no I picture or P picture in the vicinity, the immediately preceding image acquisition unit 105 is turned on.
The neighboring I / P picture acquisition unit 107 acquires a neighboring I / P picture from the frame memory 102 and outputs the image to the high resolution image generation unit 401.

Next, an operation flow of the reference image selection unit 100 according to the first embodiment will be described with reference to FIG.
The reference image selection unit 100 stores, in the frame memory 102, encoding parameter information and a moving image signal that are input to the resolution enhancement unit 203 by the input unit 101 (601). Next, the image quality determination feature generation unit 103 acquires the immediately preceding image and encoding parameter with respect to the reference image from the frame memory 102 (602). Next, the image quality determination feature generation unit 103 calculates feature amounts used for image quality determination, and outputs the feature amounts together as feature vectors to the image quality determination unit 104 (603).

  Next, the image quality determination unit 104 inputs the feature vector output from the image quality determination feature generation unit 103, and inputs it to the identification function for determining the class to which the feature vector belongs, in this case, two classes of low image quality and high image quality. It is determined whether the image quality is low or high (604). If it is determined that the image quality is low (605, Y), the same image as the temporally previous image acquired in 602 is deleted from the frame memory (606), and the neighboring I / P picture search unit 106 uses the frame memory 102 as a reference. A search is made as to whether there is an I picture or P picture in the vicinity of the image (607). However, when the frame memory is sufficient, the process 606 may not be performed.

  On the other hand, if it is determined that the image quality is high (605, N), the immediately preceding image acquisition unit 105 acquires the immediately preceding image with respect to the reference image from the frame memory 102 and uses the image as the high resolution image generating unit 401. (608). As a result of the processing in 607, when an I / P picture exists immediately before in the vicinity (609, Y), the I / P picture is acquired from the frame memory 102 (610), and the image is converted into a high-resolution image generation unit 401. Output to. On the other hand, if no I / P picture exists in the vicinity (609, N), 608 is processed.

  Next, details of the high-resolution image generation unit 401 according to the first embodiment will be described. Hereinafter, in the description of the high resolution image generation unit 401, the image decoded by the decoding unit 202 will be described as an image in units of frames.

  First, the high resolution image generation unit 401 performs high resolution by, for example, three processes of (1) position estimation, (2) wideband interpolation, and (3) weighted sum. Here, (1) position estimation is to estimate the difference in sampling phase (sampling position) of each image data using each image data of a plurality of input image frames. (2) Wideband interpolation increases the image data density by interpolating and increasing the number of pixels (sampling points) using a wide-band low-pass filter that transmits all high-frequency components of the original signal, including aliasing components. It is to become. (3) The weighted sum is a weighted sum corresponding to the sampling phase of each densified data, canceling out aliasing components generated during pixel sampling and simultaneously removing the high-frequency components of the original signal. Is to restore.

  FIG. 7 shows an outline of this high resolution technology. As shown in FIG. 6A, frame # 1 (701), frame # 2 (702), and frame # 3 (703) on different time axes are input, and these are combined to form an output frame (706). Assume that you get. For simplicity, first consider the case where the subject has moved (704) in the horizontal direction, and consider increasing the resolution by one-dimensional signal processing on the horizontal line (705).

  At this time, as shown in (b) and (d) in the same figure, in the frame # 2 (702) and the frame # 1 (701), the position of the signal waveform is shifted according to the amount of movement (704) of the subject. Occurs. The position deviation amount is obtained by (1) position estimation. As shown in FIG. 5C, the frame # 2 (702) is motion-compensated (707) so that the position deviation is eliminated, and the pixel ( The phase difference θ (711) between the sampling phases (709) and (710) of 708) is obtained.

  Based on this phase difference θ (711), by performing the above (2) wideband interpolation and (3) weighted sum, as shown in FIG. = Pi), a new pixel (712) is generated at a position to achieve high resolution. (3) The weighted sum will be described later. Actually, the movement of the subject may be accompanied by movements such as rotation and enlargement / reduction as well as parallel movement, but if the time interval between frames is very small or the movement of the subject is slow, These movements can also be considered by approximating local translation.

  At this time, the first configuration example of the high-resolution image generation unit 401 is configured to perform the high-resolution processing described in Patent Literature 3, Patent Literature 4, or Non-Patent Literature 2. In this case, when performing the weighted sum of the above (3), as shown in FIG. 8, it is possible to increase the resolution twice in the one-dimensional direction by using signals of at least three frame images.

Here, the high resolution processing in the first configuration example of the high resolution image generation unit 401 will be described with reference to FIG.
FIG. 8 is a diagram showing the frequency spectrum of each component in a one-dimensional frequency region. In the figure, the distance from the frequency axis represents the signal intensity, and the rotation angle around the frequency axis represents the phase. The weighted sum of (3) above will be described in detail below.

  When the pixel interpolation is performed by the broadband low-pass filter that transmits twice the Nyquist frequency band (frequency band 0 to sampling frequency fs) in the broadband interpolation of (2) above, the same component as the original signal (hereinafter referred to as the original component) And the sum of the aliasing components according to the sampling phase is obtained. At this time, when the (2) wideband interpolation processing is performed on the signals of the three frame images, as shown in FIG. 8A, the original components (801), (802), and (803) of each frame are displayed. It is well known that the phases are all in agreement, and the phase of the aliasing components (804), (805), and (806) rotates according to the difference in the sampling phase of each frame. In order to facilitate understanding of the respective phase relationships, the phase relationship of the original components of each frame is shown in FIG. 5B, and the phase relationship of the folded components of each frame is shown in FIG.

  Here, with respect to the signals of the three frame images, by appropriately selecting the coefficients to be multiplied and performing the above (3) weighted sum, the aliasing components (804), (805), and (806) of each frame are mutually connected. It can be canceled out and only the original components can be extracted. At this time, the vector sum of the folded components (804), (805) and (806) of each frame is set to 0, that is, both the Re axis (real axis) component and the Im axis (imaginary axis) component are set to 0. For this purpose, at least three folding components are required. Therefore, by using signals of at least three frame images, it is possible to achieve double the resolution, that is, to remove one aliasing component.

Next, FIG. 15 shows a second configuration example of the high-resolution image generation unit 401. In this case, if the signals of at least two frame images are used, the resolution can be increased to twice that in the one-dimensional direction. Details will be described below.
First, a plurality of decoded images are input from the decoding unit 202 to the input unit 1500 of the high-resolution image generation unit 401.

  Next, the position estimation unit 1501 estimates the position of the corresponding pixel on the frame # 2 based on the sampling phase (sampling position) of the pixel to be processed on the frame # 1 input to the input unit 1500. The sampling phase difference θ1502 is obtained. The sampling phase difference θ1502 is information equivalent to the motion vector information. The sampling phase difference θ1502 can be calculated, for example, by performing a motion search process.

  Next, by using the up-raters 1503 and 1504 of the motion compensation / up-rate unit 1515, the frame # 2 is motion-compensated using the information of the phase difference θ1502, and aligned with the frame # 1, and the frame # 1 and the frame # The number of pixels of 2 is doubled to increase the density. The phase shift unit 1516 shifts the phase of the densified data by a certain amount. Here, π / 2 phase shifters 1506 and 1508 can be used as means for shifting the data phase by a certain amount. In addition, in order to compensate for the delay caused by the π / 2 phase shifters 1506 and 1508, the signals of the frame # 1 and the frame # 2 that have been densified by the delay units 1505 and 1507 are delayed.

  In the aliasing component removal unit 1517, the coefficients C0, C2, and the coefficients C0, C2 generated based on the phase difference θ1502 by the coefficient determiner 1509 for the output signals of the delay units 1505 and 1507 and the π / 2 phase shifters 1506 and 1508, respectively. C1 and C3 are multiplied by multipliers 1510, 1511, 1512 and 1513, respectively, and these signals are added by an adder 1514 to obtain an output. This output is output from the output unit 1518.

  Note that the position estimation unit 1501 can be realized using the above conventional technique as it is. Details of the up-raters 1503 and 1504, the π / 2 phase shifters 1506 and 1508, and the aliasing component removing unit 1517 will be described later.

  FIG. 9 shows an operation in the second configuration example of the high-resolution image generation unit 401 in FIG. This figure shows the outputs of the delay units 1505 and 1507 and the π / 2 phase shifters 1506 and 1508 shown in FIG. 15 in a one-dimensional frequency domain. In FIG. 9A, the signals of the frame # 1 and the frame # 2 after the up-rate output from the delay units 1505 and 1507 are returned from the original components 901 and 902 and the original sampling frequency (fs), respectively. A signal obtained by adding folded components 905 and 906 is obtained. At this time, the folded component 906 is rotated in phase by the above-described phase difference θ1502. On the other hand, the up-rate frame # 1 and frame # 2 signals output from the π / 2 phase shifters 1506 and 1508 are respectively the original components 903 and 904 after the π / 2 phase shift and the π / 2 phase shift. The signal is obtained by adding the subsequent folding components 907 and 908.

  (B) and (c) show the original component and the aliasing component extracted for easy understanding of the phase relationship between the components shown in (a). Here, when the vector sum of the four components shown in FIG. 4B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and the four components shown in FIG. When the vector sum is taken, the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are set to 0, and the weighted sum is taken. Only the components can be extracted. In other words, it is possible to realize an image signal processing apparatus that uses only two frame images to increase the resolution twice in the one-dimensional direction. Details of this coefficient determination method will be described later.

  FIG. 10 shows operations of the up-raters 1503 and 1504 used in the second configuration example of the high-resolution image generation unit 401 in FIG. In the figure, the horizontal axis represents frequency, and the vertical axis represents gain (the value of the ratio of the output signal amplitude to the input signal amplitude), indicating the “frequency-gain” characteristics of the up-raters 1503 and 1504.

  Here, in the up-raters 1503 and 1504, a new sampling frequency is set to a frequency (2fs) that is twice the sampling frequency (fs) of the original signal, and a new pixel is located at a position just in the middle of the original pixel interval. In addition, the number of pixels is doubled to increase the density by inserting a sanding point (= zero point), and a filter with a frequency between −fs and + fs all having a gain of 2.0 is applied. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 11 shows a specific example of the up-raters 1503 and 1504 used in the second configuration example of the high-resolution image generation unit 401 in FIG. This figure shows the filter tap coefficients obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck (where k is an integer) becomes a generally known sinc function, and is shifted by (−θ) to compensate for the sampling phase difference θ1502, and Ck = 2 sin (πk + θ). / (Πk + θ) may be used. In the up-rater 1503, the phase difference θ1502 may be set to 0 and Ck = 2sin (πk) / (πk).

  Further, by expressing the phase difference θ (1502) as a phase difference in integer pixel units (2π) + a phase difference in decimal pixel units, the phase difference compensation in integer pixel units is realized by a simple pixel shift. For compensation of the phase difference in units of pixels, the filters of the up-raters 1503 and 1504 may be used.

  FIG. 12 shows an operation example of the π / 2 phase shifters 1506 and 1508 used in the second configuration example of the high-resolution image generation unit 401 in FIG. As the π / 2 phase shifters 1506 and 1508, generally known Hilbert transformers can be used. In FIG. 5A, the horizontal axis represents frequency, and the vertical axis represents gain (value of the ratio of output signal amplitude to input signal amplitude), indicating the “frequency-gain” characteristics of the Hilbert transformer. Here, in the Hilbert transformer, the frequency (2fs) that is twice the sampling frequency (fs) of the original signal is used as a new sampling frequency, and all frequency components except -0 between -fs and + fs are gained. A pass band of 1.0.

  In FIG. 2B, the horizontal axis represents frequency, and the vertical axis represents phase difference (difference in output signal phase with respect to input signal phase), indicating the “frequency-phase difference” characteristic of the Hilbert transformer. Here, the phase of the frequency component between 0 and fs is delayed by π / 2, and the phase of the frequency component between 0 and −fs is advanced by π / 2. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 13 shows an example in which the π / 2 phase shifters 1506 and 1508 used in the second configuration example of the high resolution image generation unit 401 in FIG. 15 are configured with Hilbert transformers. This figure shows the filter tap coefficients obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck may be Ck = 0 when k = 2m (where m is an integer), and Ck = −2 / (πk) when k = 2m + 1.

  The π / 2 phase shifters 1506 and 1508 used in the first embodiment of the present invention can also use differentiators. In this case, if the general expression cos (ωt + α) representing a sine wave is differentiated by t and multiplied by 1 / ω, d (cos (ωt + α)) / dt * (1 / ω) = − sin (ωt + α) = cos ( ωt + α + π / 2), and the function of π / 2 phase shift can be realized. In other words, after taking the difference between the value of the target pixel and the value of the adjacent pixel, a π / 2 phase shift function is realized by applying a filter having a “frequency-amplitude” characteristic of 1 / ω. May be.

  FIG. 14 shows an operation and a specific example of the coefficient determiner (1509) used in the second configuration example of the high-resolution image generation unit 401 in FIG. As shown in FIG. 9A, when the vector sum of the four components shown in FIG. 9B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and FIG. If the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are 0 when the vector sum of the four components shown in (c) is taken, two frame images Can be used to realize an image signal processing apparatus that achieves a resolution twice as high as that in the one-dimensional direction.

  As shown in FIG. 15, the coefficient for the output of the delay unit (1505) (the sum of the original component and the folded component of frame # 1 after the up-rate) is C0, and the output of the π / 2 phase shifter 1506 (after the up-rate) The coefficient for C1 is the sum of the π / 2 phase shift results of the original and folded components of frame # 1, and the coefficient for the output of delay device 1507 (the sum of the original and folded components of frame # 2 after the update) C2 and C3 as the coefficient for the output of the Hilbert transformer 1506 (the sum of the π / 2 phase shift results of the original component and the aliasing component of the frame # 2 after the up-rate), satisfying the condition of FIG. Then, the simultaneous equations shown in FIG. 14 (b) can be obtained from the phase relationships of the components shown in FIG. 9 (b) and FIG. 9 (c). The results shown can be derived.

  The coefficient determiner 1509 may output the coefficients C0, C1, C2, and C3 obtained in this way. As an example, the values of the coefficients C0, C1, C2, and C3 when the phase difference θ1502 is changed from 0 to 2π every π / 8 are shown in FIG. This corresponds to a case where the position of the signal of the original frame # 2 is estimated with an accuracy of 1/16 pixel and motion compensation is performed on the frame # 1.

  Note that the up-raters 1503 and 1504 and the π / 2 phase shifters 1506 and 1507 require an infinite number of taps in order to obtain ideal characteristics. But there is no practical problem. At this time, a general window function (such as a Hanning window function or a Hamming window function) may be used. If the coefficient of each tap of the simplified Hilbert transformer is the value of the left and right points centered on C0, that is, C (-k) = -Ck (k is an integer), the phase can be shifted by a certain amount. it can.

  As described above, the configuration of the high-resolution image generation unit 401 in FIG. 4 is the same as the configuration described in FIGS. 7 to 15, so that an image with high resolution can be generated from a plurality of images.

  In particular, according to the second configuration of the high-resolution image generation unit 401 illustrated in FIG. 15, it is possible to generate one high-resolution image from two images.

  According to the video display apparatus according to the first embodiment of the present invention described above, when the encoded stream is decoded and the high resolution processing is performed using a plurality of images included in the decoded moving image, the decoding is performed. It is determined whether the image quality of the plurality of images included in the moving image is low or high image quality, and a plurality of images to be subjected to high resolution processing are selected using the determination result. As a result, it is possible to prevent high resolution processing using low quality images and to generate high quality high resolution images.

  The video display device according to the second embodiment of the present application differs from the video display device according to the first embodiment in the configuration and operation flow of the reference image selection unit 100. Accordingly, since the configuration other than the configuration and operation flow of the reference image selection unit 100 is the same, the description thereof is omitted.

FIG. 16 illustrates a detailed configuration of the reference image selection unit 100 according to the second embodiment.
The reference image selection unit 100 includes an input unit 1601, a frame memory 1602, an image quality measurement unit 1603, and an image acquisition unit 1604. The input unit 101 stores coding parameter information and a moving image signal, which are inputs to the resolution enhancement unit 203, in the frame memory 1602.

  The image quality measurement unit 1603 takes out the image immediately before the reference image and the encoding parameter information from the frame memory 1602, calculates the feature amount used for image quality measurement, obtains the image quality value, and corresponds to the extracted image. In addition, it is stored in the frame memory 1602. This process is repeated until an image traced back a predetermined number of times. Next, details of the image quality measurement unit 1603 will be described with reference to FIG.

  The image quality measurement unit 1603 includes a feature amount generation unit 1700 and an image quality measurement 1705. The feature amount generation unit 1700 includes a picture information acquisition unit 1701, a quantization information acquisition unit 1702, a picture coding size acquisition unit 1703, a block noise amount calculation unit 1704, and an execution control unit 1707. Picture information acquisition unit 1701, quantization information acquisition unit 1702, picture coding size acquisition unit 1703, block noise amount calculation unit 1704, execution control unit 1707, picture information acquisition unit 501, quantization information acquisition unit 502, picture coding Since it is the same as that of the size acquisition part 503, the block noise amount calculation part 504, and the execution control part 507, description is abbreviate | omitted.

  The image quality measurement 1705 determines an image quality evaluation value from the feature value generated by the feature value generation unit 1700 and outputs the image quality evaluation value to the frame memory 1602 so as to be associated with the image input to the image quality measurement unit 1603. To do. The evaluation value of the image quality is used in the following description on the assumption that a larger value represents higher image quality and a smaller value represents lower image quality. The method for determining the evaluation value of the image quality from the feature amount is, for example, that for each of the N encoded images, a plurality of people evaluate the quality of the image with K-level values, and the average of the evaluated values A value is prepared as teacher data, and a method similar to the design of the discrimination function of the image quality determination unit 104 of the first embodiment is used by using the N images and the teacher data.

  The image acquisition unit 1604 acquires, from the frame memory 1602, the image having the highest image quality evaluation value obtained in the image quality measurement 1705 from a predetermined number of images temporally before the reference image. Are output to the high-resolution image generation unit 401.

  Next, an operation flow of the reference image selection unit 100 according to the second embodiment will be described with reference to FIG. The reference image selection unit 100 stores, in the frame memory 1602, the encoding parameter information and the moving image signal, which are input to the resolution enhancement unit 203 by the input unit 1601 (1801). Next, the image quality measurement unit 1603 acquires the immediately preceding image and encoding parameter with respect to the reference image from the frame memory 1602 (1802).

  Next, the image quality measurement unit 1603 calculates a feature amount used for image quality measurement, and calculates an image quality evaluation value using the feature amount (1803). Next, the image acquisition unit 1604 acquires, from the frame memory 1602, the image having the largest image quality value obtained in the image quality measurement 1705 from a predetermined number of images temporally before the reference image. The image is output to the high resolution image generation unit 401 (1804).

  According to the video display apparatus according to the second embodiment of the present invention described above, when the encoded stream is decoded and the high resolution processing is performed using a plurality of images included in the decoded moving image, the decoding is performed. Image quality values of a plurality of images included in the moving image are obtained, and a plurality of images to be subjected to the resolution enhancement process are selected using the image quality values. As a result, it is possible to prevent high resolution processing using low quality images and to generate high quality high resolution images.

It is an example of the functional block which the reference image selection part which concerns on one Example of this invention has. It is a figure which shows an example of the video display apparatus which concerns on one Example of this invention. It is an example of the functional block which the decoding part which concerns on one Example of this invention has. It is an example of the functional block which the high resolution part which concerns on one Example of this invention has. It is an example of the functional block which the image quality determination characteristic production | generation part which concerns on one Example of this invention has. It is an example of the flowchart which shows operation | movement of the reference image selection part which concerns on one Example of this invention. It is explanatory drawing of the high resolution process in one Example of this invention. It is explanatory drawing of the high resolution process in one Example of this invention. It is explanatory drawing of the high resolution process in one Example of this invention. It is explanatory drawing of the up-rate device in one Example of this invention. It is explanatory drawing of the up-rate device in one Example of this invention. It is explanatory drawing of the phase shifter in one Example of this invention. It is explanatory drawing of the phase shifter in one Example of this invention. It is explanatory drawing of the coefficient determiner in one Example of this invention. It is explanatory drawing of an example of the high resolution part in one Example of this invention. It is an example of the functional block which the reference image selection part in one Example of this invention has. It is an example of the functional block which the image quality measurement part in one Example of this invention has. It is an example of the flowchart which shows operation | movement of the reference image selection part in one Example of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Input part, 102 ... Frame memory, 103 ... Image quality determination feature production | generation part, 104 ... Image quality determination part, 105 ... Immediate image acquisition part, 106 ... Neighboring I / P picture search part, 107 ... Neighboring I / P picture acquisition part , 201 ... input unit, 202 ... decoding unit, 203 ... high resolution unit, 204 ... display unit, 205 ... output unit, 206 ... storage medium, 1600 ... input unit, 1501 ... position estimation unit, 1503 ... up-rater, 1504: Up-rater, 1505 ... Delayer, 1506 ... Hilbert transformer, 1507 ... Delayer, 1508 ... Hilbert transformer, 1509 ... Coefficient determiner, 1510 ... Multiplier, 1511 ... Multiplier, 1512 ... Multiplier, 1513 ... multiplier, 1514 ... adder, 1515 ... motion compensation / up-rate unit, 1516 ... phase shift unit, 1517 ... folding component removal unit

Claims (20)

A decoding unit that decodes the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality determination feature generation unit that generates a feature amount used to determine image quality from a plurality of images included in the decoded moving image;
An image quality determination unit that determines whether the image quality determination feature generation unit uses low or high image quality using the feature amount;
A high-resolution unit that performs high-resolution processing of the decoded moving image using the image quality determination result determined by the image quality determination unit;
A video display device comprising: a display unit configured to display a moving image that has been subjected to high resolution processing in the high resolution unit.   The image processing apparatus includes a reference image selection unit that selects an image to be used for the resolution enhancement processing among a plurality of images included in the moving image using the image quality determination result determined by the image quality determination unit. Item 2. The video display device according to Item 1.   The reference image selection unit includes an I / P picture selection unit that preferentially selects an image of an I picture or a P picture over a picture determined to have low image quality by the image quality determination unit. The video display device described. The image quality determination feature generation unit includes a picture information acquisition unit that acquires picture information from the encoding parameter information;
A quantization accuracy acquisition unit that acquires the quantization accuracy included in the encoding parameter information;
A picture coding size acquisition unit for acquiring a coding size of a picture included in the coding parameter information;
A block noise amount calculation unit that calculates a block noise amount from a plurality of images included in the moving image;
The image quality determination unit determines whether the image quality is low or high image quality by using one or a combination of the picture type information, the quantization accuracy information, the picture coding size, and the block noise amount. The video display device according to claim 1, wherein the video display device is a video display device.   5. The image quality determination feature generation unit includes an execution control unit that controls execution of the picture information acquisition unit, the quantization accuracy acquisition unit, a picture coding size acquisition unit, and a block noise amount calculation unit. Video display device according to   The image quality determination unit removes an image determined to have low image quality among a plurality of images included in the decoded moving image as an image to be used for the high resolution processing in the reference image selection unit. The video display device according to claim 3. A decoding step of decoding the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality determination feature generation step for generating a feature amount used for determining image quality from a plurality of images included in the decoded moving image;
An image quality determination step of determining whether the image quality determination feature step is a low image quality or a high image quality using the feature amount generated;
A resolution enhancement step for performing a resolution enhancement process on the decoded moving image using the image quality determination result determined in the image quality step;
A video display method comprising: a display step of displaying a moving image that has been subjected to the high resolution processing in the high resolution step.   A reference image selection step of selecting an image to be used for the high resolution processing among a plurality of images included in the moving image using the image quality determination result determined in the image quality determination step. Item 8. The video display method according to Item 7.   The video display method according to claim 8, wherein the reference image selection step includes an I / P picture selection step of preferentially selecting an I picture or P picture over a picture determined to have low image quality in the image quality step. .   The image quality determination step is characterized in that an image determined as low image quality among a plurality of images included in the decoded moving image is excluded as an image used in the resolution enhancement process in the reference image step. The video display method according to claim 7. A decoding unit that decodes the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality measuring unit for obtaining an evaluation value of image quality from a plurality of images included in the decoded moving image;
A high-resolution part that performs high-resolution processing of the decoded moving image using the evaluation value of the image quality measured in the image quality measurement part;
A video display device comprising: a display unit configured to display a moving image that has been subjected to high resolution processing in the high resolution unit.   The image processing apparatus includes a reference image selection unit that selects an image to be used for the resolution enhancement processing among a plurality of images included in the moving image using an image quality measurement result measured by the image quality measurement unit. Item 12. The video display device according to Item 11.   The video display device according to claim 12, wherein the reference image selection unit preferentially selects an image having a higher image quality measured by the image quality measurement unit. The image quality measuring unit acquires picture information from the encoding parameter information;
A quantization accuracy acquisition unit that acquires the quantization accuracy included in the encoding parameter information;
A picture coding size acquisition unit for acquiring a coding size of a picture included in the coding parameter information;
A block noise amount calculation unit that calculates a block noise amount from a plurality of images included in the moving image;
12. The image quality is measured by using any one or a combination of the picture type information, the quantization accuracy information, the picture coding size, and the block noise amount. Item 14. The video display device according to any one of Item 13. The video according to claim 14, wherein the image quality measurement unit includes an operation execution control unit that controls execution of operations of the picture information acquisition unit, the quantization accuracy acquisition unit, a picture coding size acquisition unit, and a block noise amount calculation unit. Display device A decoding step of decoding the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality measurement step of obtaining an evaluation value of image quality from a plurality of images included in the decoded moving image;
A resolution enhancement step for performing a resolution enhancement process on the decoded moving image using the evaluation value of the image quality measured in the image quality measurement step;
A video display method comprising: a display step of displaying a moving image that has been subjected to resolution enhancement processing in the resolution enhancement step.   And a reference image selection step of selecting an image to be used for the resolution enhancement processing from a plurality of images included in the moving image using the image quality measurement result measured in the image quality measurement step. Item 17. The video display method according to Item 16.   The video display method according to claim 17, wherein the reference image selection step preferentially selects an image having a higher image quality measured in the image quality measurement step. A decoding step of decoding the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality determination feature generation step for generating a feature amount used for determining image quality from a plurality of images included in the decoded moving image;
An image quality determination step for determining whether the image quality determination feature generation step uses low or high image quality using the feature amount;
An image resolution enhancement method comprising: a resolution enhancement step of performing a resolution enhancement process on the decoded moving image using the image quality determination result determined in the image quality determination step. A decoding step of decoding the encoded moving image and the encoding parameter information encoded together with the encoded moving image;
An image quality measurement step of obtaining an evaluation value of image quality from a plurality of images included in the decoded moving image;
An image resolution enhancement method comprising: a resolution enhancement step of performing resolution enhancement processing of the decoded moving image using the evaluation value of the image quality measured in the image quality measurement step.

Images