US20100183232A1 - Image processor, image processing method and program - Google Patents

Image processor, image processing method and program Download PDF

Info

Publication number
US20100183232A1
US20100183232A1 US12/691,651 US69165110A US2010183232A1 US 20100183232 A1 US20100183232 A1 US 20100183232A1 US 69165110 A US69165110 A US 69165110A US 2010183232 A1 US2010183232 A1 US 2010183232A1
Authority
US
United States
Prior art keywords
image
phase
pixels
pixel
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/691,651
Other languages
English (en)
Inventor
Hiroshi Oryoji
Takefumi Nagumo
Hideyuki Ichihashi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of US20100183232A1 publication Critical patent/US20100183232A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0125—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level one of the standards being a high definition standard

Definitions

  • the present invention relates to an image processor, an image processing method, and a program. More particularly, the present invention relates to an image processor, an image processing method, and a program, which are suitable for use when converting an input image to an image with a higher resolution.
  • a method called super-resolution is known as a method of obtaining an image with a higher resolution from an input image (for example, see Japanese Unexamined Patent Application Publication No. 2008-140012)
  • a consecutive frame image such as a moving image, which is input to an image processor 11 , is converted to an image with a higher resolution and output, as illustrated in FIG. 1 .
  • the image with a lower resolution being input to the image processor 11 will be referred to as an LR (low resolution) image
  • the image with a higher resolution being output from the image processor 11 will be referred to as an SR (super resolution) image.
  • the input LR image is up-sampled to an image with the same resolution as the SR image by an up-sampler 21 , and the up-sampled image is supplied to a motion vector detector 22 , a mask generator 23 , and a mixer 24 .
  • the image processor 11 includes a buffer 25 storing an SR image which is obtained from an LR image of a previous frame immediately before the current frame being processed at the current point of time.
  • the SR image supplied to the buffer 25 is stored only for one frame period and is supplied to the motion vector detector 22 and a motion compensator 26 .
  • the motion vector detector 22 detects a motion vector from the LR image supplied from the up-sampler 21 and the SR image supplied from the buffer 25 .
  • the motion compensator 26 performs motion compensation using the motion vector supplied from the motion vector detector 22 and the SR image supplied from the buffer 25 . Specifically, the motion compensator 26 predicts an SR image obtained from the LR image of the current frame using the SR image of the previous frame immediately before the current frame being processed and supplies the image obtained thus to the mask generator 23 and the mixer 24 as a prediction image.
  • the mask generator 23 generates a motion mask using the LR image supplied from the up-sampler 21 and the prediction image supplied from the motion compensator 26 .
  • the motion mask is information used for specifying a region of the LR image in which a moving subject is displayed, and which is generated by calculating a difference between the LR image and the prediction image.
  • the mixer 24 combines the prediction image supplied from the motion compensator 26 and the LR image supplied from the up-sampler 21 using the motion mask supplied from the mask generator 23 and supplies a combined image obtained thus to a down-sampler 27 and an adder 28 .
  • the LR image and the prediction image are subjected to weighted addition with weighting factors determined by the motion mask, thus obtaining the combined image.
  • the weighting factors are determined such that the contribution ratio of the LR image becomes larger in the region where a motion occurs, whereby image quality deterioration of the SR image resulting from prediction errors occurring in the region with a motion is suppressed.
  • the combined image obtained thus is down-sampled by the down-sampler 27 , and a reduced image obtained thus is supplied to a subtractor 29 .
  • the reduced image is an image with the same resolution as the LR image.
  • the subtractor 29 generates a differential image by calculating a difference between the LR image supplied to the image processor 11 and the reduced image.
  • the up-sampler 30 up-samples the differential image to an image with the same resolution as the SR image, thus obtaining an enlarged image.
  • an adder 28 adds the enlarged image and the combined image, outputs an image obtained thus as an SR image of the current frame, and supplies the SR image to the buffer 25 to be stored therein.
  • a combined image P 11 which is obtained by predicting the SR image of the current frame is down-sampled to obtain a reduced image P 12 , and a difference between the reduced image P 12 and an LR image P 13 is calculated to obtain a differential image P 14 .
  • the differential image P 14 is an image which is indicative of an error in the reduced image P 12 used as the LR image of the current frame obtained by prediction with respect to the LR image P 13 of the current frame. That is to say, the differential image P 14 can be said to be an error in the combined image P 11 used as the predicted SR image with respect to a correct SR image of the current frame which should have been obtained if there was no error.
  • the obtained SR image will become an image in which the LR image is more faithfully enlarged without any image quality deterioration.
  • an enlarged image which is indicative of the calculated error is added to the combined image obtained by prediction, thus obtaining the SR image.
  • a signal at the Nyquist frequency is not properly preserved in terms of its amplitude and phase. Therefore, depending on the phase of the pixels of the combined image being sampled, there may be a case where the waveform of the combined image is not properly preserved at the time of generating the reduced image.
  • the waveform of the combined image that is, the change in the pixel values of the pixels of the combined image in a predetermined direction
  • the vertical direction represents the pixel values of pixels of an image
  • the horizontal direction represents a predetermined direction of the image.
  • each circle represents each pixel on the image.
  • a waveform indicated by an arrow A 11 represents the waveform of a combined image. That is, a curve that connects adjacent pixels (i.e., the pixel values thereof) of the combined image forms the waveform of the combined image.
  • the combined image is composed of pixels which are repeatedly arranged in the horizontal direction in the figure in the order of a pixel having the intermediate value, a pixel having the maximum value, a pixel having the intermediate value, and a pixel having the minimum value. That is to say, in the example of FIG. 3 , the waveform of the combined image has a sinusoidal shape.
  • the pixels of the combined image are sampled every other pixel in the horizontal direction in the figure so as to generate a reduced image.
  • the pixels of the combined image are sampled every other pixel in the horizontal direction so as to generate a reduced image by starting with the second pixel from the left in the figure
  • the pixels of the combined image indicated by an arrow A 12 are sampled.
  • the pixels of the combined image indicated by the arrow A 12 have pixel values of either the maximum value or the minimum value, and the waveform of the reduced image will have the same sinusoidal shape as the waveform of the combined image.
  • the pixels of the combined image indicated by the arrow A 11 are sampled every other pixel in the horizontal direction so as to generate a reduced image by starting with the first pixel from the left in the figure
  • the pixels of the combined image indicated by an arrow A 13 are sampled.
  • the pixels of the combined image indicated by the arrow A 13 have pixel values of the intermediate value, and the waveform of the reduced image will have a flat shape without amplitude differently from the waveform of the combined image.
  • the pixels of the combined image are sampled at the Nyquist frequency of the combined image (i.e., the half-frequency of the combined image) so as to generate the reduced image, depending on a sampling position, there may be a case where the original waveform of the combined image is not properly preserved in the obtained reduced image.
  • the vertical direction represents the pixel values of pixels of an image
  • the horizontal direction represents a predetermined direction of the image.
  • arrows A 21 to A 27 represent a combined image, the pixels of the combined image being sampled, a reduced image, an LR image, a differential image, an enlarged image, and an SR image, respectively.
  • each circle represents each pixel on the image.
  • a reduced image as indicated by the arrow A 23 is obtained having the same waveform as the original waveform of the combined image, in which the sinusoidal waveform of the combined image is preserved.
  • the waveform of the LR image has a flat shape without any change in the horizontal direction of the figure, as indicated by the arrow A 24 .
  • the pixels of the LR image have pixel values of an intermediate value between the maximum value and the minimum value of the pixel values of the pixels of the reduced image.
  • a differential image having a waveform as indicated by the arrow A 25 is obtained.
  • the waveform of the differential image has a shape such that the waveform of the reduced image is reversed in the vertical direction of the figure with respect to the position of the intermediate value.
  • an enlarged image as indicated by the arrow A 26 is obtained in which the original waveform of the differential image is preserved.
  • the enlarged image is added to the combined image indicated by the arrow A 21 by the adder 28 , the errors generated at the time of generating the combined image are corrected, whereby an SR image as indicated by the arrow A 27 can be obtained having the same waveform as the waveform of the LR image.
  • the vertical direction represents the pixel values of pixels of an image
  • the horizontal direction represents a predetermined direction of the image.
  • arrows A 31 to A 37 represent a combined image, the pixels of the combined image being sampled, a reduced image, an LR image, a differential image, an enlarged image, and an SR image, respectively.
  • each circle in the figure represents each pixel on the image.
  • a reduced image as indicated by the arrow A 33 is obtained having a flat shape without any change in pixel values, in which the sinusoidal waveform of the combined image is not preserved.
  • the waveform of the LR image has a flat shape without any change in the horizontal direction of the figure, as indicated by the arrow A 34 .
  • the pixels of the LR image have pixel values of an intermediate value.
  • a differential image having a waveform as indicated by the arrow A 35 is obtained.
  • the waveform of the differential image has a shape such that it is flat in the horizontal direction in the figure, and the pixel values of the pixels thereof have the same value.
  • a difference between the maximum value or minimum value and the intermediate value should be detected as an error.
  • the sampling position at the time of generating the reduced image is not proper, the above-mentioned error is not detected.
  • an enlarged image as indicated by the arrow A 36 is obtained in which the original waveform of the differential image is preserved.
  • the enlarged image is added to the combined image indicated by the arrow A 31 by the adder 28 , the errors generated at the time of generating the combined image are not corrected, whereby an SR image as indicated by the arrow A 37 can be obtained having a different waveform from that of the LR image. That is to say, in the SR image, the errors at the certain phase generated at the time of generating the combined image remain unremoved.
  • the obtained SR image is used for the prediction of an SR image of a subsequent frame as it is.
  • errors generated in each frame namely noise components generated by the prediction are accumulated in the SR image.
  • the back-projection since pixels at a predetermined phase of the combined image of each frame are sampled at the time of generating the reduced image, there is a concern that errors are accumulated at positions (phases) of the pixels of the combined image, which are not sampled.
  • Map method a technique of suppressing accumulation of noise in an SR image.
  • the Map method utilizes characteristics of images having a strong spatial correlation to apply feedback to an image obtained by predicting an SR image with spatial constraints placed on the errors relative to an LR image. Therefore, accumulation of noise is prevented.
  • the Map method is able to suppress the accumulation of noise, since the edge portions are lost, the SR image will become blurred, resulting in image quality deterioration.
  • an image processor performing a super-resolution process of converting an input image with a first resolution to an output image with a second resolution higher than the first resolution with respect to a plurality of input images which are consecutive in time, including: prediction means for predicting the output image with the second resolution of a time being processed using the input image of the time being processed and the output image obtained by performing the super-resolution process on an input image of a time earlier than the time being processed; generation means for generating a reduced image with the first resolution composed of pixels at different phases of the prediction image, the phases being different from time to time, using a prediction image obtained by the prediction of the prediction means; difference calculation means for calculating a difference between the input image of the time being processed and the reduced image; and addition means for adding the difference which is up-sampled to the second resolution to the prediction image, thus generating the output image with the second resolution of the time being processed.
  • the generation means may change a phase of each pixel of the prediction image to be used for generating the reduced image for each time in accordance with a predetermined pattern.
  • the generation means may include selection means for selecting a phase of each pixel of the prediction image, and sampling means for generating the reduced image by sampling pixels at the phase selected by the selection means from the prediction image.
  • the generation means may include selection means for selecting a phase of each pixel of the prediction image, and filtering means for generating pixels at the selected phase by a filtering process using several pixels around a pixel of the prediction image which is positioned at the phase selected by the selection means, thus generating the reduced image.
  • the generation means may include phase control means for generating the reduced image composed of pixels which are positioned at a phase separated by a predetermined distance in a predetermined direction from a predetermined reference phase of the prediction image, and shifting a phase of the difference up-sampled to the second resolution by the predetermined distance in the predetermined direction.
  • the input image may be an image of an interlaced format.
  • the generation means may include: switching means for changing an output destination of the prediction image depending on whether the input image of the time being processed is a top-field image or a bottom-field image; first selection means for selecting a phase of each pixel of the prediction image obtained from the input image of a top field which is output from the switching means; first sampling means for generating the reduced image by sampling a pixel at the phase selected by the first selection means from the prediction image; second selection means for selecting a phase of each pixel of the prediction image obtained from the input image of a bottom field which is output from the switching means; and second sampling means for generating the reduced image by sampling pixels at the phase selected by the second selection means from the prediction image.
  • the first and second selection means may independently change the phase of each pixel of the prediction image used for generating the reduced image from field to field in accordance with a predetermined pattern.
  • the generation means may include phase control means for moving each pixel of the prediction image by a predetermined distance in a predetermined direction to shift a phase of each pixel of the prediction image; and reduced image generation means for generating the reduced image composed of pixels at a predetermined phase of the prediction image in which the phase is shifted by the phase control means.
  • the phase control means may change a direction of shifting the phase of each pixel of the prediction image in each time in accordance with a predetermined pattern.
  • an image processing method or program for performing a super-resolution process of converting an input image with a first resolution to an output image with a second resolution higher than the first resolution with respect to a plurality of input images which are consecutive in time, the method including the steps of: predicting the output image with the second resolution of a time being processed using the input image of the time being processed and the output image obtained by performing the super-resolution process on an input image of a time earlier than the time being processed; generating a reduced image with the first resolution composed of pixels at different phases of the prediction image, the phases being different from time to time, using a prediction image obtained by the prediction of the prediction unit; calculating a difference between the input image of the time being processed and the reduced image; and adding the difference which is up-sampled to the second resolution to the prediction image, thus generating the output image with the second resolution of the time being processed.
  • the output image with the second resolution of a time being processed is predicted using the input image of the time being processed and the output image obtained by performing the super-resolution process on an input image of a time earlier than the time being processed.
  • a reduced image with the first resolution composed of pixels at different phases of the prediction image, the phases being different from time to time, is generated using a prediction image obtained by the prediction of the prediction unit.
  • a difference between the input image of the time being processed and the reduced image is calculated.
  • the difference which is up-sampled to the second resolution is added to the prediction image, whereby the output image with the second resolution of the time being processed is generated.
  • FIG. 1 is a diagram illustrating a configuration of an image processor according to related art.
  • FIG. 2 is a diagram illustrating the generation of a differential image in the image processor of the related art.
  • FIG. 3 is a diagram illustrating the generation of a reduced image according to the related art.
  • FIG. 4 is a diagram illustrating an example where the waveform of an image is preserved at the time of generating the reduced image according to the related art.
  • FIG. 5 is a diagram illustrating an example where the waveform of an image is not preserved at the time of generating the reduced image according to the related art.
  • FIG. 6 is a diagram illustrating an outline of the processing of an image processor according to an embodiment of the present invention.
  • FIG. 7 is a diagram illustrating an outline of the processing of the image processor according to the embodiment of the present invention.
  • FIG. 8 is a diagram illustrating an exemplary configuration of the image processor according to the embodiment of the present invention.
  • FIG. 9 is a diagram illustrating the phases of the pixels used for generation of a reduced image.
  • FIG. 10 is a diagram illustrating changes in the phases of the pixels used for the generation of the reduced image.
  • FIG. 11 is a flowchart illustrating an image conversion process.
  • FIG. 12 is a diagram illustrating another exemplary configuration of an image processor according to an embodiment of the present invention.
  • FIG. 13 is a diagram illustrating a filtering process performed at the time of generating a reduced image.
  • FIG. 14 is a diagram illustrating a filtering process performed at the time of generating a reduced image.
  • FIG. 15 is a diagram illustrating a filtering process performed at the time of generating a reduced image.
  • FIG. 16 is a flowchart illustrating an image conversion process.
  • FIG. 17 is a diagram illustrating new errors generated at the time of generating an SR image.
  • FIG. 18 is a diagram illustrating the suppression of the generation of new errors at the time of generation of the SR image.
  • FIG. 19 is a diagram illustrating another exemplary configuration of an image processor of an embodiment of the present invention.
  • FIG. 20 is a flowchart illustrating an image conversion process.
  • FIG. 21 is a diagram illustrating another exemplary configuration of an image processor according to an embodiment of the present invention.
  • FIG. 22 is a diagram illustrating the phases of the pixels used for generation of a reduced image.
  • FIG. 23 is a diagram illustrating changes in the phases of the pixels used for the generation of the reduced image.
  • FIG. 24 is a flowchart illustrating an image conversion process.
  • FIG. 25 is a diagram illustrating another exemplary configuration of an image processor of an embodiment of the present invention.
  • FIG. 26 is a diagram illustrating a 2-dimensional filtering process.
  • FIG. 27 is a flowchart illustrating an image conversion process.
  • FIG. 28 is a diagram illustrating a shift in the phases of each pixel of a combined image.
  • FIG. 29 is a diagram illustrating another exemplary configuration of an image processor according to an embodiment of the present invention.
  • FIG. 30 is a flowchart illustrating an image conversion process.
  • FIG. 31 is a diagram illustrating an exemplary configuration of a computer.
  • a time-consecutive image (hereinafter referred to as an LR image), such as a moving image, is input.
  • the LR image is converted to an image (hereinafter referred to as an SR image) with a higher resolution than the LR image by a super-resolution process and output.
  • the SR image is an enlarged image of the LR image.
  • the LR image may not be a moving image and may be a plurality of images which are time-consecutive and contain the same subject captured at different times.
  • the image processor predicts an SR image of the current frame using an LR image of the current frame (time) being processed and an SR image obtained by the LR image of a previous frame (time) earlier than the current frame and outputs an image obtained by prediction as a combined image. Then, the image processor down-samples the combined image to obtain a reduced image, up-samples a differential image obtained by calculating a difference between the reduced image and the LR image, and adds the up-sampled differential image to the combined image, thus obtaining an SR image of the current frame.
  • the image processor when the reduced image is generated, the pixels at different phases of the combined image are selected for each frame to be used for the generation of the reduced image, thus suppressing accumulation of noise in the SR image.
  • the vertical direction represents the pixel values of pixels of an image
  • the horizontal direction represents a predetermined direction of the image.
  • each circle represents each pixel on the image.
  • a waveform indicated by an arrow A 51 represents the waveform of a combined image of the current frame. That is, a curve that connects adjacent pixels (i.e., the pixel values thereof) of the combined image forms the waveform of the combined image.
  • the combined image is composed of pixels which are repeatedly arranged in the horizontal direction in the figure in the order of a pixel having the intermediate value, a pixel having the maximum value, a pixel having the intermediate value, and a pixel having the minimum value. That is to say, in the example of FIG. 6 , the waveform of the combined image has a sinusoidal shape.
  • the pixels of the combined image are sampled every other pixel in the horizontal direction so as to generate a reduced image.
  • the pixels of the combined image indicated by the arrow A 51 are sampled every other pixel in the rightward direction by starting with the third pixel from the left in the figure, whereby the pixels of the combined image indicated by an arrow A 52 are sampled and a reduced image is generated.
  • the waveform of the obtained reduced image has a flat shape without any change in the horizontal direction of the figure, and the original waveform of the combined image is not properly preserved.
  • a differential image having a waveform indicated by an arrow A 54 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a shape such that it is flat in the horizontal direction of the figure, and errors generated in the combined image are not detected in the differential image.
  • the differential image is up-sampled and added to the combined image as a correction value of the errors of the combined image relative to the LR image
  • the errors i.e., noise components are not removed from the SR image obtained by addition.
  • the direction in each image corresponding to the rightward direction of the figure will be also referred to as an x-direction.
  • the SR image obtained thus is used for generation of a combined image of a subsequent frame.
  • a combined image of a subsequent frame is generated using the SR image obtained from the differential image indicated by the arrow A 54 , whereby a combined image having a waveform indicated by an arrow A 55 is obtained.
  • the combined image indicated by the arrow A 55 has the same waveform as the waveform of the combined image indicated by the arrow A 51 .
  • a reduced image which is composed of the pixels at a predetermined phase of the combined image as indicated by an arrow A 56 is generated.
  • the dotted circles of the waveform of the combined image indicated by the arrow A 56 represent the positions of the pixels which are sampled at the time of generating the reduced image in the previous frame.
  • the dotted circles represent the pixels at the same phase as the pixels of the combined image indicated by the arrow A 51 , which are used for generating the reduced image.
  • the left adjacent pixels in the figure of the pixels of the combined image used at the time of generating the reduced image of the previous frame are used for generation of a reduced image of the current frame being processed.
  • the reduced image composed of such pixels is generated, and the waveform of the reduced image has the same shape as the waveform of the combined image indicated by the arrow A 55 , namely, a sinusoidal shape.
  • the original waveform of the combined image is properly preserved.
  • a differential image having a waveform indicated by an arrow A 58 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a shape such that the waveform of the combined image indicated by the arrow A 55 is reversed in the vertical direction of the figure with respect to the central position in the pixel value direction (vertical direction) of the waveform.
  • the waveform of the differential image indicated by an arrow A 58 has a shape which is indicative of an error between the combined image and the LR image. Therefore, when the differential image is added to the combined image indicated by the arrow A 55 , an SR image having higher image quality is obtained in which errors (noise components) are removed.
  • the SR image is used for generating a combined image of a subsequent frame. Since noise components have been removed from the SR image, the combined image obtained from the SR image will be a combined image having a waveform similar to the original waveform of the SR image as indicated by an arrow A 59 , for example.
  • the waveform of the combined image indicated by the arrow A 59 has a flat shape without any change in the x-direction.
  • the pixels at the same phase as the pixels of the combined image of the frame indicated by the arrow A 52 which are used for generating the reduced image, are used and a reduced image composed of the pixels is generated.
  • the obtained reduced image will be an image having a waveform in which the original waveform of the combined image is properly preserved.
  • a differential image having a waveform indicated by an arrow A 62 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a flat shape without any change in the x-direction.
  • a combined image having a waveform having a shape such that it is flat in the x-direction is obtained in a subsequent frame as indicated by an arrow A 63 .
  • a reduced image is generated which is composed of the pixels at a predetermined phase of the combined image indicated by the arrow A 63 .
  • the dotted circles of the waveform of the combined image indicated by the arrow A 64 represent the positions of the pixels of the combined image of the frame indicated by the arrow A 52 , which are sampled at the time of generating the reduced image.
  • the right adjacent pixels in the figure of the pixels of the combined image used for generating the reduced image from the combined image of the frame indicated by the arrow A 52 are used for generation of a reduced image.
  • the reduced image composed of such pixels is generated, and the waveform of the reduced image has the same shape as the waveform of the combined image indicated by the arrow A 63 .
  • a differential image having a waveform indicated by an arrow A 66 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a flat shape without any change in the x-direction.
  • each frame time
  • the phases of the pixels used for generation of the reduced image are changed for each frame (time)
  • the noise in the combined image is temporarily amplified as illustrated in FIG. 7 .
  • the vertical direction represents the pixel values of pixels of an image
  • the rightward direction represents the x-direction.
  • each circle represents each pixel on the image.
  • a waveform indicated by an arrow A 81 represents the waveform of a combined image of the current frame.
  • the waveform of the combined image indicated by the arrow A 81 has the same shape as the waveform of the combined image indicated by the arrow A 51 of FIG. 6 .
  • the pixels of the combined image indicated by the arrow A 81 are sampled every other pixel by starting with the third pixel from the left in the figure so as to generate a reduced image in the x-direction indicated by an arrow A 82 .
  • the waveform of the obtained reduced image has a flat shape without any change in the x-direction, and the original waveform of the combined image is not properly preserved.
  • a differential image having a waveform indicated by an arrow A 84 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a shape such that it is flat in the x-direction, and errors generated in the combined image are not detected in the differential image. Therefore, the errors, i.e., noise components are not removed from the SR image obtained by addition of the differential image and the combined image.
  • the SR image obtained thus is used for generation of a combined image of a subsequent frame.
  • a combined image of a subsequent frame is generated using the SR image obtained from the differential image indicated by the arrow A 84 , whereby a combined image having a waveform indicated by an arrow A 85 is obtained.
  • the combined image indicated by the arrow A 85 has the same waveform as the waveform of the combined image indicated by the arrow A 81 .
  • a reduced image which is composed of the pixels at a predetermined phase of the combined image as indicated by an arrow A 86 is generated.
  • the dotted circles of the waveform of the combined image indicated by the arrow A 86 represent the positions of the pixels which are sampled at the time of generating the reduced image in the previous frame.
  • the dotted circles represent the pixels at the same phase as the pixels of the combined image indicated by the arrow A 81 , which are used for generating the reduced image.
  • the left adjacent pixels in the figure of the pixels of the combined image used at the time of generating the reduced image of the previous frame are used for generation of a reduced image of a current frame being processed.
  • the reduced image composed of such pixels is generated, and the waveform of the reduced image has substantially the same shape as the waveform of the combined image indicated by the arrow A 85 , namely, a sinusoidal shape.
  • the subject on the combined image is shifted leftward in the figure by one pixel from the center of the reduced image.
  • a differential image having a waveform indicated by an arrow A 88 is obtained.
  • the waveform of the differential image has the same shape as the waveform of the combined image indicated by the arrow A 85 .
  • an SR image is obtained in which errors (noise components) are amplified.
  • a combined image of a subsequent frame is generated using such an SR image, a combined image having a waveform as indicated by an arrow A 89 is obtained in which the noise components of the combined image indicated by the arrow A 85 are amplified.
  • a reduced image is generated which is composed of the pixels at the same phase as the pixels of the combined image, which are used for generation of the reduced image in the frame indicated by the arrow A 82 .
  • the obtained reduced image will be an image having a waveform in which the original waveform of the combined image is not properly preserved.
  • a differential image having a waveform indicated by an arrow A 92 is obtained.
  • the waveform of the differential image has a shape such that it is flat in the x-direction.
  • the differential image is added to a combined image indicated by an arrow A 89 to obtain an SR image; however, the SR image will be an image in which the amplified noise components are accumulated as they are.
  • a combined image having a waveform having a sinusoidal shape as indicated by an arrow A 93 is obtained in a subsequent frame.
  • a reduced image is generated which is composed of the pixels at a predetermined phase of the combined image indicated by the arrow A 93 .
  • the dotted circles of the waveform of the combined image indicated by the arrow A 94 represent the positions of the pixels of the combined image of the frame indicated by the arrow A 82 , which are sampled at the time of generating the reduced image.
  • the left adjacent pixels in the figure of the pixels of the combined image used for generating the reduced image from the combined image of the frame indicated by the arrow A 82 are used for generation of a reduced image in the current frame being processed.
  • the reduced image composed of such pixels is generated, and the waveform of the reduced image has the same shape as the waveform of the combined image indicated by the arrow A 93 .
  • a differential image having a waveform indicated by an arrow A 96 is obtained by calculating a difference between the reduced image and the LR image.
  • the waveform of the differential image has a shape such that the waveform of the combined image indicated by the arrow A 93 is reversed in the vertical direction of the figure with respect to the central position in the pixel value direction of the waveform.
  • the waveform of the differential image indicated by an arrow A 96 has a shape which is indicative of an error between the combined image and the LR image. Therefore, when this differential image is added to the combined image indicated by the arrow A 93 , an SR image having higher image quality is obtained in which errors (noise components) are removed as indicated by an arrow A 97 . As seen from the waveform of the SR image indicated by the arrow A 97 , the waveform has the same shape as the waveform of the LR image indicated by the arrow A 95 , and the noise components which have been accumulated without being detected are removed.
  • FIG. 8 is a diagram illustrating an exemplary configuration of an image processor according to an embodiment of the present invention.
  • the image processor 61 includes an up-sampler 71 , a motion vector detector 72 , a motion compensator 73 , a mask generator 74 , a mixer 75 , a phase controller 76 , down-samplers 77 - 1 to 77 - 3 , a phase controller 78 , down-samplers 79 - 1 to 79 - 3 , a subtractor 80 , an up-sampler 81 , an adder 82 , and a buffer 83 .
  • the image processor 61 receives an LR image of one frame to be processed, and the input LR image is supplied to the up-sampler 71 and the subtractor 80 .
  • the up-sampler 71 up-samples the supplied LR image to an image with the same resolution as an SR image, which will be generated from now on, and supplies the up-sampled image to the motion vector detector 72 , the mask generator 74 , and the mixer 75 .
  • the motion vector detector 72 calculates a motion vector of the whole SR image from the LR image supplied from the up-sampler 71 and an SR image of a previous frame earlier than the current frame being processed, supplied from the buffer 83 , and supplies the motion vector to the motion compensator 73 .
  • the motion compensator 73 performs motion compensation using the motion vector supplied from the motion vector detector 72 and an SR image supplied from the buffer 83 to generate a prediction image. Specifically, the motion compensator 73 moves the whole SR image by a distance the same as the magnitude of the motion vector in a direction indicated by the motion vector, thus generating an image obtained thus as the prediction image.
  • the prediction image is an image obtained by predicting an SR image of the current frame by motion compensation using the SR image of the previous frame to the current frame.
  • the prediction image is supplied from the motion compensator 73 to the mask generator 74 and the mixer 75 .
  • the mask generator 74 generates a motion mask using the LR image supplied from the up-sampler 71 and the prediction image supplied from the motion compensator 73 and supplies the motion mask to the mixer 75 .
  • the motion mask is information used for specifying a region of the LR image in which a moving subject is displayed, and which is generated by calculating a difference between the LR image and the prediction image.
  • the mixer 75 performs weighted addition of the prediction image supplied from the motion compensator 73 and the LR image supplied from the up-sampler 71 using the motion mask supplied from the mask generator 74 and supplies a combined image obtained thus to the phase controller 76 and the adder 82 .
  • the phase controller 76 selects a phase of the combined image to be used for generating the reduced image and changes an output destination of the combined image supplied from the mixer 75 in accordance with the selected phase. That is to say, the phase controller 76 supplies the supplied combined image to any one of the down-samplers 77 - 1 to 77 - 3 .
  • Each of the down-samplers 77 - 1 to 77 - 3 samples the pixels at a predetermined phase of the combined image supplied from the phase controller 76 and generates an image composed of such pixels, thus down-sampling the combined image in the x-direction.
  • Each of the down-samplers 77 - 1 to 77 - 3 supplies the generated image to the phase controller 78 .
  • a down-sampler 77 when it is not necessary to distinguish particularly between the down-samplers 77 - 1 to 77 - 3 , they will be simply referred to as a down-sampler 77 .
  • the phase controller 78 selects a phase of the combined image to be used for generating the reduced image and changes an output destination of the image supplied from the down-sampler 77 in accordance with the selected phase. That is to say, the phase controller 78 supplies the supplied image to any one of the down-samplers 79 - 1 to 79 - 3 .
  • Each of the down-samplers 79 - 1 to 79 - 3 samples the pixels at a predetermined phase of the image supplied from the phase controller 78 and generates a reduced image composed of such pixels. In this way, the combined image is down-sampled in a direction (hereinafter referred to as a y-direction) perpendicular to the x-direction, and thus, a reduced image with the same resolution as the LR image is obtained.
  • Each of the down-samplers 79 - 1 to 79 - 3 supplies the generated reduced image to the subtractor 80 .
  • a down-sampler 79 when it is not necessary to distinguish particularly between the down-samplers 79 - 1 to 79 - 3 , they will be simply referred to as a down-sampler 79 .
  • the subtractor 80 subtracts the reduced image supplied from the down-sampler 79 from the LR image supplied thereto to calculate a difference between the LR image and the reduced image and supplies a differential image obtained thus to the up-sampler 81 .
  • the up-sampler 81 up-samples the differential image supplied from the subtractor 80 to an image with the same resolution as the SR image and supplies an enlarged image obtained thus to the adder 82 .
  • the adder 82 adds the combined image supplied from the mixer 75 and the enlarged image supplied from the up-sampler 81 to generate an SR image and supplies the SR image to a subsequent stage while supplying the SR image to the buffer 83 to be stored therein.
  • the buffer 83 stores the SR image supplied from the adder 82 only for one frame period and supplies the SR image stored therein to the motion vector detector 72 and the motion compensator 73 when a subsequent frame is processed.
  • the down-sampler 77 and the down-sampler 79 are configured to down-sample the combined image by generating an image which is composed of the pixels at a predetermined phase (position) of the combined image.
  • the down-samplers 77 and 79 sample pixels at different phases based on a predetermined reference phase.
  • the rightward direction represents the x-direction
  • the downward direction represents the y-direction.
  • each circle represents each pixel on the combined image.
  • the down-sampler 77 - 1 samples a left adjacent pixel G 12 of a reference pixel G 11 which serves as a reference pixel on the combined image, as depicted on the top left of the figure.
  • the reference pixel G 11 will be assumed as being a plurality of pixels which are arranged at predetermined intervals on the combined image. For example, when the combined image is down-sampled in the x-direction to half its pixel count, a predetermined number of pixels on the combined image, which are arranged every other pixel in the x-direction, will be assumed as being the reference pixel G 11 .
  • the down-sampler 77 - 2 samples the reference pixel G 11 on the combined image, as depicted on the top center of the figure.
  • the down-sampler 77 - 3 samples a right adjacent pixel G 13 of the reference pixel G 11 on the combined image, as depicted on the top right of the figure.
  • the down-sampler 79 - 1 samples an upper adjacent pixel G 22 of a reference pixel G 21 which serves as a reference pixel on the combined image which is down-sampled in the x-direction, as depicted on the bottom left of the figure.
  • the reference pixel G 21 will be assumed as being a plurality of pixels which are arranged at predetermined intervals on the combined image which is down-sampled in the x-direction. For example, when the combined image is down-sampled in the y-direction to half its pixel count, a predetermined number of pixels on the combined image, which are arranged every other pixel in the y-direction, will be assumed as being the reference pixel G 21 .
  • the down-sampler 79 - 2 samples the reference pixel G 21 on the combined image, as depicted on the bottom center of the figure.
  • the down-sampler 79 - 3 samples a lower adjacent pixel G 23 of the reference pixel G 21 on the combined image, as depicted on the bottom right of the figure.
  • Each of the phase controllers 76 and 78 selects for each frame as to which one of the down-samplers 77 and 79 will be used for down-sampling the combined image.
  • each of the phase controllers 76 and 78 selects the output destination of the combined image so that the phases of the pixels to be sampled are changed in a predetermined pattern.
  • the rightward direction represents time
  • the number above each phase controller 76 represents a number that identifies the frame of the combined image to be processed. Specifically, the frames are processed in the order from the 0th frame indicated by “0” to the 7th frame indicated by “7”.
  • the bold line extending from the phase controller 76 to the down-sampler 79 represents a path along which the combined image is supplied.
  • the combined image is supplied from the phase controller 76 to the down-sampler 77 - 2 , and the combined image supplied from the down-sampler 77 - 2 to the phase controller 78 is then supplied to the down-sampler 79 - 2 .
  • the combined image is supplied from the phase controller 76 to the down-sampler 77 - 1 , and the combined image supplied from the down-sampler 77 - 1 to the phase controller 78 is then supplied to the down-sampler 79 - 2 .
  • the combined image is supplied to the down-sampler 77 - 2 and the down-sampler 79 - 2 . That is to say, the reference pixel G 11 of the combined image is sampled by the down-sampler 77 - 2 , and the reference pixel G 21 of the combined image is sampled by the down-sampler 79 - 2 , whereby a reduced image is generated.
  • the combined image is supplied to the down-sampler 77 - 2 and the down-sampler 79 - 1 .
  • the combined image is supplied to the down-sampler 77 - 2 and the down-sampler 79 - 2 .
  • the combined image is supplied to the down-sampler 77 - 3 and the down-sampler 79 - 2 .
  • the combined image is supplied to the down-sampler 77 - 2 and the down-sampler 79 - 2 .
  • the combined image is supplied to the down-sampler 77 - 2 and the down-sampler 79 - 3 .
  • the same pattern as the 0th to 7th frames is repeated, whereby the combined image is supplied to each down-sampler 77 and each down-sampler 79 . Therefore, in the (1+8i)th frame (where i is a natural number), the combined image is supplied to the down-sampler 77 - 1 and the down-sampler 79 - 2 .
  • the amount of shifting in the phase of the pixels of the combined image used for generation of the reduced image is increased, the amount of a positional shift between the position of the subject in the reduced image and the position of the subject in the combined image will increase, whereby new errors (noise components) different from the errors generated at the time of generating the combined image will be generated in the SR image. Therefore, it is preferable that the amount of shifting in the phase of the pixels of the combined image is set as small as possible.
  • the phases of the pixels of the combined image to be used for generating the reduced image may be changed in a predetermined pattern and may be randomly selected for each frame.
  • the image processor 61 When an LR image is supplied to the image processor 61 , the image processor 61 starts an image conversion process which is a process of converting the LR image of each frame to an SR image by a super-resolution process.
  • the image conversion process by the image processor 61 will be described.
  • the up-sampler 71 up-samples an LR image supplied thereto to an image with the same resolution as an SR image, which will be generated from now on, and supplies the up-sampled image to the motion vector detector 72 , the mask generator 74 , and the mixer 75 .
  • the motion compensator 73 performs motion compensation to generate a prediction image.
  • the motion vector detector 72 calculates one motion vector of the whole SR image from the LR image supplied from the up-sampler 71 and an SR image supplied from the buffer 83 and supplies the motion vector to the motion compensator 73 .
  • the motion compensator 73 moves the whole SR image by a distance the same as the magnitude of the motion vector in a direction indicated by the motion vector supplied from the motion vector detector 72 to generate a prediction image and supplies the generated prediction image to the mask generator 74 and the mixer 75 .
  • the mask generator 74 generates a motion mask using the LR image supplied from the up-sampler 71 and the prediction image supplied from the motion compensator 73 and supplies the motion mask to the mixer 75 .
  • the mask generator 74 sequentially uses each pixel of a motion mask, which will be obtained from now on, as a target pixel, and calculates a difference between the pixel value of a pixel of the LR image and the pixel value of a pixel of the prediction image, the pixels being located at the same position as the target pixel.
  • the mask generator 74 uses the pixel value of the target pixel to a value which is determined with respect to the absolute value of the calculated difference. For example, the pixel value of the target pixel has a larger value as the absolute value of the difference decreases.
  • a region of the motion mask where the pixel value of a pixel is small can be said to be a region of the LR image where the subject shows a motion different from the motion of the whole LR image; that is, a region where the subject is moving relative to the whole LR image.
  • the mixer 75 performs weighted addition of the prediction image supplied from the motion compensator 73 and the LR image supplied from the up-sampler 71 using the motion mask supplied from the mask generator 74 to generate a combined image and supplies the combined image to the phase controller 76 and the adder 82 .
  • the mixer 75 uses, as the pixel value of the target pixel, a value obtained by the addition of the pixel value of the pixel of the LR image at the same position as the target pixel, multiplied by the weighting factor Wi, and the pixel value of the pixel of the prediction image at the same position as the target pixel, multiplied by the weighting factor Wj. In this way, the mixer 75 uses each pixel of the combined image as the target pixel and calculates the pixel values of such pixels, thus generating a combined image.
  • the weighting factor Wi is set to be larger as the pixel value of the pixel of the motion mask decreases. This is to increase the contribution ratio of the LR image to the generation of the combined image in the region where the pixel value of the pixel of the motion mask is small, namely a region where the accuracy of the prediction based on the motion compensation is low, thus preventing image quality deterioration of the combined image.
  • the phase controller 76 selects the x-directional phase of the combined image to be used for generating a reduced image. For example, when the x-directional phase is changed in the pattern described with reference to FIG. 10 , the phase controller 76 selects the phase based on the frame number in FIG. 10 of the current frame being processed.
  • the phase controller 76 supplies the combined image supplied from the mixer 75 to the down-sampler 77 corresponding to the selected phase and causes x-directional down-sampling to be performed. For example, when the current frame corresponds to the 5th frame in FIG. 10 , the combined image is supplied to the down-sampler 77 - 3 .
  • the down-sampler 77 performs x-directional down-sampling on the combined image supplied from the phase controller 76 and supplies an image obtained thus to the phase controller 78 . Specifically, the down-sampler 77 samples the pixels at a predetermined phase of the supplied combined image to generate an image composed of the sampled pixels.
  • the phase controller 78 selects the y-directional phase of the combined image to be used for generating a reduced image. For example, when the y-directional phase is changed in the pattern described with reference to FIG. 10 , the phase controller 78 selects the phase based on the frame number in FIG. 10 of the current frame being processed.
  • the phase controller 78 supplies the combined image supplied from the down-sampler 77 to the down-sampler 79 corresponding to the selected phase and causes y-directional down-sampling to be performed.
  • the down-sampler 79 performs y-directional down-sampling on the combined image supplied from the phase controller 78 and supplies an image obtained thus to the subtractor 80 . Specifically, the down-sampler 79 samples the pixels at a predetermined phase of the supplied combined image to generate a reduced image composed of the sampled pixels.
  • the subtractor 80 calculates a difference between the LR image supplied thereto and the reduced image supplied from the down-sampler 79 to generate a differential image.
  • the pixel values of the pixels of the differential image are used as the values of the difference between the pixel values of the pixels of the LR image and the pixel values of the pixels of the reduced image at the same positions as the pixels.
  • the pixel values of the pixels of the differential image generated thus represent the difference between the reduced image obtained by predicting the LR image (SR image) and the real LR image. Therefore, when the combined image is corrected by the amount of the difference, an SR image will be obtained in which the LR image is more faithfully enlarged; that is to say, an SR image will be obtained in which the waveform of the LR image is preserved as it is.
  • the subtractor 80 supplies the generated differential image to the up-sampler 81 .
  • the up-sampler 81 up-samples the differential image supplied from the subtractor 80 to an image with the same resolution as the SR image and supplies an enlarged image obtained thus to the adder 82 .
  • the adder 82 adds the combined image supplied from the mixer 75 and the enlarged image supplied from the up-sampler 81 to generate an SR image of the current frame and supplies the SR image to a subsequent stage while supplying the SR image to the buffer 83 to be stored therein.
  • the image processor 61 determines whether or not the process should be terminated. For example, the process is determined to be terminated when the supply of the LR image to the image processor 61 stops and a process termination command is received.
  • step S 22 When it is determined at step S 22 that the process should not be terminated, the process returns to step S 11 and the above-described processes are repeated. That is to say, the LR image of a subsequent frame is used as the LR image of the current frame being processed and converted to an SR image by a super-resolution process.
  • each part of the image processor 61 terminates its pending processes, and the image conversion process ends.
  • the image processor 61 performs a super-resolution process on the input LR image and converts the LR image to the SR image.
  • the image processor 61 changes the phases of the pixels to be used for generating the reduced image from frame to frame.
  • the phases of the pixels to be used for generating the reduced image are shifted slightly for each frame in accordance with a predetermined pattern, it is not necessary to specify the phase of the pixels of the combined image so that the original waveform of the combined image is preserved in the reduced image. Therefore, it is possible to obtain the SR image more quickly.
  • the reduced image is generated by sampling the pixels of the combined image
  • the reduced image may be generated by performing a filtering process on the combined image.
  • the image processor may be configured as illustrated in FIG. 12 , for example.
  • the same or corresponding portions as those in FIG. 8 will be denoted by the same reference numerals, and description thereof will be appropriately omitted.
  • the image processor 121 of FIG. 12 includes a phase controller 131 , filtering processors 132 - 1 to 132 - 3 , a phase controller 133 , and filtering processors 134 - 1 to 134 - 3 , in lieu of the phase controller 76 , the down-samplers 77 - 1 to 77 - 3 , the phase controller 78 , and the down-samplers 79 - 1 to 79 - 3 of the image processor 61 .
  • the phase controller 131 selects the phase of the combined image to be used for generating the reduced image and changes an output destination of the combined image supplied from the mixer 75 in accordance with the selected phase. That is to say, the phase controller 131 supplies the supplied combined image to any one of the filtering processors 132 - 1 to 132 - 3 .
  • Each of the filtering processors 132 - 1 to 132 - 3 calculates the pixels at a predetermined phase of the combined image supplied from the phase controller 131 by a filtering process and generates an image composed of the calculated pixels, thus down-sampling the combined image in the x-direction.
  • Each of the filtering processors 132 - 1 to 132 - 3 supplies the generated image to the phase controller 133 .
  • a filtering processor 132 when it is not necessary to distinguish particularly between the filtering processors 132 - 1 to 132 - 3 , they will be simply referred to as a filtering processor 132 .
  • the phase controller 133 selects a phase of the combined image to be used for generating the reduced image and changes an output destination of the combined image supplied from the filtering processor 132 in accordance with the selected phase. That is to say, the phase controller 133 supplies the supplied combined image to any one of the filtering processors 134 - 1 to 134 - 3 .
  • Each of the filtering processors 134 - 1 to 134 - 3 calculates the pixels at a predetermined phase of the combined image supplied from the phase controller 133 by a filtering process and generates a reduced image composed of the calculated pixels, thus down-sampling the combined image in the y-direction.
  • Each of the filtering processors 134 - 1 to 134 - 3 supplies the generated reduced image to the subtractor 80 .
  • a filtering processor 134 when it is not necessary to distinguish particularly between the filtering processors 134 - 1 to 134 - 3 , they will be simply referred to as a filtering processor 134 .
  • the filtering processor 132 or 134 the pixel values of the pixels are calculated by a filtering process. Therefore, the phase of the combined image selected by the phase controller 131 or 133 is not necessarily the position of the pixel on the combined image, but the phase may be shifted with an accuracy of a pixel or lower.
  • the filtering processor 132 - 2 calculates pixels at the phase of the reference pixel of the combined image
  • the filtering processors 132 - 1 and 132 - 3 calculate pixels at a phase shifted by 1 ⁇ 2 pixel from the reference pixel of the combined image. More specifically, although no pixel exists at positions of the combined image separated by a distance of a pixel of smaller from the reference pixel, it is assumed that pixels exist at such positions, and the pixel values of the pixels are calculated.
  • the filtering processor 132 - 2 calculates pixels at the phase of the reference pixel by a 3-tap filtering process as illustrated in FIG. 13 .
  • the rightward direction represents the x-direction.
  • the filtering processor 132 - 2 multiplies three coefficients W 1 to W 3 as indicated by an arrow Q 11 with the pixel values of three pixels around the reference pixel arranged in the x-direction on the combined image, respectively, thus calculating the sum of the pixel values.
  • the coefficient W 2 represents a coefficient multiplied with the reference pixel
  • the coefficients W 1 and W 3 represent coefficients multiplied with the adjacent pixels of the reference pixel, respectively.
  • the vertical direction represents the magnitude of each coefficient, and the coefficient has a larger value as it is located at the higher position. Specifically, the coefficients W 1 and W 3 have the same value, and the coefficients W 1 and W 3 have the smaller value than the coefficient W 2 .
  • the filtering processor 132 - 2 multiplies the coefficients W 1 to W 3 with the pixel values of the three pixels arranged in the x-direction to calculate a pixel value as indicated by an arrow Q 12 .
  • each circle represents each pixel.
  • the upper line of circles arranged in the horizontal direction (x-direction) in the figure represents a combined image
  • the lower line of circles arranged in the horizontal direction in the figure represents an image obtained by performing x-directional down-sampling.
  • a pixel G 42 positioned at the center of the pixels G 41 to G 43 on the combined image, which are arranged adjacent to each other in the x-direction is a reference pixel
  • the pixel value of a pixel P 11 is calculated using the pixels G 41 to G 43 .
  • This pixel P 11 is a pixel of the combined image which will be calculated from now on, the combined image being down-sampled in the x-direction, and the pixel P 11 is positioned at the same phase as the pixel G 42 which is the reference pixel.
  • the filtering processor 132 - 2 multiplies the coefficients W 1 to W 3 , respectively, with the pixel values of the pixels G 41 to G 43 , calculates the sum of the pixel values multiplied with the coefficients W 1 to W 3 , and uses a value obtained thus as the pixel value of the pixel P 11 .
  • Each of the filtering processors 132 - 1 and 132 - 3 calculates the pixel at a phase shifted by 1 ⁇ 2 phase from the reference pixel by a 4-tap filtering process as illustrated in FIG. 14 .
  • the rightward direction represents the x-direction.
  • Each of the filtering processors 132 - 1 and 132 - 3 multiplies four coefficients W 11 to W 14 as indicated by an arrow Q 21 with the pixel values of four pixels including the reference pixel arranged in the x-direction on the combined image, respectively, thus calculating the sum of the pixel values.
  • the vertical direction represents the magnitude of each coefficient, and the coefficient has a larger value as it is located at the higher position.
  • the coefficients W 12 and W 13 which are multiplied with adjacent pixels have the same value.
  • the coefficients W 11 and W 14 which are multiplied with pixels located so as to surround the pixels multiplied with the coefficients W 12 and W 13 have the same value, and the coefficients W 11 and W 14 have the smaller value than the coefficients W 12 and W 13 .
  • the filtering processor 132 - 1 multiplies the coefficients W 11 to W 14 with the pixel values of the four pixels arranged in the x-direction to calculate a pixel value as indicated by an arrow Q 22 .
  • each circle represents each pixel.
  • the upper line of circles arranged in the horizontal direction (x-direction) in the figure represents a combined image
  • the lower line of circles arranged in the horizontal direction in the figure represents an image obtained by performing x-directional down-sampling.
  • the pixel value of a pixel P 21 is calculated using the pixels G 51 to G 54 .
  • This pixel P 21 is a pixel of the combined image which will be calculated from now on, the combined image being down-sampled in the x-direction, and the pixel P 21 is positioned at a position (phase) separated leftward (in the opposite direction of the x-direction) in the figure by a distance of 1 ⁇ 2 pixel from the pixel G 53 which is the reference pixel.
  • the filtering processor 132 - 1 multiplies the coefficients W 11 to W 14 , respectively, with the pixel values of the pixels G 51 to G 54 , calculates the sum of the pixel values multiplied with the coefficients W 11 to W 14 , and uses a value obtained thus as the pixel value of the pixel P 21 .
  • the filtering processor 132 - 3 multiplies the coefficients W 11 to W 14 with the pixel values of the four pixels arranged in the x-direction to calculate a pixel value as indicated by an arrow Q 23 .
  • the pixel value of a pixel P 22 is calculated using the pixels G 52 to G 55 .
  • This pixel P 22 is a pixel of the combined image which will be calculated from now on, the combined image being down-sampled in the x-direction, and the pixel P 22 is positioned at a position (phase) separated rightward (in the x-direction) in the figure by a distance of 1 ⁇ 2 pixel from the pixel G 53 which is the reference pixel.
  • the filtering processor 132 - 3 multiplies the coefficients W 11 to W 14 , respectively, with the pixel values of the pixels G 52 to G 55 , calculates the sum of the pixel values multiplied with the coefficients W 11 to W 14 , and uses a value obtained thus as the pixel value of the pixel P 22 .
  • the filtering processor 132 performs the filtering process by using different pixels depending on whether it calculates a pixel at a phase shifted leftward or rightward from the reference pixel.
  • a pixel at a desired phase is calculated using the reference pixel, two left adjacent pixels of the reference pixel, and one right adjacent pixel of the reference pixel in the figure.
  • a pixel at a desired phase is calculated using the reference pixel, one left adjacent pixel of the reference pixel, and two right adjacent pixels of the reference pixel in the figure.
  • the filtering processor 134 calculates a pixel at a desired phase using pixels arranged consecutively in the y-direction.
  • the filtering processor 134 - 2 calculates the pixel value of a pixel at the same phase as the reference pixel of the combined image by a filtering process.
  • the filtering processor 134 - 1 calculates the pixel value of a pixel at a phase (position) separated by a predetermined distance in the opposite direction of the y-direction from the reference pixel of the combined image by a filtering process.
  • the filtering processor 134 - 3 calculates the pixel value of a pixel at a phase (position) separated by a predetermined distance in the y-direction from the reference pixel of the combined image by a filtering process.
  • the phase of the pixel calculated by the filtering processor 132 - 1 or 132 - 3 is not limited to a position shifted by 1 ⁇ 2 pixel from the reference pixel but may be any position.
  • a pixel at a position shifted by 1 ⁇ 4 pixel from the reference pixel may be generated.
  • the rightward direction represents the x-direction.
  • each pixel of the combined image is first shifted rightward in the figure by 1 ⁇ 4 pixel as indicated by an arrow Q 31 , and then, a filtering process is performed.
  • each circle in the figure represents one pixel.
  • the upper line of circles represents a combined image
  • the lower line of circles represents an image obtained by shifting the phase of each pixel of the combined image rightward in the figure by 1 ⁇ 4 pixel.
  • the pixel values of the three pixels of the phase-shifted combined image which are arranged in the x-direction are multiplied with coefficients W 21 to W 23 indicated by an arrow Q 32 , and the sum of the pixel values multiplied with the coefficients is calculated.
  • the value of the calculated sum is used as the pixel value of the pixel at the position shifted rightward in the figure by 1 ⁇ 4 pixel from the reference pixel.
  • the vertical direction in the figure represents the magnitude of each coefficient, and the coefficient has a larger value as it is located at the higher position.
  • the coefficient W 22 represents a coefficient multiplied with the phase-shifted reference pixel
  • the coefficients W 21 and W 23 represent coefficients multiplied with the horizontally adjacent pixels of the phase-shifted reference pixel.
  • the pixel values of four pixels which include the reference pixel of the combined image and are arranged in the x-direction, are multiplied with coefficients W 31 to W 34 , and the sum of the pixel values multiplied with the coefficients is used as the pixel value of a pixel which is to be calculated. That is to say, the sum is used as the pixel value of the pixel at a position shifted by 1 ⁇ 4 pixel in the x-direction from the reference pixel.
  • the coefficients have different values which can be arranged in the descending order of W 32 , W 33 , W 31 , and W 34 .
  • the values of the coefficients W 31 to W 34 may be changed.
  • the filtering processors 132 and 134 it is possible to calculate the pixel value of a pixel at a phase separated by a certain distance in a certain direction from the reference pixel of the combined image by a filtering process without shifting the phase of each pixel of the combined image.
  • steps S 51 to S 54 are the same as the processes of steps S 11 to S 14 in FIG. 11 , and description thereof will be omitted.
  • the phase controller 131 selects the x-directional phase of the combined image to be used for generating a reduced image. For example, when the x-directional phase is changed in a predetermined pattern for each frame, the phase controller 131 selects a phase specified by the pattern and the current frame being processed.
  • the phase controller 131 supplies the combined image supplied from the mixer 75 to the filtering processor 132 corresponding to the selected phase and causes x-directional down-sampling to be performed.
  • the filtering processor 132 performs x-directional down-sampling on the combined image supplied from the phase controller 131 by a filtering process and supplies an image obtained thus to the phase controller 133 .
  • the filtering processor 132 multiplies the pixel value of three or four pixels, which include the reference pixel on the combined image and are arranged consecutively in the x-direction, with predetermined coefficients and calculates the sum of the pixel values multiplied with the coefficients. Then, the filtering processor 132 uses the value of the calculated sum as the value of the pixel value of the pixel at the selected phase of the combined image and generates an image composed of the pixels at the selected phase of the combined image, thus down-sampling the combined image.
  • the phase controller 133 selects the y-directional phase of the combined image to be used for generating a reduced image. For example, when the y-directional phase is changed in a predetermined pattern for each frame, the phase controller 133 selects a phase specified by the pattern and the current frame being processed.
  • the phase controller 133 supplies the combined image supplied from the filtering processor 132 to the filtering processor 134 corresponding to the selected phase and causes y-directional down-sampling to be performed.
  • the filtering processor 134 performs y-directional down-sampling on the combined image supplied from the phase controller 133 by a filtering process and supplies an image obtained thus to the subtractor 80 .
  • the filtering processor 134 multiplies the pixel value of three or four pixels, which include the reference pixel on the combined image and are arranged consecutively in the y-direction, with predetermined coefficients and calculates the sum of the pixel values multiplied with the coefficients. Then, the filtering processor 134 uses the value of the calculated sum as the value of the pixel value of the pixel at the selected phase of the combined image and generates a reduced image composed of the pixels at the selected phase of the combined image, thus down-sampling the combined image.
  • the image processor 121 performs a super-resolution process on the input LR image and converts the LR image to the SR image.
  • the image processor 121 when generating the reduced image from the combined image obtained by prediction by the filtering process, the image processor 121 generates the reduced image so that the reduced image is composed of pixels at different phases of the combined image for each frame.
  • the reduced image is generated by the filtering process, it is not only possible to shift the phases of the pixels of the combined image used for generating the reduced image with an accuracy of a pixel or lower, but also to decrease further the amount of a phase shift in the subject between the reduced image and the combined image. In this way, it is possible to decrease further the errors caused by the phase shift in the subject generated at the time of generating the SR image. That is to say, it is possible to detect more accurately the errors between the combined image and the LR image and improve the image quality of the SR image. Moreover, it is possible to obtain the reduced image more simply and quickly by the filtering process.
  • the phases of the pixels to be used for generating the reduced image are changed from frame to frame, it is possible to prevent accumulation of errors (noise components) generated at the time of generating the combined image, whereby the image quality of the SR image can be improved.
  • the vertical direction represents the pixel values of pixels of an image
  • the rightward direction represents the x-direction in the image.
  • each circle in the figure represents one pixel on the image.
  • a waveform indicated by an arrow A 121 represents the waveform of a combined image of the current frame. That is, a curve that connects adjacent pixels of the combined image indicated by the arrow A 121 forms the waveform of the combined image.
  • the pixel values of the pixels located slightly close to the right side from the center of the combined image are larger than the pixel values of other pixels and are protruding upward in the figure.
  • the pixels of the combined image are sampled every other pixel in the x-direction in the figure so as to generate a reduced image.
  • the sampling positions of the pixels used for generating the reduced image are shifted by 2 pixels rightward in the figure from a reference pixel
  • the position of a subject in the reduced image will be shifted by 2 pixels leftward in the figure from the center of the image as indicated by an arrow A 122 , when compared with the position of the subject on the combined image.
  • a differential image having a waveform indicated by an arrow A 124 is obtained by calculating a difference between the reduced image and the LR image.
  • an enlarged image having a waveform indicated by an arrow A 125 is obtained.
  • the waveform of the enlarged image has a shape such that the pixel values of the pixels located slightly close to the left side from the center of the enlarged image are smaller than the pixel values of other pixels and are protruding downward in the figure.
  • a SR image having a waveform indicated by an arrow A 126 is obtained.
  • some errors remain unremoved from the combined image and new errors generated by the addition of the enlarged image are included.
  • the errors (noise components) included in the combined image are not removed in a portion of the SR image which is located slightly close to the right side from the center of the SR image, but remain as they are. That is to say, in an ideal case, the waveform of the SR image should have a shape such that it is flat in the x-direction like the LR image indicated by the arrow A 123 . However, the portion of the SR image which in the figure is located slightly close to the right side from the center protrude upward in the figure like the combined image indicated by the arrow A 121 .
  • the vertical direction represents the pixel values of pixels of an image
  • the rightward direction represents the x-direction in the image.
  • each circle represents each pixel on the image.
  • arrows A 131 to A 135 represent the waveforms of a combined image, a reduced image, an LR image, a differential image, and an enlarged image, respectively, and these waveforms are the same as the case indicated by the arrows A 121 to A 125 in FIG. 17 , and description thereof will be omitted.
  • a reduced image having a waveform indicated by the arrow A 132 is obtained from a combined image having a waveform indicated by the arrow A 131 .
  • a difference between the reduced image and an LR image having a waveform indicated by the arrow A 133 is calculated, whereby a differential image having a waveform indicated by the arrow A 134 is obtained.
  • the differential image is up-sampled, an enlarged image having a waveform indicated by the arrow A 135 is obtained.
  • the pixels of the enlarged image are shifted rightward in the figure, whereby a final enlarged image is obtained.
  • the waveform of the SR image indicated by the arrow A 137 has a shape such that it is the same shape as the waveform of the LR image indicated by the arrow A 133 and it is flat in the x-direction.
  • the image processor adjusting the phases of the pixels of the enlarged image is configured as illustrated in FIG. 19 , for example.
  • FIG. 19 the same or corresponding portions as those in FIG. 8 will be denoted by the same reference numerals, and description thereof will be appropriately omitted.
  • the image processor 161 of FIG. 19 further includes a phase controller 171 in the image processor 61 of FIG. 8 .
  • the phase controller 171 shifts the phase of each pixel of the enlarged image supplied from the up-sampler 81 in the direction of shifting the phase by the down-sampler 77 or 79 by the same shift distance and supplies an image obtained thus to the adder 82 .
  • steps S 91 to S 100 are the same as the processes of steps S 11 to S 20 of FIG. 11 , and description thereof will be omitted.
  • step S 101 the phase controller 171 shifts the phase of each pixel of the enlarged image supplied from the up-sampler 81 and supplies an image obtained thus to the adder 82 .
  • the down-sampler 77 - 3 when the reduced image is generated, the down-sampler 77 - 3 generates an image composed of pixels on the combined image which are located at positions separated by 2 pixels in the x-direction from the reference pixel. Moreover, it will be assumed that the down-sampler 79 - 3 generates a reduced image composed of pixels on the image generated by the down-sampler 77 - 3 , which are located at positions separated by 2 pixels in the y-direction from the reference pixel.
  • the phase controller 171 moves each pixel of the enlarged image by a distance of 2 pixels in the x and y-directions and uses an image obtained thus as a final enlarged image. That is to say, the pixel values of the pixels of the final enlarged image obtained after the phase adjustment will have the pixel values of pixels which are located at positions separated by 2 pixels in the opposite direction of the x-direction and by 2 pixels in the opposite direction of the y-direction from the pixels of the enlarged image before the phase adjustment, which are located at the same position.
  • step S 102 the adder 82 adds the enlarged image supplied from the phase controller 171 to the combined image supplied from the mixer 75 to generate a SR image. Subsequently, the process of step S 103 is performed, and the image conversion process ends. Since the process of step S 103 is the same as the process of step S 22 of FIG. 11 , description thereof will be omitted.
  • the image processor 161 shifts the phase of the enlarged image obtained by up-sampling the differential image and then adds the phase-shifted enlarged image to the combined image, thus generating a SR image.
  • the phase of the enlarged image is first shifted so as to cancel the phase shift generated at the time of generating the reduced image and the phase-shifted enlarged image is added to the combined image, it is not only possible to remove more certainly the errors included in the combined image but also to suppress generation of new errors, whereby the image quality of the SR image can be improved.
  • an LR image of a top field and an LR image of a bottom field are alternately input to the image processor.
  • an LR image of one frame is obtained from two LR images which are consecutive in time, that is, an LR image of a top field and an LR image of a bottom field which are consecutive.
  • phase change pattern of the pixels on the combined image which are sampled at the time of generating the reduced image there may occur a case where the phase of the pixels being sampled is biased in the top field or the bottom field.
  • the reference pixel when the phases of the pixels of the combined image which are sampled every other field are shifted from the position of the reference pixel, the reference pixel will be sampled in any one of the top field and the bottom field which appear every other field. In such a case, there is a concern that errors (noise components) are not detected at a certain phase of the combined image but are accumulated.
  • a phase change pattern for the top field and a phase change pattern for the bottom field may be prepared independently. In this way, it is possible to suppress accumulation of errors in the SR image of a particular kind of field, namely the top field or the bottom field.
  • the image processor may be configured as illustrated in FIG. 21 , for example.
  • the same or corresponding portions as those in FIG. 8 will be denoted by the same reference numerals, and description thereof will be appropriately omitted.
  • the image processor 201 of FIG. 21 includes a switch 211 , a phase controller 212 , down-samplers 213 - 1 to 213 - 3 , a phase controller 214 , down-samplers 215 - 1 to 215 - 3 , a phase controller 216 , down-samplers 217 - 1 to 217 - 3 , a phase controller 218 , and down-samplers 219 - 1 to 219 - 3 , in lieu of the phase controller 76 , the down-samplers 77 - 1 to 77 - 3 , the phase controller 78 , and the down-samplers 79 - 1 to 79 - 3 of the image processor 61 .
  • the switch 211 changes the output destination of the combined image supplied from the mixer 75 in accordance with the kind of the field (frame) of the LR image to be processed. That is to say, when the current field being processed is a top field, the switch 211 supplies the supplied combined image to the phase controller 212 . When the current field is a bottom field, the switch 212 supplies the supplied combined image to the phase controller 216 .
  • the phase controller 212 selects the phase of the combined image to be used for generating the reduced image and changes the output destination of the combined image supplied from the switch 211 in accordance with the selected phase. That is to say, the phase controller 212 supplies the supplied combined image to any one of the down-samplers 213 - 1 to 213 - 3 .
  • Each of the down-samplers 213 - 1 to 213 - 3 samples the pixels at a predetermined phase of the combined image supplied from the phase controller 212 and generates an image composed of such pixels, thus down-sampling the combined image in the x-direction.
  • Each of the down-samplers 213 - 1 to 213 - 3 supplies the generated image to the phase controller 214 .
  • a down-sampler 213 when it is not necessary to distinguish particularly between the down-samplers 213 - 1 to 213 - 3 , they will be simply referred to as a down-sampler 213 .
  • the phase controller 214 selects a phase of the combined image to be used for generating the reduced image and changes an output destination of the image supplied from the down-sampler 213 in accordance with the selected phase. That is to say, the phase controller 214 supplies the supplied image to any one of the down-samplers 215 - 1 to 215 - 3 .
  • Each of the down-samplers 215 - 1 to 215 - 3 samples the pixels at a predetermined phase of the image supplied from the phase controller 214 and generates a reduced image composed of such pixels. In this way, the combined image is down-sampled in the y-direction.
  • Each of the down-samplers 215 - 1 to 215 - 3 supplies the generated reduced image to the subtractor 80 .
  • a down-sampler 215 when it is not necessary to distinguish particularly between the down-samplers 215 - 1 to 215 - 3 , they will be simply referred to as a down-sampler 215 .
  • the phase controller 216 selects the phase of the combined image to be used for generating the reduced image and changes an output destination of the image supplied from the switch 211 in accordance with the selected phase. That is to say, the phase controller 216 supplies the supplied combined image to any one of the down-samplers 217 - 1 to 217 - 3 .
  • Each of the down-samplers 217 - 1 to 217 - 3 samples the pixels at a predetermined phase of the combined image supplied from the phase controller 216 and generates an image composed of such pixels, thus down-sampling the combined image in the x-direction.
  • Each of the down-samplers 217 - 1 to 217 - 3 supplies the generated image to the phase controller 218 .
  • a down-sampler 217 when it is not necessary to distinguish particularly between the down-samplers 217 - 1 to 217 - 3 , they will be simply referred to as a down-sampler 217 .
  • the phase controller 218 selects a phase of the combined image to be used for generating the reduced image and changes an output destination of the image supplied from the down-sampler 217 in accordance with the selected phase. That is to say, the phase controller 218 supplies the supplied image to any one of the down-samplers 219 - 1 to 219 - 3 .
  • Each of the down-samplers 219 - 1 to 219 - 3 samples the pixels at a predetermined phase of the image supplied from the phase controller 218 and generates a reduced image composed of such pixels. In this way, the combined image is down-sampled in the y-direction.
  • Each of the down-samplers 219 - 1 to 219 - 3 supplies the generated reduced image to the subtractor 80 .
  • a down-sampler 219 when it is not necessary to distinguish particularly between the down-samplers 219 - 1 to 219 - 3 , they will be simply referred to as a down-sampler 219 .
  • the buffer 83 may be configured to store the SR image of the previous two fields so that an SR image which is two fields earlier than the current field being processed is supplied to the motion vector detector 72 and the motion compensator 73 by the buffer 83 .
  • a prediction image is generated from the LR image of the top field and the SR image or from the LR image of the bottom field and the SR image.
  • the down-samplers 213 , 215 , 217 , and 219 are configured to down-sample the combined image by generating an image composed of pixels at a predetermined phase (position) of the combined image.
  • the down-samplers 213 and 215 sample pixels at different phases based on a predetermined reference pixel.
  • the rightward direction represents the x-direction
  • the downward direction represents the y-direction.
  • each circle represents each pixel on the combined image.
  • the down-sampler 213 - 1 samples a left adjacent pixel G 82 of a reference pixel G 81 which serves as a reference pixel on the combined image, as depicted on the top left of the figure.
  • the down-sampler 213 - 2 samples the reference pixel G 81 on the combined image, as depicted on the top center of the figure.
  • the down-sampler 213 - 3 samples a right adjacent pixel G 83 of the reference pixel G 81 on the combined image, as depicted on the top right of the figure.
  • the down-sampler 215 - 1 samples an upper adjacent pixel G 92 of a reference pixel G 91 which serves as a reference pixel on the combined image which is down-sampled in the x-direction, as depicted on the bottom left of the figure.
  • the down-sampler 215 - 2 samples the reference pixel G 91 on the combined image, as depicted on the bottom center of the figure.
  • the down-sampler 215 - 3 samples a lower adjacent pixel G 93 of the reference pixel G 91 on the combined image, as depicted on the bottom right of the figure.
  • the down-samplers 217 - 1 to 217 - 3 are configured to down-sample the combined image by sampling the same pixels as the pixels which are sampled from the combined image by the down-samplers 213 - 1 to 213 - 3 .
  • the down-samplers 219 - 1 to 219 - 3 are configured to down-sample the combined image by sampling the same pixels as the pixels which are sampled from the combined image by the down-samplers 215 - 1 to 215 - 3 .
  • each of the phase controllers 212 , 214 , 216 , and 218 selects the output destination of the combined image so that the phases of the pixels to be sampled are changed in a predetermined pattern.
  • the rightward direction represents time
  • the number above each phase controller 212 or 216 represents a number that identifies the field (frame) of the combined image to be processed.
  • the frames are processed in the order from the 0th frame indicated by “0” to the 7th frame indicated by “7”.
  • the characters “T” or “B” to the right of the field number stands for the top field and the bottom field, respectively, representing the kind of the field.
  • the bold line extending from the phase controller 212 to the down-sampler 215 and the bold line extending from the phase controller 216 to the down-sampler 219 represent paths along which the combined images are supplied.
  • the combined image of the 0th field is supplied to the phase controller 212 . Subsequently, the combined image is supplied from the phase controller 212 to the down-sampler 213 - 2 , and the combined image supplied from the down-sampler 213 - 2 to the phase controller 214 is then supplied to the down-sampler 215 - 2 .
  • the combined image of the 1st field is supplied from the switch 211 to the phase controller 216 Subsequently, the combined image of the 1st field is supplied from the phase controller 216 to the down-sampler 217 - 2 , and the combined image supplied from the down-sampler 217 - 2 to the phase controller 218 is then supplied to the down-sampler 219 - 2 .
  • the 2nd field is the top field
  • the combined image of the 2nd field is supplied to the down-samplers 213 - 1 and 215 - 2 .
  • the 3rd field is the bottom field, the combined image of the 3rd field is supplied to the down-samplers 217 - 2 and 219 - 1 .
  • the 4th field is the top field
  • the combined image of the 4th field is supplied to the down-samplers 213 - 2 and 215 - 2 .
  • the 5th field is the bottom field, the combined image of the 5th field is supplied to the down-samplers 217 - 2 and 219 - 2 .
  • the combined image of the 6th field is supplied to the down-samplers 213 - 2 and 215 - 3 . Since the 7th field is the bottom field, the combined image of the 7th field is supplied to the down-samplers 217 - 3 and 219 - 2 .
  • the output destination of the combined image is changed in accordance with a predetermined pattern for each kind of the field depending on whether a field being processed is the top field or the bottom field.
  • a reduced image composed of the reference pixels of the combined image is generated every other field (frame). That is to say, the phases of the pixels sampled from the combined image are shifted from the reference pixel every other field. Moreover, on the bottom fields, a reduced image composed of the reference pixels of the combined image is generated every other field (frame).
  • steps S 131 to S 134 are the same as the processes of steps S 11 to S 14 in FIG. 11 , and description thereof will be omitted.
  • the switch 211 changes the output destination of the combined image supplied from the mixer 75 depending on whether a field being processed is the top field or the bottom field. Specifically, when the field being processed is the top field, the switch 211 supplies the combined image to the phase controller 212 . When the field being processed is the bottom field, the switch 211 supplies the combined image to the phase controller 216 .
  • the phase controller 212 or 216 selects the x-directional phase of the combined image to be used for generating the reduced image in accordance with a predetermined pattern, for example, the pattern described with reference to FIG. 23 .
  • the phase controller 212 or 216 supplies the combined image supplied from the switch 211 to the down-sampler 213 or 217 corresponding to the selected phase and causes x-directional down-sampling to be performed.
  • the down-sampler 213 or 217 performs x-directional down-sampling on the combined image supplied from the phase controller 212 or 216 and supplies an image obtained thus to the phase controller 214 or 218 .
  • the phase controller 214 or 218 selects the y-directional phase of the combined image to be used for generating the reduced image in accordance with a predetermined pattern, for example, the pattern described with reference to FIG. 23 .
  • the phase controller 214 or 218 supplies the combined image supplied from the down-sampler 213 or 217 to the down-sampler 215 or 219 corresponding to the selected phase and causes y-directional down-sampling to be performed.
  • the down-sampler 215 or 219 performs y-directional down-sampling on the combined image supplied from the phase controller 214 or 218 and supplies an image obtained thus to the subtractor 80 .
  • the down-sampler 215 or 219 performs y-directional down-sampling on the combined image supplied from the phase controller 214 or 218 and supplies an image obtained thus to the subtractor 80 .
  • the down-sampler 215 or 219 performs y-directional down-sampling on the combined image supplied from the phase controller 214 or 218 and supplies an image obtained thus to the subtractor 80 .
  • steps S 140 to S 143 are performed, and the image conversion process ends. Since the processes of steps S 140 to S 143 are the same as the processes of steps S 19 to S 22 of FIG. 11 , description thereof will be omitted.
  • the image processor 201 changes the output destination of the combined image in accordance with the kind of a field to be processed and changes the phases of the pixels to be used for generating the reduced image in accordance with a pattern prepared for each kind of the field.
  • the output destination of the combined image is changed depending on the kind of the field, and the phases of the pixels to be used for generating the reduced image are changed in accordance with a pattern for each kind of the field. Therefore, it is possible to prevent the phases of the pixels used for generating the reduced image from being biased to a certain phase in a certain kind of field. Accordingly, it is possible to suppress accumulation of errors in the SR image of a certain kind of field and improve the image quality of the SR image.
  • the down-samplers 213 , 215 , 217 , and 219 down-sample the combined image by sampling certain pixels
  • the combined image may be down-sampled by a filtering process.
  • the combined image is first subjected to the x-directional down-sampling and then to the y-directional down-sampling, the x-directional down-sampling and the y-directional down-sampling may be performed simultaneously.
  • the image processor may be configured as illustrated in FIG. 25 , for example.
  • the same or corresponding portions as those in FIG. 8 will be denoted by the same reference numerals, and description thereof will be appropriately omitted.
  • the image processor 251 of FIG. 25 includes a phase controller 261 and filtering processors 262 - 1 to 262 - 5 , in lieu of the phase controller 76 , the down-samplers 77 - 1 to 77 - 3 , the phase controller 78 , and the down-samplers 79 - 1 to 79 - 3 of the image processor 61 .
  • the phase controller 261 selects the phase of the combined image to be used for generating the reduced image and changes an output destination of the combined image supplied from the mixer 75 in accordance with the selected phase. That is to say, the phase controller 261 supplies the supplied combined image to any one of the filtering processors 262 - 1 to 262 - 5 .
  • Each of the filtering processors 262 - 1 to 262 - 5 uses the combined image supplied from the phase controller 261 to calculate the pixels at a predetermined phase of the combined image supplied from the phase controller 261 by a 2-dimensional filtering process and generates a reduced image composed of the calculated pixels. By this filtering process, the combined image is down-sampled in the x and y-directions.
  • Each of the filtering processors 262 - 1 to 262 - 5 supplies the generated reduced image to the subtractor 80 .
  • a filtering processor 262 when it is not necessary to distinguish particularly between the filtering processors 262 - 1 to 262 - 5 , they will be simply referred to as a filtering processor 262 .
  • the filtering process of the filtering processor 262 uses a plurality of pixels on the combined image which are arranged in the x and y-directions as illustrated in FIG. 26 , for example.
  • the rightward direction in FIG. 26 represents the x-direction.
  • the image processor 121 described with reference to FIG. 13 , it has been described that when the pixels at the phase of the reference pixel are calculated by the filtering process, the pixels which are arranged in the x-direction (or the y-direction) are multiplied respectively with coefficients corresponding to the distance from the reference pixel, as indicated by an arrow Q 61 in FIG. 26 .
  • the image processor 121 it has been described that when pixels at a phase separated by a distance of a pixel or smaller from the reference pixel are calculated by the filtering process, the pixels which are arranged in the x-direction (or the y-direction) are multiplied respectively with coefficients having different values for each pixel, as indicated by an arrow Q 62 .
  • each circle represents the coefficient multiplied with one pixel
  • the vertical direction represents the magnitude of the coefficients. That is to say, the coefficients indicated by the arrows Q 61 and Q 62 are the same as the coefficients indicated by the arrow Q 11 in FIG. 13 and the arrow Q 33 in FIG. 15 , respectively.
  • the pixels which are arranged in the x and y-directions are multiplied with coefficients as indicated by arrows Q 63 and Q 64 , whereby the pixel values of the pixels of the reduced image are calculated.
  • each shadowed circle represents one coefficient multiplied with the pixel on the combined image, and circles representing such coefficients are arranged with the same positional relationship as the pixels on the combined image.
  • the rightward direction and the downward direction in the left figure of the drawing represent the x-direction and the y-direction, respectively.
  • the rightward direction and the frontward direction in the right figure of the drawing represent the x-direction and the y-direction, respectively, and the vertical direction represents the magnitude of the coefficients.
  • the example indicated by the arrow Q 63 represents the coefficients used for calculating the pixels at the phase of the reference pixel of the combined image, and the three pixels by three pixels, which are arranged in the x and y-directions, respectively, are multiplied with the coefficients, whereby pixel values are calculated.
  • the coefficient at the center represents the coefficient multiplied with the reference pixel of the combined image and has the largest value among the 9 coefficients.
  • the coefficients have values corresponding to the distance of the pixels being multiplied from the reference pixel, and the coefficients multiplied with the pixels have the larger values as the pixels are located closer to the reference pixel.
  • the example indicated by the arrow Q 64 represents the coefficients used for calculating the pixels at phases separated by a distance of a pixel or smaller from the reference pixel of the combined image, and the four pixels by four pixels, which are arranged in the x and y-directions, respectively, are multiplied with the coefficients, whereby pixel values are calculated.
  • the central position of the 16 coefficients in total is the phase of the pixels of a reduced image to be calculated, and the pixel multiplied with the coefficient which is located the closest to the phase of the pixels of the reduced image is the reference pixel of the combined image.
  • the coefficients multiplied with the pixels of the combined image have the larger values as the pixels are located closer to the phase of the pixels of the reduced image which is to be calculated.
  • the filtering processor 262 multiplies a plurality of pixels, which are located near the reference pixel and arranged in the x and y-direction, with coefficients, whereby the pixel value of each pixel of the reduced image is calculated by a 2-dimensional filtering process. In this way, a reduced image composed of pixels at phases which are separated by a distance of a pixel or smaller from the reference pixel in the x and y-direction can be obtained more simply and quickly by one filtering process.
  • the filtering processor 262 - 1 generates a reduced image composed of pixels at phases which are shifted by 1 ⁇ 2 pixel in the opposite direction of the x-direction from the reference pixel of the combined image.
  • the filtering processor 262 - 2 generates a reduced image composed of pixels at phases which are shifted by 1 ⁇ 2 pixel in the x-direction from the reference pixel of the combined image.
  • the filtering processor 262 - 3 generates a reduced image composed of pixels at the phase of the reference pixel of the combined image.
  • the filtering processor 262 - 4 generates a reduced image composed of pixels at phases which are shifted by 1 ⁇ 2 pixel in the y-direction from the reference pixel of the combined image.
  • the filtering processor 262 - 5 generates a reduced image composed of pixels at phases which are shifted by 1 ⁇ 2 pixel in the opposite direction of the y-direction from the reference pixel of the combined image.
  • steps S 171 to S 174 are the same as the processes of steps S 11 to S 14 in FIG. 11 , and description thereof will be omitted.
  • the phase controller 261 selects the phase of the combined image to be used for generating a reduced image. For example, when the phase is changed in at least one of the x and y-directions in a predetermined pattern, the phase controller 261 selects a phase which is determined for the current frame being processed in accordance with the pattern.
  • the phase controller 261 supplies the combined image supplied from the mixer 75 to the filtering processor 262 corresponding to the selected phase and causes a reduced image to be generated.
  • the filtering processor 262 performs a 2-dimensional filtering process on the combined image supplied from the phase controller 261 to generate a reduced image. Specifically, the filtering processor 262 multiplies the pixel values of several pixels around the reference pixel of the combined image supplied from the phase controller 261 with coefficients stored in advance and calculates the sum of the pixel values multiplied with the coefficients, thus calculating the pixel value of each pixel of the reduced image.
  • the filtering processor 262 supplies the generated reduced image to the subtractor 80 . Subsequently, the processes of steps S 177 to S 180 are performed, and the image conversion process ends. Since such processes are the same as the processes of steps S 19 to S 22 of FIG. 11 , description thereof will be omitted.
  • the image processor 251 performs a super-resolution process on the input LR image and converts the LR image to the SR image.
  • the image processor 251 performs the 2-dimensional filtering process on the combined image to generate the reduced image while changing the phases of the pixels of the combined image to be used for generating the reduced image from frame to frame.
  • the combined image is generated by the 2-dimensional filtering process, it is not only possible to shift the phases with an accuracy of a pixel or lower, but also to decrease further the amount of a phase shift in the subject between the reduced image and the combined image. In this way, it is possible to decrease further the errors generated at the time of generating the SR image and improve the image quality of the SR image. Moreover, it is possible to obtain the reduced image more simply and quickly by the 2-dimensional filtering process.
  • the phases of the pixels of the combined image to be used for generating the reduced image are shifted for each frame by a distance of smaller than 2 pixels, such as 1 pixel, 1 ⁇ 2 pixel, or 1 ⁇ 4 pixel.
  • the phases of the pixels of the combined image used for generating the reduced image may be shifted for each frame by a distance in units of pixels. For example, when a reduced image having pixels corresponding to 1 ⁇ 8 of the pixel count of the combined image is generated, the phases may be shifted by a distance of smaller than 8 pixels. Therefore, when the phases are shifted by a distance in units of pixels such as 4 pixels, 2 pixels, or 1 pixel, the reduced image can be generated by simpler processing.
  • each circle represents each pixel on the combined image.
  • the phase of each pixel can be shifted by moving each pixel of the combined image in the x-direction by 2 pixels as indicated by an arrow C 11 .
  • the upper line of plural circles represents the combined image before the phases are shifted
  • the lower line of plural circles represents the combined image after the phases are shifted.
  • the rightward direction represents the x-direction.
  • the phase of each pixel of the combined image is shifted by a distance of 2 pixels in the x-direction. Therefore, by generating an image composed of pixels of the phase-shifted combined image which are located at the same positions as the reference pixel of the combined image before the phase shift, it is possible to obtain an image composed of pixels which are located at positions separated by 2 pixels in the opposite direction of the x-direction from the reference pixel of the combined image.
  • the phase of each pixel can be shifted by moving each pixel of the combined image in the y-direction by 2 pixels as indicated by an arrow C 12 .
  • the left line of plural circles represents the combined image before the phases are shifted
  • the right line of plural circles represents the combined image after the phases are shifted.
  • the downward direction represents the y-direction.
  • the phase of each pixel of the combined image is shifted by a distance of 2 pixels in the opposite direction of the y-direction. Therefore, by generating an image composed of pixels of the phase-shifted combined image which are located at the same positions as the reference pixel of the combined image before the phase shift, it is possible to obtain an image composed of pixels which are located at positions separated by 2 pixels in the opposite direction of the y-direction from the reference pixel of the combined image.
  • the reduced image when a reduced image composed of pixels at phases shifted by a distance of a predetermined number of pixels from the reference pixel of the combined image is generated, the reduced image can be obtained more simply by shifting first the phase of each pixel of the combined image in accordance with the phase and sampling the pixels at the same positions as the original reference pixel.
  • the image processor may be configured as illustrated in FIG. 29 .
  • FIG. 29 the same or corresponding portions as those in FIG. 8 will be denoted by the same reference numerals, and description thereof will be appropriately omitted.
  • the image processor 291 of FIG. 29 includes a phase controller 301 , a down-sampler 302 , a phase controller 303 , and a down-sampler 304 , in lieu of the phase controller 76 , the down-samplers 77 - 1 to 77 - 3 , the phase controller 78 , and the down-samplers 79 - 1 to 79 - 3 of the image processor 61 .
  • the phase controller 301 selects the phase of the combined image to be used for generating the reduced image and shifts appropriately the phase of each pixel of the combined image supplied from the mixer 75 in the x-direction (more specifically, the direction parallel to the x-direction) in accordance with the selected phase.
  • the phase controller 301 supplies the phase-shifted combined image to the down-sampler 302 as necessary.
  • the down-sampler 302 samples the pixels at a predetermined phase of the combined image supplied from the phase controller 301 , that is, the pixels at the same positions as the reference pixel of the original combined image, thus generating an image composed of such pixels. In this way, the combined image is down-sampled in the x-direction.
  • the down-sampler 302 supplies the image (combined image) generated by the down-sampling to the phase controller 303 .
  • the phase controller 303 selects the phase of the image to be used for generating the reduced image and shifts appropriately the phase of each pixel of the image supplied from the down-sampler 302 in the y-direction (more specifically, the direction parallel to the y-direction) in accordance with the selected phase.
  • the phase controller 303 supplies the phase-shifted image to the down-sampler 304 as necessary.
  • the down-sampler 304 samples the pixels at a predetermined phase of the combined image supplied from the phase controller 303 , that is, the pixels at the same positions as the reference pixel of the original image before the phase shift, thus generating a reduced image composed of such pixels. In this way, the combined image is down-sampled in the y-direction.
  • the down-sampler 304 supplies the reduced image generated by the down-sampling to the subtractor 80 .
  • steps S 211 to S 214 are the same as the processes of steps S 11 to S 14 in FIG. 11 , and description thereof will be omitted.
  • the phase controller 301 selects the phase of the combined image to be used for generating the reduced image in accordance with a predetermined pattern and shifts appropriately the phase of each pixel of the combined image supplied from the mixer 75 in the x-direction in accordance with the selected phase.
  • the phase controller 301 when the selected phase is the same as the phase of the reference pixel, the phase controller 301 does not shift the phase of each pixel of the combined image but supplies the combined image to the down-sampler 302 as it is. For example, when the selected phase is different from the phase of the reference pixel, the phase controller 301 shifts the phase of each pixel by moving each pixel of the combined image in accordance with the selected phase and supplies the phase-shifted combined image to the down-sampler 302 .
  • the down-sampler 302 samples the pixels at the same position as the reference pixel before the phase shift, of the combined image supplied from the phase controller 301 , to generate an image composed of such pixels, whereby the combined image is down-sampled in the x-direction.
  • the down-sampler 302 supplies the image obtained by the down-sampling to the phase controller 303 .
  • the phase controller 303 selects the phase of the combined image to be used for generating the reduced image in accordance with a predetermined pattern and shifts appropriately the phase of each pixel of the combined image supplied from the down-sampler 302 in the y-direction in accordance with the selected phase.
  • the phase controller 303 when the selected phase is the same as the phase of the reference pixel, the phase controller 303 does not shift the phase of each pixel of the combined image but supplies the combined image to the down-sampler 304 as it is. For example, when the selected phase is different from the phase of the reference pixel, the phase controller 303 shifts the phase of each pixel by moving each pixel of the combined image in accordance with the selected phase and supplies the phase-shifted combined image to the down-sampler 304 .
  • the down-sampler 304 samples the pixels at the same position as the reference pixel before the phase shift, of the combined image supplied from the phase controller 303 , to generate an image composed of such pixels, whereby the combined image is down-sampled in the y-direction.
  • the down-sampler 304 supplies the image obtained by the down-sampling to the subtractor 80 .
  • steps S 219 to S 222 are performed, and the image conversion process ends. Since such processes are the same as the processes of steps S 19 to S 22 of FIG. 11 , description thereof will be omitted.
  • the image processor 291 performs a super-resolution process on the input LR image and converts the LR image to the SR image.
  • the image processor 291 changes the phases of the pixels to be used for generating the reduced image from frame to frame and shifts the phase of each pixel of the combined image in accordance with the phase, thus down-sampling the phase-shifted combined image and generating the reduced image.
  • the down-samplers 302 and 304 generate the reduced image by sampling the pixels at a predetermined position of the image
  • the reduced image may be generated by a filtering process using the pixels at the predetermined position.
  • the pixel values of the pixels at a predetermined position of the combined image having been subjected to the phase shift by the phase controller 301 are multiplied with predetermined coefficients, and the value of the sum of the pixel values multiplied with the coefficients is calculated. Then, the value of the calculated sum is used as the pixel value of the pixel at the selected phase, and an image composed of the pixels at the selected phase of the combined image is generated. The image generated thus is used as the combined image which is down-sampled in the x-direction.
  • the above-described series of processing may be executed by hardware or may be executed by software.
  • a program constituting the software is installed from a program recording medium in a computer integrated into an exclusive hardware or a general-personal computer which can execute various functions by installing various programs in the computer.
  • FIG. 31 is a block diagram illustrating an exemplary hardware configuration of a computer executing the above-described series of processing by a program.
  • a CPU central processing unit
  • a ROM read only memory
  • RAM random access memory
  • the bus 504 is connected to an input/output interface 505 .
  • the input/output interface 505 is connected to an input unit 506 including a keyboard, a mouse, and a microphone; an output unit 507 including a display and a speaker, a recording unit 508 including a hard disk and a nonvolatile memory, a communication unit 509 including a network interface, and a drive 510 driving a removable medium 511 such as a magnetic disc, an optical disc, an opto-magnetic disc, or a semiconductor memory.
  • the CPU 501 loads the program recorded in the recording unit 508 into the RAM 503 via the input/output interface 505 and the bus 504 and executes the program, whereby the above-described series of processing is executed.
  • the program executed by the computer may be provided by being recorded on the removable medium 511 which is a package medium including a magnetic disc (including a flexible disk), an optical disc (e.g., CD-ROM (compact disc-read only memory) and DVD (digital versatile disc)), an opto-magnetic disc, and a semiconductor memory.
  • the program may be provided via wired or wireless transmission media such as local area network, the Internet, digital satellite broadcasting.
  • the program is installed in the recording unit 508 via the input/output interface 505 when the removable medium 511 is mounted on the drive 510 . Furthermore, the program may be received by the communication unit 509 via wired or wireless transmission media and installed in the recording unit 508 . In addition, the program may be installed in advance in the ROM 502 or the recording unit 508 .
  • the program executed by the computer may be a program executing processing in a time-sequential manner in accordance with the procedures described in this specification and may be a program executing the processing in a parallel manner or at necessary times such as in response to calls.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Editing Of Facsimile Originals (AREA)
  • Studio Circuits (AREA)
  • Television Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
US12/691,651 2009-01-22 2010-01-21 Image processor, image processing method and program Abandoned US20100183232A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2009012350A JP5146335B2 (ja) 2009-01-22 2009-01-22 画像処理装置および方法、並びにプログラム
JPP2009-012350 2009-01-22

Publications (1)

Publication Number Publication Date
US20100183232A1 true US20100183232A1 (en) 2010-07-22

Family

ID=42336996

Family Applications (2)

Application Number Title Priority Date Filing Date
US12/691,651 Abandoned US20100183232A1 (en) 2009-01-22 2010-01-21 Image processor, image processing method and program
US12/691,652 Expired - Fee Related US8447138B2 (en) 2009-01-22 2010-01-21 Image processor, image processing method and program

Family Applications After (1)

Application Number Title Priority Date Filing Date
US12/691,652 Expired - Fee Related US8447138B2 (en) 2009-01-22 2010-01-21 Image processor, image processing method and program

Country Status (3)

Country Link
US (2) US20100183232A1 (enrdf_load_stackoverflow)
JP (1) JP5146335B2 (enrdf_load_stackoverflow)
CN (1) CN101882434B (enrdf_load_stackoverflow)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8805113B2 (en) 2011-11-11 2014-08-12 Mitsubishi Electric Corporation Image processing device and method and image display device
US20160217552A1 (en) * 2015-01-22 2016-07-28 Samsung Electronics Co., Ltd. Video super-resolution by fast video segmentation for boundary accuracy control

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5975791B2 (ja) * 2012-08-27 2016-08-23 三菱電機株式会社 画像処理装置及び方法、並びに画像表示装置及び方法
KR101467370B1 (ko) 2013-09-04 2014-12-02 부경대학교 산학협력단 자가 열화 복원을 이용한 영상 보간 방법
CN103501401B (zh) * 2013-10-01 2016-09-14 中国人民解放军国防科学技术大学 面向超大噪声基于预滤波的实时视频去噪方法
US9691133B1 (en) * 2013-12-16 2017-06-27 Pixelworks, Inc. Noise reduction with multi-frame super resolution
CN106165389B (zh) * 2014-03-18 2020-02-14 索尼公司 图像处理装置、图像处理方法以及程序
US10229478B2 (en) * 2014-09-26 2019-03-12 Samsung Electronics Co., Ltd. Image processing apparatus and image processing method
JP6377258B2 (ja) * 2015-04-16 2018-08-22 三菱電機株式会社 画像処理装置及び方法、並びにプログラム及び記録媒体
JP6719104B2 (ja) * 2015-08-28 2020-07-08 パナソニックＩｐマネジメント株式会社 画像出力装置、画像送信装置、画像受信装置、画像出力方法および記録媒体
CN106846241B (zh) * 2015-12-03 2020-06-02 阿里巴巴集团控股有限公司 一种图像融合的方法、装置和设备
CN107124534B (zh) * 2017-03-29 2020-11-27 歌尔股份有限公司 一种三轴云台、拍摄设备、拍摄系统及无人拍摄系统
CN107290700B (zh) * 2017-08-08 2020-12-04 上海联影医疗科技股份有限公司 一种相位校正方法、装置及磁共振系统
KR102326165B1 (ko) * 2017-08-23 2021-11-16 엘지디스플레이 주식회사 영상 처리 방법 및 이를 이용한 표시 장치
KR102508984B1 (ko) * 2017-09-22 2023-03-13 엘지이노텍 주식회사 어라운드뷰 영상 제어장치 및 그 어라운드뷰 영상 처리방법
JP7302596B2 (ja) 2018-05-15 2023-07-04 ソニーグループ株式会社 画像処理装置、および画像処理方法、並びにプログラム
JP2021061501A (ja) * 2019-10-04 2021-04-15 シャープ株式会社 動画像変換装置及び方法
CN112700368B (zh) * 2019-10-22 2025-05-30 华为技术有限公司 图像处理方法、装置及电子设备
CN111179883B (zh) * 2020-01-03 2022-06-03 云谷（固安）科技有限公司 图像显示方法和装置、移动终端、计算机设备、存储介质
CN111914764B (zh) * 2020-08-05 2023-09-15 杭州睿琪软件有限公司 图像处理方法、图像处理装置、电子设备和存储介质
US11671714B1 (en) * 2022-01-24 2023-06-06 Qualcomm Incorporated Motion based exposure control

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090238535A1 (en) * 2008-03-21 2009-09-24 Sony Corporation Merging video with time-decimated high-resolution imagery to form high-resolution video frames
US20100149338A1 (en) * 2008-12-16 2010-06-17 Mamigo Inc Method and apparatus for multi-user user-specific scene visualization

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3583831B2 (ja) * 1995-06-09 2004-11-04 株式会社リコー 信号処理装置
US7382937B2 (en) * 2003-03-07 2008-06-03 Hewlett-Packard Development Company, L.P. Method and apparatus for re-constructing high-resolution images
JP2006202168A (ja) * 2005-01-24 2006-08-03 Seiko Epson Corp 低解像度な複数の画像を用いた高解像度画像の生成
CN100413316C (zh) * 2006-02-14 2008-08-20 华为技术有限公司 一种视频图像超分辨率重构方法
JP4518043B2 (ja) * 2006-05-31 2010-08-04 株式会社日立製作所 画像信号処理装置、及び画像信号を高解像度化するための方法、及びそれを実行するためのプログラム
JP4993403B2 (ja) * 2006-06-02 2012-08-08 富士ゼロックス株式会社 画像処理装置、画像符号化装置、画像復号化装置、画像処理システム及びプログラム
JP4646146B2 (ja) * 2006-11-30 2011-03-09 ソニー株式会社 画像処理装置、画像処理方法、およびプログラム
JP4877090B2 (ja) * 2007-06-18 2012-02-15 ソニー株式会社 画像処理装置、画像処理方法、及びプログラム
JP4793339B2 (ja) * 2007-07-09 2011-10-12 ソニー株式会社 動画像変換装置、動画像復元装置、および方法、並びにコンピュータ・プログラム
JP5076755B2 (ja) * 2007-09-07 2012-11-21 ソニー株式会社 画像処理装置、および画像処理方法、並びにコンピュータ・プログラム
JP5306061B2 (ja) * 2009-06-01 2013-10-02 キヤノン株式会社 画像処理装置、画像処理方法、プログラム及び記憶媒体

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090238535A1 (en) * 2008-03-21 2009-09-24 Sony Corporation Merging video with time-decimated high-resolution imagery to form high-resolution video frames
US20100149338A1 (en) * 2008-12-16 2010-06-17 Mamigo Inc Method and apparatus for multi-user user-specific scene visualization

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8805113B2 (en) 2011-11-11 2014-08-12 Mitsubishi Electric Corporation Image processing device and method and image display device
US20160217552A1 (en) * 2015-01-22 2016-07-28 Samsung Electronics Co., Ltd. Video super-resolution by fast video segmentation for boundary accuracy control
US9652829B2 (en) * 2015-01-22 2017-05-16 Samsung Electronics Co., Ltd. Video super-resolution by fast video segmentation for boundary accuracy control

Also Published As

Publication number Publication date
JP2010170340A (ja) 2010-08-05
JP5146335B2 (ja) 2013-02-20
CN101882434A (zh) 2010-11-10
US8447138B2 (en) 2013-05-21
CN101882434B (zh) 2013-03-27
US20100183245A1 (en) 2010-07-22

Similar Documents

Publication Publication Date Title
US8447138B2 (en) Image processor, image processing method and program
US8913822B2 (en) Learning apparatus and method, image processing apparatus and method, program, and recording medium
US8254726B2 (en) Image processing apparatus, image processing method, and program
US7012649B2 (en) Image processing apparatus and method, program, and recording medium
US6987884B2 (en) Image processing device and method, and recorded medium
US9407931B2 (en) Motion vector estimator
KR101318586B1 (ko) 화상 처리 장치 및 방법과 기록 매체
US8264600B2 (en) Image processing apparatus for converting a lower resolution image into a higher resolution image using cyclic coefficients
JP2011504682A (ja) 画像シーケンスのサイズ変更
JP2010093672A (ja) 映像変換装置および方法、並びにプログラム
US8682110B2 (en) Method and apparatus for converting resolution of block based image
KR20070074781A (ko) 프레임율 변환장치
JP4216830B2 (ja) 映像処理装置及び方法並びにコンピュータが読出可能な記憶媒体
US20090185756A1 (en) Image processing device and image processing method
JP2012213136A (ja) 映像処理装置及び映像処理方法
JP5333791B2 (ja) 画像処理方法、画像処理装置及び画像処理プログラム
JP2005236937A (ja) 画像処理装置、画像処理方法および画像処理プログラム
JP4650683B2 (ja) 画像処理装置および方法、プログラム並びに記録媒体
JP2014027405A (ja) フレーム補間装置及びプログラム
JP2005217478A (ja) 画素信号処理方法及び装置
JP2006186504A (ja) 画像処理装置および方法、記録媒体、並びにプログラム
JP2011170488A (ja) 画像処理装置、画像処理方法、画像処理プログラム、及び、電子機器
KR101107255B1 (ko) 움직임이 보상된 보간 화소를 이용한 영상 보간 방법 및 그장치
JP4650684B2 (ja) 画像処理装置および方法、プログラム並びに記録媒体
JP4182574B2 (ja) 画像情報変換装置および変換方法

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION