WO2012132183A1 - 画像処理装置、画像処理方法、画像処理のためのコンピュータプログラム及び記録媒体 - Google Patents
画像処理装置、画像処理方法、画像処理のためのコンピュータプログラム及び記録媒体 Download PDFInfo
- Publication number
- WO2012132183A1 WO2012132183A1 PCT/JP2012/000604 JP2012000604W WO2012132183A1 WO 2012132183 A1 WO2012132183 A1 WO 2012132183A1 JP 2012000604 W JP2012000604 W JP 2012000604W WO 2012132183 A1 WO2012132183 A1 WO 2012132183A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- resolution image
- pixel
- image
- value
- low
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 98
- 238000004590 computer program Methods 0.000 title claims description 24
- 238000003672 processing method Methods 0.000 title claims description 6
- 238000012937 correction Methods 0.000 claims description 62
- 238000000034 method Methods 0.000 claims description 59
- 238000004364 calculation method Methods 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 23
- 230000006866 deterioration Effects 0.000 claims description 18
- 230000002093 peripheral effect Effects 0.000 claims description 16
- 230000015572 biosynthetic process Effects 0.000 claims description 11
- 238000003786 synthesis reaction Methods 0.000 claims description 11
- 230000007423 decrease Effects 0.000 claims description 5
- 238000006731 degradation reaction Methods 0.000 abstract description 33
- 230000015556 catabolic process Effects 0.000 abstract description 26
- 230000008859 change Effects 0.000 description 22
- 230000006870 function Effects 0.000 description 13
- 102100022002 CD59 glycoprotein Human genes 0.000 description 12
- 101000897400 Homo sapiens CD59 glycoprotein Proteins 0.000 description 12
- 101100129496 Arabidopsis thaliana CYP711A1 gene Proteins 0.000 description 8
- 101100083446 Danio rerio plekhh1 gene Proteins 0.000 description 8
- 101100382854 Arabidopsis thaliana CCD7 gene Proteins 0.000 description 6
- 101100327165 Arabidopsis thaliana CCD8 gene Proteins 0.000 description 6
- 101100129499 Arabidopsis thaliana MAX2 gene Proteins 0.000 description 6
- 230000001965 increasing effect Effects 0.000 description 5
- 230000002194 synthesizing effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/40—Picture signal circuits
- H04N1/40068—Modification of image resolution, i.e. determining the values of picture elements at new relative positions
Definitions
- the present invention relates to an image processing technique for generating a high resolution image from a low resolution image.
- Non-Patent Document 1 discloses a representative technique of the super-resolution technique.
- Equation 1 an equation representing the relationship between a plurality of low resolution images and high resolution images is defined, and Equation 1 is solved to generate a high resolution image.
- iterative processing is performed until the error becomes a certain value or less by iterative processing using a gradient method or the like.
- h is the vector representation of the high-resolution image
- ⁇ is the noise deviation of the observed value
- b (xi, yi) is the vector representation of the PSF (point spread function) kernel corresponding to the position (xi, yi).
- C represents a matrix representing prior information of the high resolution image
- ⁇ represents a constraint parameter representing the strength of the constraint.
- Nl represents the total number of pixels of a plurality of low resolution images.
- the relationship between the desired high-resolution image and the low-resolution image is described as an equation, and by solving this, the high-resolution image Get.
- an equation that models this degradation is important, and the key to super-resolution technology is how to create a degradation model that estimates the degradation process of low resolution.
- Non-Patent Document 2 introduces a super-resolution technique in which robustness is improved by adding a system noise term generated in a deterioration model other than an image input process such as PSF.
- a degradation model such as Equation 2 is defined, and a high-resolution image is generated by solving this equation.
- X is an input image
- Hatm is blur due to the atmosphere
- Fk is degradation due to A / D conversion
- Hcam is blur due to the camera
- Dk is degradation due to downsampling
- Vk is system noise. Since Formula 2 is modeled including system noise, robustness is secured against noise other than normal image degradation.
- the present invention provides noise generated due to a difference between an actual degradation process and a degradation model even when the actual degradation process and the degradation model of the acquired low-resolution image are misaligned.
- An object is to provide an image processing apparatus, an image processing method, a computer program for image processing, and a recording medium that can generate a high-resolution image while suppressing the above-described problem.
- the present invention provides an image processing apparatus for generating a high resolution image from a low resolution image, an acquisition means for acquiring a low resolution image, and performing interpolation on the acquired low resolution image.
- the low-resolution image by enlarging interpolation means for generating a first high-resolution image having a higher resolution than the low-resolution image, and a degradation model that estimates a process in which the high-resolution image is degraded to become a low-resolution image in the reverse direction.
- Super resolution processing means for generating a second high resolution image having the same resolution as the first high resolution image from the resolution image, and the position of each pixel of the first high resolution image using the low resolution image Difference between a value of the pixel and a value of the corresponding pixel in the second high-resolution image for each pixel in the first high-resolution image; Difference calculating means for calculating the difference, the adjusting means for correcting the calculated difference value of each pixel using the generated feature value, and calculating the corrected difference value, and each pixel of the first high-resolution image Adding a correction difference value corresponding to the pixel to the value to generate a third high-resolution image.
- the image processing apparatus of the present invention performs different enlargement processing on the low resolution image by the enlargement interpolation means and the super resolution processing means, and applies each of the low resolution image to each pixel of the first high resolution image.
- a corresponding feature amount is generated, and a difference value between pixels of the first high-resolution image and the second high-resolution image is corrected using the generated feature amount.
- the difference value between the first high-resolution image and the second high-resolution image includes a noise component generated by super-resolution processing, and the difference value is corrected using a feature amount generated based on the low-resolution image.
- An image processing apparatus includes a display device that increases the resolution of an input low-resolution image in accordance with the resolution of the display device, an image output device such as a printer that expands and outputs a low-resolution image,
- the present invention is effectively used in an apparatus for increasing the resolution of a low-resolution image, such as an imaging apparatus that generates a still image by enlarging a single image.
- FIG. 1 is a block diagram showing a configuration of an image processing apparatus 10 as an embodiment according to the present invention.
- 2 is a block diagram illustrating a configuration of an image processing unit 100.
- FIG. An example of the arrangement of one central pixel 401 to be arranged in the first high-resolution image and 36 peripheral pixels 411 to 415 in the low-resolution image is shown.
- Four regions 502, 503, 504, and 505 centered on the pixel 501 are shown.
- the relationship between the positions of MAX1, MAX2, MAX3, MAX4 and the position of the pixel of interest 501 is shown.
- the relationship between the positions of MIN1, MIN2, MIN3, and MIN4 and the position of the pixel of interest 501 is shown.
- An example of a pixel value change 1101, a maximum allowable amplitude change 1102, and a minimum value change 1103 of a plurality of pixels arranged in the x-axis direction in the second high-resolution image is shown.
- An example of a change 1201 of pixel values of a plurality of arranged pixels is shown.
- 4 is a flowchart showing mainly the operation of the image processing unit 100 among the operations of the image processing apparatus 10.
- 5 is a flowchart showing the operation of an enlargement interpolation unit 105.
- One embodiment of the present invention is an image processing device that generates a high-resolution image from a low-resolution image, the acquisition means for acquiring a low-resolution image, and enlargement interpolation of the acquired low-resolution image so that the low-resolution image is obtained.
- interpolation means for generating a first high-resolution image having a resolution higher than that of the resolution image, and a degradation model that estimates a process in which the high-resolution image is degraded to become a low-resolution image, in the reverse direction
- the super-resolution processing means for generating a second high-resolution image having the same resolution as the first high-resolution image, and using the low-resolution image, the feature at the position of each pixel of the first high-resolution image. For each pixel in the first high-resolution image, a difference value between the value of the pixel and the value of the corresponding pixel in the second high-resolution image is calculated.
- a fraction calculation means an adjustment means for correcting the calculated difference value of each pixel using the generated feature value, and calculating a corrected difference value; and a value of each pixel of the first high-resolution image. Adding a correction difference value corresponding to the pixel to generate a third high-resolution image.
- the feature amount generation unit generates, as the feature amount, gradient information indicating a gradient of a value of a pixel in the image region of the low resolution image corresponding to a peripheral pixel of each pixel of the first high resolution image. Then, the adjustment means may correct the difference value using the generated gradient information.
- the adjustment means may calculate the correction difference value by calculating a gain that increases or decreases depending on the gradient information, and multiplying the difference value by the calculated gain.
- the feature amount analyzing means generates, as the feature amount, an allowable range that the value of the pixel can take from the image region of the low resolution image corresponding to the peripheral pixel of each pixel of the first high resolution image.
- the adjusting unit may correct the difference value for each pixel of the second high-resolution image having a value exceeding the allowable range.
- Another embodiment of the present invention is an image processing method used in an image processing apparatus that generates a high resolution image from a low resolution image, an acquisition step of acquiring a low resolution image, and the acquired low resolution image Is expanded and interpolated to generate a first high-resolution image having a higher resolution than the low-resolution image, and a deterioration model that estimates the process of deterioration of the high-resolution image to become a low-resolution image is traced in the reverse direction.
- a difference calculating step for calculating a difference value between pixel values in the resolution image; an adjustment step for calculating a corrected difference value by correcting the calculated difference value of each pixel using the generated feature amount;
- Another embodiment of the present invention is a computer program for image processing used in an image processing apparatus that generates a high-resolution image from a low-resolution image, the acquisition step of acquiring a low-resolution image in a computer; Magnifying and interpolating the acquired low-resolution image to generate a first high-resolution image having a higher resolution than the low-resolution image, and estimating a process in which the high-resolution image deteriorates to become a low-resolution image
- a feature amount generating step for generating a feature amount indicating a feature at a position of each pixel of the first high-resolution image; and each pixel in the first high-resolution image.
- a difference calculation step for calculating a difference value between the pixel value and the corresponding pixel value in the second high-resolution image, and using the generated feature value, the calculated difference value of each pixel.
- An adjustment step for correcting and calculating a correction difference value, and a synthesis step for generating a third high-resolution image by adding a correction difference value corresponding to the pixel to the value of each pixel of the first high-resolution image Is a computer program for executing.
- Another embodiment of the present invention is a computer-readable recording medium recording a computer program for image processing used in an image processing apparatus that generates a high-resolution image from a low-resolution image.
- an acquisition step for acquiring a low resolution image an enlargement interpolation step for enlarging and interpolating the acquired low resolution image to generate a first high resolution image having a higher resolution than the low resolution image, and deterioration of the high resolution image
- the super-resolution for generating the second high-resolution image having the same resolution as the first high-resolution image from the low-resolution image by tracing the degradation model in which the process of becoming the low-resolution image is estimated in the reverse direction
- a feature quantity generator that generates a feature quantity indicating a feature at the position of each pixel of the first high resolution image using the processing step and the low resolution image.
- a difference calculation step for calculating a difference value between the pixel value in the first high-resolution image and a corresponding pixel value in the second high-resolution image, and a difference between the calculated pixels An adjustment step for correcting the value using the generated feature value to calculate a correction difference value, and adding a correction difference value corresponding to the pixel to the value of each pixel of the first high-resolution image
- Image processing apparatus 10 As shown in FIG. 1, the image processing apparatus 10 includes an input unit 201, a tuner 202, an external terminal 203, a memory 205, a processor 206, a decoding unit 207, a display control unit 208, a video driver 209, and a hard disk 210. .
- a display device 211 using liquid crystal, plasma, CRT or the like is connected to the image processing apparatus 10, and a memory card 204 is attached to the image processing apparatus 10.
- the image processing apparatus 10 is, for example, a hard disk recorder, stores programs received by digital broadcasting therein, reproduces the stored programs, and outputs them to the display device 211 in accordance with a user instruction.
- the display device 211 is a large television display device having a high resolution, and displays a program.
- the tuner 202 outputs a TV signal extracted from the received broadcast wave to the input unit 201.
- This TV signal includes image data.
- the external terminal 203 receives a video signal from another video playback device (not shown) such as a DVD playback device, and outputs the received video signal to the input unit 201.
- This video signal includes image data.
- the memory card 204 records moving images and still images, and is an SD card as an example. These moving images and still images include image data.
- the input unit 201 receives image data (hereinafter referred to as a low resolution image) from the tuner 202, the external terminal 203, or the memory card 204.
- the low resolution image means an image having a lower resolution than the resolution of the display device 211.
- the memory 205 is used as a primary storage device of the image processing apparatus 10 and is composed of a DRAM or the like.
- the processor 206 controls the entire image processing apparatus 10.
- the hard disk 210 is a secondary storage device for accumulating images and the like, and accumulates low-resolution images input by the input unit 201.
- the decoding unit 207 decodes the compressed image when the low resolution image received by the input unit 201 is a compressed image, or when the low resolution image stored in the hard disk 210 is a compressed image.
- the decoded image is sent to the display control unit 208 directly or via the memory 205.
- the display control unit 208 converts the received image into an image having a resolution matching the display device 211 (hereinafter referred to as a third high resolution image), and outputs the third high resolution image to the video driver 209.
- the video driver 209 outputs the third high resolution image to the display device 211 and controls the display of the third high resolution image on the display device 211.
- the image processing apparatus 10 includes the tuner 202, but is not limited to this.
- the image processing apparatus 10 may not include the tuner 202, and a digital broadcast receiving apparatus having a tuner may be connected to the image processing apparatus 10.
- the display control unit 208 includes the image processing unit 100 shown in FIG. 2 in order to generate a third high resolution image by increasing the resolution of the received low resolution image in accordance with the resolution of the display device 211.
- the display control unit 208 extracts the pixel number inXS in the horizontal direction and the pixel number inYS in the vertical direction of the low-resolution image from the decoded low-resolution image. Further, the video driver 209 receives the horizontal pixel number outXS and the vertical pixel number outYS of the third high-resolution image to be displayed on the display device 211.
- the image processing unit 100 is implemented as hardware, but may be implemented as software processing in the processor 206. In either case, the generated third high-resolution image can be encoded again and written back to the hard disk 210 or the like.
- Image processing unit 100 As illustrated in FIG. 2, the image processing unit 100 includes a feature amount generation unit 103, a super-resolution enlargement unit 104, an enlargement interpolation unit 105, a difference calculation unit 106, an adjustment unit 107, and a synthesis unit 108.
- the image processing unit 100 receives a low resolution image.
- the enlargement interpolation unit 105 enlarges and interpolates the received low resolution image to generate a first high resolution image having a higher resolution than the low resolution image.
- the super-resolution enlargement unit 104 traces the degradation model in which the high-resolution image is deteriorated to become a low-resolution image in the reverse direction, and thus the same as the first high-resolution image from the received low-resolution image.
- a second high-resolution image having a resolution of is generated.
- the feature amount generation unit 103 generates a feature amount indicating a feature at the position of each pixel of the first high resolution image, using the received low resolution image.
- the difference calculation unit 106 calculates, for each pixel in the first high resolution image, a difference value between the pixel value and the corresponding pixel value in the second high resolution image.
- the adjustment unit 107 corrects the calculated difference value of each pixel using the generated feature amount, and calculates a corrected difference value.
- the synthesizing unit 108 adds the correction difference value corresponding to the pixel to the value of each pixel of the first high resolution image to generate and output a third high resolution image.
- the image processing unit 100 receives the horizontal pixel number inXS and the vertical pixel number inYS of the low resolution image, and receives the horizontal pixel number outXS and the vertical pixel number outYS of the high resolution image.
- the enlargement interpolation unit 105 generates a first high-resolution image as described below.
- the resolution of the first high-resolution image, the number of pixels in the horizontal direction, and the number of pixels in the vertical direction are respectively the resolution of the third high-resolution image to be displayed on the display device 211, the number of pixels in the horizontal direction, and the number of pixels in the vertical direction. Equal to the number of pixels.
- the enlargement interpolation unit 105 receives a low resolution image.
- the horizontal pixel number inXS and the vertical pixel number inYS of the low resolution image are received, and the horizontal pixel number outXS and the vertical pixel number outYS of the third high resolution image are received.
- the enlargement interpolation unit 105 uses the number of pixels inXS, inYS, outXS, and outYS to calculate the horizontal interpolation interval dx and the vertical interpolation interval dy according to Equations 3 and 4.
- Horizontal interpolation interval dx inDX / outDX (Formula 3)
- Vertical interpolation interval dy inDY / outDY (Formula 4)
- the thus calculated horizontal interpolation interval dx and vertical interpolation interval dy define the positions of the pixels included in the first high-resolution image. That is, the pixels included in the first high-resolution image are arranged at a position that is an integral multiple of the interpolation interval dx in the horizontal direction in the horizontal direction, and at a position that is an integral multiple of the interpolation interval dy in the vertical direction in the vertical direction. Arranged.
- dx and dy are values less than 1.
- the enlargement interpolation unit 105 repeats the following processes (a) to (c) for all pixel positions to be arranged in the first high resolution image.
- the enlargement interpolation unit 105 includes a plurality of pixels included in the low resolution image.
- a total of 36 pixels (referred to as peripheral pixels) arranged in rows and columns in the horizontal direction and in the vertical direction are referred to as peripheral pixels, and the above-mentioned center position is approximately at the center of the peripheral pixels.
- the pixel values of the selected peripheral pixels are acquired.
- FIG. 3 shows an arrangement of one central pixel 401 to be arranged in the first high-resolution image and 36 peripheral pixels 411, 412, 413,..., 414,. An example is shown. As shown in this figure, there is a central pixel 401 at a substantially central position of peripheral pixels 411, 412, 413,..., 414,. .
- the coordinates (x, y) of the center pixel 401 are (3.4, 3.5) as an example.
- the decimal points px and py are components in the x-axis direction and the y-axis direction of the distance between the peripheral pixel 414 closest to the central pixel 401 and the central pixel 401.
- the pixel value of the center pixel at the center position is calculated using the enlargement interpolation method using the pixel values of the acquired peripheral pixels and the decimal points px and py, and the calculated pixel value is set to the first high value. Store at the center of the resolution image.
- the conventional bilinear method As the enlargement interpolation method, the conventional bilinear method, bicubic convolution method, Lanczos method, or the like can be used.
- the bicubic convolution method uses 4 ⁇ 4 pixels around the point (x, y). Interpolation calculation is performed.
- the pixel value out of the center pixel is calculated by the following expression 5.
- Equation 6 is the weighting factor in the x direction at the l location, and wym is the weighting factor in the y direction. is there.
- Equation 6 Each weighting coefficient is expressed by the following Equation 6 in the case of bicubic interpolation, where d is the distance in the x or y direction from the interpolation pixel position indicated by px, py.
- w is expressed by the following formula 7.
- the degradation function H includes a reduction process for converting the second high resolution image into a low resolution image, a PSF (point spread function), and the like.
- the super-resolution enlargement unit 104 performs processing for obtaining X, that is, the second high-resolution image by performing the inverse operation of Expression 8.
- Y HX + V (Formula 9)
- the second high-resolution image X is provisionally determined
- Y ′ is calculated according to Equation 8 or Equation 9, and these calculations are performed until the difference from the input image is less than or equal to the threshold value. repeat.
- As a simple method there is a method of obtaining X by deconvolution.
- image degradation is estimated by some method, and the inverse function is obtained to generate a second high-resolution image.
- the pixel positions of the first and second high-resolution images generated by the enlargement interpolation unit 105 and the super-resolution enlargement unit 104 are replaced with the pixel positions of the low-resolution image (X and Y in FIG. 3). , Expressed as phase.
- the pixel position (interpolation position x, y, dx, dy) of the first high-resolution image of the enlargement interpolation unit 105 and the pixel position of the second high-resolution image of the super-resolution enlargement unit 104 Should be the same.
- the feature quantity generation unit 103 receives a low resolution image, calculates a feature quantity corresponding to each pixel of the first high resolution image, and outputs it. Therefore, the feature amount output as a result of the feature amount analysis by the feature amount generation unit 103 is also in phase with the first and second high-resolution images that are the outputs of the enlargement interpolation unit 105 and the super-resolution enlargement unit 104. It becomes a shape.
- the feature value generation unit 103 repeats the following processes (a) to (b) for all pixel positions to be arranged in the first high-resolution image.
- the feature amount generation unit 103 has a plurality of positions included in the low-resolution image with respect to the position (referred to as the target position) of one pixel (referred to as the target pixel) to be arranged in the first high-resolution image.
- the target position the position of one pixel (referred to as the target pixel) to be arranged in the first high-resolution image.
- N pixels in the horizontal direction, M pixels in the vertical direction, and a total of N ⁇ M pixels arranged in a matrix (a region including these pixels is referred to as a region S) is described above.
- the attention position is selected so as to be arranged in the approximate center of the region S, and the pixel values of the selected peripheral pixels are acquired.
- the feature value generation unit 103 calculates the gradient of the region S centering on the target pixel and the allowable amplitude of the region S as the feature value.
- the region S is N ⁇ M pixels centered on the interpolation phase (x, y) of the low-resolution image, and is 2 ⁇ 2 pixels in the minimum.
- N and M are preferably about 3 to 6.
- ⁇ u ⁇ can be calculated by, for example, the following Expression 10 or Expression 11.
- the feature amount generation unit 103 may obtain a difference between the maximum value and the minimum value in the region S.
- the feature quantity generation unit 103 calculates the gradient in the region S and outputs this as one of the feature quantities. (Calculation of allowable amplitude)
- the allowable amplitude indicates the allowable amount of the change range of the target pixel.
- the feature quantity generation unit 103 calculates the allowable amplitude from the gradient information in the region S and the like.
- the feature value generation unit 103 calculates the maximum value in the region S as the maximum value of the allowable amplitude, and calculates the minimum value in the region S as the minimum value of the allowable amplitude.
- a pixel 501 is a pixel (target pixel) of the first high-resolution image that is a target for calculating the allowable amplitude. Further, a total of 36 low-resolution image pixels 511, 512, 513, 514,. , 519,..., 519, and a total of 36 low-resolution image pixels 511, 512, 513, 514,..., 519 constitute a region S (500).
- the feature amount generation unit 103 includes a total of 36 low-resolution image pixels 511, 512, 513, 514,... Included in the region S (500) so that the pixel 501 that is the target pixel is arranged at the center. 519 is selected. Next, the region S (500) is divided into four regions 502, 503, 504, and 505. The areas 502, 503, 504, and 505 each include a total of nine low-resolution image pixels arranged in a matrix form with three in the horizontal direction and three in the vertical direction.
- the region 502 is adjacent to the upper left side of the pixel 501
- the region 503 is adjacent to the upper right side of the pixel 501
- the region 504 is adjacent to the lower left side of the pixel 501
- the region 505 is adjacent to the lower right side of the pixel 501.
- the pixel 515 is disposed at a position closest to the pixel 501
- the pixel 516 includes Among the nine low-resolution image pixels included in the region 504 that are arranged closest to the pixel 501, the nine low-resolution images included in the region 505 where the pixel 517 is arranged closest to the pixel 501.
- the pixel 518 is disposed at a position closest to the pixel 501.
- the feature value generation unit 103 selects a pixel having the maximum pixel value from nine low-resolution image pixels included in the region 502, and sets the pixel value to MAX1. In addition, a pixel having the minimum pixel value is selected from the nine low-resolution image pixels included in the region 502, and the pixel value is set to MIN1. In this way, a set of the maximum value MAX1 and the minimum value MIN1 is calculated from the area 502. Similarly, the feature quantity generation unit 103 calculates a set of the maximum value MAX2 and the minimum value MIN2 from the area 503, calculates a set of the maximum value MAX3 and the minimum value MIN3 from the area 504, and sets the maximum value MAX4 from the area 505. A set of minimum values MIN4 is calculated.
- the set of the maximum value MAX1 and the minimum value MIN1 is considered to represent the region 502, and the maximum value MAX1 and the minimum value MIN1 are the same positions in the region 502 as the positions where the pixels 515 are arranged. It shall be arranged in.
- the maximum value MAX2 and the minimum value MIN2 are arranged at the same position in the region 503 as the pixel 516 is arranged, and the maximum value MAX3 and the minimum value MIN3 are inside the region 504.
- the maximum value MAX4 and the minimum value MIN4 are arranged in the same position as the position where the pixel 518 is arranged in the region 505. It shall be.
- the positional relationship between the maximum values MAX1, MAX2, MAX3, MAX4 and the pixel 501 is as shown in FIG. 5, and the positional relationship between the minimum values MIN1, MIN2, MIN3, MIN4 and the pixel 501 is as shown in FIG. As shown.
- the feature amount generation unit 103 uses the positional relationship between the maximum values MAX1, MAX2, MAX3, MAX4, the pixel 501, and the decimal point components px, py of the interpolation phase of the pixel of interest 501 as MAX1.
- MAX2, MAX3, MAX4 are linearly interpolated to calculate the maximum allowable value MAX of the target pixel 501.
- the feature value generation unit 103 uses the positional relationship between the minimum values MIN1, MIN2, MIN3, MIN4 and the pixel 501, and the decimal point components px and py of the interpolation phase of the target pixel 501.
- MIN1, MIN2, MIN3, and MIN4 are linearly interpolated to calculate the minimum value MIN of the allowable amplitude of the target pixel 501.
- the maximum value MAX and the minimum value MIN of the allowable amplitude are calculated for all the pixels in the first high-resolution image.
- the feature amount generation unit 103 corrects the maximum value MAX and the minimum value MIN of the amplitude calculated in the first example of the allowable amplitude by the convexity determination of the region S.
- FIGS. 7 and 8 Examples of changes in the pixel value in the x-axis direction of the region S are shown in FIGS.
- FIG. 7 in the region S, the pixel values of a plurality of pixels arranged in the x-axis direction change in a concave shape along the x-axis.
- FIG. 8 in the region S, the pixel values of a plurality of pixels arranged in the x-axis direction change in a convex shape along the x-axis. 7 and 8, only the change in the pixel value in the X-axis direction is shown for simplification, but in each case, the pixel values of a plurality of pixels arranged in the Y-axis direction are also shown. It may be changed as well. Further, in the space of the entire region S, the pixel values of a plurality of pixels in the region S may be changed to a concave shape or a convex shape.
- the feature value generation unit 103 determines whether the pixel values of a plurality of pixels arranged in the region S are changed in a concave shape or a convex shape in the entire space of the region S.
- the feature amount generation unit 103 When it is determined that the pixel value changes in a concave shape in the entire space of the region S, the feature amount generation unit 103, as shown in Expression 12, allows the allowable amplitude calculated in the first example of the allowable amplitude.
- the minimum value MIN ′ after correction is calculated by multiplying the minimum value MIN of the amplitude by a coefficient ⁇ of 1 or less.
- the feature amount generation unit 103 calculates with the first example of the allowable amplitude as shown in Expression 13.
- the corrected maximum value MAX ′ is calculated by multiplying the maximum value MAX of the allowable amplitude by a coefficient ⁇ of 1 or more.
- ⁇ f (u) (Formula 14)
- ⁇ may be obtained from the Laplacian result of the region S.
- ⁇ may be obtained from the ratio between the pixel value obtained by applying the low-pass filter to the pixel value in the region S and each pixel value in the region S.
- the coefficient ⁇ may be determined by multiplying the result of the Laplacian filter by an appropriate coefficient ⁇ .
- the pixel value (when expressing the pixel value, any value from 0 to 255 may be taken.
- the pixel value is normalized and expressed, it is expressed by 0 to 1.0. , 0 indicates black, 1.0 indicates white) and ⁇ may be too large depending on the coefficient ⁇ . If ⁇ is too large or too small, ⁇ is clipped by multiplying ⁇ by the clip value.
- the clip value for example, 0.5 times to 1.5 times, 0.25 times to 1.75 times, and the like can be considered. The clip value is not limited to this range.
- the ratio (B / A) may be ⁇ .
- clipping equivalent to Laplacian may be performed.
- the corrected minimum value and maximum value are calculated as the allowable amplitude based on the unevenness degree of the region S.
- the difference calculation unit 106 receives the first high-resolution image from the enlargement interpolation unit 105 and receives the second high-resolution image from the super-resolution enlargement unit 104.
- the difference calculation unit 106 calculates the pixel values of the corresponding pixels of the first high-resolution image and the second high-resolution image according to Equation 13 for all the pixels included in the first high-resolution image and the second high-resolution image. And the difference value obtained by the calculation is output to the adjustment unit 107.
- Difference value (x, y) pixel value of pixel (x, y) of second high-resolution image ⁇ pixel value of pixel (x, y) of first high-resolution image (Formula 15)
- the difference calculation unit 106 outputs the received second high resolution image to the adjustment unit 107.
- Adjustment unit 107 receives all the calculated difference values (x, y) from the difference calculation unit 106, receives the second high-resolution image from the difference calculation unit 106, and receives the feature amount from the feature amount generation unit 103.
- the feature amount is gradient information, allowable amplitude, or gradient information and allowable amplitude for all pixels of the first high-resolution image.
- the adjustment unit 107 corrects the difference value (x, y) calculated by the difference calculation unit 106 by using the following equation 16 for all received difference values (x, y) to obtain a corrected difference value ( x, y).
- Correction difference value (x, y) difference value (x, y) ⁇ gain ⁇ (Expression 16)
- the adjustment unit 107 outputs all the calculated correction difference values (x, y) to the synthesis unit 108.
- the adjustment unit 107 may correct the difference value using gradient information that is a feature amount received from the feature amount generation unit 103.
- the difference value is corrected according to the magnitude of the gradient.
- the adjustment unit 107 stores a correspondence table (not shown) indicating the correspondence between the gradient and the gain ⁇ in advance.
- the correspondence table includes a plurality of correspondence data, and each correspondence data includes a gradient and a gain ⁇ .
- the gradient included in the same correspondence data corresponds to the gain ⁇ , and when the gradient is designated, the gain ⁇ corresponding to the designated gradient is used.
- the gain ⁇ is set within a range of about 0 to 16 times.
- the adjustment unit 107 acquires the gain ⁇ corresponding to the gradient indicated by the gradient information that is the feature amount by reading it from the correspondence table.
- the gain ⁇ corresponding to the gradient indicated by the gradient information may be calculated by linear interpolation using a plurality of correspondence data included in the correspondence table. .
- the adjustment unit 107 does not have a correspondence table, may hold an arithmetic expression indicating the correspondence between the gradient and the gain ⁇ , and may calculate the gain ⁇ corresponding to the gradient using this arithmetic expression.
- the adjustment unit 107 acquires the gain ⁇ in this way, and calculates the corrected difference value by multiplying the acquired gain ⁇ by the difference value as shown in Expression 16.
- FIG. 9A shows a first specific example of the correspondence between the gradient and the gain ⁇ in the correspondence table.
- a graph 600 shown in FIG. 9A represents the correspondence between the gradient and the gain ⁇ , the horizontal axis represents the gradient, and the vertical axis represents the gain.
- the gain ⁇ is set to be smaller than the gain value 612 and the gain ⁇ is set to a substantially constant value.
- the gain ⁇ is set to be larger than the gain value 612, and the gain ⁇ increases as the gradient increases.
- the gain ⁇ is reduced in the small gradient range, and the gain ⁇ is set larger as the gradient increases in the large gradient range.
- the small gain ⁇ is used to correct the difference value, in other words, the difference value correction is extremely small.
- the stronger edge component can be corrected using a larger gain ⁇ , in other words, only the strong edge component can be corrected. In this way, only the edge component can be emphasized in the image without enhancing the noise component included in the small amplitude.
- FIG. 9B shows a second specific example of the correspondence relationship between the gradient and the gain ⁇ in the correspondence table.
- a graph 620 illustrated in FIG. 9B represents the correspondence between the gradient and the gain ⁇ , the horizontal axis represents the gradient, and the vertical axis represents the gain.
- the gain ⁇ is set to be smaller than the gain value 641, and the gain ⁇ is set to a substantially constant value.
- the gain ⁇ is set to increase once and then decrease.
- the gain beta increases sharply from the gain value 641 to the gain value 643, the gain value 643 to the peak value, then decreases rapidly from the gain value 643 to the gain value 642 ing.
- the gain ⁇ is thus being set so as to decrease gradually, in FIG. 9 (b), in the small range of the gradient The gain ⁇ is decreased, the gain ⁇ is increased for a range where there is a little gradient, and the gain ⁇ is set again smaller for a range where the gradient is large.
- the image includes a fine texture in the range 622. Therefore, the gain in the range 622 is increased. Thus, the effect of the fine texture portion can be left, and the effects in the range 621 and the range 623 other than the range 622 can be reduced.
- the gain ⁇ is determined according to the gradient of the region S, that is, the flatness of the region S, and the corrected difference value is obtained by multiplying the difference value by the gain ⁇ . In this way, the difference value is corrected.
- the gain ⁇ is basically a value of 1 or less, but by setting it to 1 or more, it is possible to emphasize a region of a certain gradient.
- the gain ⁇ may be set within a range of about 0 to 16 times.
- the adjustment unit 107 may correct the difference value using the allowable amplitude that is the feature amount received from the feature amount generation unit 103.
- correction is performed for a portion exceeding the allowable amplitude.
- the adjustment unit 107 repeats the following (i) to (v) for all the pixels of the received second high resolution image.
- the horizontal axis represents the x coordinate and the vertical axis represents the pixel value.
- a change 1101 is shown.
- the pixel values of the pixels indicated by the specific y-coordinate values and x-coordinate values 701, 702, 703, 706, and 707 are within the range of the maximum value and the minimum value of the corresponding position. . However, the pixel value of the pixel at the position indicated by the specific y coordinate value and x coordinate value 704 and 705, respectively, is greater than the maximum value of the corresponding position.
- the adjustment unit 107 calculates the correction difference value by multiplying the calculated difference value by the gain ⁇ for the position indicated by the specific y coordinate value and the x coordinate values 704 and 705, respectively.
- the corrected difference value is output.
- the adjustment unit 107 outputs the original difference value as the correction difference value for the positions indicated by the specific y coordinate value and x coordinate value 701, 702, 703, 706 and 707, respectively.
- the composition unit 108 described later uses the correction difference value calculated in this way to generate and outputs a second high-resolution image.
- the horizontal axis represents the x coordinate
- the vertical axis represents the pixel value.
- specific y coordinate values and x coordinate values 701, 702, ..., 707 shows a change 1201 in the pixel value of the pixel arranged at each position.
- the third high-resolution image is compared with the second high-resolution image at the pixel values of the third high-resolution image at positions indicated by specific y-coordinate values and x-coordinate values 704 and 705, respectively. Has been revised downward. That is, the pixel value is corrected to be small.
- the adjustment unit 107 corrects the difference value for the portion exceeding the allowable amplitude.
- the gain ⁇ 1 and the gain ⁇ 2 are calculated individually by the first correction method and the second correction method, respectively, and the gain ⁇ 1 and the gain ⁇ 2 are added.
- Gain ⁇ Gain ⁇ 1 + Gain ⁇ 2
- a correction difference value is calculated by multiplying the gain ⁇ obtained by the difference value.
- correction may be made as follows.
- the correction difference value is calculated by the first correction method described above, the calculated correction difference value is output, and the synthesis unit 108 generates a third high-resolution image using the correction difference value and the first high-resolution image.
- the adjustment unit 107 determines whether or not the allowable amplitude is exceeded for the third high-resolution image by the second correction method, and further adds a second correction difference value to the pixel that exceeds the allowable amplitude.
- the double correction difference value is calculated by multiplying the gain of the correction method.
- the synthesizing unit 108 adds the double correction difference value to the third high-resolution image for the pixels exceeding the allowable amplitude, generates a fourth high-resolution image, and outputs the generated fourth high-resolution image. To do.
- Synthesis unit 108 receives the first high resolution image from the enlargement interpolation unit 105, receives the correction difference value from the adjustment unit 107, and adds the corresponding correction difference value to the pixel values of all the pixels of the first high resolution image. A third high resolution image is generated, and the generated third high resolution image is output.
- Third high-resolution image (x, y) first high-resolution image (x, y) + corrected difference value (x, y)
- the correction difference value is obtained by correcting the difference value of each pixel of the first high-resolution image and the second high-resolution image. Therefore, the correction difference value is added to the pixel value of each pixel of the first high-resolution image. The result of the corrected super-resolution processing can be obtained.
- the input unit 201 acquires a low-resolution image (step S101), the enlargement interpolation unit 105 performs enlargement interpolation processing to generate a first high-resolution image (step S102), and the super-resolution enlargement unit 104
- a resolution enlargement process is performed to generate a second high-resolution image (step S103)
- the feature amount generation unit 103 generates a feature amount (step S104)
- the difference calculation unit 106 calculates a difference value (step S104).
- the adjustment unit 107 calculates a correction difference value (step S106)
- the synthesis unit 108 adds the correction difference value to the first high resolution image to generate a third high resolution image (step S107).
- the generated third high-resolution image is output (step S108).
- the enlargement interpolation unit 105 determines the horizontal and vertical directions from the horizontal and vertical pixel numbers inXS and inYS of the low-resolution image and the horizontal and vertical pixel numbers outXS and outYS of the third high-resolution image to be output.
- the interpolation intervals dx and dy are calculated according to equations 3 and 4 (S302).
- the enlargement interpolation unit 105 initializes the variable y to 0 (S303), initializes the variable x to 0 (S304), and acquires the values of peripheral pixels centered on the coordinates (x, y) from the low-resolution image ( S305). Next, the decimal point parts px and py of x and y are calculated (S306).
- the enlargement interpolation unit 105 performs enlargement interpolation processing using pixel values of pixels adjacent to the low resolution pixel centered at the point (x, y) and px, py (S307), and is output by the enlargement interpolation processing.
- the obtained pixel value is stored as an output pixel value (S308), and then dx is added to the variable x (S309). If the variable x does not exceed the horizontal pixel count outXS (No in S310), the process returns to S305 and the process is repeated. If the variable x exceeds outXS (Yes in S310), dy is added to the variable y (S311).
- variable y does not exceed outYS (No in S312), the process returns to S304 for processing. repeat. If exceeded (Yes in S312), the pixel values of all the pixels of the first high-resolution image have been output, and the process is terminated.
- the super-resolution processing result is corrected.
- correction using gradient information for example, noise generated in a flat portion of a pixel value included in an image or noise generated in a portion with a large gradient, such as an edge, can be suppressed. it can.
- the correction using the allowable amplitude it is possible to correct a portion where noise that protrudes from the pixel value of the pixel of the low-resolution image that is the input image has occurred.
- noise generated in the super-resolution process can be suppressed and a visually good image can be obtained.
- the difference calculation unit 106 may calculate the difference value (x, y) by Expression 17 instead of Expression 15.
- Difference value (x, y) pixel value of pixel (x, y) of first high-resolution image ⁇ pixel value of pixel (x, y) of second high-resolution image (Expression 17)
- the synthesizing unit 108 generates a third high-resolution image using Expression 18.
- Third high-resolution image (x, y) first high-resolution image (x, y) ⁇ correction difference value (x, y) (Formula 18) (2)
- the super-resolution enlargement unit 104 may use the super-resolution enlargement processing method disclosed in Non-Patent Document 1 or Non-Patent Document 2.
- One embodiment of the present invention is an image processing device that generates a high-resolution image from a low-resolution image, an acquisition circuit that acquires the low-resolution image, and an enlarged interpolation of the acquired low-resolution image, and the low-resolution image From the low resolution image, by following an enlargement interpolation circuit that generates a first high resolution image having a higher resolution than the image, and a degradation model that estimates a process in which the high resolution image is degraded to become a low resolution image, in the reverse direction, A super-resolution processing circuit that generates a second high-resolution image having the same resolution as the first high-resolution image, and a feature at the position of each pixel of the first high-resolution image using the low-resolution image A feature amount generation circuit that generates a feature amount, and a difference for calculating a difference value between the pixel value and the corresponding pixel value in the second high-resolution image for each pixel in the first high-resolution image An arithmetic circuit,
- Another embodiment of the present invention is an image processing apparatus that generates a high-resolution image from a low-resolution image, and a memory unit that stores a computer program configured by combining a plurality of computer instructions; A computer instruction is read from the computer program stored in the memory unit one by one, decoded, and a processor that operates according to the decoded result.
- the computer program has an acquisition step of acquiring a low resolution image in the image processing apparatus, which is a computer, and enlargement interpolation of the acquired low resolution image to obtain a first high resolution image having a higher resolution than the low resolution image.
- the same interpolation resolution step as the first high-resolution image is obtained from the low-resolution image by tracing in a reverse direction an enlargement interpolation step to be generated and a degradation model that estimates a process in which the high-resolution image deteriorates to become a low-resolution image.
- another embodiment of the present invention is a computer-readable non-transitory recording medium that records a computer program used in an image processing apparatus that generates a high-resolution image from a low-resolution image.
- the computer program generates a first high-resolution image having a higher resolution than the low-resolution image by acquiring an acquisition step of acquiring a low-resolution image in an image processing apparatus that is a computer and enlarging and interpolating the acquired low-resolution image.
- a first model having the same resolution as the first high-resolution image is obtained from the low-resolution image.
- a super-resolution processing step for generating a high-resolution image a feature amount generation step for generating a feature amount indicating a feature at a position of each pixel of the first high-resolution image using the low-resolution image, For each pixel in one high-resolution image, a difference calculation that calculates a difference value between the value of the pixel and the corresponding pixel value in the second high-resolution image The step of correcting the difference value of each calculated pixel using the generated feature value, and calculating the corrected difference value; and the value of each pixel of the first high-resolution image, A correction step of adding a correction difference value corresponding to a pixel to generate a third high-resolution image is performed.
- the above apparatus may be a computer system including a microprocessor, a ROM, a RAM, a hard disk unit, and the like.
- a computer program is stored in the RAM or the hard disk unit.
- Each device achieves its function by the microprocessor operating according to the computer program.
- the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.
- the present invention may be an image processing method used in an image processing apparatus. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.
- the present invention also provides a computer-readable recording medium such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray Disc). ), Recorded in a semiconductor memory or the like. Further, the present invention may be the computer program or the digital signal recorded on these recording media.
- the computer program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.
- the present invention may also be a computer system including a microprocessor and a memory.
- the memory may store the computer program, and the microprocessor may operate according to the computer program.
- the program or the digital signal is recorded on the recording medium and transferred, or the program or the digital signal is transferred via the network or the like, and is executed by another independent computer system. It is good.
- the image processing apparatus suppresses the occurrence of noise when the deterioration model used in the super-resolution processing is different from the actual deterioration process of the acquired low-resolution image, and provides a high resolution that is visually good.
- An image can be obtained, which is suitable for an image processing technique for generating a high resolution image from a low resolution image.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Editing Of Facsimile Originals (AREA)
Abstract
Description
以下、本発明に係る一の実施の形態としての画像処理装置10について、図面を参照しながら説明する。
画像処理装置10は、図1に示すように、入力部201、チューナ202、外部端子203、メモリ205、プロセッサ206、復号部207、表示制御部208、ビデオドライバ209及びハードディスク210から構成されている。画像処理装置10には、液晶、プラズマ、CRTなどを利用した表示デバイス211が接続され、また、画像処理装置10には、メモリカード204が装着される。
画像処理部100は、図2に示すように、特徴量生成部103、超解像拡大部104、拡大補間部105、差分演算部106、調整部107及び合成部108から構成される。
拡大補間部105は、以下に示すようにして、第1高解像度画像を生成する。なお、第1高解像度画像の解像度、水平方向の画素数及び垂直方向の画素数は、それぞれ、表示デバイス211に表示されるべき第3高解像度画像の解像度、水平方向の画素数及び垂直方向の画素数に等しい。
垂直方向の補間間隔dy=inDY/outDY (式4)
こうして算出された水平方向の補間間隔dx及び垂直方向の補間間隔dyは、第1高解像度画像に含まれる画素の位置を規定することとなる。つまり、第1高解像度画像に含まれる画素は、水平方向には、水平方向の補間間隔dxの整数倍の位置に配され、垂直方向には、垂直方向の補間間隔dyの整数倍の位置に配される。
Yを入力される低解像度画像、Xを未知の正しい第2高解像度画像としたとき、入力画像Yは劣化関数Hによって劣化したものと考えることができる。この関係を次の式8に示す。
なお、劣化関数Hには、第2高解像度画像を低解像度画像に変換する縮小処理やPSF(点広がり関数)等を含む。
この逆問題を解くため、一般的には、第2高解像度画像Xを仮に決め、式8又は式9に従って、Y'を計算し、入力画像との差が閾値以下になるまで、これらの演算を繰り返す。また、低解像度画像として入力される動画像の複数のフレームを用いXを計算する方法などもある。また、単純な方法としてはデコンボリューションによりXを求める方法などもある。
上記の劣化関数Hを正しく推定することは重要であり、Hにより出来上がった第2高解像度画像の品質が変化する。なお、放送などの場合、劣化関数Hを正しく推定することは難しく、関数Hが異なることで、意図しないノイズ成分等が発生してしまう。本実施の形態の以降の処理では、この意図しないノイズ成分を抑制する。
(領域Sの勾配の算出)
特徴量生成部103は、領域Sの勾配を、領域S内の画素値を、領域S内の離散値ではなく、領域S内で連続して変化する値とみなして、関数uと表現する場合、∥∇u∥を計算することにより算出する。
(許容振幅の算出)
許容振幅は、注目画素の変化の範囲の許容量を示している。特徴量生成部103は、許容振幅を、領域S内の勾配情報などから計算する。
特徴量生成部103は、領域S内の最大値を許容振幅の最大値として算出し、領域S内の最小値を許容振幅の最小値として算出する。
ここでは、許容振幅の別の計算方法について説明する。
許容振幅の計算の第1の例について、図4~図6を用いて説明する。
許容振幅の計算の第2の例について、図7~図8を用いて説明する。
また、領域S全体の空間において、画素値が凸状に変化していると判断される場合には、特徴量生成部103は、式13に示すように、許容振幅の第1の例で計算された許容振幅の最大値MAXに対して、1以上の係数αを乗じて、補正後の最大値MAX’を算出する。
このように、領域S全体の空間における画素値の凹凸に合わせて、MIN及びMAXを補正することで、許容振幅を低解像度画像の特徴に合わせて調整する。これにより、低解像度画像において、画素値が凸状に変化する部分について、最大値を大きく、画素値が凹状に変化する部分について、最小値を小さくし、低解像度画像の注目画素の周辺画素値の最大値及び最小値を超えた振幅制限を実現することができる。
関数fでは、例えば、領域Sのラプラシアンの結果からαを求めるとしてもよい。また、領域S内の画素値にローパスフィルタを施した結果の画素値と、領域Sの各画素値の比からαを求めるとしてもよい。
差分演算部106は、拡大補間部105から第1高解像度画像を受け取り、超解像拡大部104から第2高解像度画像を受け取る。
また、差分演算部106は、受け取った第2高解像度画像を調整部107へ出力する。
調整部107は、差分演算部106から、算出された全ての差分値(x,y)を受け取り、差分演算部106から第2高解像度画像を受け取り、特徴量生成部103から特徴量を受け取る。ここで、特徴量は、上述したように、第1高解像度画像の全ての画素についての勾配情報、許容振幅、又は勾配情報及び許容振幅である。
次に、調整部107は、算出した全ての補正差分値(x,y)を合成部108へ出力する。
調整部107は、第1の補正方法として、特徴量生成部103から受け取った特徴量である勾配情報を用いて差分値を補正してもよい。勾配情報を用いた補正では、勾配の大きさにより差分値の補正を行う。
ここで、対応テーブルにおける勾配とゲインβの対応関係についての第1の具体例を図9(a)に示す。図9(a)に示すグラフ600は、勾配とゲインβの対応関係を表しており、横軸は、勾配を表し、縦軸は、ゲインを表している。グラフ600においては、勾配値611より小さい勾配の範囲601においては、ゲインβがゲイン値612より小さく、かつ、ゲインβがほぼ一定値となるように設定されている。一方、勾配値611より大きい勾配の範囲602においては、ゲインβがゲイン値612より大きく、かつ、勾配が大きくなるに従って、ゲインβが大きくなるように設定されている。
次に、対応テーブルにおける勾配とゲインβの対応関係についての第2の具体例を図9(b)に示す。図9(b)に示すグラフ620は、勾配とゲインβの対応関係を表しており、横軸は、勾配を表し、縦軸は、ゲインを表している。グラフ620において、勾配値631より小さい勾配の範囲621においては、ゲインβがゲイン値641より小さく、かつ、ゲインβがほぼ一定値となるように設定されている。また、勾配値631より大きく、勾配値632より小さい勾配の範囲622においては、ゲインβが一旦増加し、その後減少するように設定されている。つまり、範囲622において、勾配の増加に従って、ゲインβは、ゲイン値641からゲイン値643まで急激に増加し、ゲイン値643をピーク値とし、その後、ゲイン値643からゲイン値642まで急激に減少している。さらに、勾配値632より大きい勾配の範囲623においては、勾配の増加に従って、ゲインβがなだらかに減少するように設定されている
このように、図9(b)においては、勾配の小さな範囲においては、ゲインβを小さくし、勾配が少し存在する範囲については、ゲインβを大きくし、勾配の大きい範囲については、再び、ゲインβを小さく設定している。
調整部107は、第2の補正方法として、特徴量生成部103から受け取った特徴量である許容振幅を用いて差分値を補正してもよい。許容振幅を用いた補正では、許容振幅を超えた部分について補正を行う。
次に示すようにして補正してもよい。
次に、差分値に得られたゲインβを乗じて、補正差分値を算出する。
合成部108は、拡大補間部105から第1高解像度画像を受け取り、調整部107から補正差分値を受け取り、第1高解像度画像の全ての画素の画素値に、対応する補正差分値を加算して、第3高解像度画像を生成し、生成した第3高解像度画像を出力する。
補正差分値は、第1高解像度画像と第2高解像度画像の各画素の差分値を補正したものであるため、補正差分値を第1高解像度画像の各画素の画素値に加算することで、補正された超解像処理の結果を得ることができる。
画像処理装置10の動作について説明する。
ここでは、画像処理装置10の動作について、特に画像処理部100の動作を中心として、図12に示すフローチャートを用いて説明する。
ここでは、拡大補間部105の動作について、図13に示すフローチャートを用いて説明する。
以上の処理を行うことで、超解像処理結果が補正される。勾配情報を用いた補正では、例えば画像に含まれる画素の画素値の変化が平坦な部分で発生しているノイズや、勾配の大きな部分、例えばエッジなどで発生しているノイズを抑制することができる。また、許容振幅を用いた補正により、入力画像である低解像度画像の画素の画素値に対して突出したノイズが発生した部分について補正を行うことができる。これらの補正処理により、超解像処理で発生したノイズを抑制し、視覚的に良好な画像を得ることができる。
なお、本発明を上記の実施の形態に基づいて説明してきたが、本発明は、上記の実施の形態に限定されないのはもちろんである。以下のような場合も本発明に含まれる。
この場合に、合成部108は、式18により、第3高解像度画像を生成する。
(式18)
(2)超解像拡大部104は、非特許文献1又は非特許文献2により開示された超解像拡大処理方法を用いるとしてもよい。
102 第3高解像度画像
103 特徴量生成部
104 超解像拡大部
105 拡大補間部
106 差分演算部
107 調整部
108 合成部
201 入力部
202 チューナ
203 外部端子
204 メモリカード
205 メモリ
206 プロセッサ
207 復号部
208 表示制御部
209 ビデオドライバ
210 ハードディスク
211 表示デバイス
Claims (7)
- 低解像度画像から高解像度画像を生成する画像処理装置であって、
低解像度画像を取得する取得手段と、
取得した前記低解像度画像を拡大補間して、前記低解像度画像より高い解像度の第1高解像度画像を生成する拡大補間手段と、
高解像度画像が劣化して低解像度画像となる過程を推定した劣化モデルを逆方向に辿ることにより、前記低解像度画像から、前記第1高解像度画像と同一の解像度を有する第2高解像度画像を生成する超解像処理手段と、
前記低解像度画像を用いて、前記第1高解像度画像の各画素の位置における特徴を示す特徴量を生成する特徴量生成手段と、
前記第1高解像度画像における各画素について、当該画素の値と、対応する前記第2高解像度画像における画素の値の差分値を算出する差分演算手段と、
算出された各画素の差分値を、生成された前記特徴量を用いて補正して、補正差分値を算出する調整手段と、
前記第1高解像度画像の各画素の値に、当該画素に対応する補正差分値を加算して、第3高解像度画像を生成する合成手段と
を備えることを特徴とする画像処理装置。 - 前記特徴量生成手段は、前記特徴量として、前記第1高解像度画像の各画素の周辺画素に対応する前記低解像度画像の画像領域内の画素の値の勾配を示す勾配情報を生成し、
前記調整手段は、生成された前記勾配情報を用いて、前記差分値を補正する
ことを特徴とする請求項1に記載の画像処理装置。 - 前記調整手段は、前記勾配情報に依存して増減する利得を算出し、算出した利得を前記差分値に乗ずることにより、前記補正差分値を算出する
ことを特徴とする請求項2に記載の画像処理装置。 - 前記特徴量解析手段は、前記特徴量として、前記第1高解像度画像の各画素の周辺画素に対応する前記低解像度画像の画像領域から、当該画素の値の取り得る許容範囲を生成し、
前記調整手段は、前記許容範囲を超える値を有する前記第2高解像度画像の各画素について、前記差分値を補正する
ことを特徴とする請求項1又は2に記載の画像処理装置。 - 低解像度画像から高解像度画像を生成する画像処理装置で用いられる画像処理方法であって、
低解像度画像を取得する取得ステップと、
取得した前記低解像度画像を拡大補間して、前記低解像度画像より高い解像度の第1高解像度画像を生成する拡大補間ステップと、
高解像度画像が劣化して低解像度画像となる過程を推定した劣化モデルを逆方向に辿ることにより、前記低解像度画像から、前記第1高解像度画像と同一の解像度を有する第2高解像度画像を生成する超解像処理ステップと、
前記低解像度画像を用いて、前記第1高解像度画像の各画素の位置における特徴を示す特徴量を生成する特徴量生成ステップと、
前記第1高解像度画像における各画素について、当該画素の値と、対応する前記第2高解像度画像における画素の値の差分値を算出する差分演算ステップと、
算出された各画素の差分値を、生成された前記特徴量を用いて補正して、補正差分値を算出する調整ステップと、
前記第1高解像度画像の各画素の値に、当該画素に対応する補正差分値を加算して、第3高解像度画像を生成する合成ステップと
を含むことを特徴とする画像処理方法。 - 低解像度画像から高解像度画像を生成する画像処理装置で用いられる画像処理のためのコンピュータプログラムであって、
コンピュータに、
低解像度画像を取得する取得ステップと、
取得した前記低解像度画像を拡大補間して、前記低解像度画像より高い解像度の第1高解像度画像を生成する拡大補間ステップと、
高解像度画像が劣化して低解像度画像となる過程を推定した劣化モデルを逆方向に辿ることにより、前記低解像度画像から、前記第1高解像度画像と同一の解像度を有する第2高解像度画像を生成する超解像処理ステップと、
前記低解像度画像を用いて、前記第1高解像度画像の各画素の位置における特徴を示す特徴量を生成する特徴量生成ステップと、
前記第1高解像度画像における各画素について、当該画素の値と、対応する前記第2高解像度画像における画素の値の差分値を算出する差分演算ステップと、
算出された各画素の差分値を、生成された前記特徴量を用いて補正して、補正差分値を算出する調整ステップと、
前記第1高解像度画像の各画素の値に、当該画素に対応する補正差分値を加算して、第3高解像度画像を生成する合成ステップと
を実行させるためのコンピュータプログラム。 - 低解像度画像から高解像度画像を生成する画像処理装置で用いられる画像処理のためのコンピュータプログラムを記録しているコンピュータ読み取り可能な記録媒体であって、
コンピュータに、
低解像度画像を取得する取得ステップと、
取得した前記低解像度画像を拡大補間して、前記低解像度画像より高い解像度の第1高解像度画像を生成する拡大補間ステップと、
高解像度画像が劣化して低解像度画像となる過程を推定した劣化モデルを逆方向に辿ることにより、前記低解像度画像から、前記第1高解像度画像と同一の解像度を有する第2高解像度画像を生成する超解像処理ステップと、
前記低解像度画像を用いて、前記第1高解像度画像の各画素の位置における特徴を示す特徴量を生成する特徴量生成ステップと、
前記第1高解像度画像における各画素について、当該画素の値と、対応する前記第2高解像度画像における画素の値の差分値を算出する差分演算ステップと、
算出された各画素の差分値を、生成された前記特徴量を用いて補正して、補正差分値を算出する調整ステップと、
前記第1高解像度画像の各画素の値に、当該画素に対応する補正差分値を加算して、第3高解像度画像を生成する合成ステップと
を実行させるためのコンピュータプログラムを記録している記録媒体。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/983,171 US9153010B2 (en) | 2011-03-28 | 2012-01-31 | Image processing device and image processing method generating a high-resolution image from a low-resolution image |
JP2013507087A JP5837572B2 (ja) | 2011-03-28 | 2012-01-31 | 画像処理装置、画像処理方法、画像処理のためのコンピュータプログラム及び記録媒体 |
CN201280010784.5A CN103403758B (zh) | 2011-03-28 | 2012-01-31 | 图像处理装置、图像处理方法、用于图像处理的计算机程序及记录介质 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011-069668 | 2011-03-28 | ||
JP2011069668 | 2011-03-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012132183A1 true WO2012132183A1 (ja) | 2012-10-04 |
Family
ID=46929980
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/000604 WO2012132183A1 (ja) | 2011-03-28 | 2012-01-31 | 画像処理装置、画像処理方法、画像処理のためのコンピュータプログラム及び記録媒体 |
Country Status (4)
Country | Link |
---|---|
US (1) | US9153010B2 (ja) |
JP (1) | JP5837572B2 (ja) |
CN (1) | CN103403758B (ja) |
WO (1) | WO2012132183A1 (ja) |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102881000B (zh) * | 2012-09-07 | 2016-05-18 | 华为技术有限公司 | 一种视频图像的超分辨率方法、装置和设备 |
US10303945B2 (en) | 2012-12-27 | 2019-05-28 | Panasonic Intellectual Property Corporation Of America | Display method and display apparatus |
US10410398B2 (en) * | 2015-02-20 | 2019-09-10 | Qualcomm Incorporated | Systems and methods for reducing memory bandwidth using low quality tiles |
JP6545013B2 (ja) * | 2015-06-17 | 2019-07-17 | キヤノン株式会社 | 画像形成方法、画像形成装置、および画像形成プログラム |
WO2017104666A1 (ja) * | 2015-12-17 | 2017-06-22 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | 表示方法および表示装置 |
WO2017120266A1 (en) * | 2016-01-08 | 2017-07-13 | Flir Systems, Inc. | Systems and methods for image resolution enhancement |
US9848141B2 (en) * | 2016-05-10 | 2017-12-19 | Semiconductor Components Industries, Llc | Image pixels having processed signal storage capabilities |
JP6668278B2 (ja) * | 2017-02-20 | 2020-03-18 | 株式会社日立ハイテク | 試料観察装置および試料観察方法 |
US10715727B2 (en) * | 2017-05-16 | 2020-07-14 | Apple Inc. | Synthetic long exposure image with optional enhancement using a guide image |
WO2019008692A1 (ja) * | 2017-07-05 | 2019-01-10 | オリンパス株式会社 | 画像処理装置、撮像装置、画像処理方法、画像処理プログラムおよび記憶媒体 |
WO2019008693A1 (ja) * | 2017-07-05 | 2019-01-10 | オリンパス株式会社 | 画像処理装置、撮像装置、画像処理方法、画像処理プログラムおよび記憶媒体 |
US10540749B2 (en) * | 2018-03-29 | 2020-01-21 | Mitsubishi Electric Research Laboratories, Inc. | System and method for learning-based image super-resolution |
CN110070489A (zh) * | 2019-04-30 | 2019-07-30 | 中国人民解放军国防科技大学 | 一种基于视差注意力机制的双目图像超分辨方法 |
CN110889809B9 (zh) * | 2019-11-28 | 2023-06-23 | RealMe重庆移动通信有限公司 | 图像处理方法及装置、电子设备、存储介质 |
WO2021111542A1 (ja) | 2019-12-04 | 2021-06-10 | オリンパス株式会社 | 撮像装置 |
CN111932462B (zh) * | 2020-08-18 | 2023-01-03 | Oppo(重庆)智能科技有限公司 | 图像降质模型的训练方法、装置和电子设备、存储介质 |
CN114612294A (zh) * | 2020-12-08 | 2022-06-10 | 武汉Tcl集团工业研究院有限公司 | 一种图像超分辨率处理方法和计算机设备 |
JP2022117177A (ja) * | 2021-01-29 | 2022-08-10 | 富士フイルム株式会社 | 情報処理装置、情報処理方法及び情報処理プログラム |
CN115578260B (zh) * | 2022-10-08 | 2023-07-25 | 苏州大学 | 针对图像超分辨率的方向解耦的注意力方法和系统 |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001177719A (ja) * | 1999-12-21 | 2001-06-29 | Hitachi Ltd | 画像処理装置と印刷システムおよび画像表示システム |
JP2006222493A (ja) * | 2005-02-08 | 2006-08-24 | Seiko Epson Corp | 複数の低解像度画像を用いた高解像度画像の生成 |
JP2009093442A (ja) * | 2007-10-10 | 2009-04-30 | Hitachi Ltd | 画像信号処理方法、装置、及びそれを用いた画像表示装置 |
WO2009078437A1 (ja) * | 2007-12-17 | 2009-06-25 | Sharp Kabushiki Kaisha | 画像処理装置及び撮像制御装置 |
JP2009217658A (ja) * | 2008-03-12 | 2009-09-24 | Hitachi Ltd | 表示装置および画像処理回路 |
JP2010063088A (ja) * | 2008-08-08 | 2010-03-18 | Sanyo Electric Co Ltd | 撮像装置 |
WO2010073582A1 (ja) * | 2008-12-22 | 2010-07-01 | パナソニック株式会社 | 画像拡大装置、方法、集積回路及びプログラム |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4600011B2 (ja) | 2004-11-29 | 2010-12-15 | ソニー株式会社 | 画像処理装置および方法、記録媒体、並びにプログラム |
US8422788B2 (en) * | 2008-08-26 | 2013-04-16 | Microsoft Corporation | Automatic image straightening |
-
2012
- 2012-01-31 WO PCT/JP2012/000604 patent/WO2012132183A1/ja active Application Filing
- 2012-01-31 JP JP2013507087A patent/JP5837572B2/ja not_active Expired - Fee Related
- 2012-01-31 US US13/983,171 patent/US9153010B2/en not_active Expired - Fee Related
- 2012-01-31 CN CN201280010784.5A patent/CN103403758B/zh not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001177719A (ja) * | 1999-12-21 | 2001-06-29 | Hitachi Ltd | 画像処理装置と印刷システムおよび画像表示システム |
JP2006222493A (ja) * | 2005-02-08 | 2006-08-24 | Seiko Epson Corp | 複数の低解像度画像を用いた高解像度画像の生成 |
JP2009093442A (ja) * | 2007-10-10 | 2009-04-30 | Hitachi Ltd | 画像信号処理方法、装置、及びそれを用いた画像表示装置 |
WO2009078437A1 (ja) * | 2007-12-17 | 2009-06-25 | Sharp Kabushiki Kaisha | 画像処理装置及び撮像制御装置 |
JP2009217658A (ja) * | 2008-03-12 | 2009-09-24 | Hitachi Ltd | 表示装置および画像処理回路 |
JP2010063088A (ja) * | 2008-08-08 | 2010-03-18 | Sanyo Electric Co Ltd | 撮像装置 |
WO2010073582A1 (ja) * | 2008-12-22 | 2010-07-01 | パナソニック株式会社 | 画像拡大装置、方法、集積回路及びプログラム |
Also Published As
Publication number | Publication date |
---|---|
CN103403758A (zh) | 2013-11-20 |
JP5837572B2 (ja) | 2015-12-24 |
US20130308877A1 (en) | 2013-11-21 |
CN103403758B (zh) | 2018-01-05 |
JPWO2012132183A1 (ja) | 2014-07-24 |
US9153010B2 (en) | 2015-10-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5837572B2 (ja) | 画像処理装置、画像処理方法、画像処理のためのコンピュータプログラム及び記録媒体 | |
WO2011033619A1 (ja) | 画像処理装置、画像処理方法、画像処理プログラム、及び、記憶媒体 | |
US20060033936A1 (en) | Resolution-converting apparatus and method | |
US9245316B2 (en) | Image processing device, image processing method and non-transitory computer readable medium | |
JP2008091979A (ja) | 画像高画質化装置、及びその方法、並びに画像表示装置 | |
JP2012060517A (ja) | 画像処理装置、画像処理方法 | |
KR100860968B1 (ko) | 해상도 향상 장치 및 방법 | |
JP2010034964A (ja) | 画像合成装置、画像合成方法及び画像合成プログラム | |
JP2009212969A (ja) | 画像処理装置、画像処理方法、及び画像処理プログラム | |
WO2011018878A1 (ja) | 画像処理システム、画像処理方法および画像処理用プログラム | |
JP5968088B2 (ja) | 画像処理装置、画像処理方法、および、プログラム | |
JP2010212782A (ja) | 画像鮮鋭化装置、画像鮮鋭化方法 | |
JP5316385B2 (ja) | 画像処理装置及び画像処理プログラム | |
JPWO2017203941A1 (ja) | 画像処理装置、および画像処理方法、並びにプログラム | |
US20130120461A1 (en) | Image processor and image processing method | |
US10424055B2 (en) | Image processing apparatus and method for controlling the same | |
US20070121126A1 (en) | Image conversion apparatus and image conversion method | |
JP4598115B2 (ja) | 画像処理方法および装置並びに記録媒体 | |
JP4720626B2 (ja) | 画像処理システム、画像処理方法及びプログラム | |
US20150063726A1 (en) | Image processing method and image processing apparatus | |
JP4714062B2 (ja) | 高解像度画像を生成する画像拡大装置、その画像拡大方法及びその画像拡大プログラムを記録した記録媒体 | |
JP4267159B2 (ja) | 画像処理方法および装置並びに記録媒体 | |
JP4827137B2 (ja) | 解像度変換処理方法、画像処理装置、画像表示装置及びプログラム | |
KR101332030B1 (ko) | 영상 확대방법 및 이를 수행하는 프로그램을 기록한 컴퓨터로 읽을 수 있는 기록매체 | |
JP5410231B2 (ja) | 不等間隔標本化装置、そのプログラム、及び、多次元不等間隔標本化装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12763601 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13983171 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2013507087 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12763601 Country of ref document: EP Kind code of ref document: A1 |