US20160027148A1 - Image enhancement using a patch based technique - Google Patents
Image enhancement using a patch based technique Download PDFInfo
- Publication number
- US20160027148A1 US20160027148A1 US14/341,617 US201414341617A US2016027148A1 US 20160027148 A1 US20160027148 A1 US 20160027148A1 US 201414341617 A US201414341617 A US 201414341617A US 2016027148 A1 US2016027148 A1 US 2016027148A1
- Authority
- US
- United States
- Prior art keywords
- image
- high resolution
- resolution
- images
- patches
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 57
- 238000006731 degradation reaction Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000009499 grossing Methods 0.000 description 3
- 125000001475 halogen functional group Chemical group 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 230000002146 bilateral effect Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 208000004547 Hallucinations Diseases 0.000 description 1
- 241000898323 Microtus irani Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4076—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution using the original low-resolution images to iteratively correct the high-resolution images
-
- G06T5/001—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
Definitions
- the present invention relates to image enhancement using a patch based technique.
- Digital images are typically represented as an array of pixels.
- digital video is typically represented as a series of images or frames, each of which contains an array of pixels.
- Each pixel includes information, such as intensity and/or color information.
- each pixel is represented as a set of three colors, each of which is defined by eight bit color values.
- image and/or video encoding and/or transmission systems degrade the quality of the image content in order to reduce the storage requirements and/or the bandwidth requirements for transmission.
- a restoration technique is used on the image and/or video to attempt to recover the high-quality original image content from the degraded version.
- the degradation of the image content may occur as a result of numerous reasons, such as for example, image transmission, image coding, limitations of capture or display devices, etc.
- the enhancement of a degraded image attempts to improve the appearance of the image and/or video.
- the image content is provided at a first lower resolution, such as a progressive or interlaced scanning (e.g., 720 ⁇ 480 pixels).
- the image content may be provided in a non-degraded manner or in a degraded manner
- the lower resolution image content may be enhanced in some manner to be suitable for displaying on a display having a resolution greater than the lower resolution image content, such as a 4K display (e.g., 3840 by 2160 pixels).
- Restoration and/or enhancement of the image and/or video is often a processing step in an image/video display system, especially in large-sized displays.
- One of the goals may be to restore and enhance the visual appearance of important components of the image and/or video, for example edges, textures and other detail.
- Another goal is to limit the introduction of undesirable visual artifacts and/or amplification of existing artifacts during restoration and enhancement.
- a specific example is to limit the introduction or amplification of existing noise in the image and/or video, such as camera noise or compression noise.
- Another example is to limit introduction of artifacts near edges and contours known as “halo”, “undershoot” and “overshoot”.
- Another technique to perform image detail enhancement involves applying a bilateral filter to get different components of the image under multiple lighting conditions and enhance the details of the image by combining these components.
- the range of the bilateral filter may be modified to simultaneously perform both detail enhancement and noise removal.
- Another technique includes acquiring information about oscillations of the image from local extrema at multiple scales and using this information to build a hierarchy which is used to enhance details of the image.
- Yet another technique involves using wavelets to build a multi-resolution analysis framework to decompose an image into smooth and its detail components, where the wavelets are specifically constructed according to the edge content of the image to reduce halo effects.
- WLS Weighted Least Squares
- g is the input image
- u is the output image
- subscript p is the spatial location of the pixel.
- This function tries to maintain u as close as possible to g and achieves smoothness by minimizing the partial derivatives of u.
- the smoothness weight is determined by a x and a y , while ⁇ controls the amount of smoothing. Greater ⁇ implies more smoothing.
- this technique may be used in a Laplacian pyramid framework to obtain abstraction at different levels.
- Tappen Context constrained hallucination for image super-resolution, CVPR 2010, describe an image super-resolution technique formulated as an energy minimization framework that enforces different criteria such as fidelity of the high-resolution image to the input image, the fidelity of pixel to discrete candidate examples and the smoothness of edge. This method analyzes textural characteristics of a region surrounding each pixel to search database for segments with similar characteristics. All references described herein are incorporated by reference in their entirety.
- FIG. 1 illustrates an image degradation technique
- FIG. 2 illustrates a residual image calculation technique
- FIG. 3 illustrates an input image upsampling technique
- FIG. 4 illustrates a patch determination process
- FIG. 5 illustrates a patch selection process
- FIG. 6 illustrates a high resolution image reconstruction process
- the focus of a person is different for different regions of the image, depending on the content depicted therein.
- the various components of the image content drive people's attention to different regions of the image.
- the texture in the image also draws the attention of a person to particular portions of the image and determines whether the image is visually pleasing, such as for example, highly textured based regions of the image and edge based features of the image.
- the image enhancement technique may be based upon a set of high resolution training images 100 . It is desirable to degrade the high resolution training images in a manner that is similar in nature of the enhancement to be provided to an input image. To achieve a suitable degradation of the high resolution images, each of the high resolution training images may be downsampled to a lower resolution training image 110 . Then each of the downsampled lower resolution training images 110 may be upsampled to an upsampled higher resolution training image 120 , such as the same resolution of the original high resolution training images or otherwise to the anticipated output resolution of the input image. Other statistical measures may likewise be used to suitably degrade the image quality of the high resolution training images 100 .
- a database of high resolution training images may be used.
- the database may primarily consists of high-resolution images.
- the system Given a high resolution training image I_h, the system may generate upsampled higher resolution training images I — 1 as follows:
- the system use the differences (e.g., residue images) between each corresponding high-resolution training image and upsampled higher resolution training images.
- the residue images are used to focus on the relationship between the low-resolution images and the edges and texture content within the corresponding high-resolution images. Using residue images instead of directly using high-resolution images improves the reconstruction results.
- high-resolution is used in general to mean high-quality image and/or video data, for example image data containing high-frequency components, and/or image data containing fine details and textures, and/or image data with sharp edges and contours and/or data with more pixels than the input data.
- low-resolution is used in general to mean image and/or video data that may have been degraded during capture, encoding or transmission, and generally contains fewer high-frequency components, and/or fewer detail and/or compression artifacts and/or fewer pixels than output data.
- the low resolution input image L —I 200 is preferably increased in resolution to a predetermined resolution for subsequent processing.
- the low resolution input image L_I 200 may be upsampled 210 using any suitable technique to determine an intermediate resolution test image T_I 220 .
- the size of the intermediate resolution test image T_I 220 is the same as the size of the upsampled higher resolution training images 120 , although different sizes may be used, if desired.
- the size of the intermediate resolution test image T_I 220 is the same size as the desired output from the upscaling of the low resolution input image L_I 200 .
- patches 250 A- 250 P of the intermediate resolution test image T_I 220 with each of the upsampled higher resolution training images 120 .
- patch 250 F may be matched to each of the upsampled higher resolution training images 120 to determine a best match 260 in each of the higher resolution training images 120 .
- the patches 250 A- 250 P are overlapping with one another so that a reduced number of edges of the patches are in common between any two adjacent patches. Any suitable technique may be used to determine the best match for each of the upsampled higher resolution training images 120 .
- the matching process may use Patchmatch or Coherency Sensitive hashing (CSH).
- CSH Coherency Sensitive hashing
- the CSH technique combines two different techniques, namely, PatchMatch and Locality Sensitive Hashing (LSH) to find matching patches between two images.
- the Patchmatch method uses information about the coherency between the images to propagate good matches to the neighbors. It uses random assignment of patches for initialization.
- LSH uses hashing technique to map similar patches to the same bin in order to find matching patches.
- CSH combines the concepts of image coherency from PatchMatch and the concepts of image appearance from LSH to search for matching patches.
- due to the presence of random initialization of PatchMatch it is beneficial to have some redundancy amongst the patches that are present in the training database.
- the system While finding the best matching patch, the system considers patches from the intermediate resolution training images I_I. This is done in order to maintain the consistency of the degradation process. For every over-lapping patch y present in T_I, the system seeks the best matching patch from each image present in the database.
- the best matched patch for a particular training image will not be very representative of the patch 250 in the intermediate resolution test image T_I 220 .
- it is desirable to determine a similarity measure 300 such as an Euclidean distance measure, between the test patch 250 and the best match patch in each upsampled higher resolution training image 260 . Then the system selects the top ‘k’ patches 310 .
- each of the corresponding residuals 130 for each of the selected top ‘k’ patches 310 it is desirable to select each of the corresponding residuals 130 for each of the selected top ‘k’ patches 310 .
- the residuals 130 belong to the same spatial location in the residue images as the corresponding matched patch in the upsampled higher resolution training image 260 .
- a final estimated residue value 330 of a patch for an image location in the input image may be determined using any statistical measure based upon the selected top ‘k’ patches 310 .
- the overlapping high-resolution patches for the input image L_I 200 are combined together using any suitable technique to get a high-resolution output image.
- This reconstruction constraint may measure the difference between the low resolution input image L_I and the smoothed and the downsampled version of high resolution image I_H. For example, this may be obtained by minimizing the following function:
- G is the spatial filter and is the downsampling operator.
- the system may use a gradient descent technique such as,
- I H t+1 I H t ⁇ (( I H *G ) ⁇ L I ) ⁇ * G,
- G is the spatial filter with kernel standard variance of 1.6 for scaling factor of 4 and ⁇ is the upsampling operator. This is an iterative process and ⁇ is the step size of the gradient descent.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
Description
- None.
- The present invention relates to image enhancement using a patch based technique.
- Digital images are typically represented as an array of pixels. Similarly, digital video is typically represented as a series of images or frames, each of which contains an array of pixels. Each pixel includes information, such as intensity and/or color information. In many cases, each pixel is represented as a set of three colors, each of which is defined by eight bit color values.
- In many cases, image and/or video encoding and/or transmission systems degrade the quality of the image content in order to reduce the storage requirements and/or the bandwidth requirements for transmission. After encoding and/or transmitting an image and/or video, a restoration technique is used on the image and/or video to attempt to recover the high-quality original image content from the degraded version. The degradation of the image content may occur as a result of numerous reasons, such as for example, image transmission, image coding, limitations of capture or display devices, etc. On the other hand, the enhancement of a degraded image attempts to improve the appearance of the image and/or video.
- In other cases, the image content is provided at a first lower resolution, such as a progressive or interlaced scanning (e.g., 720×480 pixels). The image content may be provided in a non-degraded manner or in a degraded manner The lower resolution image content may be enhanced in some manner to be suitable for displaying on a display having a resolution greater than the lower resolution image content, such as a 4K display (e.g., 3840 by 2160 pixels).
- Restoration and/or enhancement of the image and/or video is often a processing step in an image/video display system, especially in large-sized displays. One of the goals may be to restore and enhance the visual appearance of important components of the image and/or video, for example edges, textures and other detail. Another goal is to limit the introduction of undesirable visual artifacts and/or amplification of existing artifacts during restoration and enhancement. A specific example is to limit the introduction or amplification of existing noise in the image and/or video, such as camera noise or compression noise. Another example is to limit introduction of artifacts near edges and contours known as “halo”, “undershoot” and “overshoot”.
- Many different techniques have been used to attempt to perform image (inclusive of video) detail enhancement in order to restore an image. Many such techniques are based upon a hierarchical framework using a Laplacian pyramid to decompose the image into multiple levels, including a smooth low frequency image and other high frequency components. Each level is then enhanced and combined together to form the enhanced image. While decomposing the image, edge preservation techniques may be used to reduce halo effects.
- Another technique to perform image detail enhancement involves applying a bilateral filter to get different components of the image under multiple lighting conditions and enhance the details of the image by combining these components. The range of the bilateral filter may be modified to simultaneously perform both detail enhancement and noise removal. Another technique includes acquiring information about oscillations of the image from local extrema at multiple scales and using this information to build a hierarchy which is used to enhance details of the image. Yet another technique involves using wavelets to build a multi-resolution analysis framework to decompose an image into smooth and its detail components, where the wavelets are specifically constructed according to the edge content of the image to reduce halo effects.
- Another technique to perform image detail enhancement uses a filter to perform multi-scale decomposition of images. The filter is edge-preserving and the smoothing is based on a Weighted Least Squares (i.e., WLS) optimization framework. This may be mathematically represented as calculating the minimum of,
-
- where g is the input image, u is the output image and subscript p is the spatial location of the pixel. This function tries to maintain u as close as possible to g and achieves smoothness by minimizing the partial derivatives of u. The smoothness weight is determined by ax and ay, while λ controls the amount of smoothing. Greater λ implies more smoothing. For example, this technique may be used in a Laplacian pyramid framework to obtain abstraction at different levels.
- As previously described there are many different techniques to provide image enhancement together with increased resolution. For example, D. Glasner, S. Bagon, M. Irani, Super-resolution from a single image, ICCV 2009, describe the use of redundancies in the input image to construct a pyramid having low-res/high-res image pairs and uses a learning-based method to perform super-resolution of the input image. For example, J. Sun, J. Sun, Z. Xu, H. Y. Shum, Gradient Profile Prior, CVPR 2008, describe the use of a large database of natural images to learn the distribution of gradient profiles and modifies the gradient information of the input image to fit this distribution in order to obtain sharp edges and consequently perform super-resolution of images. For example, Yang, J. Wright, T. Huang and Y. Ma., Image super-resolution via sparse representation, IEEE TIP 2010, describe the use of a dictionary-based super-resolution method based on ideas in sparse signal processing and show how a joint compact dictionary can be trained to learn the correspondence between high-resolution and low-resolution training image patches. H. He, W. C. Siu., Single image super-resolution using Gaussian Process Regression, CVPR 2010, describe a super-resolution technique using a Gaussian Process Regression model without any training dataset. R. Fattal, Upsampling via imposed edge statistics, SIGGRAPH 2007, describe the use of a super-resolution method based on the relationship between the edge statistics based on local intensity continuity of low-resolution and the high-resolution images. W. Freeman, T. Jones, E. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications 2002, describe using a technique to hallucinate high frequency details from a training set of high-resolution and low-resolution image pairs. Y. W. Tai, S. Liu, M. Brown, S. Lin, Super resolution using edge prior and single image detail synthesis, CVPR 2010, describe the use of an extended gradient profile technique using exemplar texture patches to get improved details in the image. J. Sun, J. Zhu, M. Tappen, Context constrained hallucination for image super-resolution, CVPR 2010, describe an image super-resolution technique formulated as an energy minimization framework that enforces different criteria such as fidelity of the high-resolution image to the input image, the fidelity of pixel to discrete candidate examples and the smoothness of edge. This method analyzes textural characteristics of a region surrounding each pixel to search database for segments with similar characteristics. All references described herein are incorporated by reference in their entirety.
- Many existing techniques for detail enhancement, such as those mentioned above, are effective in enhancing the image and/or video. However, such techniques may still result in images that are not aesthetically pleasing to the viewer.
- The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
-
FIG. 1 illustrates an image degradation technique. -
FIG. 2 illustrates a residual image calculation technique. -
FIG. 3 illustrates an input image upsampling technique. -
FIG. 4 illustrates a patch determination process. -
FIG. 5 illustrates a patch selection process. -
FIG. 6 illustrates a high resolution image reconstruction process. - While viewing an image or a video, the focus of a person is different for different regions of the image, depending on the content depicted therein. The various components of the image content drive people's attention to different regions of the image. In addition, the texture in the image also draws the attention of a person to particular portions of the image and determines whether the image is visually pleasing, such as for example, highly textured based regions of the image and edge based features of the image.
- Referring to
FIG. 1 , the image enhancement technique may be based upon a set of highresolution training images 100. It is desirable to degrade the high resolution training images in a manner that is similar in nature of the enhancement to be provided to an input image. To achieve a suitable degradation of the high resolution images, each of the high resolution training images may be downsampled to a lowerresolution training image 110. Then each of the downsampled lowerresolution training images 110 may be upsampled to an upsampled higher resolution training image 120, such as the same resolution of the original high resolution training images or otherwise to the anticipated output resolution of the input image. Other statistical measures may likewise be used to suitably degrade the image quality of the highresolution training images 100. - Referring to
FIG. 2 , it was determined that to maintain and/or emphasize the texture and edge based aspects of a lower resolution input image when its resolution is substantially increased, it is desirable to base the upscaling on a set ofresiduals 130 between the upsampled higher resolution training images 120 and the highresolution training images 100. Accordingly, it is desirable to determine the difference 140 between the pixels of each the upsampled higher resolution training images 120 and the corresponding highresolution training images 100. Other statistical measures may be used to characterize the relationship between the high resolution training images and corresponding degraded higher resolution training images. - In one embodiment, a database of high resolution training images may be used. The database may primarily consists of high-resolution images. Given a high resolution training image I_h, the system may generate upsampled higher resolution training images I—1 as follows:
-
I l=((I h *G)↓)↑, (1) - where * is the convolution operator, G is a Gaussian kernel, ↓ is the downsampling operator and ↑ is the upsampling operator. The downsampling and the upsampling operator may be performed with the desired scaling factor and may be used to mimic the degradation process. Instead of using the high-resolution images during training, the system use the differences (e.g., residue images) between each corresponding high-resolution training image and upsampled higher resolution training images. Residue images for training may be denoted by R=I_h−I—1. The residue images are used to focus on the relationship between the low-resolution images and the edges and texture content within the corresponding high-resolution images. Using residue images instead of directly using high-resolution images improves the reconstruction results.
- The term “high-resolution” is used in general to mean high-quality image and/or video data, for example image data containing high-frequency components, and/or image data containing fine details and textures, and/or image data with sharp edges and contours and/or data with more pixels than the input data. The term “low-resolution” is used in general to mean image and/or video data that may have been degraded during capture, encoding or transmission, and generally contains fewer high-frequency components, and/or fewer detail and/or compression artifacts and/or fewer pixels than output data.
- Referring to
FIG. 3 , the low resolutioninput image L —I 200 is preferably increased in resolution to a predetermined resolution for subsequent processing. The low resolutioninput image L_I 200 may be upsampled 210 using any suitable technique to determine an intermediate resolutiontest image T_I 220. Preferably, the size of the intermediate resolutiontest image T_I 220 is the same as the size of the upsampled higher resolution training images 120, although different sizes may be used, if desired. In addition, preferably the size of the intermediate resolutiontest image T_I 220 is the same size as the desired output from the upscaling of the low resolutioninput image L_I 200. - Referring to
FIG. 4 , it is desirable to match patches 250A-250P of the intermediate resolutiontest image T_I 220 with each of the upsampled higher resolution training images 120. For example, patch 250F may be matched to each of the upsampled higher resolution training images 120 to determine abest match 260 in each of the higher resolution training images 120. Preferably, the patches 250A-250P are overlapping with one another so that a reduced number of edges of the patches are in common between any two adjacent patches. Any suitable technique may be used to determine the best match for each of the upsampled higher resolution training images 120. - For example, the matching process may use Patchmatch or Coherency Sensitive hashing (CSH). The CSH technique combines two different techniques, namely, PatchMatch and Locality Sensitive Hashing (LSH) to find matching patches between two images. The Patchmatch method uses information about the coherency between the images to propagate good matches to the neighbors. It uses random assignment of patches for initialization. LSH uses hashing technique to map similar patches to the same bin in order to find matching patches. CSH combines the concepts of image coherency from PatchMatch and the concepts of image appearance from LSH to search for matching patches. However, due to the presence of random initialization of PatchMatch, it is beneficial to have some redundancy amongst the patches that are present in the training database. While finding the best matching patch, the system considers patches from the intermediate resolution training images I_I. This is done in order to maintain the consistency of the degradation process. For every over-lapping patch y present in T_I, the system seeks the best matching patch from each image present in the database.
- Referring to
FIG. 5 , for some training images even the best matched patch for a particular training image will not be very representative of thepatch 250 in the intermediate resolutiontest image T_I 220. To reduce the number of poor matches, it is desirable to determine asimilarity measure 300, such as an Euclidean distance measure, between thetest patch 250 and the best match patch in each upsampled higherresolution training image 260. Then the system selects the top ‘k’patches 310. - Referring to
FIG. 6 , it is desirable to select each of thecorresponding residuals 130 for each of the selected top ‘k’patches 310. Theresiduals 130 belong to the same spatial location in the residue images as the corresponding matched patch in the upsampled higherresolution training image 260. A final estimatedresidue value 330 of a patch for an image location in the input image may be determined using any statistical measure based upon the selected top ‘k’patches 310. - For example, the final estimation of the
final residue value 330 of a patch for an image location may be obtained as the convex combination of residue patches given in the form: {circumflex over (r)}=Σi=1, . . . , k wiri - where wi=(di)−1/Σj=1, . . . ,k(dj)−1 and di=∥y−xi∥2 for i=1, . . . ,k and {d—1, . . . , d_k} are the estimation of distances between the input patch and the top k best matching patches from the database. This estimation may be calculated using an Euclidean distance.
- The estimated
residual value 330 is then combined 340 with thetest patch 250 in some manner, such as by summation, to obtain a high resolution patch, such as, ŷ=y+{circumflex over (r)}. The overlapping high-resolution patches for theinput image L_I 200 are combined together using any suitable technique to get a high-resolution output image. - Because the input low resolution image and the output high resolution image obtained by overlapping high resolution patches do not demand exact equality it is desirable to enforce a
global reconstruction constraint 350. This reconstruction constraint may measure the difference between the low resolution input image L_I and the smoothed and the downsampled version of high resolution image I_H. For example, this may be obtained by minimizing the following function: -
E(I H /L I)=|(I H *G)↓−L I|2, - where G is the spatial filter and is the downsampling operator. To minimize the above function, the system may use a gradient descent technique such as,
-
I H t+1 =I H t−τ((I H *G)↓−L I)↑*G, - where G is the spatial filter with kernel standard variance of 1.6 for scaling factor of 4 and ↓ is the upsampling operator. This is an iterative process and τ is the step size of the gradient descent.
- The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
Claims (17)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/341,617 US9256920B1 (en) | 2014-07-25 | 2014-07-25 | Image enhancement using a patch based technique |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/341,617 US9256920B1 (en) | 2014-07-25 | 2014-07-25 | Image enhancement using a patch based technique |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160027148A1 true US20160027148A1 (en) | 2016-01-28 |
US9256920B1 US9256920B1 (en) | 2016-02-09 |
Family
ID=55167101
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/341,617 Expired - Fee Related US9256920B1 (en) | 2014-07-25 | 2014-07-25 | Image enhancement using a patch based technique |
Country Status (1)
Country | Link |
---|---|
US (1) | US9256920B1 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150332435A1 (en) * | 2014-05-16 | 2015-11-19 | Naoki MOTOHASHI | Image processing apparatus, image processing method, and computer-readable recording medium |
US20160328630A1 (en) * | 2015-05-08 | 2016-11-10 | Samsung Electronics Co., Ltd. | Object recognition apparatus and method |
WO2019147693A1 (en) * | 2018-01-23 | 2019-08-01 | Insurance Services Office, Inc. | Computer vision systems and methods for machine learning using image hallucinations |
US10909700B2 (en) * | 2019-04-02 | 2021-02-02 | Samsung Electronics Co., Ltd. | Display apparatus and image processing method thereof |
US10929952B2 (en) * | 2016-01-19 | 2021-02-23 | Google Llc | Image upscaling |
US11336943B2 (en) * | 2020-02-24 | 2022-05-17 | Realtek Semiconductor Corporation | Method and electronic device for detecting resolution |
CN114529464A (en) * | 2022-01-14 | 2022-05-24 | 电子科技大学 | Underwater image recovery method based on deep learning |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9799102B2 (en) * | 2015-12-02 | 2017-10-24 | Adobe Systems Incorporated | Smoothing images using machine learning |
US10410084B2 (en) | 2016-10-26 | 2019-09-10 | Canon Virginia, Inc. | Devices, systems, and methods for anomaly detection |
US10997712B2 (en) | 2018-01-18 | 2021-05-04 | Canon Virginia, Inc. | Devices, systems, and methods for anchor-point-enabled multi-scale subfield alignment |
US10997462B2 (en) | 2018-04-04 | 2021-05-04 | Canon Virginia, Inc. | Devices, systems, and methods for clustering reference images for non-destructive testing |
US11429806B2 (en) | 2018-11-09 | 2022-08-30 | Canon Virginia, Inc. | Devices, systems, and methods for anomaly detection |
US11321846B2 (en) | 2019-03-28 | 2022-05-03 | Canon Virginia, Inc. | Devices, systems, and methods for topological normalization for anomaly detection |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7218796B2 (en) * | 2003-04-30 | 2007-05-15 | Microsoft Corporation | Patch-based video super-resolution |
US7715658B2 (en) * | 2005-08-03 | 2010-05-11 | Samsung Electronics Co., Ltd. | Apparatus and method for super-resolution enhancement processing |
JP5706177B2 (en) * | 2010-02-09 | 2015-04-22 | パナソニック インテレクチュアル プロパティ コーポレーション オブアメリカPanasonic Intellectual Property Corporation of America | Super-resolution processing apparatus and super-resolution processing method |
US9324133B2 (en) * | 2012-01-04 | 2016-04-26 | Sharp Laboratories Of America, Inc. | Image content enhancement using a dictionary technique |
US8620073B2 (en) * | 2012-02-24 | 2013-12-31 | Mitsubishi Electric Research Laboratories, Inc. | Upscaling natural images |
US9117262B2 (en) * | 2012-11-30 | 2015-08-25 | Adobe Systems Incorporated | Learned piece-wise patch regression for image enhancement |
US9123138B2 (en) * | 2013-06-18 | 2015-09-01 | Adobe Systems Incorporated | Adaptive patch-based image upscaling |
US9208539B2 (en) * | 2013-11-30 | 2015-12-08 | Sharp Laboratories Of America, Inc. | Image enhancement using semantic components |
-
2014
- 2014-07-25 US US14/341,617 patent/US9256920B1/en not_active Expired - Fee Related
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150332435A1 (en) * | 2014-05-16 | 2015-11-19 | Naoki MOTOHASHI | Image processing apparatus, image processing method, and computer-readable recording medium |
US9697583B2 (en) * | 2014-05-16 | 2017-07-04 | Ricoh Company, Ltd. | Image processing apparatus, image processing method, and computer-readable recording medium |
US20160328630A1 (en) * | 2015-05-08 | 2016-11-10 | Samsung Electronics Co., Ltd. | Object recognition apparatus and method |
US9940539B2 (en) * | 2015-05-08 | 2018-04-10 | Samsung Electronics Co., Ltd. | Object recognition apparatus and method |
US10929952B2 (en) * | 2016-01-19 | 2021-02-23 | Google Llc | Image upscaling |
WO2019147693A1 (en) * | 2018-01-23 | 2019-08-01 | Insurance Services Office, Inc. | Computer vision systems and methods for machine learning using image hallucinations |
US10902294B2 (en) | 2018-01-23 | 2021-01-26 | Insurance Services Office, Inc. | Computer vision systems and methods for machine learning using image hallucinations |
US10909700B2 (en) * | 2019-04-02 | 2021-02-02 | Samsung Electronics Co., Ltd. | Display apparatus and image processing method thereof |
US11336943B2 (en) * | 2020-02-24 | 2022-05-17 | Realtek Semiconductor Corporation | Method and electronic device for detecting resolution |
CN114529464A (en) * | 2022-01-14 | 2022-05-24 | 电子科技大学 | Underwater image recovery method based on deep learning |
Also Published As
Publication number | Publication date |
---|---|
US9256920B1 (en) | 2016-02-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9256920B1 (en) | Image enhancement using a patch based technique | |
US9734566B2 (en) | Image enhancement using semantic components | |
CN111047516B (en) | Image processing method, image processing device, computer equipment and storage medium | |
Wang et al. | Robust object removal with an exemplar-based image inpainting approach | |
US8675999B1 (en) | Apparatus, system, and method for multi-patch based super-resolution from an image | |
CN102902961B (en) | Face super-resolution processing method based on K neighbor sparse coding average value constraint | |
Singh et al. | Super-resolving noisy images | |
US9589323B1 (en) | Super resolution image enhancement technique | |
US20120328210A1 (en) | Method and system for generating an output image of increased pixel resolution from an input image | |
US20160335748A1 (en) | Method for inpainting a target area in a target video | |
US20230051960A1 (en) | Coding scheme for video data using down-sampling/up-sampling and non-linear filter for depth map | |
Fang et al. | Rapid image completion system using multiresolution patch-based directional and nondirectional approaches | |
CN114170088A (en) | Relational reinforcement learning system and method based on graph structure data | |
Zhang et al. | Deep tone mapping network in HSV color space | |
US20130084025A1 (en) | Method for Brightness Correction of Defective Pixels of Digital Monochrome Image | |
Kang et al. | Self-learning-based single image super-resolution of a highly compressed image | |
Chudasama et al. | RSRGAN: computationally efficient real-world single image super-resolution using generative adversarial network | |
Lin et al. | No-reference quality assessment for low-light image enhancement: Subjective and objective methods | |
Zhang et al. | BE-ACGAN: Photo-realistic residual bit-depth enhancement by advanced conditional GAN | |
Imani et al. | Pansharpening optimisation using multiresolution analysis and sparse representation | |
Liu et al. | Arbitrary-scale super-resolution via deep learning: A comprehensive survey | |
Hesabi et al. | Structure and texture image inpainting | |
Hesabi et al. | A modified patch propagation-based image inpainting using patch sparsity | |
Zhou et al. | Improving low-resolution image classification by super-resolution with enhancing high-frequency content | |
Kang et al. | Self-learning-based low-quality single image super-resolution |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SHARP LABORATORIES OF AMERICA, INC., WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CHOUDHURY, ANUSTUP KUMAR;REEL/FRAME:033623/0620 Effective date: 20140822 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SHARP KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHARP LABORATORIES OF AMERICA, INC.;REEL/FRAME:037838/0570 Effective date: 20160226 |
|
CC | Certificate of correction | ||
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20200209 |