KR101723738B1 - Apparatus and method for resolution enhancement based on dictionary learning - Google Patents

Apparatus and method for resolution enhancement based on dictionary learning Download PDF

Info

Publication number
KR101723738B1
KR101723738B1 KR1020150118114A KR20150118114A KR101723738B1 KR 101723738 B1 KR101723738 B1 KR 101723738B1 KR 1020150118114 A KR1020150118114 A KR 1020150118114A KR 20150118114 A KR20150118114 A KR 20150118114A KR 101723738 B1 KR101723738 B1 KR 101723738B1
Authority
KR
South Korea
Prior art keywords
resolution
dimensional
image
patch
light field
Prior art date
Application number
KR1020150118114A
Other languages
Korean (ko)
Other versions
KR20170022736A (en
Inventor
박인규
이승재
Original Assignee
인하대학교 산학협력단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 인하대학교 산학협력단 filed Critical 인하대학교 산학협력단
Priority to KR1020150118114A priority Critical patent/KR101723738B1/en
Publication of KR20170022736A publication Critical patent/KR20170022736A/en
Application granted granted Critical
Publication of KR101723738B1 publication Critical patent/KR101723738B1/en

Links

Images

Classifications

    • H04N13/0003—
    • H04N13/0007—
    • H04N13/0018—
    • H04N13/0029—

Landscapes

  • Image Processing (AREA)

Abstract

A dictionary learning based resolution enhancement apparatus and method are disclosed. A method of improving resolution includes: defining an input image obtained through a microlens array and a two-dimensional image sensor as an initial high-resolution image, and converting the initial high-resolution image into a high-resolution four-dimensional light field image; Generating a low-resolution four-dimensional light field image having a lower resolution than the converted high-resolution four-dimensional light field image; Generating a 4-dimensional high resolution patch (HR patch) and a 4-dimensional low resolution patch (LR patch) using the high resolution 4-dimensional light field image and the low resolution 4-dimensional light field image; Learning a dictionary using the generated 4-dimensional high resolution patch and 4-dimensional low resolution patch; And improving the resolution of the target image using the learned dictionary.

Description

[0001] APPARATUS AND METHOD FOR RESOLUTION ENHANCEMENT BASED ON DICTIONARY LEARNING [0002]

The present invention relates to a dictionary learning-based resolution enhancement apparatus and method, and more particularly, to a dictionary learning enhancement apparatus and method for enhancing resolution of a target image using a dictionary learning based super resolution algorithm suitable for a 4-dimensional light field image Apparatus and method.

The 4-dimensional light field image generated by acquiring the amount of light traveling in various directions in space includes the direction information of the light ray in comparison with the existing two-dimensional image. Therefore, the 4-D light field image can perform various image processing such as re-focus image and 3-dimensional depth information estimation by using such information. The information of the light field image has the resolution of the spatial domain of the existing two-dimensional image and the resolution of the angular domain having the information of the direction. However, due to these characteristics, the biggest disadvantage of the light field image occurs. That is, since the four-dimensional information is acquired by the two-dimensional sensor in the image acquisition step, the light field image has a resolution limitation due to the trade-off between the resolution of the spatial domain and the resolution of each domain. These correlations lead to a reduction in resolution in commercial light field acquisition equipment with limited number of lenses and lens size. That is, as the resolution of each domain increases with a limited amount of information, the resolution of the spatial domain is inevitably reduced. This results in lowering user satisfaction in various application programs such as refocusing images and 3D depth information estimation.

Among the resolution enhancement algorithms to solve this problem, the super resolution algorithm has a merit that the execution time is longer than that of the general linear interpolation method, but the accuracy and precision are high. There are two types of super resolution algorithms in 2D images. The first is a multi-image-based super resolution algorithm that improves the resolution of a single image using multiple images. The second one is the resolution of the input image using the information stored in the single input image and the dictionary through learning. Resolution super-resolution algorithm.

The present invention proposes a single-image-based super resolution algorithm for improving the resolution of each sub-image of a light field image without application of a single resolution enhancement using a plurality of images and applying it to various fields in the future.

The present invention provides an apparatus and method for improving the resolution of a target image by using a dictionary learning-based super resolution algorithm suitable for a 4-dimensional light field image.

According to an embodiment of the present invention, an input image obtained through a microlens array and a 2D image sensor is defined as an initial high-resolution image, and the initial high-resolution image is converted into a high-resolution 4-dimensional light field image Converting; Generating a low-resolution four-dimensional light field image having a lower resolution than the converted high-resolution four-dimensional light field image; Generating a 4-dimensional high resolution patch (HR patch) and a 4-dimensional low resolution patch (LR patch) using the high resolution 4-dimensional light field image and the low resolution 4-dimensional light field image; Learning a dictionary using the generated 4-dimensional high resolution patch and 4-dimensional low resolution patch; And improving the resolution of the target image using the learned dictionary.

The step of generating the 4-dimensional high-resolution patch (HR patch) and the 4-dimensional low-resolution patch (LR patch) may include a step of generating a high-resolution 4-dimensional light field image and a low- Generating a set of patches at the same position in a spatial domain; And storing the set of generated patches in the form of column vectors to extract a four-dimensional high-resolution patch and a four-dimensional low-resolution patch.

Wherein the learning step comprises: applying a clustering operation to the generated four-dimensional low resolution patch; Extracting a four-dimensional center vector corresponding to a cluster determined by applying the clustering operation; Confirming a regression coefficient corresponding to the determined cluster using the generated 4-dimensional high-resolution patch and 4-dimensional low-resolution patch; And storing the four-dimensional center vector and the regression coefficient corresponding to the determined cluster as a component of the dictionary.

The step of enhancing the resolution of the target image may include extracting a four-dimensional patch vector of the target image; Determining a cluster having a four-dimensional center vector corresponding to the extracted four-dimensional patch vector using the learned dictionary; And improving the resolution of the target image using a regression coefficient corresponding to the determined cluster.

The step of improving the resolution of the target image may include: evenly dividing the target image into a plurality of regions; Enhancing resolution of each of the uniformly divided regions; And reconstructing a high-resolution target image by merging a plurality of regions having the improved resolution.

The resolution enhancement apparatus according to an embodiment of the present invention defines an input image obtained through a microlens array and a two-dimensional image sensor as an initial high-resolution image, and converts the initial high-resolution image into a high-resolution 4-dimensional light field image An image converting unit for converting the image; An image generator for generating a low-resolution four-dimensional light field image having a lower resolution than the converted high-resolution four-dimensional light field image; A patch generator for generating a 4-dimensional high resolution patch (HR patch) and a 4-dimensional low resolution patch (LR patch) using the high resolution 4-dimensional light field image and the low resolution light field image; A dictionary learning unit for learning a dictionary using the generated 4-dimensional high-resolution patch and 4-dimensional low-resolution patch; And a resolution enhancement unit for improving the resolution of the target image using the learned dictionary.

The patch generation unit generates a set of patches existing at the same position in a spatial domain with respect to the absurd open images constituting the high-resolution 4-dimensional light field image and the low-resolution 4-dimensional light field image, A set of patches can be stored in the form of column vectors to extract four-dimensional high-resolution patches and four-dimensional low-resolution patches.

The dictionary learning unit applies a clustering operation to the generated four-dimensional low-resolution patch, extracts a four-dimensional center vector corresponding to the determined clustering operation by applying the clustering operation, and generates the generated four- A 4-dimensional center vector and a regression coefficient corresponding to the determined cluster may be stored as a component of the dictionary by checking a regression coefficient corresponding to the determined cluster using a low resolution patch.

Wherein the resolution enhancement unit extracts a four-dimensional patch vector of the target image, determines a cluster having a four-dimensional center vector corresponding to the extracted four-dimensional patch vector using the learned dictionary, The resolution of the target image can be improved by using the regression coefficient.

The resolution enhancement unit may uniformly divide the target image into a plurality of regions, improve resolution of each of the uniformly divided regions, and reconstruct a high-resolution target image by merging a plurality of regions having the improved resolution .

According to an embodiment of the present invention, resolution of a target image can be improved by using a dictionary learning-based super resolution algorithm suitable for a 4-dimensional light field image.

1 is a diagram illustrating a resolution enhancement apparatus according to an embodiment of the present invention.
2 is a diagram showing a result of converting raw data acquired through a commercial light field camera into a four-dimensional light field image according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating generation of a 4-dimensional high-resolution patch and a 4-dimensional low-resolution patch according to an embodiment of the present invention.
4 is a diagram illustrating a dictionary learning process according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a method of dividing a target image in order to unify a resolution of a dictionary and a resolution of a target image according to an embodiment of the present invention.
6 is a diagram illustrating an example of improving a resolution by applying a post-processing algorithm to a target image according to an exemplary embodiment of the present invention.
FIG. 7 is a diagram illustrating a method of a post-processing algorithm applied to a target image according to an embodiment of the present invention.
8 is a diagram illustrating a resolution improvement result in a spatial domain according to an embodiment of the present invention.
FIG. 9 is a diagram illustrating a four-dimensional light field image photographed in various environments according to an embodiment of the present invention.
FIG. 10 is a diagram illustrating a qualitative example of a resolution improvement result according to an embodiment of the present invention.
11 is a diagram illustrating a resolution enhancement method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a resolution enhancement apparatus according to an embodiment of the present invention.

The resolution enhancement apparatus 100 may include an image conversion unit 110, an image generation unit 120, a patch generation unit 130, a dictionary learning unit 140, and a resolution enhancement unit 150. The image converting unit 110 can convert various real-world input images acquired through the microlens array and the two-dimensional image sensor into a high-resolution four-dimensional light field image. At this time, various real-world input images acquired through the microlens array and the two-dimensional image sensor can be defined as the initial high-resolution images.

A commercial light field camera with a microlens arrangement allows light rays coming through the main lens to pass through a small lens array and be recorded at different positions of the sensor by the direction of light. As a result, a commercial light field camera can store an image in an initial data format that can not be directly used as shown in FIG. 2 (a). 2 (a) is enlarged, it can be confirmed that it is composed of a set of small micro images. Therefore, it may be necessary to transform the coordinate system so that the raw data shown in FIG. 2 (a) is suitable for the experiment.

To this end, the image transform unit 110 may correct the color information of the original data as shown in FIG. 2A and classify the light according to the directions to generate the sub-aperture images that can be directly used have. 2 (b), the image transform unit 110 combines the absurd open images into a single image organized by each domain (angular domain) to represent a 4-dimensional light field image And can develop a super resolution algorithm using it. In the present invention, the coordinate system of each angular domain is denoted by u and v, and the coordinate system of the spatial domain is denoted by x and y.

The image generation unit 120 lowers the resolution of the high resolution 4-dimensional light field image acquired through the image conversion unit 110 by half in both the spatial domain and the angular domain, Can be generated.

The patch generator 130 generates a 4-dimensional high resolution patch (HR patch) 350 and a 4-dimensional low resolution patch (LR patch) 360 using the 4-dimensional light field images 330 and 340 as shown in FIG. Respectively. Since the object of the present invention is to improve both the spatial domain resolution and the resolution of each domain constituting the 4-dimensional light field image, the patch has a 4-dimensional structure including information of the spatial domain and information of each domain Lt; / RTI >

The patch generation unit 130 can generate a set of two-dimensional patches existing in the same position in the spatial domain with respect to a plurality of absurd open images constituting the four-dimensional light field images 330 and 340. [ In addition, the patch generator 130 stores the generated set of patches in the form of column vectors in a form of column vectors so that mathematical operations can be easily performed in the future reconstruction step, thereby generating a 4-dimensional high resolution patch 350 and a 4-dimensional low resolution patch 360 Can be extracted.

The dictionary learning unit 140 can learn a dictionary using the 4-dimensional high-resolution patch and the 4-dimensional low-resolution patch generated by the patch generation unit 130. [

In order to define the relation between two patches using 4-dimensional high-resolution patches and 4-dimensional low-resolution patches generated based on various real-world input images acquired by commercial light field cameras and to apply them to the resolution restoration step of an efficient dictionary, And learning algorithms may be required. At this time, since a patch is generated in various image coordinates for various input images of the real world, a very large amount of patches may exist. However, all of these can be inefficient in terms of processing speed and memory usage due to repetitive patches and meaningless patches.

In order to solve such a problem, the present invention provides a method of grouping similar patches into a single cluster using a K-means clustering algorithm in the configuration and learning stages of the dictionary. Since the K-average clustering algorithm can not achieve good results if the amount of data to be clustered is small or the kind of data is limited, the construction and learning of the dictionary are performed with a large number of images acquired in various environments.

The dictionary learning step may be configured as shown in FIG. The dictionary learning unit 140 uses the four-dimensional low-resolution patches generated by the patch generation unit 130 as input vectors of the K-average clustering algorithm 410, and iteratively clusters the K clusters and corresponding centers A cluster center 420 may be extracted. Also belonging to each cluster

Figure 112015081389302-pat00001
Dimensional low-resolution patch vectors,
Figure 112015081389302-pat00002
Resolution patch vectors are used to form a matrix < RTI ID = 0.0 >
Figure 112015081389302-pat00003
. ≪ / RTI > here
Figure 112015081389302-pat00004
Denotes the index of the cluster, and n and m denote the dimensions of the 4-dimensional high-resolution patch vector and the 4-dimensional low-resolution patch vector, respectively. Finally, the linear approximation of the low-resolution patch matrix and the high-resolution patch matrix is performed by the least-squares method as shown in Equation 1 to obtain a regression coefficient 430 matrix
Figure 112015081389302-pat00005
Can be obtained.

Equation 1

Figure 112015081389302-pat00006

The dictionary learning unit 140 may then store the center vector 420 for each cluster and the regression coefficient 430 of the corresponding cluster together as a component of the dictionary 440. At this time, the center vector 420 for each cluster to be stored is compared with a 4-dimensional low resolution patch for a target image in the future, and the regression coefficient 430 can be used to improve the resolution of the target image after the comparison.

The resolution enhancing unit 150 can improve the resolution of the target image using the dictionary 440 learned through the dictionary learning unit 140. [ The resolution enhancement apparatus 100 of the present invention converts a high-resolution 4-dimensional light field image acquired by a commercial light field camera into a low-resolution four-dimensional light field image by reducing the resolution in each of the spatial domain and each domain by half, 4-dimensional low resolution patches were generated. However, in the present invention, it is aimed to improve the resolution of the target image acquired by the commercial light field camera, so that the resolution used by the dictionary learning unit 140 and the resolution used by the resolution enhancing unit 150 differ. Therefore, according to the present invention, even if the same feature exists in one image, there is a feature of various scales, and even if a 4-dimensional low resolution patch and a 4-dimensional high resolution patch are learned for a specific scale, The algorithm is designed based on.

For this purpose, the resolution enhancement unit 150 may equally divide the target image into a plurality of regions. That is, in order to use the dictionary learning result for improving the resolution of the target image, it is possible to make the resolution of the target image coincide with the resolution of the low resolution image of the dictionary through image segmentation. At this time, the target image needs a 4-dimensional division process. As shown in FIG. 5, the target image can be divided into half in the horizontal direction and in the vertical direction in each absurd image in the spatial domain. In addition, the resolution enhancing unit 150 may divide the segment of the absurd divided images into halves for each domain, thereby dividing the target image into 16 low-resolution images that match the size of the dictionary low-resolution image.

The resolution enhancement unit 150 extracts a four-dimensional patch vector from each of the uniformly divided target images, compares the four-dimensional center vectors stored in the dictionary with reference to the four-dimensional patch vectors, A cluster can be determined.

At this time, the index of the determined cluster

Figure 112015081389302-pat00007
, The regression coefficient stored together with the four-dimensional center vector as shown in Equation 2 below
Figure 112015081389302-pat00008
And a 4-dimensional high-resolution patch h determined through multiplication of 1, which is a 4-dimensional patch vector of the target image.

Equation 2

Figure 112015081389302-pat00009

Based on this, it is possible to improve the resolution of the entire target image by repeatedly applying Equation 2 using all the four-dimensional patch vectors of the target image.

According to the present invention, the resolution of the target image segmented into 16 segments is improved, and it is synthesized again in the reverse order of the segmentation process, thereby finally generating a high-resolution target image having a spatial domain and an improved resolution for each domain. At this time, due to the characteristics of the resolution algorithm based on the patch of a predetermined size, an incorrect result may be generated due to the absence of the pixel value at the edge portion of the image as shown in FIG. 6 (a). The super-resolution algorithms based on two-dimensional images process this problem with a linear interpolation technique or remove edge portions. However, since the present invention performs a post-segmentation synthesis process, an algorithm for removing edge portions of an image can not be applied . Therefore, a new post-processing algorithm is needed to compensate the edges for 4-D images. The simple linear interpolation method can be applied in the spatial domain, but it can not be applied in each domain.

In order to solve the above problem, the resolution enhancing unit 150 may perform linear interpolation by converting the target image into an image of an EPI (Epi Plane Image) as shown in FIG. Specifically, the resolution enhancement unit 150 converts the divided target image into an image of an EPI image on a ux plane, and then performs a high-order interpolation (Bicubic) It is possible to improve the resolution of the EPI image. The resolution enhancement unit 150 may then convert the EPI image into a segmented target image that is a general image.

The resolution enhancement unit 150 may convert the divided target image into the v-y plane EPI image in the same manner as the method of improving the resolution of the u-x plane EPI image. After applying the Bicubic technique, which is a linear interpolation technique in the EPI image, the resolution of the EPI image can be improved in the vy plane and the EPI image can be converted into a divided target image, which is a general image.

The resolution enhancement unit 150 may use the high-resolution image generated by using the linear interpolation technique as a post-processing algorithm that compensates for edge portions having insufficient pixel values as shown in FIG. 6 (a). 6 (b) shows the result of applying the higher-order interpolation technique to the EPI image. As a result, the resolution enhancement unit 150 applies a post-processing algorithm to the divided target image, c).

8 is a diagram illustrating a resolution improvement result in a spatial domain according to an embodiment of the present invention.

The results of the resolution enhancement experiments performed by the resolution enhancing apparatus 100 of the present invention are as follows. The resolution enhancement experiment of the present invention was performed on a computer equipped with Intel i7-3770K 3.5GHz CPU and 16G RAM, and the experimental image was obtained from Lytro, a commercial light field camera based on a microlens array. The resolution of a 4-dimensional light field image used as a high-resolution image of an input image and a dictionary to improve resolution is 360

Figure 112015081389302-pat00010
360
Figure 112015081389302-pat00011
8
Figure 112015081389302-pat00012
8. In addition, the resolution of a low-resolution image that reduces the resolution of the input image by half is 180
Figure 112015081389302-pat00013
180
Figure 112015081389302-pat00014
4
Figure 112015081389302-pat00015
4. This is the resolution of the divided target image when the resolution algorithm is applied after dividing the low resolution image and the target image of the dictionary. The final result image is the spatial domain of the target image and 720
Figure 112015081389302-pat00016
720
Figure 112015081389302-pat00017
16
Figure 112015081389302-pat00018
16.

Also, the sizes of the 4-dimensional high-resolution patches and the 4-dimensional low-resolution patches extracted from the 4-dimensional light field image are 8

Figure 112015081389302-pat00019
8
Figure 112015081389302-pat00020
, 4
Figure 112015081389302-pat00021
4
Figure 112015081389302-pat00022
4
Figure 112015081389302-pat00023
4, which are 4096
Figure 112015081389302-pat00024
, 256
Figure 112015081389302-pat00025
In the form of a column vector. In the experiment of the present invention, as shown in FIG. 9, 200,000 patches were randomly selected from 40 4-dimensional light field images taken in various environments to perform dictionary learning. The value of K applied to the K-mean clustering algorithm affects the completeness and the execution time of the result image. In this experiment, the value of K is set to 512.

The present invention can improve the resolution of each domain as well as improve the resolution of the spatial domain, unlike the conventional resolution enhancement algorithms that only perform resolution enhancement in the existing spatial domain. Therefore, it is impossible to directly compare with the resolution enhancement algorithms for existing two-dimensional images. In addition, since Lytro, a commercial camera, is used to acquire an input image, there is no true value as a reference for quantitative evaluation. Therefore, the quantitative evaluation of the algorithm is based on the input image acquired by Lytro (360

Figure 112015081389302-pat00026
360
Figure 112015081389302-pat00027
8
Figure 112015081389302-pat00028
8) is set to a true value and the resolution of the spatial domain and each domain direction is lowered (180
Figure 112015081389302-pat00029
180
Figure 112015081389302-pat00030
4
Figure 112015081389302-pat00031
4) as the input image and compared with the existing resolution enhancement algorithm only in the spatial domain. On the other hand, the qualitative evaluation is based on the video acquired by Lytro (360
Figure 112015081389302-pat00032
360
Figure 112015081389302-pat00033
8
Figure 112015081389302-pat00034
8) as the target image and the resultant image (720
Figure 112015081389302-pat00035
720
Figure 112015081389302-pat00036
16
Figure 112015081389302-pat00037
16) were compared with existing resolution enhancement algorithms in the spatial domain and performance evaluation was performed by verifying whether appropriate values were generated for each domain resolution. In FIG. 8 (b), it can be seen that the proposed method generates a clearer high-resolution image than the conventional high-order bicubic technique (a). 8 (c) shows the result of measuring the peak signal to noise ratio (PSNR) with respect to the experimental results shown in FIGS. 8 (a) and 8 (b) We can confirm that the proposed algorithm produces a result that is improved by up to 2dB.

For the qualitative evaluation of the algorithm proposed in the present invention, the obtained light field image 360

Figure 112015081389302-pat00038
360
Figure 112015081389302-pat00039
8
Figure 112015081389302-pat00040
8) as a target image, we performed a qualitative comparison in the spatial domain of the bicubic method, which is a linear interpolation method and the method using a single image based super resolution method, among the existing resolution enhancement methods. As a result, it has been confirmed that the algorithm proposed by the present invention produces a clearer result in edges, textures, character regions, and the like. The resolution enhancement in each domain direction which can not be performed in the existing resolution enhancement technique is qualitatively evaluated in FIG. (1, 2), (3, 12), (10, 10), (14, 6), (16, 16) of each domain restored after applying the proposed algorithm in actual images It is possible to confirm that the image of the dog is generated correctly.

11 is a diagram illustrating a resolution enhancement method according to an embodiment of the present invention.

In step 1110, the resolution enhancement apparatus 100 may convert various real-world input images acquired through the microlens array and the two-dimensional image sensor into a high-resolution four-dimensional light field image. At this time, various real-world input images obtained through the microlens array and the two-dimensional image sensor by the resolution enhancing apparatus 100 can be defined as initial high-resolution images.

In a commercial light field camera with a microlens arrangement, light rays coming through the main lens can be passed through a small lens array and recorded at different positions of the sensor by the direction of light. As a result, a commercial light field camera can store an image in an initial data format that can not be directly used as shown in FIG. 2 (a). 2 (a) is enlarged, it can be confirmed that it is composed of a set of small micro images. Therefore, it may be necessary to transform the coordinate system so that the raw data shown in FIG. 2 (a) is suitable for the experiment.

To this end, the resolution enhancement apparatus 100 corrects the color information of the original data as shown in FIG. 2A and generates sub-aperture images so that the light can be directly used by classifying the light according to directions . 2 (b), the resolution enhancing apparatus 100 combines the absurd open images into an image arranged by each domain (angular domain) to represent the 4-dimensional light field image And can develop a super resolution algorithm. In the present invention, the coordinate system of each angular domain is denoted by u and v, and the coordinate system of the spatial domain is denoted by x and y.

In step 1120, the resolution enhancement apparatus 100 generates a low-resolution 4-dimensional light field image by lowering resolution of the acquired high-resolution 4-dimensional light field image into a spatial domain and an angular domain, can do.

In step 1130, the resolution enhancement apparatus 100 generates a 4-dimensional high-resolution patch (HR patch) 350 and a 4-dimensional low-resolution patch (LR patch) 360 Can be generated. Since the object of the present invention is to improve both the spatial domain resolution and the resolution of each domain constituting the 4-dimensional light field image, the patch has a 4-dimensional structure including information of the spatial domain and information of each domain Lt; / RTI >

The resolution enhancement apparatus 100 may generate a set of two-dimensional patches existing in the same position in the spatial domain with respect to a plurality of absurd open images constituting the four-dimensional light field images 330 and 340. [ In addition, the resolution enhancement apparatus 100 stores the generated set of patches in the form of column vectors so that mathematical operations can be easily performed in the future reconstruction step, thereby generating a 4-dimensional high resolution patch 350 and a 4-dimensional low resolution patch 360 Can be extracted.

In step 1140, the resolution enhancement apparatus 100 may learn a dictionary using the 4-dimensional high-resolution patch generated in step 1130 and the 4-dimensional low-resolution patch. In order to define the relation between two patches using 4-dimensional high-resolution patches and 4-dimensional low-resolution patches generated based on various real-world input images acquired by commercial light field cameras and to apply them to the resolution restoration step of an efficient dictionary, And learning algorithms may be required. At this time, since a patch is generated in various image coordinates for various input images of the real world, a very large amount of patches may exist. However, all of these can be inefficient in terms of processing speed and memory usage due to repetitive patches and meaningless patches.

In order to solve such a problem, the present invention provides a method of grouping similar patches into a single cluster using a K-means clustering algorithm in the configuration and learning stages of the dictionary. Since the K-average clustering algorithm can not achieve good results if the amount of data to be clustered is small or the kind of data is limited, the construction and learning of the dictionary are performed with a large number of images acquired in various environments.

The dictionary learning step may be configured as shown in FIG. The resolution enhancement apparatus 100 uses the four-dimensional low-resolution patches generated by the resolution enhancement apparatus 100 as input vectors of the K-average clustering algorithm 410, and performs K clusters and corresponding centers A cluster center 420 may be extracted. Also belonging to each cluster

Figure 112015081389302-pat00041
Dimensional low-resolution patch vectors,
Figure 112015081389302-pat00042
Resolution patch vectors are used to form a matrix < RTI ID = 0.0 >
Figure 112015081389302-pat00043
. ≪ / RTI > here
Figure 112015081389302-pat00044
Denotes the index of the cluster, and n and m denote the dimensions of the 4-dimensional high-resolution patch vector and the 4-dimensional low-resolution patch vector, respectively. Finally, the linear approximation of the low-resolution patch matrix and the high-resolution patch matrix is performed by the least-squares method as shown in Equation 1 to obtain a regression coefficient 430 matrix
Figure 112015081389302-pat00045
Can be obtained.

Equation 1

Figure 112015081389302-pat00046

The resolution enhancer 100 may then store the center vector 420 for each cluster and the regression coefficient 430 of the cluster together as a component of the dictionary 440. At this time, the center vector 420 for each cluster to be stored is compared with a 4-dimensional low resolution patch for a target image in the future, and the regression coefficient 430 can be used to improve the resolution of the target image after the comparison.

In step 1150, the resolution enhancement apparatus 100 may improve the resolution of the target image using the dictionary 440 learned in step 1140. [ The resolution enhancement apparatus 100 of the present invention converts a high-resolution 4-dimensional light field image acquired by a commercial light field camera into a low-resolution four-dimensional light field image by reducing the resolution in each of the spatial domain and each domain by half, 4-dimensional low resolution patches were generated. However, in the present invention, it is aimed to improve the resolution of the input image acquired by the commercial light field camera, so that there is a problem that the resolution used in the dictionary learning step differs from the resolution used in the resolution improving step. Accordingly, in the present invention, even if the same feature exists in one image, there are various scale features. Even if a 4-dimensional low-resolution patch and a 4-dimensional high-resolution patch are learned for a specific scale, this learning is still effective for other scales The algorithm is designed based on

To this end, the resolution enhancement apparatus 100 may equally divide the target image into a plurality of regions. That is, in order to use the dictionary learning result for improving the resolution of the target image, it is possible to make the resolution of the target image coincide with the resolution of the low resolution image of the dictionary through image segmentation. At this time, the target image needs a 4-dimensional division process. As shown in FIG. 5, the target image can be divided into half in the horizontal direction and in the vertical direction in each absurd image in the spatial domain. In addition, the resolution enhancing apparatus 100 may divide the segment of the absurd open image into halves for each domain to divide the target image into 16 low resolution images matching the size of the low resolution image of the dictionary.

The resolution enhancement apparatus 100 extracts a four-dimensional patch vector from each target image that is uniformly divided, compares the four-dimensional center vector stored in the dictionary with the extracted four-dimensional patch vector, and determines a minimum distance corresponding to the four- A cluster can be determined.

At this time, the index of the determined cluster

Figure 112015081389302-pat00047
, The regression coefficient stored together with the four-dimensional center vector as shown in Equation 2 below
Figure 112015081389302-pat00048
And a 4-dimensional high-resolution patch h determined through multiplication of 1, which is a 4-dimensional patch vector of the target image.

Equation 2

Figure 112015081389302-pat00049

Based on this, it is possible to improve the resolution of the entire target image by repeatedly applying Equation 2 using all the four-dimensional patch vectors of the target image.

According to the present invention, the resolution of the target image segmented into 16 segments is improved, and it is synthesized again in the reverse order of the segmentation process, thereby finally generating a high-resolution target image having a spatial domain and an improved resolution for each domain. At this time, due to the characteristics of the resolution algorithm based on the patch of a predetermined size, an incorrect result may be generated due to the absence of the pixel value at the edge portion of the image as shown in FIG. 6 (a). The super-resolution algorithms based on two-dimensional images process this problem with a linear interpolation technique or remove edge portions. However, since the present invention performs a post-segmentation synthesis process, an algorithm for removing edge portions of an image can not be applied . Therefore, a new post-processing algorithm is needed to compensate the edges for 4-D images. The simple linear interpolation method can be applied in the spatial domain, but it can not be applied in each domain.

In order to solve the above problem, the resolution enhancement apparatus 100 may perform linear interpolation by converting a target image into an EPI image as shown in FIG. Specifically, the resolution enhancement apparatus 100 converts the divided target image into an image of an EPI image on a ux plane, and then performs a high-order interpolation (Bicubic) technique It is possible to improve the resolution of the EPI image. Then, the resolution enhancement apparatus 100 may convert the EPI image into a divided target image, which is a general image.

The resolution enhancement apparatus 100 may convert the divided target image into the v-y plane EPI image in the same manner as the method of improving the resolution of the u-x plane EPI image. After applying the Bicubic technique, which is a linear interpolation technique in the EPI image, the resolution of the EPI image can be improved in the vy plane and the EPI image can be converted into a divided target image, which is a general image.

The resolution enhancement apparatus 100 may use the high-resolution image generated by using the linear interpolation technique as a post-processing algorithm that compensates for the edge portion lacking the pixel value as shown in FIG. 6 (a). 6 (b) shows the result of applying the higher-order interpolation technique to the EPI image. As a result, the resolution enhancing apparatus 100 applies a post-processing algorithm to the divided target image, c).

The methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: resolution enhancement device
110:
120:
130:
140: Dictionary Learning Section
150: resolution enhancement unit

Claims (10)

The input image obtained through the microlens array and the 2D image sensor is defined as an initial high resolution image,
Converting the initial high-resolution image into a high-resolution four-dimensional light field image;
Generating a low-resolution four-dimensional light field image having a lower resolution than the converted high-resolution four-dimensional light field image;
Generating a 4-dimensional high resolution patch (HR patch) and a 4-dimensional low resolution patch (LR patch) using the high resolution 4-dimensional light field image and the low resolution 4-dimensional light field image;
Learning a dictionary using the generated 4-dimensional high resolution patch and 4-dimensional low resolution patch; And
A step of improving the resolution of the target image using the learned dictionary
Lt; / RTI >
The step of generating the 4-dimensional high-resolution patch (HR patch) and the 4-dimensional low-resolution patch (LR patch) may include a step of generating a high-resolution 4-dimensional light field image and a low- Dimensional patches and 4-dimensional low-resolution patches by generating a set of patches existing at the same position in a spatial domain and storing the generated sets of patches in the form of column vectors. delete The method according to claim 1,
Wherein the learning comprises:
Applying a clustering operation to the generated four-dimensional low resolution patch;
Extracting a four-dimensional center vector corresponding to a cluster determined by applying the clustering operation;
Confirming a regression coefficient corresponding to the determined cluster using the generated 4-dimensional high-resolution patch and 4-dimensional low-resolution patch; And
Storing a four-dimensional center vector and a regression coefficient corresponding to the determined community as elements of a dictionary
/ RTI > The method according to claim 1,
Wherein the step of enhancing the resolution of the target image comprises:
Extracting a four-dimensional patch vector of the target image;
Determining a cluster having a four-dimensional center vector corresponding to the extracted four-dimensional patch vector using the learned dictionary; And
Enhancing the resolution of the target image using a regression coefficient corresponding to the determined cluster
/ RTI > The method according to claim 1,
Wherein the step of enhancing the resolution of the target image comprises:
Dividing the target image into a plurality of regions;
Enhancing resolution of each of the uniformly divided regions; And
And reconstructing a high-resolution target image by merging a plurality of regions having the enhanced resolution
/ RTI > The input image obtained through the microlens array and the 2D image sensor is defined as an initial high resolution image,
An image converting unit for converting the initial high-resolution image into a high-resolution four-dimensional light field image;
An image generator for generating a low-resolution four-dimensional light field image having a lower resolution than the converted high-resolution four-dimensional light field image;
A patch generator for generating a 4-dimensional high-resolution patch (HR patch) and a 4-dimensional low-resolution patch (LR patch) using the high-resolution 4-dimensional light field image and the low-resolution 4-dimensional light field image;
A dictionary learning unit for learning a dictionary using the generated 4-dimensional high-resolution patch and 4-dimensional low-resolution patch; And
A resolution enhancement unit for improving a resolution of a target image using the learned dictionary,
Lt; / RTI >
The patch generation unit generates,
Generating a set of patches existing at the same position in a spatial domain with respect to the absurd open images constituting the high-resolution 4-dimensional light field image and the low-resolution 4-dimensional light field image, Resolution enhancement device that extracts four-dimensional high-resolution patches and four-dimensional low-resolution patches by storing them in the form of column vectors. delete The method according to claim 6,
The dictionary learning unit,
Dimensional clustering method, a clustering operation is applied to the generated four-dimensional low-resolution patches, a four-dimensional center vector corresponding to the determined clusters is extracted by applying the clustering operation, and the generated four-dimensional high- And determining a regression coefficient corresponding to the determined cluster, and storing a four-dimensional center vector and a regression coefficient corresponding to the determined cluster as a component of a dictionary. The method according to claim 6,
Wherein the resolution enhancing unit comprises:
Dimensional patch vector of the target image, using the learned dictionary to determine a cluster having a four-dimensional center vector corresponding to the extracted four-dimensional patch vector, and using a regression coefficient corresponding to the determined cluster Thereby improving the resolution of the target image. The method according to claim 6,
Wherein the resolution enhancing unit comprises:
Wherein the target image is equally divided into a plurality of regions and resolution of each of the plurality of equally divided regions is improved, and a plurality of regions having the improved resolution are merged to restore a high-resolution target image.
KR1020150118114A 2015-08-21 2015-08-21 Apparatus and method for resolution enhancement based on dictionary learning KR101723738B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150118114A KR101723738B1 (en) 2015-08-21 2015-08-21 Apparatus and method for resolution enhancement based on dictionary learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150118114A KR101723738B1 (en) 2015-08-21 2015-08-21 Apparatus and method for resolution enhancement based on dictionary learning

Publications (2)

Publication Number Publication Date
KR20170022736A KR20170022736A (en) 2017-03-02
KR101723738B1 true KR101723738B1 (en) 2017-04-18

Family

ID=58426876

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150118114A KR101723738B1 (en) 2015-08-21 2015-08-21 Apparatus and method for resolution enhancement based on dictionary learning

Country Status (1)

Country Link
KR (1) KR101723738B1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020040521A1 (en) * 2018-08-21 2020-02-27 삼성전자 주식회사 Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field
US11195288B2 (en) 2017-11-15 2021-12-07 Interdigital Ce Patent Holdings, Sas Method for processing a light field video based on the use of a super-rays representation
US11533464B2 (en) 2018-08-21 2022-12-20 Samsung Electronics Co., Ltd. Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field
US11861808B2 (en) 2018-02-20 2024-01-02 Samsung Electronics Co., Ltd. Electronic device, image processing method, and computer-readable recording medium

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102661826B1 (en) * 2018-02-27 2024-04-26 엘지전자 주식회사 Signal processing device and image display apparatus including the same
KR102246110B1 (en) * 2019-04-02 2021-04-29 삼성전자주식회사 Display apparatus and image processing method thereof
US10909700B2 (en) 2019-04-02 2021-02-02 Samsung Electronics Co., Ltd. Display apparatus and image processing method thereof
KR102155381B1 (en) * 2019-09-19 2020-09-11 두에이아이(주) Method, apparatus and software program for cervical cancer decision using image analysis of artificial intelligence based technology
CN111951159B (en) * 2020-07-02 2024-04-26 西安理工大学 Processing method for super-resolution of light field EPI image under strong noise condition
CN112288654A (en) * 2020-11-09 2021-01-29 珠海市润鼎智能科技有限公司 Method for enhancing fine particles in image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Donghyeon Cho et al, "Modeling the calibration pipeline of the Lytro camera for high quality light-field image reconstruction"(2013.12.)*
Kshitij Marwah et al, "Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections", ACM Transactions on Graphics(2013.07.)*

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11195288B2 (en) 2017-11-15 2021-12-07 Interdigital Ce Patent Holdings, Sas Method for processing a light field video based on the use of a super-rays representation
US11861808B2 (en) 2018-02-20 2024-01-02 Samsung Electronics Co., Ltd. Electronic device, image processing method, and computer-readable recording medium
WO2020040521A1 (en) * 2018-08-21 2020-02-27 삼성전자 주식회사 Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field
US11533464B2 (en) 2018-08-21 2022-12-20 Samsung Electronics Co., Ltd. Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field

Also Published As

Publication number Publication date
KR20170022736A (en) 2017-03-02

Similar Documents

Publication Publication Date Title
KR101723738B1 (en) Apparatus and method for resolution enhancement based on dictionary learning
JP7159057B2 (en) Free-viewpoint video generation method and free-viewpoint video generation system
Li et al. Joint example-based depth map super-resolution
Lu et al. Depth enhancement via low-rank matrix completion
Pan et al. Super-resolution based on compressive sensing and structural self-similarity for remote sensing images
CN108961180B (en) Infrared image enhancement method and system
JP6512100B2 (en) INFORMATION PROCESSING APPARATUS AND IMAGE PROCESSING METHOD FOR EXECUTING IMAGE PROCESSING
JP2007000205A (en) Image processing apparatus, image processing method, and image processing program
KR101567792B1 (en) System and method for describing image outlines
CN110490924B (en) Light field image feature point detection method based on multi-scale Harris
JP2016509805A (en) High frame rate of image stream
Almasri et al. Rgb guided thermal super-resolution enhancement
Zhou et al. PADENet: An efficient and robust panoramic monocular depth estimation network for outdoor scenes
Wang et al. An ensemble multi-scale residual attention network (EMRA-net) for image Dehazing
CN114758145A (en) Image desensitization method and device, electronic equipment and storage medium
Shyam et al. Lightweight hdr camera isp for robust perception in dynamic illumination conditions via fourier adversarial networks
JP2014112362A (en) Image processing method and device
CN113298187A (en) Image processing method and device, and computer readable storage medium
KR102617023B1 (en) System and method for improving light field spatial resolution
Yeganli et al. Image super-resolution via sparse representation over coupled dictionary learning based on patch sharpness
Oh et al. Fpanet: Frequency-based video demoireing using frame-level post alignment
Mandal et al. Hierarchical example-based range-image super-resolution with edge-preservation
CN108801457A (en) Three-dimensional collection of illustrative plates based on the design of coded sample plate and second energy about beam alignment obtains and method for reconstructing
Mandal et al. Super-resolving a single intensity/range image via non-local means and sparse representation
Nguyen et al. Super resolution face image based on locally linear embedding and local correlation

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right