JP2013127718A - Superresolution image processing device and codebook creation device for superresolution image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a practical superresolution image processing device providing high reproducibility by a relatively small hardware resource, and a codebook creation device for superresolution image processing that creates a codebook used by the superresolution image processing device.SOLUTION: For executing highly versatile superresolution image processing in a short time, a superresolution image processing device 101 according to the present embodiment first, creates an original codebook 202 from an image as a processing object, and creates a codebook 106 formed by selecting only records suited for the superresolution processing, from the original codebook 202. A norm is used for effectively reducing the number of records for use in the codebook 106. The adoption of the techniques can materialize the superresolution image processing device 101 achieving a high recall ratio of high-frequency components by a relatively small calculation amount.

Description

The present invention relates to a technique suitably applied to a super-resolution image processing device and a super-resolution image processing codebook creation device.
More specifically, a super-resolution image processing device that outputs a high-resolution image from a low-resolution original image, and a code book for super-resolution image processing that creates a code book used by the super-resolution image processing device It relates to a creation device.

The super-resolution technique is a technique for obtaining a magnified image with little discomfort by generating a high-resolution image from a low-resolution original image. This super-resolution technique has attracted attention and has been adopted in some consumer devices and the like as the resolution of image browsing devices such as televisions and personal computers increases.
For example, when viewing images taken with a relatively low-resolution camera attached to a mobile phone on a high-resolution display, or viewing low-resolution video content created in the analog TV era on a high-resolution television, Alternatively, this super-resolution technique is used when browsing low-resolution video streaming data on a high-resolution display.
Furthermore, by increasing the resolution of the captured image data of a surveillance camera or the like, there is a possibility that it can contribute to improving the safety of the social infrastructure.

As described above, the super-resolution technique is a technique for obtaining an enlarged image without a sense of incongruity, but this is a technique that in turn is a technique for “analyzing a lost high-frequency component”.
For example, when the pixel size of a certain image data is reduced to ¼ times in vertical and horizontal directions, an average value of two adjacent pixels is calculated and converted into one pixel. This average value calculation is a kind of integration. When the enumeration of pixels is regarded as a signal, thinning out the pixels by calculating the average value is equivalent to removing the high-frequency component of the signal.
On the other hand, when the pixel size of a certain image data is enlarged by 4 times by 2 times vertically and horizontally, the simplest method is to obtain an intermediate pixel by calculating the intermediate value between two adjacent pixels. However, this calculation method does not reproduce lost high frequency components. For this reason, the enlarged image obtained by the intermediate value calculation is an image in which the boundary line is ambiguous as if it is out of focus.
In other words, super-resolution technology is a technology that "analyzes lost high-frequency components" by analyzing pixel patterns, calculating appropriate pixels, and filling between adjacent pixels in the original image. It is.

  Until now, many researchers have developed super-resolution image processing technology. Non-patent documents referred to by the inventors as prior art documents of the present invention are listed in Non-patent documents 1 to 5. These techniques create a code book that can be called a “high-frequency component dictionary file” in advance, extract a microscopic local region from the input image data, and search the code book with the local region data. In common.

WT Freeman, EC Pasztor, OT Carmichael, Learning Low-Level Vision, International Journal of Computer Vision, 40 (1), pp. 25-47, 2000. [Searched November 21, 2011], Internet <URL: http: //www.stat.ucla.edu/~sczhu/workshops/sctv99/papers/freeman_sctv99.pdf> J. Sun, NN Zheng, H. Tao, and SH-Y. Generic image hallucination with primal sketch prior. CVPR, 2003. [Searched on November 21, 2011], Internet <URL: http://research.microsoft. com / en-us / um / people / jiansun / papers / Hallucination_CVPR03.pdf> “Image Enlargement Method Using Self-Decomposed Codebook and Mahalanobis Distance” Hideaki Kawano, Noriaki Suetake, Kitsuki Kuruma, Takashi Aso, IEICE Transactions, J91-D, vol. 8, pp. 1983-1985, [Search December 16, 2011], Internet <URL: http://ci.nii.ac.jp/naid/110007385952/> Super-Resolution from a Single Image, D. Glasner, S. Bagon, and M. Irani, ICCV 2009. [Searched November 21, 2011], Internet <URL: http://www.wisdom.weizmann.ac. il / ~ vision / single_image_SR / files / single_image_SR.pdf> J. Sun, J. Sun, ZB Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in Proceedings of IEEE Computer Society Conference Computer Vision and Pattern Recogition, 2008. [November 2011 21 days search], Internet <URL: http: //nguyendangbinh.org/Proceedings/CVPR/2008/data/papers/319.pdf>

  The prior arts disclosed in Non-Patent Documents 1 to 5 have difficulties in super-resolution image reproducibility and codebook data size. If the reproducibility is low, the image quality is degraded. If the data size of the code book is large, it will lead to a decrease in calculation speed and an increase in the size of the device, which will reduce the practicality.

  The present invention has been made in view of the above problems, and provides a practical super-resolution image processing apparatus that realizes high reproducibility with relatively few hardware resources, and a code used by the super-resolution image processing apparatus. It is an object of the present invention to provide a code book creation device for super-resolution image processing that creates a book.

  In order to solve the above problems, a super-resolution image processing apparatus of the present invention includes an enlargement processing unit that enlarges input image data, and an input local area that is extracted from the input image data enlarged by the enlargement processing unit within a predetermined rectangular range. A local region extraction unit that outputs region low frequency components, a norm calculation unit that calculates the norm of the input local region low frequency component and outputs the input local region norm, and the input local region low frequency component is divided by the input local region norm And a divider that outputs the input local region normalized low-frequency component, and a normalized low-frequency component that stores the normalized low-frequency component obtained by normalizing the low-frequency component of the local region extracted from the image data in a predetermined rectangular range A component field and a normalized high-frequency component field that stores a normalized high-frequency component obtained by normalizing a high-frequency component in a local region, and records with a principal component value based on the normalized low-frequency component. Are grouped, a tree search unit that searches the codebook based on the input local region normalized low frequency component to identify a predetermined record group, an input local region normalized low frequency component, and a predetermined A vector similarity calculation unit that calculates the similarity between the normalized low frequency component of the record group and identifies the most similar record, a normalized high frequency component of the record specified by the vector similarity calculation unit, and an input local region A multiplier that multiplies the norm and outputs a local region high-frequency component; and an adder that adds the local region high-frequency component and the input local region low-frequency component.

  Further, in order to solve the above-described problem, the super-resolution image processing codebook creation apparatus of the present invention performs normalized low frequency components and normalization for each local region obtained by extracting sample image data within a predetermined rectangular range. Codebook record that creates an original codebook consisting of a normalized low-frequency component field that stores normalized high-frequency components and a normalized high-frequency component field that stores normalized high-frequency components After creating a super-resolution image process by identifying the most similar record to the original codebook for each local region where the image data for evaluation and the image data for evaluation are extracted within a predetermined rectangular range, the similarity is calculated and the original code A record selection processing unit that selects a record of the book and creates a code book.

  When creating a codebook used for super-resolution image processing, only records that can provide high-quality output are selected to improve search speed and image reproducibility. In addition, by grouping the code book records using principal component analysis, the number of vector calculation steps is reduced as much as possible.

  According to the present invention, a practical super-resolution image processing device that realizes high reproducibility with relatively few hardware resources and a super-resolution image processing for creating a codebook used by the super-resolution image processing device A code book creation device can be provided.

1 is a schematic diagram of a super-resolution image processing apparatus and a code book creation apparatus that are examples of embodiments of the present invention. It is a functional block diagram of a code book creation apparatus. It is a flowchart of the code book creation process by a code book creation apparatus. It is a functional block diagram of a code book record creation part. It is a flowchart which shows the flow of the code book record creation process by a code book record creation part. It is a conceptual diagram explaining the acquisition method of a local region. It is the schematic explaining the relationship between the low frequency component and high frequency component of an image. It is a functional block diagram of a record selection processing unit. It is a flowchart which shows the flow of the record selection process by a record selection process part. It is a flowchart which shows the flow of the record selection process by a record selection process part. It is the schematic which shows the relationship between the process and data which a principal component analysis process part implements. It is a flowchart which shows the flow of a principal component analysis process by a principal component analysis process part. It is the schematic explaining the processing content of the principal component analysis process by a principal component analysis process part. It is the schematic explaining the processing content of the principal component analysis process by a principal component analysis process part. It is a functional block diagram of a super-resolution image processing apparatus. It is a flowchart which shows the flow of the super-resolution image processing by a super-resolution image processing apparatus. It is a flowchart which shows the flow of the super-resolution image processing by a super-resolution image processing apparatus.

In the present embodiment, a super-resolution image processing device and a code book creation device will be described.
The super-resolution image processing apparatus is an apparatus that enlarges input image data at a predetermined magnification. When enlarging an image, the super-resolution image processing apparatus estimates a high-frequency component that does not exist in the input image data. This estimation is realized by searching a dictionary file called a code book for each local region obtained by subdividing input image data within a rectangular range.
The performance of the super-resolution image processing apparatus is evaluated by the time required for the super-resolution image processing and the reproducibility of the super-resolution image data.
The super-resolution image processing apparatus according to the present embodiment uses principal component analysis for indexing to reduce the number of calculation steps in order to reduce the time required for super-resolution image processing.
In addition, in order for the super-resolution image processing apparatus according to the present embodiment to improve the reproducibility of the super-resolution image data, the code book creation apparatus executes a process of selecting only high-quality records in the process of creating the code book. doing.

[overall structure]
FIG. 1 is a schematic diagram of a super-resolution image processing apparatus and a code book creation apparatus, which are examples of embodiments of the present invention.
When the super-resolution image processing apparatus 101 receives input image data 104 of vertical m pixels × horizontal n pixels from the digital camera 102, the image file 103, or the like, output image data of vertical k × m pixels × horizontal k × n pixels. 105 is generated. Here, k is an integer greater than 1, and m and n are natural numbers of 2 or more.
The image processed by the super-resolution image processing apparatus 101 may be a still image or a moving image. In the present embodiment, for the sake of simplicity, a description will be given centering on still images.

When the super-resolution image processing apparatus 101 performs super-resolution image processing on the input image data 104 to generate the output image data 105, in order to analogize the lost high-frequency component, One principal component basis vector 107, second principal component basis vector 108, and third principal component basis vector 109 are used. These data are generated by the code book creation device 110 and incorporated in the super-resolution image processing device 101 in advance.
Upon receiving one or more pieces of sample image data 111 and one or more pieces of evaluation image data 112, the code book creation device 110 performs a predetermined calculation process to be described later, whereby the code book 106 and the first principal component basis are obtained. A vector 107, a second principal component basis vector 108, and a third principal component basis vector 109 are generated.
Here, the code book 106, the first principal component basis vector 107, the second principal component basis vector 108, and the third principal component basis vector 109 are one set of these four. Hereinafter, the code book 106, the first principal component basis vector 107, the second principal component basis vector 108, and the third principal component basis vector 109 are collectively referred to as “code book 106”.

In FIG. 1, a specific implementation form of the super-resolution image processing apparatus 101 is not specified, but the super-resolution image processing apparatus 101 is an arithmetic unit that processes image data, and thus can be implemented in various forms. is there.
For example, it can be realized as a program for installing the super-resolution image processing apparatus 101 in a known personal computer. At that time, the code book 106 and the like will be stored in a non-volatile storage such as a hard disk device.
Further, it can be incorporated into the digital camera 102 as firmware of the digital camera 102.
Further, it can be realized by using a programmable logic device (PLD) such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like.

In FIG. 1, a specific implementation form of the code book creation device 110 is not specified, but since the code book creation device 110 is an arithmetic device that generates the code book 106 and the like from the sample image data 111, the super solution described above is used. Similar to the image processing apparatus 101, it can be implemented in various forms. The code book creation device 110 is realized by a program that is installed and executed on a personal computer, for example, and is provided at a production site such as a factory where the super-resolution image processing device 101 is manufactured. Is generated.
Here, the scene refers to the spot color of the image. For example, it is a photographed image of a landscape, a medical X-ray image, or an animation. In the case of a landscape image, many intermediate colors and intermediate brightness appear in the pixel pattern. On the other hand, in the case of an animation image, there are many pixel patterns with clear outlines and sharply changing (edge-effected) borders with certain colors and brightness. Thus, the image has a special color according to the scene.
Therefore, the sample image data 111 is a collection of images corresponding to the scene, and serves as a basis for the code book 106 and the like. Although it is difficult to create a universal code book 106 that can perform optimum super-resolution processing for any scene, super-resolution can be achieved by creating a code book 106 optimized for a scene. A code book 106 suitable for a scene to which the image processing apparatus 101 is applied can be provided. Of course, by preparing a plurality of code books 106 and the like, and the super resolution image processing apparatus 101 switching the code book 106 and the like according to the scene, it is possible to realize the super resolution image processing optimum for the scene. Further, as the scene is subdivided, the code book 106 and the like more suitable for the scene can be created, and super-resolution image processing with higher reproducibility can be realized.

  The code book 106, the first principal component basis vector 107, the second principal component basis vector 108, and the third principal component basis vector 109 are formed through well-known principal component analysis, as is apparent from their names. For this reason, the code book 106, the first principal component basis vector 107, the second principal component basis vector 108, and the third principal component basis vector 109 have a close relationship. Details of this relationship will be described later in the description of the code book creation device 110.

  For ease of explanation, in the following description of the embodiments, the image handled by the codebook creation device 110 and the super-resolution image processing device 101 is a monochrome still image, and one pixel is 8 bits. To do. Therefore, one pixel is 0 to 255 luminance data.

[Codebook creation device 110: overall configuration]
FIG. 2 is a functional block diagram of the code book creation device 110.
The code book record creation unit 201 reads one or more sample image data 111 and creates an original code book 202. This is the basis of the code book 106 used for super-resolution image processing.
Next, the record selection processing unit 203 reads the evaluation image data 112, selects only records satisfying a predetermined condition from the original code book 202, and copies them to the code book 106.
Next, the principal component analysis processing unit 204 reads the normalized low frequency component field data of all the records of the code book 106, performs a known principal component analysis, and performs the first principal component basis vector 107 and the second principal component analysis. A basis vector 108 and a third principal component basis vector 109 are calculated.
The principal component analysis processing unit 204 groups each record of the code book 106 by the first principal component value, the second principal component value, and the third principal component value.
The principal component analysis processing unit 204 assigns a group number serving as index information for search processing to the record of the code book 106.
The field configuration of the code book 106 will be described later with reference to FIG.

[Codebook creation apparatus 110: flowchart]
FIG. 3 is a flowchart of code book creation processing by the code book creation apparatus 110.
When the process is started (S301), the code book record creation unit 201 reads the sample image data 111 and creates the original code book 202 (S302).
Next, the record selection processing unit 203 reads the evaluation image data 112, selects only records satisfying a predetermined condition from the original code book 202, and copies them to the code book 106 (S303).
Next, the principal component analysis processing unit 204 reads the normalized low frequency component field data of all the records of the code book 106, performs a known principal component analysis, and performs the first principal component basis vector 107 and the second principal component analysis. After calculating the basis vector 108 and the third principal component basis vector 109, each record of the code book 106 is grouped by the first principal component value, the second principal component value, and the third principal component value (S304). The series of processes is terminated (S305).

[Codebook record creation unit 201: overall configuration and flowchart]
FIG. 4 is a functional block diagram of the code book record creation unit 201.
FIG. 5 is a flowchart showing the flow of code book record creation processing by the code book record creation unit 201.
Hereinafter, the block diagram of FIG. 4 will be described while explaining the flowchart of FIG.
When the processing is started (S501), the reduction processing unit 401 reads the sample image data 111, reduces the sample image data 111 to 1 / (k × k) times, and obtains reduced image data 402 (S502).

Next, the Bicubic enlargement processing unit 403 reads the reduced image data 402 and performs an enlargement process using a known Bicubic algorithm (S503). Then, the edge region determination unit 404 extracts a local region from the enlarged image data generated by the Bicubic enlargement processing unit 403, determines whether or not there is a variation component in the pixel data in the local region, and the local region with the variation component Only the sample Bicubic image data 405 is generated (S504).
Various algorithms for enlarging an image exist in addition to the Bicubic method, but in this embodiment, the Bicubic method is adopted because the sharpest image can be obtained among the currently widely used algorithms.

  The local area is a unit of image processing that is common to the super-resolution image processing apparatus 101 and the code book creation apparatus of the present embodiment. Image data, which is an aggregate of enormous pixel data, is divided by a small rectangular area to form a local region of vertical p pixels × horizontal q pixels. The super-resolution image processing apparatus 101 and the code book creation apparatus 110 according to the present embodiment perform various arithmetic processes on the pixel data included in the rectangular range. As an example, in the case of the present embodiment, the local region is configured by a total of 25 pixels, that is, 5 pixels vertically and 5 pixels horizontally.

FIG. 6 is a conceptual diagram illustrating a local area acquisition method.
In the code book record creation process, the code book record creation unit 201 generates the local area by moving the local area vertically and horizontally by one pixel.

Returning to FIG.
Next, the norm data calculation unit 406 reads the sample Bicubic image data 405 and calculates a norm for each local region. The norm is calculated by the following well-known formula.

That is, the value (luminance value) of each pixel is squared, and all the squared values are added, and then the square root of the added value is obtained. This norm is calculated for all local regions that may exist in the sample Bicubic image data 405 to obtain norm data 407 (S505).
When the local area of the sample Bicubic image data 405 is divided by the norm, the obtained value falls within a value between 0 and 1. That is, a change in absolute value is converted into a change in relative value. By converting the change in absolute value into the change in relative value, the change in luminance with different absolute values is also converted into the relative change in luminance, thereby improving the versatility of the data. Hereinafter, data obtained by dividing the image data by the norm is referred to as normalized image data.
The divider 408 reads the sample Bicubic image data 405 and norm data 407, and divides by the norm corresponding to each local region to obtain a normalized low frequency component 409 (S506).

The super-resolution image processing apparatus 101 of the present embodiment estimates a lost high-frequency component for image data to be processed and applies it to each local region. Therefore, the code book 106 needs to contain a high frequency component as a premise.
FIG. 7 is a schematic diagram illustrating the relationship between the low frequency component and the high frequency component of an image.
When the image 701 is reduced, high frequency components included in the image 701 are lost. When the reduced image 702 is enlarged again to the original size, an image with a blurred outline is generated because the high-frequency component is lost. This is the low frequency component 703 of the original image. By subtracting this low frequency component 703 from the original image 701, a high frequency component 704 of the original image is obtained.

Returning to FIG. 4 and FIG.
As described with reference to FIG. 7, when the sample Bicubic image data 405 is subtracted from the sample image data 111 by the adder 410, a high frequency component 411 of the sample image data 111 is obtained (S507). In this subtraction process, the area excluded by the edge area determination unit 404 is not subject to calculation.
In step S506, the low frequency component is normalized by the norm. It is necessary to perform the same process on the high frequency component. Therefore, the divider 412 reads the high frequency component 411 and the norm data 407, and divides by the norm corresponding to each local region to obtain the normalized high frequency component 413 (S508).
Finally, the raster scan processing unit 414 combines the normalized low-frequency component 409 obtained in step S506 and the normalized high-frequency component 413 obtained in step S508 as a record for each common local region. A book 202 is created (S509), and a series of processing ends (S510).

[Record Selection Processing Unit 203: Overall Configuration and Flowchart]
The original code book 202 created by the code book record creation unit 201 has an enormous number of records. However, of this number of records, there are very few records that actually obtain practical results. As a result of calculation by the inventor, about 3% of records that can withstand practical use in the original codebook were found. Thus, the super-resolution image processing is tried using the image data for evaluation, and only usable records are extracted. This is the role of the record selection processing unit 203.
FIG. 8 is a functional block diagram of the record selection processing unit 203.
9 and 10 are flowcharts showing the flow of record selection processing by the record selection processing unit 203. FIG.
Hereinafter, the block diagram of FIG. 8 will be described while explaining the flowcharts of FIGS. 9 and 10.
When the processing is started (S901), the reduction processing unit 401 reads the evaluation image data 112 different from the sample image data 111, reduces the evaluation image data 112 to 1 / (k × k) times, and reduces the evaluation. Image data 801 is obtained (S902).

  Next, the Bicubic enlargement processing unit 403 reads the evaluation reduced image data 801 and performs enlargement processing using a known Bicubic algorithm (S903). Then, the edge region determination unit 404 extracts a local region from the enlarged image data generated by the Bicubic enlargement processing unit 403, determines whether or not there is a variation component in the pixel data in the local region, and the local region with the variation component Only the image data 802 for evaluation is generated (S904). Here, the algorithm used for the enlargement processing needs to use the same algorithm as that of the Bicubic enlargement processing unit 403 of the code book record creation unit 201 described above in order to improve the reproducibility of the image.

Subsequent processing is loop processing.
First, a control unit (not shown) sets a counter variable i to 1 (S905). The counter variable i is set to the maximum number of local regions included in the evaluation bicubic image data 802.
Next, the local region extraction unit 803 extracts data of the i-th local region from the evaluation bicubic image data 802, and obtains an evaluation local region low-frequency component 804 (S906). This local area has the same form (vertical p pixels × horizontal q pixels) as the local area described above in the code book record creation process. In the case of this embodiment, it is 5 × 5 pixels.

Next, the norm calculation unit 805 reads the evaluation local region low-frequency component 804 and calculates the evaluation local region norm 806 (S907).
Next, the divider 807 divides the evaluation local region low frequency component 804 by the evaluation local region norm 806 to obtain the evaluation local region normalized low frequency component 808 (S908).

In step S 908, the evaluation local region normalized low frequency component 808 calculated by the divider 807 corresponds to the value of the normalized low frequency component field of the record included in the original codebook 202. That is, the value of the normalized high-frequency component field of the record of the value of the normalized low-frequency component field having the closest value to the local region normalized low-frequency component 808 for evaluation is used for super-resolution image processing. This is the normalized high frequency component to be obtained. Therefore, the SSD search processing unit 809 reads the evaluation local region normalized low frequency component 808, and the evaluation local region normalized low frequency component 808 and the normalized low frequency component field of all records existing in the original codebook 202. The distance to the value is calculated, and the closest record is specified (S909).
The SSD search processing unit 809 calculates image similarity using a well-known SSD (Sum of Squared Difference). Specifically, it is the following formula (2).

  That is, the luminance of pixels existing at the same position in the comparison target local areas is subtracted and squared, and the sum of the square values is calculated. If the two image data to be compared are exactly the same image, the minimum value is “0”. This calculation process is performed on the normalized low frequency component fields of all records existing in the original codebook 202, and the record having the smallest value is specified.

The description of the flowchart is continued in FIG.
Next, the multiplier 810 multiplies the value of the normalized high frequency component field in the record of the original codebook 202 specified in step S909 by the evaluation local region norm 806 calculated in step S907, and thereby local region high frequency component. Is obtained (S1010).
Next, the adder 811 adds the local region high frequency component calculated by the multiplier 810 in step S1010 and the evaluation local region low frequency component 804 obtained in step S906, and evaluates the local region super-resolution. Data 812 is obtained (S1011).

In step S 1011, evaluation local region super-resolution data 812 is obtained based on the records of the original code book 202. On the other hand, since the evaluation image data 112 exists, the real high frequency component included in the evaluation image data 112 at this time and the normalized high frequency component in the record in the original codebook 202 used in the super-resolution image processing. Similarity with the value of the field can be confirmed. Therefore, the RMSE calculation unit 813 obtains the evaluation local region super-resolution data 812, the evaluation image data 112, and the address in the evaluation image data 112 corresponding to the i-th local region from the local region extraction unit 803. Then, the similarity between the local regions is calculated (S1012).
The RMSE calculation unit 813 calculates a known RMSE calculation value (Root Mean Square Error). RMSE is the square root of the SSD described above, as shown in the following equation (3). The meaning of RMSE is the same index as the standard deviation of statistics, and represents an error in pixels from the original image.

Next, the comparator 814 compares the RMSE calculation value calculated by the RMSE calculation unit 813 with the RMSE threshold value 815, and outputs a logical value comparison result (S1014). In the present embodiment, the RMSE threshold value 815 is “5”. The RMSE of “5” means that the error between the original image and the super-resolution image is shifted by 5 (pixel value) per pixel.
If the comparison result of the RMSE threshold 815 determines that the RMSE calculation value is smaller than the RMSE threshold 815 (YES in S1014), the record copy processing unit 816 receives the logical “true” that is the comparison result of the RMSE threshold 815. Then, the record of the original code book 202 identified in step S909 is copied to the code book 106 (S1015).
Conversely, when the comparison result of the RMSE threshold value 815 determines that the RMSE operation value is equal to or greater than the RMSE threshold value 815 (NO in S1014), the record copy processing unit 816 has a logical “false” that is the comparison result of the RMSE threshold value 815. In response, does not work. Therefore, the record of the original code book 202 identified in step S909 is not copied to the code book 106.

In any case after step S1015 or when it is determined in step S1014 that the RMSE calculation value is equal to or greater than the RMSE threshold value 815 (NO in S1014), the control unit (not shown) next sets the counter variable i to 1. Increment (S1016). Then, the control unit (not shown) checks whether or not the counter variable exceeds the maximum value, that is, the maximum number of local regions (S1017). If not exceeded yet (NO in S1017), the process is repeated from step S906 in FIG.
If the counter variable has reached the maximum value, that is, the maximum number of local areas (YES in S1017), the control unit (not shown) ends a series of processes (S1018).

[Principal component analysis processing unit 204: Overview of flowchart and processing]
The super-resolution image processing apparatus 101 extracts a local area from the input image data 104 and searches the code book 106. In the search, finally, the similarity of vectors is calculated, and the most similar record is selected. In the present embodiment, 5 × 5 = 25 pixels, and the number of variables is large. A large number of variables means that there are a large number of calculation steps, which leads to a decrease in search speed.
Therefore, principal component analysis is used to reduce the number of variables and the number of calculation steps. If principal component analysis is used, a vector can be converted into a scalar value, so that indexing is facilitated and the search speed of the code book 106 can be improved.

FIG. 11 is a schematic diagram illustrating a relationship between processing performed by the principal component analysis processing unit 204 and data.
The code book 106 includes a normalized low-frequency component field copied from the original code book 202, a normalized high-frequency component field, a first principal component value field, a first principal component group number field, and a second principal component value. The field includes a field, a second principal component group number field, a third principal component value field, and a third principal component group number field. The principal component analysis processing unit 204 performs the arithmetic processing shown in FIG. 12 from the normalized low frequency components 409 of all the records of the code book 106, and a first principal component value field, a first principal component group number field, Enter values in the second principal component value field, the second principal component group number field, the third principal component value field, and the third principal component group number field, the first principal component basis vector 107, A principal component basis vector 108 and a third principal component basis vector 109 are calculated.

FIG. 12 is a flowchart showing the flow of principal component analysis processing by the principal component analysis processing unit 204.
When the process is started (S1201), first, the principal component analysis processing unit 204 performs a known principal component analysis on the normalized low frequency component field values of all the records of the code book 106 to obtain the first principal component basis vector 107. Calculate (S1202). As is well known, the first principal component basis vector 107 represents the first principal component axis, and its starting point is the basis vector that most represents the feature of the data to be subjected to principal component analysis.

Next, the principal component analysis processing unit 204 calculates an inner product operation with the inner product calculator 1101 between the normalized low frequency component field values of all the records of the code book 106 and the first principal component basis vector 107 obtained in step S1202. The first principal component value is obtained and recorded for each record (S1203).
Next, the principal component analysis processing unit 204 groups the first principal component values of all the records of the code book 106 with a predetermined number of records, and records the first principal component group number for each record (S1204).

  Next, the principal component analysis processing unit 204 acquires a second principal component basis vector 108 orthogonal to the first principal component axis from the result of the principal component analysis performed previously (S1205). Similar to the first principal component basis vector 107, the second principal component basis vector 108 also represents the second principal component axis and is the basis vector that captures the most characteristic of the data among the vectors orthogonal to the first principal component basis vector. .

Next, the principal component analysis processing unit 204 calculates an inner product operation by the inner product computing unit 1102 between the normalized low frequency component field values of all the records of the code book 106 and the second principal component basis vector 108 obtained in step S1205. Then, the second principal component value is obtained and recorded for each record (S1206).
Next, the principal component analysis processing unit 204 groups the second principal component values of the records having the same first principal component group number in the code book 106 with a predetermined number of records, and records the second principal component group number for each record. (S1207). That is, records having the same first principal component group number are further subdivided by the second principal component group number.

  Next, the principal component analysis processing unit 204 acquires a third principal component basis vector 109 orthogonal to the first principal component axis and the second principal component axis from the result of the principal component analysis performed previously (S1208). Similar to the first principal component basis vector 107, the third principal component basis vector 109 also represents the third principal component axis, and is the basis vector that captures the most characteristic of the data among the vectors orthogonal to the first and second principal component vectors. is there.

Next, the principal component analysis processing unit 204 performs an inner product operation on the normalized low frequency component field values of all the records in the code book 106 and the third principal component basis vector 109 obtained in step S1208 by the inner product calculator 1103. Then, the third principal component value is obtained and recorded for each record (S1209).
Next, the principal component analysis processing unit 204 groups the third principal component values of the records having the same first principal component group number and second principal component group number in the code book 106 by a predetermined number of records, A number is recorded for each record (S1210). That is, records having the same first principal component group number and second principal component group number are further subdivided by the third principal component group number. Then, a series of processing ends (S1211).

FIGS. 13A, 13 </ b> B, 13 </ b> C, 13 </ b> D, 14 </ b> E, and 14 </ b> F are schematic diagrams for explaining the processing content of the principal component analysis processing by the principal component analysis processing unit 204. .
FIG. 13A is a conceptual diagram of a normalized low frequency component 409 that is a target of principal component analysis. Originally, the normalized low-frequency component 409 is an aggregate of 25 scalar values of 5 × 5 pixels, that is, a 25-dimensional vector, but is assumed to be two-dimensional. Within a certain coordinate system, each of the normalized low frequency components 409 exists at an arbitrary position.
Principal component analysis finds the axis with the largest variance for these data. This axis is the first principal component axis. This corresponds to the processing in step S1202 of FIG.

FIG. 13B is a conceptual diagram when the diagram of FIG. 13A is viewed from the first principal component axis. Looking at each data from the first principal component axis, the data appears to be the most scattered. That is, the first principal component axis has the maximum distance from the minimum value to the maximum value on the first principal component axis of each data (first principal component value).
The first principal component value is obtained by calculating the inner product of each data and the first principal component basis vector 107 (step S1203 in FIG. 12). Then, when the appearance frequency of the first principal component value is taken on the vertical axis, an uneven appearance frequency can be observed as shown in FIG. Therefore, this appearance frequency graph is divided into equal areas as shown in FIG. This is the process of step S1204 in FIG.

The description of the principal component analysis process will be continued in FIG.
FIG. 14E is a conceptual diagram of the eigenspace formed by the first principal component axis, the second principal component axis, and the third principal component axis. When the processing proceeds to step S1210 in FIG. 12, the data set is divided by the first principal component axis, the second principal component axis, and the third principal component axis, and a set having the same number of elements is formed.
The group to which the record of the code book 106 belongs has a first principal component value range, a second principal component value range, and a third principal component value range. As shown in FIG. 14F, this range of principal component values is used to narrow down records by tree search.

[Super-Resolution Image Processing Device 101: Overview of Flowchart and Processing]
FIG. 15 is a functional block diagram of the super-resolution image processing apparatus 101.
16 and 17 are flowcharts illustrating the flow of super-resolution image processing by the super-resolution image processing apparatus 101.
Hereinafter, the block diagram of FIG. 15 will be described with reference to the flowcharts of FIGS. 16 and 17.
When the process is started (S1601), the Bicubic enlargement processing unit 1501 reads the input image data 104, performs an enlargement process using a known Bicubic algorithm, and generates input Bicubic image data 1502 (S1602).

Subsequent processing is loop processing.
First, a control unit (not shown) sets a counter variable i to 1 (S1603). The counter variable i has the maximum number of local regions included in the input Bicubic image data 1502. Note that the method of adopting a local region in the super-resolution image processing apparatus 101 is a tile-shaped method having no overlap unlike the method having overlap described with reference to FIG. By eliminating duplication, the number of processing steps of the super-resolution image processing apparatus 101 can be reduced.

  Next, the local region extraction unit 1503 extracts the data of the i-th local region from the input bicubic image data 1502 to obtain the input local region low frequency component 1504 (S1604). This local area has the same form (vertical p pixels × horizontal q pixels) as the local area described above in the code book record creation process. In the case of this embodiment, it is 5 × 5 pixels.

Next, the norm calculation unit 1505 reads the input local region low frequency component 1504 and calculates the input local region norm 1506 (S1605).
Next, the divider 1507 divides the input local region low frequency component 1504 by the input local region norm 1506 to obtain an input local region normalized low frequency component 1508 (S1606).

In step S 1606, the input local region normalized low frequency component 1508 calculated by the divider 1507 corresponds to the normalized low frequency component of the record included in the code book 106. That is, the normalized high-frequency component of the record of the value of the normalized low-frequency component field having the closest value to the input local region normalized low-frequency component 1508 is the normalized high-frequency to be obtained for super-resolution image processing. Become an ingredient.
As described above with reference to FIGS. 11 and 12, the code book 106 is indexed using principal component analysis. Therefore, the first principal component value, the second principal component value, and the third principal component value are calculated from the input local region normalized low frequency component 1508.
The first inner product calculator 1509 calculates the inner product of the input local region normalized low frequency component 1508 and the first principal component basis vector 107 to obtain a first principal component value (S1607).
The second inner product calculator 1510 calculates the inner product of the input local region normalized low frequency component 1508 and the second principal component basis vector 108 to obtain a second principal component value (S1608).
The third inner product calculator 1511 calculates the inner product of the input local region normalized low frequency component 1508 and the third principal component basis vector 109 to obtain a third principal component value (S1609).

The description of the flowchart is continued in FIG.
The tree search unit 1512 includes a first principal component value calculated by the first dot product calculator 1509, a second principal component value calculated by the second dot product calculator 1510, and a third principal value calculated by the third dot product calculator 1511. Using the component values, the code book 106 is tree-searched to identify a group of records (record group) having a value closest to the input local region normalized low frequency component 1508, and the vector similarity calculation unit 1513 The address information indicating the record group is delivered (S1710).

The vector similarity calculation unit 1513 obtains the address information of the record group delivered from the tree search unit 1512, the values of the normalized low frequency component fields of all records belonging to the record group, and the input local region normalized low frequency The SSD with the component 1508 is calculated to identify the closest record (S1711).
In step S1711, the vector similarity calculation unit 1513 identifies the record, whereby the normalized high-frequency component 1514 of the record necessary for super-resolution image processing can be obtained. The multiplier 1515 multiplies the normalized high frequency component 1514 of the record specified by the vector similarity calculation unit 1513 by the input local region norm 1506 calculated by the norm calculation unit 1505 in step S1605 to obtain the local region high frequency component 1516. Calculate (S1712).

  The adder 1517 adds the local region high-frequency component 1516 calculated by the multiplier 1515 and the input local region low-frequency component 1504 output from the local region extraction unit 1503 in step S1604, and local region super-resolution image data 1518. Is created (S1713). Then, the combination processing unit 1519 obtains address information corresponding to the i-th local region from the local region extraction unit 1503, and places the local region super-resolution image data 1518 on a RAM (not shown) (S1714).

Next, the control unit (not shown) increments the counter variable i by 1 (S1715). Then, the control unit (not shown) checks whether or not the counter variable i has exceeded the maximum value of the local region (S1716). If not exceeded yet (NO in S1716), the process is repeated from step S1604 of FIG.
If the counter variable i exceeds the maximum value of the local area (YES in S1716), the series of processes is terminated (S1717).
At step S1717, output image data 105, which is super-resolution image data, is formed on a RAM (not shown).

In this embodiment, the following application examples can be considered.
(1) In the above-described embodiment, an apparatus for a monochrome still image has been described. However, the input image data 104 (source) of the super-resolution image processing apparatus 101 may be a moving image. In this case, the local region to be evaluated takes into account the time axis. For example, for two images that are adjacent on the time axis, the number of vertical p pixels × horizontal q pixels × r samples is used as a local area unit.

(2) In the above-described embodiment, an apparatus for a monochrome still image has been described. However, the input image data 104 (source) of the super-resolution image processing apparatus 101 may be a color image. In this case, the local area takes into account the three primary colors. That is, the number of pixels constituting each bitmap is simply tripled.
Of course, the image to be handled may be a color moving image.

  (3) For example, in the case of image data generated by well-known chroma key composition that synthesizes a live-action person with another video such as animation or computer graphics, one image data has a plurality of completely different features. The parts will be mixed. In the case of such image data, it is desirable to separate the image portions having different features in advance and perform super-resolution image processing using the code book 106 suitable for each.

  (4) As is well known, in the principal component analysis, the number of elements of the matrix subject to principal component analysis is enormous, and the set is narrowed down only by the first principal component axis, the second principal component axis, and the third principal component axis. Is not performed satisfactorily, a fourth principal component axis that is orthogonal to both the first principal component axis, the second principal component axis, and the third principal component axis and has the largest variance is calculated. And the main component axis can be increased according to the fifth, sixth, and so on. The same applies to the super-resolution image processing apparatus 101 and the code book creation apparatus 110 of the present embodiment. That is, the basis vectors of the principal component axes included in the code book 106 and the like are not necessarily limited to the first principal component axis, the second principal component axis, and the third principal component axis, but may include the fourth principal component axis and the like as necessary. Can be included. At this time, a “fourth principal component” field or the like is added to the code book 106 accordingly.

  (5) The normalized low frequency component field of the code book 106 is based on the bitmap data of the local area enlarged by the Bicubic enlargement processing unit 403 in FIG. Instead of using the local area enlarged by the Bicubic enlargement processing unit 403, the bitmap data of the portion corresponding to the local area of the reduced image data 402 is normalized by the norm and used as a normalized low frequency component. Also good. As described above, since the high-frequency component is lost when the image is reduced, the image data at the time of the reduction process is directly used as the value of the normalized low-frequency component field of the original codebook 202 before the enlargement process. . By doing so, the number of elements stored in the normalized low frequency component field is reduced, so the SSD search processing unit 809, the RMSE calculation unit 813, the principal component analysis processing unit 204, the vector similarity calculation unit 1513, and the like. The number of calculation steps can be reduced.

  (6) Since the super-resolution image processing apparatus 101 according to the present embodiment performs normalization using the norm when creating the code book 106, the arithmetic processing includes floating-point arithmetic. By multiplying the value stored in the code book 106 by a predetermined value, it is possible to avoid floating point arithmetic and configure the apparatus only with integer arithmetic processing.

In the present embodiment, the super-resolution image processing apparatus 101 and the code book creation apparatus 110 used therefor have been disclosed.
In order to execute highly versatile super-resolution image processing in a short time, the super-resolution image processing apparatus 101 of the present embodiment first creates an original code book 202 from an image to be processed, and the original code book 202 From this, a code book 106 was selected in which only records suitable for super-resolution processing were selected.
In addition, a norm is used to effectively reduce the number of records used in the code book 106.
By adopting these techniques, it is possible to realize the super-resolution image processing apparatus 101 that achieves a high frequency component reproduction rate with a relatively small amount of calculation.

  The embodiment of the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and other modifications may be made without departing from the gist of the present invention described in the claims. Includes application examples.

DESCRIPTION OF SYMBOLS 101 ... Super-resolution image processing apparatus, 102 ... Digital camera, 103 ... Image file, 104 ... Input image data, 105 ... Output image data, 106 ... Code book, 107 ... First principal component basis vector, 108 ... Second main Component basis vectors, 109 ... third principal component basis vectors, 110 ... codebook creation device, 111 ... sample image data, 112 ... evaluation image data, 201 ... codebook record creation unit, 202 ... original codebook, 203 ... record Selection processing unit, 204 ... principal component analysis processing unit, 401 ... reduction processing unit, 402 ... reduced image data, 403 ... bicubic enlargement processing unit, 404 ... edge region determination unit, 405 ... sample bibic image data, 406 ... norm data calculation Part, 407 ... norm data, 408 ... divider, 409 ... normalized low Wave component, 410 ... adder, 411 ... high frequency component, 412 ... divider, 413 ... normalized high frequency component, 414 ... raster scan processing unit, 701 ... image, 702 ... reduced image, 703 ... low frequency component, 704 ... high frequency Components: 801... Evaluation reduced image data, 802. Evaluation evaluation Bicubic image data, 803... Local region extraction unit, 804... Evaluation local region low frequency component, 805... Norm calculation unit, 806. ... Divider, 808... Evaluation local region normalized low frequency component, 809... SSD search processing unit, 810... Multiplier, 811 ... Adder, 812 .. Evaluation local region super-resolution data, 813. 814... Comparator 815. RMSE threshold 816 Record copy processing unit 1101 Inner product calculator 1102 Inner product calculator 110 ... inner product calculator, 1501 ... Bicubic enlargement processing unit, 1502 ... input Bicubic image data, 1503 ... local region extraction unit, 1504 ... input local region low frequency component, 1505 ... norm calculation unit, 1506 ... input local region norm, 1507 ... Divider, 1508 ... input local region normalized low frequency component, 1509 ... first inner product operator, 1510 ... second inner product operator, 1511 ... third inner product operator, 1512 ... tree search unit, 1513 ... vector similarity calculation 1515 ... Normalized high frequency component, 1515 ... Multiplier, 1516 ... Local area high frequency component, 1517 ... Adder, 1518 ... Local area super-resolution image data, 1519 ... Join processing section

Claims (6)

An enlargement processing unit for enlarging input image data;
A local region extraction unit that extracts an input local region low-frequency component by extracting a predetermined rectangular range from the input image data enlarged by the enlargement processing unit;
A norm calculation unit for calculating a norm of the input local region low frequency component and outputting an input local region norm;
A divider for dividing the input local region low frequency component by the input local region norm and outputting an input local region normalized low frequency component;
Normalized low frequency component field that stores normalized low frequency components obtained by normalizing low frequency components in a local area obtained by extracting image data within a predetermined rectangular range, and normalized high frequencies obtained by normalizing the high frequency components in the local area A codebook in which records are grouped with principal component values based on the normalized low-frequency components, and a normalized high-frequency component field in which components are stored;
A tree search unit for tree-searching the codebook based on the input local region normalized low-frequency component to identify a predetermined record group;
A vector similarity calculation unit for calculating the similarity between the input local region normalized low frequency component and the normalized low frequency component of the predetermined record group to identify the most similar record;
A multiplier that multiplies the normalized high-frequency component of the record specified by the vector similarity calculation unit and the input local-region norm to output a local-region high-frequency component;
A super-resolution image processing apparatus comprising: an adder that adds the high frequency component of the local region and the low frequency component of the input local region. Furthermore,
A first inner product calculator that calculates an inner product of the input local region normalized low frequency component and the first principal component basis vector of the codebook and outputs a first principal component value;
A second inner product calculator that calculates an inner product of the input local region normalized low frequency component and the second principal component basis vector of the codebook and outputs a second principal component value;
The codebook includes the normalized low frequency component field, the normalized high frequency component field, a first principal component value field in which a first principal component value of the normalized low frequency component is stored, and the first main component value field. A first principal component group number field that stores a first principal component group number grouped for records that approximate component values, and a second principal component value field that stores a second principal component value of the normalized low-frequency component. And a second principal component group number field in which a second principal component group number grouped with respect to a record that approximates the second principal component value among records having the same first principal component group number is stored,
The tree search unit specifies a predetermined record group by performing a tree search of the codebook with the first principal component value and the second principal component value.
The super-resolution image processing apparatus according to claim 1.   The super-resolution image processing apparatus according to claim 1, wherein the enlargement processing unit executes a Bicubic enlargement process. A normalized low frequency component field in which a normalized low frequency component and a normalized high frequency component are created for each local region in which sample image data is extracted within a predetermined rectangular range, and the normalized low frequency component is stored. A code book record creation unit for creating an original code book composed of a normalized high-frequency component field in which the normalized high-frequency component is stored;
After specifying the record most similar to the original code book for each local region in which the image data for evaluation is extracted within a predetermined rectangular range and performing super-resolution image processing, the similarity is calculated and the original code book A code book creation device for super-resolution image processing comprising a record selection processing unit for selecting a record and creating a code book. Furthermore,
The code book is grouped into a first principal component value field storing a first principal component value obtained by principal component analysis of the normalized low frequency component of the code book, and a record that approximates the first principal component value. A first principal component group number field storing a first principal component group number; a second principal component value field storing a second principal component value of the normalized low frequency component; and the first principal component group number. A second principal component group number field for storing a second principal component group number grouped with respect to a record that approximates the second principal component value among records having the same value, a first principal component basis vector, and a second principal component basis vector, The super-resolution image processing codebook creation apparatus according to claim 4, further comprising a principal component analysis processing unit that calculates and outputs a component basis vector.   6. The codebook creation device for super-resolution image processing according to claim 5, wherein the codebook record creation unit executes Bicubic enlargement processing.