JP2016091060A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for reducing a processing amount when making a plurality of images high definition to achieve high definition.SOLUTION: An image processing apparatus includes: first generation means for generating a high order tensor of 3 order or more from input image data indicating an image; and second generation means for generating output image data which is image data obtained by making the input image data high definition by using the high order tensor.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for increasing the resolution of an image.

  As an image processing technology that generates high-resolution images from multiple low-resolution images, the number of pixels is increased by performing pixel interpolation through iterative operations using positional displacement in units of subpixels that exist between multiple low-resolution images. And a method for simultaneously restoring high-frequency components (Patent Document 1). Such high resolution processing algorithms include a MAP (Maximum A Poster) method and a POCS (Projection On to Convex Sets) method.

Japanese Patent No. 5197446

  However, when high resolution processing is performed using the technique described in Patent Document 1, only one high resolution image is generated, and a high resolution image corresponding to each of a plurality of low resolution images is obtained. It was necessary to perform high resolution processing multiple times. Furthermore, in the case of increasing the resolution of an image using the technique described in Patent Document 1, it is necessary to perform alignment between the images or perform an iterative calculation based on an observation model. Therefore, enormous calculation cost was necessary. Here, the observation model is a model of image degradation caused by the optical system or image sensor of the image pickup apparatus, and generally includes point image intensity distribution and downsampling information. In view of the above, an object of the present invention is to reduce the calculation cost when increasing the resolution of a plurality of images.

  In order to solve the above problems, an image processing apparatus according to the present invention uses first generation means for generating a three-dimensional or higher order tensor from input image data representing an image, and the input image data using the higher order tensor. And second generation means for generating output image data which is image data with a higher resolution.

  According to the present invention, it is possible to generate high-resolution image data using a three-dimensional or higher-order tensor generated from image data, so that it is possible to reduce the calculation cost when increasing the resolution of a plurality of images.

1 is a block diagram illustrating a configuration of an image processing apparatus according to Embodiment 1. FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to Embodiment 1. FIG. 3 is a flowchart illustrating a flow of processing performed by the image processing apparatus according to the first embodiment. The figure which shows the example of blank pixel insertion. 5 is a flowchart illustrating a flow of processing performed by a complementing unit 206 according to the first embodiment.

<Example 1>
In the present embodiment, a process for increasing the resolution of image data by applying a missing data interpolation technique in a tensor will be described. In other words, image data with improved resolution is obtained by intentionally inserting a missing pixel into a tensor generated from image data and using a data complement technique in the tensor for that pixel. A tensor is a generalization of a linear quantity. If a rule is selected, it is expressed as a multidimensional array. The number of dimensions of the tensor element spread is called the rank. For example, the scalar quantity is a rank 0 tensor, the vector quantity is a rank 1 tensor, and the matrix is a rank 2 tensor. That is, a three-dimensional array in which matrices are stacked in a direction different from the two dimensions expressed by the matrix is a rank 3 tensor (third-order tensor).

  FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus according to the present invention. The image processing apparatus 100 includes a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface 105, an output interface 106, and a system bus 107. The external storage unit 108 is connected to the input interface 105 and the output interface 106, and the display device 109 is connected to the output interface 106.

  The CPU 101 is a processing circuit that executes a program stored in the ROM 103 by using the RAM 102 as a work memory, and comprehensively controls each unit to be described later via the system bus 107. Thereby, various processes described later are executed. The secondary storage device 104 is a memory capable of accumulating data necessary for program execution via the system bus 107. The data stored in the secondary storage device 104 can be read out via the system bus 107. As the secondary storage device 104, a storage device such as an optical disk drive or a flash memory can be used in addition to the HDD.

  The input interface 105 is a serial bus interface such as USB or IEEE1394. Data can be acquired from the external storage means 108 (for example, hard disk, memory card, CF card, SD card, USB memory) via the input interface 105.

  The output interface 106 is a video output terminal such as DVI or HDMI (registered trademark). The processed image or the like can be displayed on the display device 109 (various output devices such as a liquid crystal display) via the output interface 106. In addition, the output interface 106 includes a serial bus interface such as USB or IEEE 1394, similar to the input interface 105. The image processing apparatus 100 can store data in the external storage unit 108 via the output interface 106.

  Note that the components of the image processing apparatus 100 exist in addition to those described above.

  Next, processing performed in the image processing apparatus 100 will be described with reference to a block diagram shown in FIG. 2 and a flowchart shown in FIG. In the image processing apparatus 100, the CPU 101 executes the program stored in the ROM 103 using the RAM 102 as a work memory, thereby functioning as each block illustrated in FIG. 2 and performing the process of each step illustrated in FIG. 3. Note that it is not necessary for the CPU 101 to execute all of the processes described below, and the image processing apparatus 100 may be configured such that part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. Good.

  In step S <b> 301, the acquisition unit 201 acquires input image data to be processed and outputs it to the dividing unit 202. The image data acquired here is a plurality of images obtained by viewing the same subject from a plurality of different viewpoints acquired by a multi-view camera or a plenoptic camera. In the present embodiment, a process for obtaining an image obtained by increasing the resolution of each of the plurality of images indicated by the input image data is performed. The input image data to be processed need not be data indicating a plurality of images, and may be data indicating a single image. The use of multiple images corresponding to different viewpoints improves the accuracy of super-resolution processing. However, even when only one image is used, it is common for images that have many similar regions in the image. It is possible to increase the resolution by using the characteristic.

  In step S302, the dividing unit 202 divides each image indicated by the input image data input from the obtaining unit 201 into a set of blocks (partial images) of a certain size, and the obtained set of blocks is the clustering unit 203. Output to. At this time, the dividing unit 202 stores position information indicating which position in the input image each block corresponds to in the RAM 102 or the secondary storage device 104. Note that if each block is extracted so as to have an overlapping area, the boundary between the blocks is less noticeable in the finally obtained high-resolution image. In this case, the total number of pixels of all blocks is larger than the total number of pixels of the input image.

  In step S303, the clustering unit 203 classifies the plurality of blocks input from the dividing unit 202 into a plurality of clusters according to a criterion such as similarity between the blocks (clustering is performed). As the similarity index, a known index such as SAD (Sum of Absolute Difference) or NCC (Normalized Cross Correlation) can be used. As a clustering method, a known method such as a k-means method can be used. In the present embodiment, SSD (Sum of Squared Difference) is used as an index of similarity. Further, the k-means ++ method is used as a clustering method. Note that the similarity index and the clustering method used here are not limited to those described above. For example, if the similarity criterion is changed for each cluster in consideration of the characteristics of each block, the processing cost is slightly increased, and more accurate super-resolution processing can be performed.

  Details of the clustering processing performed in this embodiment will be described. First, the clustering unit 203 converts each block into a monochrome image in order to handle each block as vector data. In the present embodiment, the clustering unit 203 converts each block into a luminance channel Y in the YCbCr space. Next, the clustering unit 203 randomly selects one of a plurality of vectors corresponding to each block obtained by this processing, and the sum of squared differences between the selected one vector and each of the remaining plurality of vectors. Is calculated as SSD. Next, the clustering unit 203 selects N vectors as the initial clustering solution for each cluster from the predetermined number N of clusters and the relative ratio of each SSD value. Next, the clustering unit repeats the k-means process until the end condition is satisfied based on the determined initial solution. That is, the SSD value is calculated for all combinations of N vectors that are the centroids of the clusters and the remaining vectors, and the clusters are updated based on the SSD values. The N vectors are newly set as the centroids of the clusters. Repeat a series of processing to replace. The number of clusters N is not limited to a fixed value, and may be adaptively adjusted according to the similarity between blocks. For example, N may be increased or decreased when the worst value of the similarity or the number of blocks in one cluster exceeds a threshold value. When the end condition of the clustering process such as the change amount of the cluster assignment before and after the processing falls below the threshold is satisfied, the clustering unit 203 outputs the block classification result to the generation unit 204.

  In step S <b> 304, the generation unit 204 generates a plurality of third-floor tensors by grouping a plurality of blocks belonging to the same cluster, and outputs the generated third-floor tensors to the insertion unit 205. Here, the third-order tensor is a three-dimensional array in which blocks that are two-dimensional images are stacked in the direction of the third dimension. In the present embodiment, as shown in FIG. 4A, similar blocks are extracted in each image to constitute one tensor. Each of the images 401 to 403 is a part of the image indicated by the input image data, and is a block in which the area surrounded by the dotted line is classified into the same cluster. They are accumulated to form a tensor 404. When the input image is color, a tensor is formed independently for each color channel. Note that it is desirable that the clustering method is common between the color channels so that a false color does not occur in the final output image. The tensor configuration is not limited to that described above, and a method as shown in FIG. That is, similar regions in one image may be classified into the same cluster to form a tensor.

  In step S <b> 305, the insertion unit 205 inserts blank pixels into the third-floor tensor input from the generation unit 204, and outputs the third-floor tensor into which the blank pixels are inserted to the complementing unit 206. Here, the blank pixel means a pixel having no pixel value, but the blank pixel is not necessarily a pixel having no value, and a pixel in which a temporary value is inserted is referred to as a blank pixel. May be. The blank pixels are inserted so that the ratio of the number of pixels in the horizontal and vertical directions (the coordinate axis direction of the two-dimensional plane image) of the third-floor tensor before and after blank pixel insertion matches the desired resolution increase rate. For example, when the resolution is increased so that the resolution in each of the horizontal and vertical directions is doubled, as shown in FIG. 4C, every other column in the horizontal and vertical directions of the third floor tensor. A blank pixel is inserted. In addition to the insertion method shown in FIG. 4 (c), for example, a method of making a blank pixel column inserted before or after a non-defective pixel column (a column of pixels that are not blank pixels) differs for each block. It may be used. Furthermore, a blank pixel may be inserted at a non-constant relative position with respect to each non-defective pixel. For example, when the resolution is increased so that the resolution in the horizontal and vertical directions is doubled, each non-defective pixel is replaced with a 2 × 2 pixel unit block, and one arbitrarily selected pixel in each unit block For example, a non-deficient pixel is inserted into the remaining three pixels and a blank pixel is inserted into the remaining three pixels. Note that the insertion unit 205 does not necessarily need to perform blank pixel insertion processing on the third-floor tensor, and may insert blank pixels into the input image or block before forming the third-floor tensor. . In this case, the generation unit 204 generates a third-floor tensor using input image data in which blank pixels are inserted.

  In step S306, the complementing unit 206 estimates the pixel value corresponding to the blank pixel using an algorithm that complements the missing element of the tensor, and outputs the third-order tensor complemented with the pixel value of the blank pixel to the decomposing unit 207. To do. Details of the processing performed here will be described later.

  In step S307, the decomposing unit 207 decomposes the third-floor tensor input from the complementing unit 206 into two-dimensional image blocks, and outputs a plurality of blocks obtained by the decomposing to the integrating unit 208. Here, each block obtained by the decomposition is a block having a higher resolution than the original block due to the result of blank pixel insertion and blank pixel interpolation processing.

  In step S308, the integration unit 208 integrates the image blocks input from the decomposition unit 207 and generates high-resolution image data. Here, the integration unit 208 generates high-resolution image data by applying each block to the corresponding position of the input image data based on the information indicating the position corresponding to each block stored in step 302. In addition, when each block has an image area | region which overlaps mutually, the pixel value of the pixel of the overlapped part is processed by applying the average value of the pixel value of the some pixel of a corresponding position. The pixel value generation method is not limited to this, and other methods may be used as necessary. As a result, high-resolution images are obtained for all the images indicated by the input image data. Note that the integration process need not be performed for all images, and only a high-resolution image corresponding to a specific image designated by the user may be generated. Thus, even when only a specific image is output, inputting another image is useful for the purpose of improving the accuracy of the complementing process in the complementing unit 206. The integration unit 208 outputs the generated high resolution image data as output image data to the secondary storage device 104 or the like, and ends the processing.

  The above is the flow of processing performed by the image processing apparatus 100 of the present embodiment. Next, details of the complementing process (step S306) performed by the complementing unit 206 will be described with reference to the flowchart shown in FIG.

  In step S501, the complementing unit 206 determines the initial value of the element corresponding to the missing pixel (blank pixel) of the tensor based on the pixel value of the element corresponding to the non-deficient pixel (non-blank pixel). As a method for determining the initial value, a known interpolation method such as a nearest neighbor method, bilinear interpolation, or bicubic interpolation can be used. In addition, when the position of the defective pixel row is different for each block, the value of the defective pixel may be determined from the pixel value of the non-defective pixel at the same position in another block.

  In step S502, the complementing unit 206 expands each third-floor tensor into a matrix (second-floor tensor) for one of the dimensions indicated by the third-floor tensor. The process performed here is an element rearrangement process in which a tensor is once decomposed into a specified dimension column vector and juxtaposed, and is also called unfolding.

In step S503, the complement unit 206 performs singular value decomposition on the matrix obtained in step S502. Singular value decomposition is a technique for decomposing an arbitrary matrix into a product of three matrices including a diagonal matrix. For example, the singular value decomposition of an n-row × m-column matrix A _nm is expressed by the following equation.

Here, U _nr is an n-row × r-column matrix, Σ _rr is an r-row × r-column diagonal matrix called a singular value matrix, and V _rm is an r-row × m-column matrix. Here, r is called the rank (rank) of the matrix A _nm .

In step S504, the complementing unit 206 performs threshold processing on the result of the singular value decomposition obtained in step S503. Specifically, among the elements of the singular value matrix, all elements having a value equal to or less than a predetermined threshold are replaced with 0. This substantially reduces the rank of the matrix A _nm to the number of non-zero elements of the singular value matrix. Since the matrix corresponding to the partial region of the natural image is generally low rank, the result of the interpolation performed in step S501 is closer to the natural image by performing correction processing for lowering the rank of the matrix Anm here. Replaced with the result. In step S505, the complementing unit 206 converts the matrix into a third-order tensor by an operation opposite to the operation performed in step S503.

  In step S506, the complementing unit 206 determines whether or not the processing from step S502 to step S505 has been performed for all three dimensions corresponding to the third-floor tensor. If it is determined that the processing from step S502 to step S505 has been performed for all dimensions, the process proceeds to step S507. If it is determined that the processing from step S502 to step S505 is not performed for all dimensions, the process returns to step S502, and the tensor is expanded for another dimension.

  In step S507, the complementing unit 206 averages the tensors obtained for each of the three dimensions as a result of the rank lowering process. The averaging process performed here is a process in which the average value of the elements existing at the same coordinates of the three tensors is used as the final tensor value. The averaging process performed here does not have to be a simple averaging process, and may be a weighted averaging process in which each tensor is weighted according to the reliability of rank reduction processing in each dimension. .

  In step S508, the complementing unit 206 replaces the value of the element corresponding to the non-missing pixel in each element of the tensor obtained in step S507 with the value before the process of step S504 is performed. This prevents the data corresponding to the pixel indicated by the input image data (the pixel indicating the actually captured information) from being rewritten by the process of lowering the tensor rank or the process of averaging the tensor. Can do. Here, without replacing the value, the value of the element corresponding to the non-missing pixel may be excluded from the data update target.

  In step S509, the complement unit 206 determines whether or not the completion condition for the complement process is satisfied. In the present embodiment, the completion condition of the complement process is that the number of iterations of the complement process has reached a predetermined number. When it is determined that the number of repetitions of the complement process has reached a predetermined number, the complement process is terminated, and the complement unit 206 outputs the third-floor tensor subjected to the complement process to the decomposition unit 207. If it is determined that the number of iterations of the complement process has not reached the predetermined number, the process returns to step 402, and the processes from step S502 to step S508 are executed again using the third-order tensor at that time as the initial solution.

  The above is the process performed by the image processing apparatus 100 of the present invention. According to the above processing, a plurality of high-resolution images obtained by increasing the resolution of a plurality of images indicated by the input image data by using the concept of tensor to increase the resolution of the image data is converted into the MAP method or POCS. Compared to the case of using the method, it can be obtained with a small processing amount.

  In the above-described embodiment, the generation unit 204 functions as a first generation unit that generates higher-order tensors on the third floor or higher from input image data indicating a plurality of images including the same subject. The integration unit 208 includes a second generation unit that generates output image data that is image data obtained by increasing the resolution of the input image data using the higher-order tensor. Function. The insertion unit 205 functions as an insertion unit that inserts a blank pixel into the higher order tensor. The complementing unit 206 functions as a complementing unit that complements the blank pixel. The clustering unit 203 functions as a clustering unit that classifies a plurality of image blocks included in the input image data based on the similarity between the image blocks.

<Other embodiments>
In the above embodiment, the super-resolution processing of the image data is performed by applying the third-layer tensor defect complement processing. However, the tensor rank is not limited to this, and any tensor of the third floor or higher may be used. Alternatively, the defect complementing process may be performed using a fourth-floor tensor that is a four-dimensional array in which a plurality of third-floor tensors are stacked in a four-dimensional direction. When a tensor of the fourth floor or higher is used, the processing amount increases, but more accurate defect complement processing can be performed. Here, a tensor having three or more floors is referred to as a higher-order tensor.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 201 Acquisition part 202 Division part 203 Clustering part 204 Generation part 205 Insertion part 206 Complement part 207 Decomposition part 208 Integration part

Claims (12)

First generation means for generating a higher-order tensor of the third floor or higher from input image data indicating an image;
An image processing apparatus comprising: a second generation unit configured to generate output image data which is image data obtained by increasing the resolution of the input image data using the higher-order tensor.   The image processing apparatus according to claim 1, wherein the input image data is data indicating a plurality of images when the same subject is viewed from a plurality of different viewpoints. Insertion means for inserting blank pixels into the higher order tensor;
Complementing means for complementing the blank pixels inserted into the higher order tensor by the insertion means;
3. The image processing apparatus according to claim 1, wherein the second generation unit generates the output image data using the higher-order tensor in which the blank pixel is complemented by the complementing unit. Inserting means for inserting blank pixels into the input image data;
Complementing means for complementing the blank pixels further,
The first generation means generates the higher order tensor using the input image data in which the blank pixel is inserted,
The complement means complements the blank pixels included in the higher order tensor,
3. The image processing apparatus according to claim 1, wherein the second generation unit generates the output image data using the higher-order tensor in which the blank pixel is complemented by the complementing unit.   The image processing apparatus according to claim 4, wherein the complementing unit complements the blank pixel so that a rank of the higher-order tensor becomes small. The complement means determines an initial value of an element corresponding to the blank pixel based on a value of an element corresponding to a pixel that is not the blank pixel of the higher-order tensor,
Further, correction is performed to set the elements having a value smaller than a predetermined threshold to 0 for the singular value matrix obtained by performing singular value decomposition on the higher-order tensor in which the initial value is given to the element corresponding to the blank pixel. The image processing apparatus according to claim 5, wherein the value of each element of the higher order tensor is corrected. The complement means replaces the value of an element corresponding to a pixel different from the blank pixel of the higher-order tensor whose value is corrected by threshold processing of the singular value matrix with a pixel value indicated by the input image data,
The image processing apparatus according to claim 6, wherein threshold processing of the singular value matrix is performed again on the higher-order tensor in which the element value is replaced.   The image processing apparatus according to claim 4, wherein the blank pixel is a pixel having no pixel value.   The image processing apparatus according to claim 4, wherein the blank pixel is a pixel having a provisional pixel value. A clustering unit that classifies a plurality of image blocks included in the input image data based on a similarity between the image blocks;
The said 2nd production | generation means produces | generates one high-order tensor using the several image block classified into the same cluster by the said clustering means, The 1st to 9 characterized by the above-mentioned. Image processing apparatus. A first generation step of generating a higher-order tensor of the third floor or higher from input image data indicating an image;
And a second generation step of generating output image data, which is image data obtained by increasing the resolution of the input image data, using the higher-order tensor.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 10.

Images