WO2014077024A1 - Image processing device, image processing method and image processing program - Google Patents

Abstract

An image forming device (1) comprises a super-resolution preprocessing unit (34), a super-resolution processing unit (36), and an image synthesis unit (38) in an image processing unit (3) for executing a process in which, from a group of multi-view input images, outputs a high resolution image as an output image, said high resolution image having more frequency information than an input image. The super-resolution preprocessing unit (34): splits an input image to establish subregions, and calculates the degree of similarity between corresponding subregions with regard to a plurality of input images; and includes an assessment unit (341) for determining whether super-resolution processing is necessary for each subregion on the basis of the degree of similarity. The super-resolution processing unit (36) executes super-resolution processing for subregions deemed to require super-resolution processing of the input image. The image synthesis unit (38) performs processing that includes a process for converting the resolution of subregions deemed not to require super-resolution processing of the input image to the resolution of the output image.

Description

Image processing apparatus, image processing method, and image processing program

The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and more particularly, to an image processing apparatus, an image processing method, and an image processing program for performing processing for improving resolution.

There is an image processing technology that generates a single high-resolution image from a low-resolution multi-viewpoint input image group. Such processing is also called super-resolution processing.

For example, Japanese Patent Laid-Open No. 2011-171843 (hereinafter referred to as Patent Document 1) extracts a high-frequency component of an input low-quality image to be restored and generates a high-frequency low-quality image when generating a high-quality image from the low-quality image. Disclosure of technology to increase the processing speed of interpolation processing by generating an image and partially determining whether or not to execute interpolation calculation processing for interpolating ultra-high frequency components and switching to execute interpolation calculation processing is doing.

Japanese Patent Laid-Open No. 2010-108161 (hereinafter referred to as Patent Document 2) acquires a plurality of low-resolution images as a device for acquiring high-resolution images, and generates a high-resolution image based on at least a part thereof. A high-resolution image generating means for performing the high-resolution processing, a setting means capable of setting the resolution enlargement ratio and the number of low-resolution images to be subjected to the high-resolution processing, and the generated high-resolution image And determining means for determining whether or not at least one of the enlargement ratio and the number of sheets needs to be changed, and determining that at least one of the enlargement ratio and the number of sheets needs to be changed, An image processing apparatus capable of acquiring a high-resolution image having a target image quality while shortening the processing time by including a control unit that controls the setting unit so as to change at least one of them. It discloses.

Japanese Patent Laid-Open No. 2007-305113 (hereinafter referred to as Patent Document 3) is an image processing apparatus that generates a high resolution image using a representative image having a low resolution and a plurality of reference images, while switching the reference image. Repeated the alignment process, repeated the update process to update the pixel estimation value of the high-resolution image to be obtained, determined the pixels satisfying the end condition from the result of the alignment process or update process, and determined to satisfy the end condition An image processing apparatus that can reduce the amount of calculation without degrading the image quality after super-resolution by excluding pixels from the alignment process or the update process is disclosed.

JP 2011-171843 A JP 2010-108161 A JP 2007-305113 A

However, when the techniques disclosed in Patent Documents 1 to 3 are employed, super-resolution processing is performed at all positions of the image. That is, since the super-resolution processing is performed even on the unfocused position in the image, there is a problem that the processing for that position is wasteful and processing for generating a high-resolution image takes time. .

The present invention has been made in view of such problems, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of speeding up a process for generating a high-resolution image.

In order to achieve the above object, according to an aspect of the present invention, an image processing apparatus executes a process of outputting a high-resolution image having more frequency information than an input image as an output image from a multi-viewpoint input image group. An image processing apparatus that divides an input image to set partial areas, calculates a similarity between corresponding partial areas for a plurality of input images, and performs super solution for each partial area based on the similarity. Determination means for determining whether image processing is necessary, first processing means for performing super-resolution processing on a partial area of the input image that is determined to be subjected to super-resolution processing in the determination means, Second processing means for executing processing for converting the resolution of the partial area of the input image determined not to be subjected to the super-resolution processing into the resolution of the output image.

Preferably, the determination unit uses at least one of the density, the color value, and the sharpness of the partial region for calculating the similarity.

Preferably, the determination unit calculates the similarity by performing template matching.
Preferably, the second processing unit synthesizes partial areas of the plurality of input images that are determined not to be subjected to super-resolution processing.

Preferably, the second processing means uses the partial area of one input image in the input image group as the partial area determined not to perform the super-resolution processing.

Preferably, the determination unit determines whether or not the super-resolution processing for the partial area is necessary for each specific pixel of the partial area.

Preferably, the input image is a color image composed of a plurality of color channels, and the determination unit determines whether or not super-resolution processing is necessary for one of the plurality of color channels.

Preferably, the input image is a color image composed of a plurality of color channels, and the determination unit determines whether or not super-resolution processing is necessary for an image converted from the plurality of color channels to one channel.

Preferably, the determination unit determines whether or not the super-resolution processing is necessary using an image obtained by reducing the input image.

Preferably, the determination unit selects a prescribed number from the input image group and divides each of the selected input images to set a partial region.

More preferably, the input image group is temporally continuous, and the determination unit selects the input image for each specific image in the order of continuous.

According to another aspect of the present invention, an image processing method is a method for generating, as an output image, a high-resolution image having more frequency information than an input image from a multi-viewpoint input image group. A step of setting a region, a step of calculating a similarity between corresponding partial regions for a plurality of input images, a step of determining whether or not super-resolution processing is necessary for each partial region based on the similarity, and an input The step of executing the super-resolution processing on the partial area determined to perform the super-resolution processing of the image, and the resolution of the partial area of the input image that is determined not to perform the super-resolution processing, And a step of executing a process of converting to resolution.

According to still another aspect of the present invention, an image processing program is a program for causing a computer to execute processing for generating, as an output image, a high-resolution image having more frequency information than an input image from a multi-viewpoint input image group. A step of setting a partial region by dividing the input image, a step of calculating a similarity between corresponding partial regions for a plurality of input images, and a step of super-resolution processing for each partial region based on the similarity. A step of determining NO, a step of executing super-resolution processing on a partial area of the input image determined to perform super-resolution processing, and a portion of the input image determined not to perform super-resolution processing And causing the computer to execute a process of converting the resolution of the region into the resolution of the output image.

According to the present invention, in a process for generating a high resolution image including a super-resolution process such as a refocus process, it is possible to speed up the process without degrading the image quality.

1 is a block diagram illustrating a basic configuration of an image processing apparatus according to an embodiment. It is a block diagram which shows the structure of the digital camera which actualized the image processing apparatus. It is a block diagram which shows the structure of the personal computer which actualized the image processing apparatus. It is a figure showing the specific example of arrangement | positioning of the lens contained in a camera. It is a figure showing the flow of the 1st refocus process. It is a figure showing the specific example of the imaging | photography condition. FIG. 7 is a diagram illustrating a part of an input image from the camera when the subject in FIG. 6 is captured by the camera. It is a figure showing operation | movement in step # 11 of FIG. It is a figure showing the synthesized image of the input image group of FIG. It is a figure showing operation | movement in step # 11 of FIG. It is a figure showing the synthesized image of the input image group of FIG. It is a figure showing a synthesized image when the input image group of FIG. 7 is synthesized without performing the process of step # 11 of FIG. It is a figure showing the flow of the 2nd refocus process. It is a figure showing the flow of the super-resolution process in step # 22 of FIG. It is a figure showing the detail of the process in step # 33 of FIG. It is a figure for demonstrating deterioration information. It is a figure showing the specific example of deterioration information. It is a figure showing the specific example of the input image (a) and the high resolution image (b) before and after super-resolution processing. It is a flowchart showing the flow of operation | movement in an image processing apparatus. It is a flowchart showing the flow of the pre-processing in step S105 of FIG. It is a figure showing the determination result in step S209 of FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, these descriptions will not be repeated.

<System configuration>
FIG. 1 is a block diagram showing a basic configuration of an image processing apparatus 1 according to the present embodiment.

Referring to FIG. 1, the image processing apparatus 1 includes an imaging unit 2, an image processing unit 3, and an image output unit 4. In the image processing apparatus 1 illustrated in FIG. 1, the image capturing unit 2 captures an image of a subject to acquire an image (hereinafter also referred to as “input image”), and the image processing unit 3 performs an operation on the acquired input image. By performing image processing as will be described later, a high-resolution output image (hereinafter also referred to as “high-resolution image”) having more frequency information than the input image is generated. Then, the image output unit 4 outputs this high resolution image to a display device or the like.

The imaging unit 2 captures an object (subject) and generates an input image. More specifically, the imaging unit 2 includes a camera 22 and an A / D (Analog to Digital) conversion unit 24 connected to the camera 22. The A / D converter 24 outputs an input image indicating the subject imaged by the camera 22.

The camera 22 is an optical system for imaging a subject, and is an array camera. In other words, the camera 22 uses N lenses 22a-1 to 22a-n (which are also referred to as lenses 22a, which are different from each other) arranged in a lattice shape, and the light collected by the lenses 22a is electrically And an image sensor 22b which is a device for converting the signal. The A / D converter 24 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 22b into a digital signal and outputs the digital signal. The imaging unit 2 may further include a control processing circuit for controlling each part.

The image processing unit 3 generates a high-resolution image by performing the image processing method according to the present embodiment on the input image acquired by the imaging unit 2. More specifically, the image processing unit 3 includes a refocus processing unit 32, a super-resolution preprocessing unit 34, a super-resolution processing unit 36, and an image composition unit 38.

The refocus processing unit 32 executes refocus processing described later on a plurality of input images from the camera 22.

The super-resolution processing unit 36 performs super-resolution processing described later on the input image during the refocus processing. The super-resolution processing is processing for generating frequency information that exceeds the Nyquist frequency of the input image. At that time, the super-resolution pre-processing unit 34 performs pre-processing including determination processing to be described later, and distinguishes a target area for super-resolution processing and a non-target area in the input image. The super-resolution preprocessing unit 34 includes a determination unit 341 for performing the determination process.

The image synthesis unit 38 performs a process of synthesizing an input image or an area of the input image where super-resolution processing is not performed.

The image output unit 4 outputs the high resolution image generated by the image processing unit 3 to a display device or the like.

The image processing apparatus 1 shown in FIG. 1 can be configured as a system in which each unit is embodied as an independent apparatus. However, for general purposes, the image processing apparatus 1 may be embodied as a digital camera or a personal computer described below. Many. Therefore, as an implementation example of the image processing apparatus 1 according to the present embodiment, an implementation example with a digital camera and an implementation example with a PC (personal computer) will be described.

FIG. 2 is a block diagram showing a configuration of a digital camera 100 that embodies the image processing apparatus 1 shown in FIG. 2, components corresponding to the respective blocks constituting the image processing apparatus 1 shown in FIG. 1 are denoted by the same reference numerals as those in FIG. 1.

Referring to FIG. 2, a digital camera 100 includes a CPU (Central Processing Unit) 102, a digital processing circuit 104, an image display unit 108, a card interface (I / F) 110, a storage unit 112, and a camera unit. 114.

The CPU 102 controls the entire digital camera 100 by executing a program stored in advance. The digital processing circuit 104 executes various digital processes including image processing according to the present embodiment. The digital processing circuit 104 is typically configured by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), or the like. The digital processing circuit 104 includes an image processing circuit 106 for realizing the functions provided by the image processing unit 3 shown in FIG.

The image display unit 108 includes an input image provided by the camera unit 114, an output image generated by the digital processing circuit 104 (image processing circuit 106), various setting information related to the digital camera 100, and a control GUI (Graphical User Interface) screen is displayed.

The card interface (I / F) 110 is an interface for writing image data generated by the image processing circuit 106 to the storage unit 112 or reading image data or the like from the storage unit 112. The storage unit 112 is a storage device that stores image data generated by the image processing circuit 106 and various types of information (setting values such as control parameters and operation modes of the digital camera 100). The storage unit 112 includes a flash memory, an optical disk, a magnetic disk, and the like, and stores data in a nonvolatile manner.

The camera unit 114 generates an input image by imaging a subject.
A digital camera 100 shown in FIG. 2 is obtained by mounting the entire image processing apparatus 1 according to the present embodiment as a single apparatus. That is, the user can visually recognize a high-resolution image on the image display unit 108 by imaging the subject using the digital camera 100.

FIG. 3 is a block diagram showing a configuration of a personal computer 200 that embodies the image processing apparatus 1 shown in FIG. In the personal computer 200 shown in FIG. 3, the imaging unit 2 for acquiring an input image is not mounted, and an input image acquired by an arbitrary imaging unit 2 is input from the outside. Even such a configuration can be included in the image processing apparatus 1 according to the embodiment of the present invention. In FIG. 3 as well, components corresponding to blocks constituting the image processing apparatus 1 shown in FIG. 1 are denoted by the same reference numerals as those in FIG.

3, the personal computer 200 includes a personal computer main body 202, a monitor 206, a mouse 208, a keyboard 210, and an external storage device 212.

The personal computer main body 202 is typically a general-purpose computer according to a general-purpose architecture, and includes a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as basic components. The personal computer main body 202 can execute an image processing program 204 for realizing a function provided by the image processing unit 3 shown in FIG. Such an image processing program 204 is stored and distributed in a storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or distributed from a server device via a network. The image processing program 204 is stored in a storage area such as a hard disk of the personal computer main body 202.

Such an image processing program 204 implements processing by calling necessary modules among program modules provided as part of an operating system (OS) executed by the personal computer main body 202 at a predetermined timing and order. It may be configured as follows. In this case, the image processing program 204 itself does not include a module provided by the OS, and image processing is realized in cooperation with the OS. Further, the image processing program 204 may be provided by being incorporated in a part of some program instead of a single program. Even in such a case, the image processing program 204 itself does not include a module that is commonly used in the program, and image processing is realized in cooperation with the program. Even such an image processing program 204 that does not include some modules does not depart from the spirit of the image processing apparatus 1 according to the present embodiment.

Of course, some or all of the functions provided by the image processing program 204 may be realized by dedicated hardware.

The monitor 206 displays a GUI screen provided by an operating system (OS), an image generated by the image processing program 204, and the like.

The mouse 208 and the keyboard 210 each accept a user operation and output the contents of the accepted user operation to the personal computer main body 202.

The external storage device 212 stores an input image acquired by some method, and outputs this input image to the personal computer main body 202. As the external storage device 212, a device that stores data in a nonvolatile manner such as a flash memory, an optical disk, or a magnetic disk is used.

A personal computer 200 shown in FIG. 3 is obtained by mounting a part of the image processing apparatus 1 according to the present embodiment as a single apparatus.

<Overview of operation>
The image processing apparatus 1 obtains a high-resolution image by performing refocus processing on a plurality of input images at different viewpoints obtained by photographing with the camera 22 that is an array camera. At this time, the image processing apparatus 1 includes a region where super-resolution processing is performed for an in-focus region in a high-resolution image (image with a large number of pixels) after refocus processing, and a region which is not performed for an out-of-focus region. By distinguishing, super-resolution processing is performed at high speed.

(Refocus processing 1)
As a first example of refocusing, first refocusing processing will be described. The first refocus processing does not include super-resolution processing.

FIG. 4 is a diagram showing a specific example of the arrangement of the lenses 22 a included in the camera 22. In the example of FIG. 4, as an example, it is assumed that the camera 22 is an array camera including 16 lenses 22a-1 to 22a-16 (lenses A to P) arranged in a grid pattern. Assume that the intervals (base line lengths) between the lenses A to P in FIG. 4 are uniform in both the vertical and horizontal directions.

FIG. 5 is a diagram showing the flow of the first refocus processing. Referring to FIG. 5, in the first example, each of the 16 input images obtained from the camera 22 of FIG. 4 has a length proportional to the baseline length from the reference image of the input images. Are moved in parallel (step # 11), and these are combined (step # 12), and the combined image is output. Since the first refocus processing does not include super-resolution processing, the resolution of the input image is equal to the resolution of the output image.

FIG. 6 is a diagram showing a specific example of the shooting situation. With reference to FIG. 6, as an example, one first subject (apple) arranged at a position near the camera 22 on the front surface (Z direction) of the camera 22 and a position far from the camera 22 and the camera 22. It is assumed that two second subjects (mandarin oranges) of the same type arranged at the same distance from the camera are photographed by the camera 22.

FIG. 7 is an input image from the camera 22 when the subject of FIG. 6 is photographed by the camera 22, and among the 16 lenses A to P, the lens A, which is an array of 3 vertical and horizontal grids. , B, C, E, F, G, I, J, and K are diagrams representing input images. Input images A, B, C, E, F, G, I, J, and K represent that they are input images from lenses A, B, C, E, F, G, I, J, and K, respectively. ing. Lenses A, B, C, lenses E, F, G, and lenses I, J, K are arranged at equal intervals in that order in the X direction in the positional relationship with the subject in FIG. The lenses B, F, J and the lenses C, G, K are arranged at equal intervals in the Y direction in that order.

Referring to FIG. 7, when the parallax of a subject whose distance from the lens 22a is infinite is 0, the first subject (apple) and the second subject (mandarin orange) in the input image A from the viewpoint of the lens A And the respective positions as reference positions, the position of the image of the apple that is the first subject in the input image is -X from the reference position by the distance c in the order of lens arrangement in the input image from the lens arranged in the X direction. Deviation in direction. In the input image from the lens arranged in the Y direction, the distance d is shifted from the reference position in the −Y direction by the distance d in the lens arrangement order. The position of the second mandarin orange image in the input image is shifted from the reference position in the −X direction by a distance a in the order of lens arrangement in the input image from the lens arranged in the X direction. In the input image from the lens arranged in the Y direction, the distance b is shifted from the reference position in the −Y direction by the distance b in the lens arrangement order.

Suppose that the mandarin orange to be focused is the second subject, the input image is translated in step # 11 so that the position of the mandarin orange in each input image is matched.

FIG. 8 is a diagram showing the operation in Step # 11 when the focus target is the orange that is the second subject. Referring to FIG. 8, when the moving distance of input image B is (a, 0), the moving distance of input image C is such that the base line length from input image A as a reference is twice that of input image B. Therefore, (2a, 0) is obtained. Thereafter, the other input images are moved by the movement distance obtained in the same manner.

When the input images other than the reference input image A are translated as described above, the position of the orange in each input image coincides with the position in the reference input image A as shown in FIG. However, the position of the apple is different for all input images.

All input images are translated by a length proportional to the baseline length from the reference input image, and then synthesized in step # 12. Specifically, an average value of 16 pixel values is output for each pixel.

FIG. 9 shows a composite image of the input image group in FIG. As shown in FIG. 9, when the mandarin oranges are translated and matched so as to match the positions of the mandarin oranges as described above, the mandarin oranges are in focus in the synthesized image, but the apples are positioned in the input images (disparity). ) Is different, the focus is blurred. That is, in the first refocus processing, as the number of input images with different viewpoints increases, a subject whose position does not match in each input image appears to be naturally blurred.

Similarly, when the focus object is the apple that is the first subject, the input image is translated in step # 11 so that the position of the apple in each input image matches.

FIG. 10 is a diagram showing the operation in step # 11 when the focus target is the first object, phosphorus. Referring to FIG. 10, when the moving distance of input image B is (c, 0), the moving distance of input image C is such that the base line length from input image A as a reference is twice that of input image B. Therefore, (2c, 0) is obtained. Thereafter, the other input images are moved by the movement distance obtained in the same manner.

When the input image other than the reference input image A is translated as described above, the position of the apple in each input image matches the position in the reference input image A as shown in FIG. However, the position of the mandarin orange is different for all input images.

FIG. 11 shows a composite image of the input image group in FIG. As shown in FIG. 11, if the apples are synthesized after being translated so as to match the positions of the apples as described above, the apples are in focus in the synthesized image, but the oranges are in the positions (parallaxes) in each input image. ) Is different, the focus is blurred.

Note that FIG. 12 is a diagram illustrating a composite image when the input image group of FIG. 7 is combined without performing the process of step # 11. Referring to FIG. 12, in this case, neither the apple nor the orange is in focus in the composite image. In the case of this example, since the orange is farther from the camera 22 than the apple, the orange has a smaller parallax than the apple. For this reason, in FIG. 12, the degree of apples is greater than the degree of oranges.

(Refocus process 2)
As a second example of refocusing, a second refocusing process will be described. The second refocus process includes a super-resolution process.

FIG. 13 is a diagram showing the flow of the second refocusing process. Referring to FIG. 13, in the second example, after each of the 16 input images obtained from the camera 22 of FIG. 4 is moved in parallel as in the first process (step # 21), Resolution processing is performed (step # 22). Since the second refocus processing includes super-resolution processing, the resolution of the output image becomes higher than the resolution of the input image. As an example, when the input image has a resolution of 800 × 600 pixels, the output image may have a resolution of 3200 × 2400 pixels.

In the parallel movement in step # 21, in the second refocusing process, as described later in the super-resolution process, a process for improving the resolution is performed using information on sub-pixels (decimal pixels). Integer pixels are translated. For example, assuming that the movement amount (a, 0) is (15.25, 0) in the input image B of FIG. 7, in step # 21, 15 pixels are translated in parallel, and the remaining 0.25 pixels will be described later. This is used as a part of deterioration information in super-resolution (FIGS. 16 and 17).

FIG. 14 is a diagram showing the flow of super-resolution processing in step # 22. In FIG. 14, as a specific example, the super processing when the processing described in the paper “Fast and Robust Multiframe Super Resolution” (IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 10, OCTOBER 2004 page.1327-1344) is performed. The flow of resolution processing is shown.

Referring to FIG. 14, in step # 31, one of the input images is subjected to an interpolation process such as a bilinear method to convert the resolution of the input image to a high resolution that is the resolution after the super-resolution process. Thus, an output candidate image as an initial image is generated.

In step # 32, a BTV (Bilateral Total Variation) amount for robust convergence to noise is calculated.

In step # 33, the generated output candidate images are compared with the 16 input images, and a residual is calculated. FIG. 15 is a diagram showing details of the process in step # 33. That is, referring to FIG. 15, in step # 33, the generated output candidate image includes each input image and its deterioration information (information indicating the relationship between the super-resolution image and the input image). Is converted into an input image size (reduction in resolution) based on (# 41), and the difference from the 16 input images is calculated and recorded (# 42). Then, the difference is returned to the size after the super-resolution processing (# 43) to be a residual.

In step # 34, the residual and the BTV amount calculated from the output candidate image generated in step # 31 are reduced, and the next output candidate image is generated.

The processes in steps # 31 to # 34 are repeated until the output candidate image converges, and the converged output candidate image is output as an output image after the super-resolution processing.

The number of iterations may be a predetermined number of times such as the number of times of convergence (for example, 200 times), or a convergence determination may be made for each series of processing, and may be repeated according to the result. .

FIG. 16 is a diagram for explaining the deterioration information used in step # 41.
Deterioration information refers to information representing the relationship of each input image with respect to a high-resolution image after super-resolution processing, and is represented, for example, in a matrix format. The deterioration information includes a shift amount (for the remaining small number of translated pixels) at each sub-pixel level of each input image, a down-sampling amount, a blur amount, and the like.

Referring to FIG. 16, the deterioration information is defined by a matrix indicating the conversion when each of the input image and the high-resolution image after super-resolution processing is expressed in a one-dimensional vector.

FIG. 17 is a diagram illustrating a specific example of deterioration information.
Referring to FIG. 17, it is assumed that the amount of pixel shift is 0.25 pixels and the downsampling amount is defined as 1/4 in the vertical and horizontal directions as deterioration information. When one location corresponding to one pixel in the input image corresponds to 16 pixels in 16 locations in the high-resolution image after super-resolution processing, the degradation information includes 1 in 16 locations. A coefficient of / 16 is listed. Therefore, when the pixel shift amount is 0.25 pixel, each 16 pixels of the high resolution image contributes by 1/16 of the shift amount.

FIG. 18 is a diagram illustrating a specific example of the input image (a) and the high-resolution image (b) before and after the super-resolution processing. In each of FIGS. 18A and 18B, the same figure is shown in which a thick band spreads from the left to the right and becomes linear, and there is a limit that can be identified as a line. It is added with a bold line.

As shown in FIG. 18, the limit that can be identified as a line in the high-resolution image of FIG. 18B is on the left side than the limit that can be identified as a line in the input image of FIG. It can be seen that 18 (b) is more detailed. That is, in the high resolution image of FIG. 18B, information on the frequency exceeding the Nyquist frequency of the input image is generated. Thus, even a portion that cannot be read as characters in the input image is reproduced so that it can be read in a high-resolution image that has been subjected to super-resolution processing.

Note that the super-resolution processing method shown in FIG. 14 is an example, and other methods may be used. For example, in the case of reconstruction type super-resolution processing that generates one high-resolution image from a plurality of input images, the BTV amount may be another term.

(Explanation of issues)
In the second refocusing process including the super-resolution process, a high-resolution image with higher image quality can be obtained by the super-resolution process than in the first refocusing process in which the super-resolution process is not performed. However, since the convergence calculation is repeatedly performed in the super-resolution processing as described above, the processing speed is very slow compared to the first refocus processing.

Therefore, the image processing apparatus 1 according to the present embodiment performs preprocessing for determining whether or not to perform super-resolution processing for each partial region.

In this example, the necessity for the super-resolution process when performing the refocus process is determined in advance, but the prior determination of the necessity of the super-resolution process is included in the super-resolution process included in the refocus process. The present invention is not limited to image processing, and whether it is super-resolution processing other than refocus processing can be similarly determined in advance.

<Operation flow>
FIG. 19 is a flowchart showing the flow of operations in the image processing apparatus 1 according to the present embodiment. Again, according to the above example, one high-resolution image of 3200 × 2400 pixels is generated from 16 input images having a low resolution of 800 × 600 pixels.

Referring to FIG. 19, first, a process for acquiring 16 input images is executed (step S101). Here, it is assumed that a low-resolution image of 800 × 600 pixels is input.

Next, a process of moving the input images other than the reference image in parallel by a length proportional to the baseline length from the reference image is performed (step S103). Then, pre-processing for super-resolution processing is performed (step S105), and it is determined for each partial region whether or not super-resolution processing is necessary.

For the partial area determined to be subjected to the super-resolution processing (YES in step S107), the super-resolution processing as described above is performed (step S109).

For the partial area determined not to be subjected to the super-resolution process (NO in step S107), the resolution is converted to the resolution of the high-resolution image (3200 × 2400 pixels) (step S111), and then the image composition process is performed. (Step S113).

In step S113, blur may be generated from one or a plurality of input images using a Gaussian filter or the like instead of the image composition processing.

Then, the partial region that has been super-resolution processed in step S109 and the partial region that has been subjected to image composition processing in step S113 after conversion to high resolution in step S111 (or in which blur is generated) are 3200 ×. A high-resolution image of 2400 pixels is output (step S115).

FIG. 20 is a flowchart showing the flow of preprocessing in step S105.
Referring to FIG. 20, first, among the 16 input images, a predetermined number for determination (for example, four input images A, C, I, K, etc.) is selected (step S201). Here, by selecting a predetermined number for determination from the input image, the determination process itself can be speeded up. On the other hand, the subsequent determination processing may be performed using all the 16 input images. Thereby, the determination accuracy can be improved.

Preferably, the image for determination is subjected to resolution conversion to a lower resolution such as 1/2 pixel (400 × 300 pixels) both vertically and horizontally (step S203). For the resolution conversion here, an interpolation process such as a bilinear method may be employed, or a nearest neighbor method corresponding to a thinning process may be employed in order to perform at high speed.

Next, each image after resolution conversion is divided into partial areas of a predetermined size, such as 10 × 10 pixels (step S205), and the similarity is calculated for each partial area (step S207). ). Based on the similarity, the presence / absence of super-resolution processing is determined for each partial region (step S209).

When a partial area is not similar to each of the judgment image groups and is clearly different, the partial area is a blurred area that is out of focus after refocusing even if super-resolution processing is performed. is there. Therefore, super-resolution processing for the partial area is not necessary. Therefore, in the image processing apparatus 1 according to the present embodiment, if the calculated similarity is equal to or less than a preset threshold value and the region of the determination image is clearly different (if not similar). Then, it is determined that the super-resolution processing is unnecessary for the partial area. In the present specification, the similarity is higher as the comparison target is similar. For example, the color value (average color), density, sharpness, or a combination of at least two of these can be used for calculating the similarity.

As an example of a method for determining similarity using sharpness, edge components are extracted using a first derivative such as Sobel, a second derivative such as Laplacian, and the like, There is a method of determining whether the similarity based on the result of adding the pixel differences is equal to or less than a threshold value. As another method, template matching such as SAD (Sum of Absolute Difference) or NCC (Normalized Cross Correlation) may be performed.

Moreover, the similarity determination may be performed for each specific pixel such as every two pixels. In this way, the determination process can be further speeded up.

Further, when the image is a color image, for example, when the super-resolution processing is individually performed for each of the RGB channels, the similarity determination may be performed on any one channel (for example, G). Alternatively, for example, it may be performed using only one channel converted by weighting RGB channels such as luminance. In this way, the determination process can be further speeded up.

In addition, when the input image is a moving image, the similarity determination may be performed for frames at a predetermined interval (for example, every three frames) according to the sequence of frames. In this case, the result of the most recent frame used for the determination may be used for a frame that is an input image that is not used for the similarity determination. In this way, the determination process can be further speeded up.

FIG. 21 is a diagram showing the determination result in step S209. In FIG. 21, the gray area has a calculated similarity less than (or less than) a preset threshold value, the area of the determination image is not similar, and the white area has a similarity value of the threshold value. This indicates that it is determined that the region of the determination image is similar as described above (or exceeds the threshold value).

If determined in this way, the image processing apparatus 1 performs super-resolution processing only on the white area in FIG. For the gray region in FIG. 21, for example, the bicubic method is adopted, and all the input images for 16 images are converted to the resolution of the high resolution image, and the converted images are synthesized. That is, pixel values are added for each pixel, and the average value is output.

<Effect of Embodiment>
In the image processing apparatus 1, by performing the above-described determination as the super-resolution pre-processing, in the processing for generating a high-resolution image including super-resolution processing such as refocus processing, the processing speed is increased without degrading the image quality. be able to.

The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 image processing device, 2 imaging unit, 3 image processing unit, 4 image output unit, 22 camera, 22a lens, 22b imaging device, 24 conversion unit, 32 refocus processing unit, 34 preprocessing unit, 36 super-resolution processing unit , 38 Image composition unit, 100 Digital camera, 104 Digital processing circuit, 106 Image processing circuit, 108 Image display unit, 112 Storage unit, 114 Camera unit, 200 Personal computer, 202 Personal computer main body, 204 Image processing program, 206 Monitor, 208 mouse, 210 keyboard, 212 external storage device, 341 determination unit.

Claims (13)

1. An image processing apparatus that executes processing for outputting, as an output image, a high-resolution image having more frequency information than the input image from a multi-viewpoint input image group,
   The input image is divided to set partial areas, the similarity between the corresponding partial areas for a plurality of the input images is calculated, and the super-resolution processing is required for each partial area based on the similarity. A determination means for determining whether or not,
   First processing means for performing super-resolution processing on a partial area of the input image that is determined to be subjected to super-resolution processing by the determination means;
   A second processing means for executing a process of converting the resolution of the partial area of the input image that is determined not to be subjected to super-resolution processing by the determination means to the resolution of the output image. Processing equipment.
2. The image processing apparatus according to claim 1, wherein the determination unit uses at least one of the density, the color value, and the sharpness of the partial region for calculating the similarity.
3. The image processing apparatus according to claim 1, wherein the determination unit performs template matching to calculate the similarity.
4. The image processing apparatus according to any one of claims 1 to 3, wherein the second processing unit synthesizes partial areas determined not to be subjected to super-resolution processing of the plurality of input images.
5. 4. The method according to claim 1, wherein the second processing unit uses the partial area of one input image of the input image group as a partial area determined not to perform super-resolution processing. The image processing apparatus according to item.
6. 6. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not super-resolution processing is required for the partial area for each specific pixel of the partial area.
7. The input image is a color image composed of a plurality of color channels,
   6. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not super-resolution processing is necessary for one of the plurality of color channels.
8. The input image is a color image composed of a plurality of color channels,
   6. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not super-resolution processing is necessary for an image converted from the plurality of color channels to one channel.
9. 9. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not super-resolution processing is necessary using an image obtained by reducing the input image.
10. The image processing according to any one of claims 1 to 9, wherein the determination unit selects a prescribed number from the input image group, divides each of the selected input images, and sets the partial region. apparatus.
11. The image processing apparatus according to claim 10, wherein the input image group is temporally continuous, and the determination unit selects the input image for each specific image in the order of the continuous.
12. A method for generating, as an output image, a high-resolution image having more frequency information than the input image from a multi-viewpoint input image group,
    Dividing the input image and setting a partial region;
    Calculating a similarity between corresponding partial regions for a plurality of the input images;
    Determining the necessity of super-resolution processing for each of the partial areas based on the similarity;
    Performing super-resolution processing on a partial area of the input image that is determined to be super-resolution processing;
    An image processing method comprising: executing a process of converting the resolution of a partial area of the input image that is determined not to be subjected to super-resolution processing into the resolution of the output image.
13. A program for causing a computer to execute a process of generating, as an output image, a high-resolution image having more frequency information than the input image from a multi-viewpoint input image group,
    Dividing the input image and setting a partial region;
    Calculating a similarity between corresponding partial regions for a plurality of the input images;
    Determining the necessity of super-resolution processing for each of the partial areas based on the similarity;
    Performing super-resolution processing on a partial area of the input image that is determined to be super-resolution processing;
    An image processing program causing the computer to execute a process of converting a resolution of a partial area of the input image that is determined not to be subjected to super-resolution processing to a resolution of the output image.

Images