JPWO2014010726A1 - Image processing apparatus and image processing program - Google Patents

Abstract

An image processing apparatus capable of obtaining good PSF estimation accuracy is provided. The image processing apparatus 1 of the present invention includes a blurred image generation unit 13 that generates a blurred image from an input image, a region identification unit 11 that divides a region in the input image into a non-effective region and a valid region, A weight mask creating unit for creating a weight mask that increases the weight in the effective region than the effective region and decreases the weight in accordance with the distance from the boundary with the non-effective region in the effective region; Analysis image generation for generating an analysis image by adding the blurred image weighted by the weight mask created by the creation unit 12 and the input image weighted by a mask obtained by inverting the weight mask And a PSF estimation unit 15 for estimating a point spread function using the analysis image.

Description

  The present invention relates to an image processing apparatus and an image processing program.

  2. Description of the Related Art Conventionally, a technique for estimating a PSF (Point Spread Function) representing blur due to camera shake or out-of-focus in an image at high speed has been proposed (see, for example, Patent Document 1).

JP 2010-157210 A

However, since many PSF estimation methods use luminance gradient information, if there is a whiteout pixel saturated with a light source or the like reflected in an image, or a blackout pixel with extremely low luminance, an accurate luminance gradient is obtained. As a result, good estimation accuracy cannot be obtained.
In addition, it is difficult to accurately estimate the PSF even when regions having different PSFs are mixed in the image due to subject blurring or the like.

  An object of the present invention is to provide an image processing apparatus and an image processing program capable of obtaining good PSF estimation accuracy.

The present invention solves the above problems by the following means.
According to a first aspect of the present invention, a blurred image generation unit that generates a blurred image from an input image, a region identification unit that divides a region in the input image into an ineffective region and an effective region, and the ineffective A weight mask creating unit that creates a weight mask that increases the weight in the region than the effective region and decreases the weight in accordance with the distance from the boundary with the ineffective region in the effective region; and the weight mask creating unit An analysis image generation unit that generates an analysis image by adding the blurred image weighted by the weight mask created by the above and an input image weighted by a mask obtained by inverting the weight mask; There is provided an image processing apparatus comprising: a PSF estimation unit that estimates a point spread function using the analysis image.
The weight mask creation unit sets the ineffective area to 1, the effective area sets the boundary with the ineffective area to 1, and is decreased according to the distance from the boundary with the ineffective area to a certain distance. In the above, a weight mask of 0 may be created.
The weight mask creation unit creates a primary spare mask with the ineffective area set to 1 and the effective area set to 0, and performs expansion processing for expanding the ineffective area with respect to the primary spare mask. A weight mask that reduces the weight of the blurred image according to the distance from the boundary with the ineffective area in the effective area can be generated by creating a mask and performing a blurring process on the secondary preliminary mask. Good.
The area identification unit may set an area where the output of the pixel in the image sensor is not less than the upper limit value or not more than the lower limit value as the ineffective area.
The area identifying unit may use an area identified by main subject recognition, face recognition and / or object recognition as an effective area.
The area identification unit may recognize a main subject, a face, and / or an object, recognize these areas as ineffective areas, and recognize other areas as effective areas.
The blurred image generation unit may use a Gaussian filter, a disk filter, or a binomial filter.
According to the second aspect of the present invention, a blurred image generation unit that generates a blurred image from an input image, a first region corresponding to at least one of a high luminance region and a low luminance region of the input image, and a first region, Corresponding to the second region, a setting unit for setting different second regions, the input image in which the second region is weighted more than the first region, and a region corresponding to the first region A calculation image generating unit that generates a calculation image using the blurred image weighted larger than a region, and a calculation unit that calculates a point spread function using the calculation image. An image processing apparatus is provided.
According to a third aspect of the present invention, a setting unit that sets a first area corresponding to at least one of a high-luminance area and a low-luminance area of the input image and a second area that is different from the first area; There is provided an image processing apparatus comprising an arithmetic unit that calculates a point spread function using two regions.
In the image processing apparatus according to the third aspect, the calculation image generation unit generates a plurality of predetermined regions having a predetermined area in the second region, and the calculation unit uses the plurality of predetermined regions. The spread function may be calculated.
According to a fourth aspect of the present invention, there is provided an image processing program for causing a computer to execute the functions of the image processing apparatus.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing program which can obtain favorable PSF estimation precision can be provided.

It is a block block diagram of one Embodiment of the image processing apparatus which concerns on this invention. It is a block block diagram of a PSF information creation part. It is a figure which shows an original image. It is a flowchart which shows the PSF information creation process by a PSF information creation part. It is explanatory drawing which shows the image and mask in a PSF information preparation process in steps. It is a figure which shows the image for analysis by a PSF information preparation part. It is a figure which shows analysis PSF. It is a figure which shows analysis PSF. It is a figure which shows analysis PSF. It is a flowchart which shows the PSF information creation process by the PSF information creation part in 2nd Embodiment. It is explanatory drawing which shows the brightness | luminances of an image, a map, etc. in a weight mask preparation part. It is explanatory drawing which shows the brightness | luminances of an image, a map, etc. in a weight mask preparation part. It is explanatory drawing which shows the brightness | luminances of an image, a map, etc. in a weight mask preparation part. It is a block block diagram of the PSF information creation part of 3rd Embodiment. It is explanatory drawing which shows an original image. It is another flowchart which shows the PSF information creation process by the PSF information creation part of 3rd Embodiment. It is the figure which classified the mesh in which the effective area | region is not included, and the mesh in which the effective area | region is included.

(First embodiment)
Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram of the PSF information creation unit. FIG. 3 is a diagram showing the original image 30. FIG. 4 is a flowchart showing a PSF information creation process by the PSF information creation unit 10. FIG. 5 is an explanatory view showing images, maps, and the like in the PSF information creation step in stages. FIG. 6 is a diagram showing an analysis image 30 ′ by the PSF information creation unit 10. FIG. 7 is a diagram showing an analysis PSF.

The image processing apparatus 1 includes a PSF information creation unit 10 and an image processing unit 20. In the present embodiment, for example, a PC (personal computer) is caused to function as the image processing apparatus 1 by a program.
An image to be processed is input to the image processing apparatus 1 (PC) from a memory card or a camera connected via a cable or a wireless transmission path.

The PSF information creation unit 10 creates an analysis image based on the input processing target image, and estimates a PSF (Point Spread Function) representing blur due to camera shake or out-of-focus in the image. .
Based on the PSF estimated by the PSF information creation unit 10, the image processing unit 20 performs image processing for correcting blur due to camera shake or out-of-focus on the input image.
Then, the image processing apparatus 1 outputs an image with the blur corrected.

Hereinafter, the PSF information creation unit 10 will be described in detail.
The PSF information creation unit 10 includes functional units such as an area identification unit 11, a weight mask creation unit 12, a blurred image generation unit 13, an analysis image generation unit 14, and a PSF estimation unit 15.
The PSF information creation unit 10 in this configuration performs high-precision PSF estimation even when there are saturated whiteout pixels such as a light source or blackout pixels with extremely low luminance in the image.
That is, for example, as shown in FIG. 3, an image 30 in which camera shake occurs during shooting (the main subject 31 is blurred) is a saturated whiteout due to a light source 32 such as a fluorescent light being reflected. Suppose there is. In this case, even if the image of the light source 32 is blurred, it cannot be recognized as a blur because the edge of its contour is sharp (the luminance gradient is large). As a result, good estimation accuracy cannot be obtained. The PSF information creation unit 10 suppresses such erroneous PSF estimation due to overexposure or underexposure, and outputs highly accurate PSF estimation information.

Next, the operation of each functional unit in the PSF information creation unit 10 and the PSF information creation process by the PSF information creation unit 10 will be described with reference to FIGS. 5A to 5F along the flowchart shown in FIG. In the drawings and the following description, step is also abbreviated as “S”.
First, the input image I is acquired (S401), and the gray scale image Ig is created (S402). The input image I is A in FIG.

Next, the blurred image generation unit 13 creates a blurred image B (S403).
The blurred image B is created by convolution of the grayscale image Ig and the blur filter B_filt1.
That is,
B_filt1 is a blur filter, and in this embodiment, a Gaussian filter represented by the following equation is used.
(Xc, yc) represents the coordinates of the center of the blur filter.
The sum of the denominators is performed in the following range, and is normalized so that the sum of the filters B_filt1 is 1.
1 ≦ x, y ≦ fs

Here, the kernel size of the PSF is m × n, ks = max (m, n), and the size of the blurring filter is fs × fs. fs is about 0.5 × ks or more and about 2 × ks or less, and is preferably an odd number of fs = ks (in order to distribute the blur equally from the center).
Also, σ is preferably ¼ of the blur filter size in order to cover almost all of the blur distribution.

The kernel size is the size of the blur locus, and is set by requesting the operator of the image processing apparatus 1 to input. The blur filter is not limited to the Gaussian filter as described above, and may be a disk type or binomial filter.
The blurred image B is shown in B of FIG.

Next, the area identifying unit 11 defines, based on the input image I, an area where the luminance gradient may be discontinuous due to overexposure or blackout as an ineffective area non_valid, and other areas Is defined as an effective area valid (S404).
The non-effective area non_valid and the effective area valid are obtained by the following method by comparing the brightness of each pixel of R, G, and B with a predetermined threshold value (H_th, L_th) for each of the light and dark.

That is, in the non-valid region non_valjd and the valid region valid, the luminance of each pixel of R, G, B
max (R (x, y), G (x, y), B (x, y))> H_th
Or
min (R (x, y), G (x, y), B (x, y)) <L_th
Then non_vaid (x, y) = 1
valid (x, y) = 0
Other than that,
non_vaid (x, y) = 0
valid (x, y) = 1
And

Here, max (A, B,...) Represents the largest value among A, B,..., And min (A, B,...) Represents the smallest value among A, B,.
Since this step 404 defines the ineffective area and the effective area based on the input image I, it may be before the above steps as long as it is after step 401. It may be performed in parallel with the steps.

Next, the distance mask dist (x, y) corresponding to the distance from the ineffective area of each pixel is obtained by the weight mask creating unit 12 (S405). FIG. 5C shows the distance map dist (x, y). Then, a weight mask w (x, y) is created based on the distance map dist (x, y) obtained in step 405 (S406).
The weight mask w (x, y) in step 405 is
w (x, y) = max (1−coef1 × dist (x, y), 0)
Ask for.
Here, a good result is obtained when coef1 is about 0.5 × fs or more and 2 × fs or less. The weight mask thus obtained is as shown in D of FIG.

Further, the analysis image generation unit 14 further forms an analysis image using the weight mask w (x, y) (S407).
In the image formation for analysis in step 407, the weight mask [D in FIG. 5] created in step 406 is subtracted from 1 (1-w (x, y)) on the input image I shown in A of FIG. An image obtained by multiplying [E] in FIG. 5 and an image obtained by multiplying the blurred image [B in FIG. 5] created in Step 403 by the weight mask [D in FIG. 5] created in Step 406 are obtained. Is added. The obtained image is F in FIG.

That is,
Ia (x, y) = (1−w (x, y)) × I (x, y) + w (x, y) × B (x, y)
As a result, as shown in FIG. 6, an analysis image 30 ′ in which a whiteout portion (light source 32) is blurred is created.

Then, the PSF estimation unit 15 obtains the PSF using the analysis image created in step 407 (S408).
FIG. 7B shows the PSF estimated in this way. FIG. 7A is the correct PSF, and FIG. 7C shows the estimated PSF by the conventional method. Thus, since the sharp edge around the whiteout part is weakened, an estimation result close to the correct PSF can be obtained.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. 8 and 9A to 9C.
FIG. 8 is a flowchart showing a PSF information creation process by the PSF information creation unit 10 of the second embodiment. FIG. 9A to FIG. 9C are explanatory diagrams showing the brightness of images, maps, and the like in the weight mask creation unit 12.

  In the first embodiment described above, after the ineffective area and the effective area are identified by the area identifying unit 11, a distance map from the ineffective area of each pixel is obtained by the weight mask creating unit 12, and based on the distance map. The analysis image generation unit 14 creates a weight mask. In contrast, the second embodiment creates a weight mask as follows without using a distance map.

  In addition, the same code | symbol is attached | subjected to the component of the same function as 1st Embodiment mentioned above, and description is abbreviate | omitted. Further, in the flowchart showing the PSF information creation process by the PSF information creation unit 10 shown in FIG. 8, the distance map creation (S405) in the first embodiment is merely replaced with the ineffective area expansion processing (S905). These steps are the same.

In step 904, the weight mask creating unit 12 in the second embodiment identifies the effective area and the ineffective area as shown in FIG. 9A, and creates a primary preliminary mask of 1 in the ineffective area and 0 in the effective area.
Next, as shown in FIG. 9B, the ineffective area is expanded (S905), and a secondary preliminary mask is created.
Then, the secondary preliminary mask is subjected to a blurring process to create a weight mask as shown in FIG. 9C.

A good result is obtained when the size of the expansion in step 905 is about 0.25 × fs or more and 2 × fs or less.
Also in the second embodiment, as shown in FIG. 9C, a weight mask substantially equivalent to the weight mask shown in FIG. 5D in the first embodiment described above can be obtained.
Hereinafter, an accurate PSF can be obtained by performing the same calculation as in the first embodiment.

As described above, this embodiment has the following effects.
(1) In the PSF information creation unit 10 of the image processing apparatus 1 according to the first embodiment, an input image and a blurred image obtained by blurring the input image according to the distance from the boundary between the region such as whiteout and the other region Create the added image for analysis. Then, by estimating the PSF from the analysis image, it is possible to minimize the deterioration in estimation accuracy.

(2) The PSF information creation unit 10 in the second embodiment creates a weight mask by performing a blurring process after performing an expansion process on an ineffective area such as a whiteout. Then, an analysis image is generated by adding the blurred image to which the weight of the weight mask is added and the input image to which the weight of the mask obtained by inverting the weight mask is added, and the PSF is estimated from the analysis image. Thereby, deterioration of estimation accuracy can be suppressed by simple processing.

(Third embodiment)
Next, a third embodiment of the present invention will be described with reference to FIGS.
FIG. 10 is another block diagram of the PSF information creation unit. FIG. 11 is an explanatory diagram showing the original image 300. FIG. 12 is another flowchart showing the PSF information creation process by the PSF information creation unit of the third embodiment.

A third embodiment of the present invention will be described with reference to FIGS. 10 and 11 along the flowchart shown in FIG.
First, the input image I is acquired (S401), and the gray scale image Ig is created (S402).
Next, a plurality of meshes Me are generated in the gray scale image Ig (S503). In the present embodiment, each of the plurality of meshes Me has the same area and is provided adjacent to each other so as not to overlap each other.
Each of the plurality of meshes may have a different area. Each of the plurality of meshes may be provided so that at least a part thereof overlaps, or may be provided at a distance from each other.

Next, the area identifying unit 11 defines, based on the input image I, an area where the luminance gradient may be discontinuous due to overexposure or blackout as an ineffective area non_valid, and other areas Is defined as an effective area valid (S404).
Next, in step 507, the analysis image generation unit 14 determines whether or not each of the plurality of meshes Me includes the ineffective area non_valid, and the mesh Me0 including the ineffective area non_valid is determined. And the mesh Me1 that does not include the non-valid region non_valid.
The analysis image generation unit 14 selects a mesh Me1 that does not include the ineffective area non_valid, and forms an analysis image.
Next, the PSF estimator 15 obtains the PSF using the analysis image created in step 507 (S408).

(Deformation)
The present invention is not limited to the embodiment described above, and various modifications and changes as shown below are possible, and these are also within the scope of the present invention.
(1) In the above-described embodiment, the region identifying unit 11 sets a region where the luminance gradient is discontinuous due to overexposure or blackout as an ineffective region, and other regions as effective regions. However, an arbitrary area such as a target subject in the image may be set as an effective area. This makes it possible to estimate the PSF limited to the target subject when areas having various different PSFs are mixed in the screen.
For example, in FIG. 13, the area identifying unit 11 includes a mesh Me2 that does not include an effective area corresponding to the face and head of the person who is the target subject, and an effective area that corresponds to the face and head of the person. The mesh Me3 is classified. The analysis image generation unit 14 selects the mesh Me3 including the effective region and forms an analysis image.

For example, when shooting a scene in which a plurality of people are moving freely, even if an attempt is made to calculate a PSF representing the blur of a specific person, various PSFs affect each other if the estimation is performed with the image as it is. It is not exactly required.
Therefore, when an analysis image is created by the above method using a specific person whose PSF is to be obtained as an effective area and other portions as non-effective areas, and the PSF is estimated from the analysis image, the PSF of the corresponding specific person is obtained. Can be requested.

As a method for identifying a person, a method such as known face recognition or having a user specify an area can be used.
In addition to a person, it is also possible to recognize a specific target such as a car or an animal (object recognition) and similarly create an image for analysis to obtain a PSF of only a part of an arbitrary shape. .

(2) In the above embodiments, the present invention has been described as an image processing apparatus, but the present invention includes what is supplied as an image processing program. In this case, the program can be provided through a recording medium such as a CD-ROM or a data signal such as the Internet. The image processing apparatus may be built in the camera.

  The above embodiments and modifications can be used in appropriate combinations, but the configuration of each embodiment is clear from the illustration and description, and thus detailed description thereof is omitted. Furthermore, the present invention is not limited by the embodiment described above.

  1: image processing device, 10: PSF information creation unit, 11: region identification unit, 12: weight mask creation unit, 13: blurred image generation unit, 14: image generation unit for analysis, 15: PSF estimation unit, 20: image Processing unit, 30: image, 31: main subject, 32: light source

Claims (11)

A blurred image generation unit that generates a blurred image from an input image;
An area identifying unit that divides an area in the input image into an ineffective area and an effective area;
A weight mask creating unit that creates a weight mask that increases the weight in the ineffective area compared to the effective area and decreases the weight in accordance with the distance from the boundary with the ineffective area in the effective area;
For analysis to generate an analysis image by adding a blurred image weighted by the weight mask created by the weight mask creation unit and an input image weighted by a mask obtained by inverting the weight mask An image generator;
A PSF estimation unit for estimating a point spread function using the analysis image;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The weight mask creating unit
A weight mask in which the ineffective area is set to 1, the effective area has a boundary with the ineffective area as 1, and is decreased according to the distance from the boundary with the ineffective area to be 0 over a certain distance. Creating
An image processing apparatus. The image processing apparatus according to claim 1,
The weight mask creating unit
A primary preliminary mask with the ineffective area set to 1 and the effective area set to 0 is created, a dilatation process for enlarging the ineffective area is performed on the primary spare mask, and a secondary spare mask is created. Creating a weight mask that reduces the weight of the blurred image in accordance with the distance from the boundary with the non-effective area in the effective area by performing a blurring process on the next preliminary mask;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 3,
The area identification unit is configured such that an area in which an output of a pixel in an image sensor is equal to or higher than an upper limit value or lower limit value is the ineffective area.
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The area identifying unit sets an area identified by main subject recognition, face recognition and / or object recognition as an effective area,
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The area identification unit recognizes a main subject, a face and / or an object, recognizes these areas as non-effective areas, and recognizes other areas as effective areas;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 6,
The blurred image generation unit uses a Gaussian filter, a disk filter or a binomial filter,
An image processing apparatus. A blurred image generator for generating a blurred image from an input image;
A setting unit for setting a first area corresponding to at least one of the high-luminance area and the low-luminance area of the input image, and a second area different from the first area;
The input image in which the second region is weighted more than the first region, and the blurred image in which the region corresponding to the first region is weighted more than the region corresponding to the second region. A calculation image generation unit that generates a calculation image using the calculation unit;
An image processing apparatus comprising: a calculation unit that calculates a point spread function using the calculation image. A setting unit that sets a first area corresponding to at least one of a high-luminance area and a low-luminance area of the input image, and a second area that is different from the first area;
An image processing apparatus comprising: an arithmetic unit that calculates a point spread function using the second region. An image processing apparatus according to claim 9, comprising:
The calculation image generation unit generates a plurality of predetermined regions having a predetermined area in the second region,
The image processing apparatus, wherein the calculation unit calculates a point spread function using the plurality of predetermined regions.   An image processing program for causing a computer to execute the function of the image processing apparatus according to claim 1.