JP2001223874A - Arbitrary focused image composite device and camera for simultaneously picking up a plurality of images, which is used for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for reconstructing an arbitrary focused image being an image where the degree of the blurring of respective depths is arbitrarily suppressed and emphasized from the plurality of pictures of different focusing. SOLUTION: There are provided a first filter for converting a first image focused to a first part on the basis of a first blurring parameter inputted from the outside, a second filter for converting a second image focused to a second part on the basis of second blurring parameter inputted from the outside, a composite part for composting the output of the first filter and the output of the second filter and generating an arbitrary focused image, and a luminance correction part for correcting luminance in the block unit of the image so that the luminance of the first image and that of the second image become the same level and supplying the image after the luminance correction to the first filter and the second filter.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an arbitrary-focus image synthesizing apparatus for reconstructing, from a plurality of images, an arbitrary-focus image which is an image in which the degree of blur at each depth is arbitrarily suppressed / emphasized, and used in the apparatus. The present invention relates to a multiple image simultaneous imaging camera.

[0002]

2. Description of the Related Art As a conventional example of an image processing method for generating a desired image from a plurality of images, there is, for example, an image processing method based on area division. In this conventional image processing method, for example, a plurality of images with different focuses are prepared, the in-focus area in each image is determined, and the area division of the plurality of images is performed based on the determination result. A desired image is generated by performing a series of processes for giving a predetermined visual effect to each area. At this time, if the above-described series of processing is automatically executed without human intervention, an image processing program describing a procedure for sequentially performing the above-described area determination, area division, and visual effect processing is used. Is common.

In order to generate a desired image from a plurality of images, first, it is necessary to obtain a plurality of images for the same object. In order to obtain a plurality of images of the same scene captured by different focusing using a conventional camera, it is necessary to perform a plurality of imagings while changing the focusing.

That is, when n types of images with different focusing are captured using a conventional camera device, the zoom lens is controlled manually or by a servo lens control device provided outside the camera, and firstly, After controlling the focusing of the zoom lens so that it is focused on the depth of
The second image is captured after the second image is captured, and then the focusing of the zoom lens is controlled so as to focus on the second depth. In the same manner, the focusing of the zoom lens 53 is controlled so that the focus is on the n-th depth, and then the n-th image is captured. As described above, when it is desired to capture an image focused on n types of depths, n times of focusing and imaging are required.

[0005]

Since the above-mentioned conventional image processing method uses the determination condition of "in-focus area", the area having a uniform luminance value or the depth in the scene to be imaged is used. If there are areas with changes, it is not possible to ensure sufficient area determination accuracy for those areas. For this reason, the applicable range of the above-described conventional image processing method is limited to sharpening of an image by integrating in-focus regions, and the defocus is arbitrarily adjusted for each region, or a pseudo It is extremely difficult to extend to higher-order image processing, such as generating a stereoscopic image by giving parallax. Further, the conventional image processing method cannot obtain an arbitrary focus image which is an image in which the degree of blur at each depth is arbitrarily suppressed / emphasized.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and is intended to reconstruct an arbitrary-focus image which is an image in which the degree of blur at each depth is arbitrarily suppressed and emphasized from a plurality of images having different focuses. It is an object of the present invention to provide an arbitrary focus image synthesizing apparatus. Another object of the present invention is to provide a multiple image simultaneous imaging camera capable of simultaneously capturing a plurality of images with different focusing.

[0007]

An arbitrary focus image synthesizing apparatus according to the present invention converts a first image focused on a first portion based on a first blur parameter input from the outside. A second filter that converts a second image focused on a second portion based on a second blur parameter input from the outside, an output of the first filter, and a second filter A synthesizing unit for synthesizing the output of the filter to generate an arbitrary focus image, and performing brightness correction in image block units so that the brightness of the first image and the brightness of the second image are substantially the same. A brightness correction unit that supplies an image to the first filter and the second filter.

An arbitrary focus image synthesizing apparatus according to the present invention comprises:
A first blur parameter based on a first externally input blur parameter;
A first filter that converts a first image that is in focus on a portion, and a second filter that converts a second image that is in focus on a second portion based on an externally input second blur parameter , A synthesizing unit that synthesizes an output of the first filter and an output of the second filter to generate an arbitrary focus image, and adjusts the position of the first image and the second image by using image data. It is provided based on a luminance distribution obtained by projecting in the horizontal and vertical directions, and includes a positioning unit that supplies a registered image to the first filter and the second filter.

An arbitrary focus image synthesizing apparatus according to the present invention comprises:
A first blur parameter based on a first externally input blur parameter;
A first filter that converts a first image that is in focus on a portion, and a second filter that converts a second image that is in focus on a second portion based on an externally input second blur parameter , A special effect filter that performs a predetermined process on the output of the second filter, and a synthesizing unit that synthesizes the output of the first filter and the output of the special effect filter to generate an arbitrary focus image Is provided.

Preferably, on the input side and the output side of the special effect filter, a rectangular coordinate-polar coordinate conversion unit for converting the coordinates of the image data from rectangular coordinates to polar coordinates, respectively, and the coordinates of the image data are returned from polar coordinates to rectangular coordinates. A polar coordinate-rectangular coordinate conversion unit.

An arbitrary focus image synthesizing apparatus according to the present invention comprises:
Based on the first to N-th blur parameters input from the outside, the first to N-th images that are respectively focused on the first to N-th parts are arranged in the order of the focus positions, and the i-th image, which is one of these images, is arranged. A determination unit that determines whether or not the part is in focus in a plurality of images before and after the i-th image with respect to a part in the image of the i-th image; And a synthesizing unit for synthesizing the first to N-th images based on the comparison result of the comparing unit to generate an all-focus image.

Preferably, the determination unit includes a Gaussian filter for performing a filtering process on the i-th image while changing a parameter, and a difference for obtaining a difference value between an output of the Gaussian filter and a plurality of images before and after the image. A processing unit; and an estimating unit that estimates the parameter by obtaining a value at which the difference value is minimized.

[0013] A multiple image simultaneous imaging camera according to the present invention includes an image sensor, a processing unit that receives a signal from the image sensor and converts the signal into image data, and a display unit that displays the image data processed by the processing unit. A focus designating unit for designating a plurality of objects in an image and requesting a plurality of images having different focuses, a focus adjusting mechanism for setting a focus position by designating the focus designating unit, and a memory for storing image data The processing unit focuses on the designated plurality of targets in order, captures the images, and stores the obtained plurality of image data in the memory.

Preferably, a plurality of images having different focal points are captured by one shutter operation.

Preferably, a first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter A second filter for converting a second image focused on a second portion based on the second filter, and a synthesis for generating an arbitrary focus image by synthesizing an output of the first filter and an output of the second filter. Unit, and performs luminance correction on an image block basis so that the luminance of the first image and the luminance of the second image are substantially the same, and outputs the image after the luminance correction to the first filter and the second filter. And an arbitrarily focused image synthesizing device including a luminance correction unit for supplying the image.

Preferably, a first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter A second filter for converting a second image focused on a second portion based on the second filter, and a synthesis for generating an arbitrary focus image by synthesizing an output of the first filter and an output of the second filter. Part, performing the alignment of the first image and the second image based on a luminance distribution obtained by projecting image data in horizontal and vertical directions,
Aligning the aligned image with the first filter and the second
And an alignment unit that supplies the images to the filters.

Preferably, a first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter A second filter that converts a second image focused on a second portion based on the first filter, a special effect filter that performs a predetermined process on an output of the second filter, A synthesizing unit for synthesizing an output and an output of the special effect filter to generate an arbitrary focus image.

Preferably, on the input side and the output side of the special effect filter, a rectangular coordinate-polar coordinate conversion unit for converting the coordinates of the image data from rectangular coordinates to polar coordinates, respectively, and the coordinates of the image data are returned from polar coordinates to rectangular coordinates. A polar coordinate-rectangular coordinate conversion unit.

A recording medium according to the present invention stores a program for causing a computer to function as the arbitrary-focus image synthesizing apparatus or the multiple image simultaneous imaging camera.

The medium includes, for example, a floppy disk,
Hard disk, magnetic tape, magneto-optical disk, CD-
Includes ROM, DVD, ROM cartridge, RAM memory cartridge with battery backup, flash memory cartridge, nonvolatile RAM cartridge, and the like.

The communication medium also includes a communication medium such as a wired communication medium such as a telephone line and a wireless communication medium such as a microwave line. The Internet is also included in the communication medium mentioned here.

A medium is a medium on which information (mainly digital data and programs) is recorded by some physical means, and which allows a processing device such as a computer or a dedicated processor to perform a predetermined function. You can do it.
In short, any method may be used as long as the program is downloaded to the computer by some means and a predetermined function is executed.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 of the Invention In the embodiment of the present invention, from a plurality of images, an all-focus image in which both the near view and the distant view are completely focused, or an arbitrary focus image which is an image in which the degree of blur at each depth is arbitrarily suppressed / emphasized. An apparatus / method for reconfiguration will be described.

A method of reconstructing a desired arbitrary-focus image f from the near-field focused image g1 and the far-field focused image g2 will be briefly described. FIG. 1 shows a schematic block diagram of an apparatus according to an embodiment of the present invention. This apparatus can reconstruct both an all-focus image and an arbitrary-focus image from the foreground image g1 and the distant view image g2. The filter 10a outputs the foreground image g1.
And the filter 10b performs a predetermined process on the distant view image g2. Details of the filter will be described later. The combining unit 11 combines the output of the filter 10a and the output of the filter 10b to output a reconstructed image f. The filters 10a and 10b each have a parameter R
a and Rb are received from outside. Parameters Ra and Rb are the blur radii of the near and distant views of the desired image, respectively. Ra
When = Rb = 0, the reconstructed image f is an all-focus image.
By adjusting the parameters Ra and Rb, an arbitrary focus image can be reconstructed.

For example, when generating an all-focus image, both the first filter for the near view and the second filter for the distant view have high-pass characteristics. An omnifocal image can be obtained by extracting and adding high-frequency components in a balanced manner to the first image and the second image using the first and second filters. Even an arbitrary focus image can be generated by devising filter characteristics. Specific filter characteristics will be described later.

The apparatus / method according to the embodiment of the present invention is based on the fact that a filter for uniquely reconstructing a target image exists in a model for acquiring a focused image and an arbitrary focused image. In the conventional iterative restoration method, the DC component of a target image is a so-called ill-conditio
n). In the embodiment of the present invention, the DC component of the reconstruction filter exists, and all frequency components can be reconstructed.

First, a model for obtaining a focus image and reconstructing an arbitrary focus image will be examined. In the method of reconstructing an arbitrary focus image, it is assumed that the depth of the target scene of the acquired image changes in a stepped manner. In the following, the target scene is
For a case with two layers of depth, the acquisition of each focus image and the reconstruction of an arbitrary focus image are modeled.

<Focal Image Acquisition Model> The focal image acquisition model is performed using superposition of images. Image f1
Is defined as an image having a luminance value focused only on the foreground region, and having a luminance value of 0 in other regions, that is, a distant region. Conversely, the image f2 is defined as an image having a luminance value focused only on a distant view area and having a luminance value of 0 in a near view area. The near-field focused image is represented by g1 and the far-field focused image is represented by g2. The blur function of the distant view area of the image g1 is represented by h2, and the blur function of the near view area of the image g2 is represented by h1. Each focus image g1
And the acquired model of g2 are represented by superposition as shown in FIG. That is, it is represented by the following equation. g1 = f1 + h2 * f2 g2 = h1 * f1 + f2 (6) where * indicates a convolution operation.

<Reconstruction Model of Arbitrary Focus Image> Similarly, a reconstruction model of an arbitrary focus image uses image superposition. A desired arbitrary focus image is represented by f, and blurs in the near and distant areas are given by blur functions ha and hb, respectively. Therefore, as shown in FIG. 2, the reconstruction model of the arbitrary focus image f is expressed by the following equation. f = ha * f1 + hb * f2 (7) The blur functions ha and hb are arbitrarily specified by the user. As the blur function, a Gaussian function expressed by the following equation is used.

[0030]

(Equation 1)

Ri (i = 1, 2, a, b) represents the blur radius and corresponds to √2 times the standard deviation of the Gaussian function. Ra
If = Rb = 0, then ha = hb = δ (delta function), so that equation (7) is a reconstruction model of the all-focus image.

Next, a reconstruction method using a filter will be described. Using a filter, a desired arbitrary focus image f can be reconstructed from the focus images g1 and g2. Simulations show that this method is more accurate and faster than the conventional iterative restoration method. The method is described below.

<Derivation of Reconstruction Filter> A reconstruction filter is derived from the obtained models of g1 and g2 in equation (7) and the reconstruction model of f in equation (7). First, each model is transformed into the frequency domain. The acquisition models of G1 and G2 can be represented by a matrix as in the following equation. G = HF (17) where each matrix is

[0034]

(Equation 2)

Is as follows. The reconstruction model of F is as follows. F = HaF1 + HbF2 (18)

Next, F is obtained from equations (17) and (18). The following cases are classified according to the value of the determinant | H |. Note that | H | = 1−H1H2, and | H | is g = h * f, where f is the image and g and the blurring function is h, and is equal to H of G = HF obtained by Fourier-transforming both sides. .

(I) In the case of | H | ≠ 0 In the case of 1−H1H2 ≠ 0, that is, at frequencies other than DC, there is an inverse matrix H− ¹ . Therefore, the matrix F is obtained as follows.

[0038]

(Equation 3)

Substituting F into equation (18) and rearranging it,

[0040]

(Equation 4) Is found.

(Ii) In the case of | H | = 0 Since (1-H1H2) = 0 in direct current, there is no inverse matrix H- ¹ . Therefore, the matrix F cannot be obtained.
However, the numerator of the coefficients of G1 and G2 in Equation (20) is also Ha
−HbH1 = 0 and Hb−HaH2 = 0. Therefore, solving the limiting values of these coefficients to DC using Lopital's theorem

[0042]

(Equation 5)

Exists. Therefore, (i) and (ii)
By

[0044]

(Equation 6)

By the filters Ka and Kb expressed by
Can be reconstructed as follows. Fab = KaG1 + KbG2 (25)

As shown in FIG. 1, an arbitrary focus image f can be obtained by passing the focus images g1 and g2 through filters Ka and Kb, respectively, and adding them. By changing Ra and Rb, the blur of the near view and the distant view can be arbitrarily set. In the known iterative restoration method, the DC component was in a bad condition, but it can be solved as an appropriate problem by using the filter method. That is, it was shown that f is uniquely present and can be determined by this method.

<Characteristics of Reconstruction Filter> FIG. 3 shows an example of frequency characteristics of the reconstruction filters Ka and Kb. First, in two images, an image focused on a distant view (g2)
In this example, the blur radius R1 of the foreground is R3 = 3, and the blur radius R2 of the distant view is R2 = 2 for the image (g1) focused on the foreground. At this time, FIG. 3 shows the characteristics of the filters Ka and Kb when Ra = 0 and Rb are changed from 0 to 4 as an arbitrary focus image. This means a process of largely changing the degree of blur in the distant view area while keeping the focus on the near view area.

Since Ra = 0, Rb = 0 indicates the characteristics of a filter for reconstructing an all-focus image. Both filters show characteristics like a high-pass filter. That is, it can be seen that the high-frequency components of the respective focus images are integrated to reconstruct the 仝 -focus image. In the case of Rb = 2, Ka has the characteristic of passing through all regions, and Kb has the characteristic of blocking all regions. The reason is that the reconstructed arbitrary focus image f is the focus image g1.
Is the same as When Rb> 2, Ka exhibits the characteristics of low-frequency emphasis while maintaining high-frequency pass. Kb shows characteristics that emphasize low frequencies negatively while keeping high frequencies cut off. It can be seen that the blur in the distant view area is enhanced by subtracting the low-frequency component emphasized in the focus image.

According to the simulation results, both the accuracy and the calculation time are improved by using the filter method as compared with the iterative method. In the iterative method, much time is required for the convolution operation of the blur function in the spatial domain. Also, as the number of convolutions of the blur function increases,
The error may propagate greatly. In the present filter method, a desired image can be directly reconstructed with a single process with high accuracy by using a filter.

Embodiment 2 of the Invention In the above-mentioned method of reconstructing and generating an arbitrary focus image by filtering two focus images, the difference in average luminance level between a plurality of images to be used. In some cases, a good image cannot be reconstructed. Since an imaging device such as a digital camera has a function of automatically adjusting the brightness (AGC), the brightness of the near view image does not always match the brightness of the far view image. Therefore, it is desirable to perform the following luminance correction.

When reconstructing a desired arbitrary-focus image f from the near-field focused image g1 and the far-field focused image g2, the following parameters A and B that minimize the cost function are estimated using the least squares method. At this time, it is desirable to evaluate the following cost function between the hierarchized images in consideration of the difference in the amount of blur between the two images. Here, g1 (k) and g2 (k) each represent an image of the k-th layer of the Gaussian villa mid (details will be described later). However, the 0th layer is the original image.

[0052]

(Equation 7)

According to the simulation results, the parameters A and B could be estimated with high accuracy by this method. In the image generated using the image g2 before the correction, it was confirmed that the brightness value was reduced over the entire screen, and that the edge was emphasized in the distant view area. On the other hand, an image generated using the corrected image was successfully generated. In the generation of an arbitrary focus image that gives a larger blur than the amount of blur in the observation image, the low-frequency components of the filters Ka and Kb for the generation become positive and negative, respectively. For this reason, it can be considered that the average luminance value difference causes such an artifact in the generated image. Therefore, by performing the brightness correction, it is possible to avoid a problem that an artifact of edge enhancement and brightness reduction occurs in a generated image in which blur is enhanced.

When the image is taken with the focus changed between the center and the end of the screen, the brightness varies within the screen. In this case, it is necessary to divide the screen into each block and obtain an appropriate correction parameter for each block. In this case, the above processing is performed for each block. In order to reduce the variation in correction between blocks, the correction parameters for each block are used for the center pixel of each block as shown in FIG. 4, and the other pixels use the correction parameters subjected to bilinear interpolation. Used. In order to stabilize the estimation accuracy of the correction parameter, the following expression may be used as the evaluation amount. [Σg1 (i, j) −A · Σg2 (i, j)] ² where (i, j) ∈B (to obtain the sum of elements in block B) In this case, so-called average luminance value in the block Is corrected by the ratio (A).

When synthesizing a large number of images, luminance correction may be performed in block units. For example, when combining N images, each of the N captured images is divided into square blocks as shown in FIG. Correction is performed sequentially so that the k + 1st image matches the kth image (k = 1, 2,
‥‥, N−1). The ratio of the average luminance value between the images is obtained for each block, and is used as a correction parameter for the central pixel of the block. Values other than the center pixel are obtained by bilinear interpolation.
Finally, the luminance correction is performed by multiplying the luminance value of the pixel by the correction parameter. When the ratio of the area where luminance saturation occurs in a block in one of the images is equal to or larger than a certain size, the correction parameter of the central pixel of the block is interpolated as an average value of the peripheral blocks.

Third Embodiment of the Invention In order to use a plurality of focus images in the reconstruction processing, registration (registration) between these images is required. When capturing a plurality of focal images, it is difficult to obtain images whose imaging positions match each other accurately, and it is conceivable that a positional shift may occur between the captured images. Further, a change in magnification due to a difference in focusing occurs. Poor alignment accuracy not only causes blurring of the reconstructed image, but also affects the accuracy of estimation of blur parameters required for the reconstruction. As a result, the accuracy of the reconstructed image is reduced. Therefore, positioning is a pre-process necessary for performing high-precision image reconstruction / generation.

<Positioning among a plurality of focus images (Part 1: Hierarchical matching method)> In order to reconstruct a desired arbitrary focus image f from the near focus image g1 and the far focus image g2, first, It is necessary to perform registration between each focus image. Hereinafter, a method of positioning between a plurality of focus images will be described.

It is assumed that an image In focused on a near view and an image If focused on a distant view are obtained. The image In is, for example, an image of a person standing in front of the flower bed, and the image If is, for example, an image of a background flower bed. Based on In, the difference in rotation, expansion, contraction, and translation (expressed as θ, s, and vector t = (u, v) in order) of If is estimated. In this case, the expansion and contraction need only consider the enlargement from the relation of the focal length. In the present method, the parameters of rotation and expansion and contraction are incorporated into the hierarchical matching method to search for each parameter sparsely and densely. Since In and If have different in-focus areas and out-of-focus areas, there is a high possibility that an error will occur if direct matching is performed. If the hierarchical matching method is used, matching can be performed by reducing the difference in blur between the two images by layering the images. As a result, it is considered that the positioning can be performed robustly with respect to the difference in blur.

The processing flow of this method is (1) hierarchization of an image, and (2) estimation of parameters in each layer. First, both images are hierarchized, and parameters are obtained in a wide search range in the uppermost layer having the lowest resolution. Hereinafter, matching is performed up to the lowermost layer while limiting the search range only to the vicinity of the parameters estimated in the upper layer, and finally the parameters between the original images are obtained. Hereinafter, the method will be specifically described along the processing flow.

(1) Hierarchization of Images Hierarchization of both images is performed by forming a Gaussian villamid. The Gaussian pyramid is formed as follows. The original image is the 0th layer, the top layer having the lowest resolution is the Lth layer, and the focus image of the kth (= 0, 1, ‥, L) layer is In.
(k) and If (k). The image of each layer is sequentially formed by the following equation.

[0061]

(Equation 8)

Here, w is an approximation of a two-dimensional Gaussian function having a standard deviation of 1. [•] ↓ 2 indicates downsampling. The image of the k-th layer is obtained by passing the image of the (k−1) -th layer below by a Gaussian filter and down-sampling. Since the Gaussian filter acts as a low-pass filter, the difference in the degree of blur between the two images is further reduced in the upper layer.

(2) Estimation of Parameters in Each Layer In this method, the mean square of the image If (x ′, y ′) obtained by rotating, expanding, contracting, and translating the image If (x, y) and In (x, y) Find each parameter that minimizes the error (MSE). Each parameter in the k-th layer is represented by θ (k), s (k), u (k),
Assuming that v (k), the evaluation function J (k) to be minimized in the k-th layer is

[0064]

(Equation 9)

Can be expressed as follows. here,

[0066]

(Equation 10)

Is as follows. Where B (k) is In (k)
Overlap region of (x, y) and If (k) (x ', y'), NB
(K) is the number of pixels. The search point of each parameter is set as follows according to the hierarchy.

[0068]

[Equation 11]

Where i, j, m and n are integers, {θ, △
s, △ u, △ v are search intervals between original images, that is, estimation accuracy of each parameter. Hat θ (k + 1), hat s (k + 1), hat u (k + 1), hat v (k
+1) represents each parameter estimated in the upper (k + 1) th layer. θmax, smax, umax, and vmax are values that limit the search range in the uppermost layer and are set in advance. However,
The values of umax and vmax are respectively set to half the size of each side of the image in the uppermost layer. The search intervals △ u and △ v of the translation parameter in each layer are constant because the translation amount in the k-th layer is twice as large as that in the k + 1-th layer.

FIG. 5 shows the flow of estimation in this method. In each layer, image In (k) (x, y) and rotation,
M with translation-converted image If (k) (x ', y')
Find each parameter that minimizes SE. In the uppermost layer (k = L) having the lowest resolution, each parameter is roughly estimated at wide intervals within a preset search range. The search interval corresponds to 2 ^L times that of the lowest layer. In layers other than the top layer, the search is performed by doubling the estimation accuracy while sequentially limiting the search range to only five points around the parameters estimated in the upper layer. This search is performed up to the lowest layer, and each parameter is finally estimated.

Finally, the images If (x ', y') and In (x, y) which have been subjected to the estimated parameters, the rotational expansion / contraction, and the translation are applied.
Is cut out to obtain respective corrected images. Even when the target scene has three or more layers, the correction can be performed by aligning the other images based on the focus image of the closest view and extracting the common area.

According to the simulation, correct estimation was performed in all cases if the blur radius was small in each expansion and contraction. The error is maximum even when the blur radius is large.
It was suppressed to 2 [pixels], and good results were obtained.
The error did not occur more than 3 [pixels].

Embodiment 4 of the Invention <Positioning between multiple focus images (Part 2: Luminance projection)
The hierarchical matching method according to the third embodiment of the present invention has no problem in accuracy, but has a problem that processing is complicated and takes time. Therefore, we propose a luminance projection method that can perform simple and high-speed processing.

In this method, the distant view image If is scaled and translated (s, vector t = (u,
v)) the difference can be estimated. FIG. 6 shows a block diagram of an apparatus for aligning a plurality of focus images by the luminance projection method. FIG. 7 shows an explanatory diagram of the operation. The average luminance value calculation units 20a and 20b for each row and each column respectively
Calculate the average luminance value of each row and each column of In and If. The luminance projection distribution creating units 21a and 21b create In and If luminance projection distributions, respectively. The distribution of the average luminance value in each row is a vertical distribution, and the distribution of the average luminance value in each column is a horizontal distribution. Through these processes, luminance distributions of In and If as shown in FIGS. 7A and 7B are obtained. Note that the dotted circle in FIG. 7B indicates a circle having the same size as the circle in FIG. 7A. Thus, each image is represented as a combination of two one-dimensional distributions, a horizontal distribution and a vertical distribution. The comparison unit 22 compares these two distributions with reference to In, and based on the comparison result, the enlargement / translation estimation unit 23 enlarges and translates If (in order, s, t = (u,
v) Estimate the difference. For example, when the subject has a circular shape, it is assumed that the luminance projection of the foreground image In in FIG. 7A has a center c of the horizontal distribution, a diameter a, a center d of the vertical distribution, and a diameter b. It is assumed that the luminance projection of the distant view image If in FIG. 7B is the center c ′ of the horizontal distribution, the diameter a ′, the center d ′ of the vertical distribution, and the diameter b ′. The expansion s is a '/
It can be estimated from a and b '/ b. The horizontal component u of the translation t can be estimated from c′−c, and the vertical component v can be estimated from d′−d.

The luminance projection method requires a very small amount of calculation and is very fast as compared with the hierarchical matching method. According to the simulation result, the processing time was reduced to about 1/200. Instead, the accuracy is slightly reduced, but the error is still suppressed to a change of about 1 pixel, and a good result is obtained.

Embodiment 5 of the Invention FIG. 1 shows the configuration of an apparatus for obtaining an all-focus image. This configuration is the most basic. By adding a filter to this configuration, a special effect can be applied to the omnifocal image.

FIG. 8A shows an example. The filters 10a and 10b are filters for focus processing, and the filter 12 is a filter for another special processing. The filter 12 is provided on the distant view image g2 side. This filter is an arbitrary filter. For example, a filter that adds pixel data in the horizontal (or vertical) direction can be considered. Data d0, d1, d in horizontal (or vertical) direction
When there are 2, d3, d4, ..., d2 = (d2 + d1 + d0) / 3, d3 =
(D3 + d2 + d1) / 3,. The data in the vertical (or horizontal) direction remains unchanged. When this filter is used, the distant view image g2 is converted into an image that flows in the horizontal direction, and the converted image is combined with the foreground image g1.
The synthesized image is an image like a “follow shot”.

As shown in FIG. 8B, a rectangular coordinate-polar coordinate converter 13 and a polar coordinate-rectangular coordinate converter 14 may be provided before and after the filter. According to this configuration, the distant view image g2 is converted into an image that flows radially, and this is combined with the near view image g1. That is, if the origin of the polar coordinates is made to coincide with the center of the foreground image, the composite image is an image having a background flowing around the foreground image (for example, a person). The filter may perform a non-linear geometric transformation process such as a logarithmic transformation. For example, the filter may have a small addition range near the center (r = 0) and a larger addition range as the distance from the center increases. According to this filter, the image largely flows as the distance from the foreground image increases, and the image has a sense of speed.

In the above description, the distant view image g2
, But the present invention is not limited to this. A filter may be applied to both the foreground image g1 and the distant view image g2, or a filter may be applied to only the foreground image g1.

Embodiment 6 of the Invention In order to obtain the foreground image g1 and the distant view image g2, a normal digital camera may be used with different focus. However, in a normal manner, the near view image g1 and the distant view image g2 often shift due to a change in the position or orientation of the camera. If the deviation is small, it can be corrected by the above-mentioned registration, but if the deviation is large, complete correction requires time. Therefore, an apparatus that can obtain two images with a small displacement by a simple operation is desired.

FIG. 9 shows a block diagram of this type of apparatus. The light passing through the lens 30 is incident on the CCD 31 and is converted into image data by the processing unit. The image is displayed on the viewer 33 and can be viewed by the user. The image displayed on the viewer 33 is divided into predetermined regions as shown in FIG. In the example of FIG. 10, the area is divided into nine areas in total. The user operates the focus specifying unit 34 while viewing the image of the viewer 33, and specifies at least two areas to be focused. For example, in order to obtain the foreground image g1, the focus is designated on the middle area (2, 2) where the subject T is shown, and on the upper left area (1, 1) in order to obtain the distant image g2.
To specify the focus. Upon receiving the signal from the focus specifying unit 34, the processing unit 32 drives the focus adjustment mechanism 36. The focus adjustment mechanism 36 focuses on the designated area and captures an image. Data of the captured image is stored in the memory 35.
Next, the focus adjustment mechanism 36 focuses on the next designated area to capture an image, and stores the image data in the memory 35.

Further, the processing as shown in FIG. 11 is also possible.
The focus is moved at high speed, and a plurality of images are acquired with a single shutter. By setting and storing data necessary for focusing when the focus is designated, high-speed focusing can be performed.

With the apparatus according to the sixth embodiment of the present invention, the near view image g1 and the far view image g2 can be photographed almost simultaneously with a simple operation. Therefore, it is possible to obtain two near-view images g1 and distant-view images g2 with small deviations in rotation, size, and position. Note that the designated area is not limited to two. Three
If one or more are specified, three near-view images and far-view images can be obtained.

Embodiment 7 of the Invention <Generation of All-In-Focus Image Based on Many Images and Acquisition of Three-Dimensional Structure> In the above description, an all-in-focus image was generated using two images of a near view and a distant view, but the present invention is not limited to this. Can be used to generate an all-focus image. For example, an all-in-focus image can be generated based on a large number of insect microscope images captured while slightly defocusing. In a normal microscope image sharpening process, focus determination is performed using a high-frequency component of a luminance level variation filter.
In the embodiment of the present invention, a focused area is determined by generating a blurred image and continuously comparing the blurred images. Also, by adding depth information to each k-image based on the in-focus position, it is possible to acquire a three-dimensional structure of the target.

In the embodiment of the present invention, the omnifocal image is reconstructed by using the selective integration method by continuous comparison. In the conventional selective integration method, a blurred image is created by repeatedly convolving a blurred function with two captured images, and the blurred image is compared with the other image. In the case of a microscopic image that changes the focus slightly, the judgment using only two images lowers the reliability.

For this reason, a plurality of front and rear images (for example, two front and rear images) of the target image are compared with the target image, and a final judgment is made using a judgment pattern sequence. For example, as shown in FIG. 12A, a plurality of images gn-2, gn-1, gn, g
n + 1 and gn + 2 are arranged in the order of focusing. Image gn-2 is far in focus and image gn + 2 is close in focus. Let the image gn be the image of interest. Next, with reference to a certain portion of the target image gn, it is determined whether the subject is in focus (focused) or out of focus (out of focus). That is,
The first part of the target image gn is divided into other images gn-2, gn-1,
gn + 1 and gn + 2 are compared with corresponding portions to determine whether these are in-focus or out-of-focus. Focused / unfocused
For example, the determination can be made based on the parameters of the Gaussian filter. For example, as shown in FIG.
A determination pattern such as “0,0” is generated. here,
0 and 1 indicate that the target image is more in-focus / out-of-focus. That is, the first portion is out of focus for the images gn-2 and gn-1, and is in focus for the images gn + 1 and gn + 2. From this, it is presumed that the first portion of the target image gn may be out of focus, and rather, the first image gn is in focus in the images gn + 1 and gn + 2. Similarly, “0,0,1,1” for the second part of the target image gn, “0,0,0,0” for the third part, and “0,0,0” for the fourth part.
0,1,0 ", the fifth part is" 0,1,0,0 "
Is obtained.

As is clear from the above, when a pattern “0, 0, 0, 0” is obtained for a certain portion of the target image gn, which means that all the images are in focus. It can be seen that the most in-focus image can be selected by using this portion.

The above processing is performed for a plurality of images,.
gn-1, gn, gn + 1, gn + 2,... Then, a pattern sequence as shown in FIG. 12B is obtained. Each pattern is obtained by setting an image on the pattern as a target image as shown in FIG.
Means the pattern obtained when performing the processing of. 1
Paying attention to the stage (first portion), it can be seen that the pattern of the image gn is the most in focus at “0, 0, 0, 0”, so the image gn may be used for that portion. . The patterns of the other images gn-2, gn-1, gn + 1, and gn + 2 are "0, 0, 1, 1", "0, 0,
1,1, "0,1,0,0", "1,1,0,0", and there is a high possibility that the image is out of focus. The same applies to the second stage (second portion). For the third stage (third part), the patterns of the images gn-2, gn-1, gn, gn + 1, and gn + 2 are "0, 0, 1, 1", "0, 0, 1", respectively. ,
0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1,
0,0 ", and there is no pattern that is most focused. However, when comparing the entirety of the images gn-2, gn-1, gn, gn + 1, and gn + 2, the patterns of the images gn-1 and gn have relatively three in-focus (0) patterns, so that It is in focus. Therefore, in the third stage, the image gn-1
Or gn. In the third stage of this example, it is considered that the image is in focus between the images gn-1 and gn.

As described above, the processing of FIG. 12A is performed on all images, and a pattern sequence as shown in FIG. 12B is obtained for each image. Then, by comparing the pattern rows at the pixels to be compared as described above, FIG.
It is determined that the image gn-1 or gn in (b) is the most focused image. As described above, in the present embodiment, the focus of each pixel is determined from the pattern sequence of the comparison result of all the images.
By using this processing, the determination can be performed with high accuracy. The processing for this is not so complicated and can be performed in a relatively short time.

Further, from the result of the above-mentioned in-focus area determination, information that the in-focus image of each pixel is the n-th image from the one with the shorter focal length can be added as depth information. For example, if the first part is adopted from the image gn, it can be determined that the first part is at the in-focus position of the image gn. The third portion can be determined to be at an intermediate position between the images gn-1 and gn. In the present embodiment, since the focus is continuously moved little by little with respect to the same object, the focus position is simple and relatively accurate based on the first focus position and the last focus position. Can be obtained.

According to the embodiment of the present invention, an omnifocal image can be obtained with high accuracy by continuously comparing a large number of microscope images. Further, the three-dimensional structure of the target can be known based on the focusing information.

Embodiment 8 of the Invention Embodiment 1 of the invention
, It is necessary to estimate the amount of blur (R1 and R2).
A Gaussian Unfilter is used for blur processing, but the amount of blur can be varied by adjusting this parameter.
Therefore, the blur amount can be estimated by estimating the parameter (number of times) of the Gaussian Unfilter.

This procedure will be described with reference to FIG. This graph shows the relationship between the number of times of the Gaussian Unfilter and the error. The vertical axis represents the squared difference value between the image without blur and the image subjected to the Gaussian filter, and the horizontal axis represents the number of Gaussian Unfilters. As can be seen from this graph, a downwardly convex curve is obtained. This curve can be approximated by a cubic curve.

It can be seen that when the parameters are set to 1, 2, 3, and 4, a minimum value is obtained between 2 and 3. In order to obtain more accurate parameters, a cubic curve approximating the graph of FIG. 13 is obtained. Next, the minimum value of the cubic curve is obtained, and the parameter at that time is obtained (about 2.4 in FIG. 13). By this procedure, the blur amount can be accurately estimated.

Note that actually, near the parameter = 0,
For example, a difference value may be obtained for 0.5, and an approximate curve may be obtained in consideration of this. According to the simulation results, better results were obtained by setting in this way.

The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention described in the appended claims, which are also included in the scope of the present invention. Needless to say, this is done.

In this specification, means does not necessarily mean physical means, but also includes a case where the function of each means is realized by software.
Further, the function of one unit may be realized by two or more physical units, or the function of two or more units may be realized by one physical unit.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of an all-focus image and / or arbitrary-focus image reconstructing apparatus according to Embodiment 1 of the present invention.

FIG. 2 is a reconstruction model of an arbitrary focus image f according to Embodiment 1 of the present invention.

FIG. 3 is a reconstruction filter K according to the first embodiment of the present invention;
It is a frequency characteristic of a and Kb.

FIG. 4 is an explanatory diagram of luminance correction in block units according to Embodiment 2 of the present invention;

FIG. 5 is a flowchart and an explanatory diagram of a positioning procedure between a plurality of focus images by a hierarchical matching method according to Embodiment 3 of the present invention.

FIG. 6 is a block diagram of an apparatus for aligning a plurality of focus images by a luminance projection method according to Embodiment 4 of the present invention.

FIG. 7 is an illustration of positioning between a plurality of focus images by a luminance projection method according to Embodiment 4 of the present invention.

FIG. 8 is a schematic block diagram of an all-focus image and / or arbitrary-focus image reconstructing apparatus including a filter for special effects according to Embodiment 5 of the present invention;

FIG. 9 is a schematic block diagram of a digital camera according to Embodiment 6 of the present invention.

FIG. 10 is an operation explanatory view of a digital camera according to Embodiment 6 of the present invention.

FIG. 11 is an operation flowchart of a digital camera according to Embodiment 6 of the present invention.

FIG. 12 is an explanatory diagram of a method of generating an all-focus image based on a large number of images according to Embodiment 7 of the present invention.

FIG. 13 is an explanatory diagram of blur amount estimation according to Embodiment 8 of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 10 Filter for image reconstruction 11 Synthesizing unit 12 Filter for special effects 13 Rectangular coordinate-polar coordinate conversion unit 14 Polar coordinate-rectangular coordinate conversion unit 20 Average luminance value calculation unit for each row / column 21 Luminance projection distribution creation unit 22 Comparison unit 23 Enlargement / Translation Estimation Unit 30 Lens 31 CCD 32 Processing Unit 33 Viewer 34 Focus Designation Unit 35 Memory 36 Focus Adjustment Mechanism

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/265 G06F 15/66 450 // H04N 101: 00 15/68 400A (72) Inventor Connie Rearignadi Tokyo 4-24-1-B 215, Kamoshidani, Setagaya-ku, Tokyo F-term (reference) 5B057 BA12 CD18 CE08 CE11 CH04 CH09 CH11 5C022 AA13 AB29 AB68 AC01 AC42 CA02 5C023 AA11 AA31 AA37 BA01 BA11 BA15 CA03 CA08 DA01 EA10 5C076 AA11 AA19 A27

Claims (12)

[Claims] 1. A first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter. A second filter for converting a second image focused on a second part based on the second filter, and a synthesizing unit for synthesizing an output of the first filter and an output of the second filter to generate an arbitrarily focused image And the first image and the second image
An arbitrary-focus image including a luminance correction unit that performs luminance correction on a block basis of an image so that the luminance of the image is substantially the same, and supplies the luminance-corrected image to the first filter and the second filter. Synthesizer. 2. A first filter for transforming a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter. A second filter for converting a second image focused on a second part based on the second filter, and a synthesizing unit for synthesizing an output of the first filter and an output of the second filter to generate an arbitrarily focused image And the first image and the second image
Alignment unit that performs image alignment based on a luminance distribution obtained by projecting image data in the horizontal and vertical directions, and supplies the aligned image to the first filter and the second filter. An arbitrary focus image synthesizing apparatus comprising: 3. A first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter. A second filter for converting a second image focused on a second portion based on the second filter, a special effect filter for performing a predetermined process on an output of the second filter, and an output of the first filter And an synthesizer for synthesizing an output of the special effect filter to generate an arbitrary focus image. 4. An orthogonal coordinate-polar coordinate conversion unit for converting the coordinates of the image data from rectangular coordinates to polar coordinates on the input side and the output side of the special effect filter, respectively, and polar coordinates for returning the coordinates of the image data from polar coordinates to rectangular coordinates. The arbitrary-focus image synthesizing apparatus according to claim 3, further comprising: an orthogonal coordinate conversion unit. 5. A method of arranging first to N-th images in focus on first to N-th parts based on first to N-th blur parameters input from the outside in the order of focal positions,
A determination unit that determines whether or not a part in the i-th image, which is one of these images, is in focus in a plurality of images before and after the i-th image; A comparison unit that compares which part of the image is focused in accordance with the determination pattern; and a combining unit that combines the first to Nth images based on the comparison result of the comparison unit to generate an all-focus image. Arbitrary focus image synthesizer. 6. A Gaussian filter for performing a filtering process on the i-th image while changing a parameter, and a difference process for obtaining a difference value between an output of the Gaussian filter and the preceding and following images. 6. An estimating unit for estimating the parameter by obtaining a value at which the difference value is minimized.
An arbitrary-focus image synthesizing apparatus as described in the above. 7. An image sensor, a processing unit for receiving a signal from the image sensor and converting the image data into image data, a display unit for displaying the image data processed by the processing unit, and specifying a plurality of targets in the image A focus specification unit that requests a plurality of images having different focuses, a focus adjustment mechanism that sets a focus position according to the specification of the focus specification unit, and a memory that stores image data. A plurality of simultaneous imaging cameras, wherein the plurality of objects are sequentially focused and imaged, respectively, and the plurality of image data obtained are stored in the memory. 8. The camera according to claim 7, wherein a plurality of images having different focal points are captured by one shutter operation. 9. A first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input blur parameter. A second filter for converting a second image focused on a second part based on the second filter, and a synthesizing unit for synthesizing an output of the first filter and an output of the second filter to generate an arbitrarily focused image And the first image and the second image
An arbitrary focus image including a luminance correction unit that performs luminance correction on a block basis of the image so that the luminance of the image is substantially the same, and supplies the luminance-corrected image to the first filter and the second filter. 8. The camera according to claim 7, further comprising a synthesizing device. 10. A first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input second blur parameter. A second filter for converting a second image focused on a second part based on the second filter, and a synthesizing unit for synthesizing an output of the first filter and an output of the second filter to generate an arbitrarily focused image And performing alignment between the first image and the second image based on a luminance distribution obtained by projecting image data in horizontal and vertical directions, and aligning the aligned image with the first filter and the first filter. The multiple-image simultaneous imaging camera according to claim 7, further comprising an arbitrary-focus image synthesizing device including a positioning unit that supplies the image to the second filter. 11. A first filter for converting a first image focused on a first portion based on an externally input first blur parameter, and a second externally input second blur parameter. A second filter for converting a second image focused on a second portion based on the second filter, a special effect filter for performing a predetermined process on an output of the second filter, and an output of the first filter The multiple image simultaneous imaging camera according to claim 7, further comprising an arbitrary focus image synthesizing device including: a synthesizing unit that synthesizes an output of the special effect filter to generate an arbitrary focus image. 12. An orthogonal coordinate-polar coordinate conversion unit for converting the coordinates of image data from rectangular coordinates to polar coordinates on an input side and an output side of the special effect filter, and polar coordinates for returning the coordinates of the image data from polar coordinates to rectangular coordinates. The multiple image simultaneous imaging type camera according to claim 11, further comprising a rectangular coordinate conversion unit.