JP4069468B2 - Image forming device - Google Patents

Description

Detailed Description of the Invention

The present invention relates to an apparatus for forming an image of each focused subject from images obtained by imaging subject groups at different distances.

When a subject is imaged with an optical or digital camera or video camera, a short-distance object is imaged using a lens with a long focal length, and the background image at a long distance is blurred when focused on it. It is copied. On the other hand, if the object is in the background at a long distance, the object at a short distance becomes a blurred image. In this way, information about the depth appears as a blur in a video or image captured by a lens having a finite aperture diameter. In order to eliminate this drawback, it is possible to focus on all short-distance and long-distance subjects using a wide-angle lens with a short focal length. The solution is not available.

In order to solve the problems described above, a method has been proposed in which the same optical system is sequentially shifted in focus and input to an image memory, and image processing is performed to synthesize an in-focus image for all regions. Yes. However, when such a method is adopted, it is necessary to develop a complicated mechanism for sequentially focusing.

On the other hand, as disclosed in Japanese Patent Laid-Open No. 3-80676, each subject is imaged using a single imaging system focused on three different distances, that is, a short distance, a medium distance, and a long distance, and then a short distance is obtained. A method of extracting and synthesizing images focused on distance, medium distance, and long distance using high frequency components of each image is also considered.
Japanese Patent Application Laid-Open No. 2001-298657 discloses images of each subject using a photographing system focused on three different distances, a short distance, a medium distance, and a long distance. A method of extracting and synthesizing images that are in focus using the difference waveform between the luminance value of each image and its moving average value is also considered.

As a method of using wavelet coefficients, wavelet transform is performed on multiple images with different focal lengths, and the characteristics of the maximum spectral amplitude of the coefficients are used, and the focused image is obtained from the inverse wavelet transform of the coefficients synthesized on the coefficient space. A method of obtaining can also be considered.

However, various problems remain in realizing such a method in either image forming apparatus. One of the problems when extracting and synthesizing a region having a high frequency component from each video is that it is difficult to obtain a video that is well focused on a video that includes disturbances such as noise. This is because the high frequency component of noise cannot be distinguished from that due to focusing, and the same drawback remains even when a difference waveform that is the difference between the luminance value of each image and its moving average value is used.
The second problem when extracting and synthesizing a region with many high-frequency components from each video is that the FFT or DCT generally used for frequency calculation is a block that is divided to improve frequency separation accuracy. The size needs to be increased to some extent, and therefore the block size cannot be reduced below a certain level. In addition, these calculation methods include problems because frequency components are calculated from a finite region.

A method of converting multiple focal length images into multi-resolution representations by wavelet transform, comparing the multi-resolution coefficients at the same position, and synthesizing the multi-resolution representations with the maximum spectral amplitude is used. There is a drawback in the phenomenon accompanying the shift and the shift invariance of the multi-resolution analysis.
The relationship between the coefficients on the multi-resolution expression is not independent, and the high-resolution level coefficient complements the low-resolution level coefficient. For this reason, if the multi-resolution expression in which each resolution expression coefficient is simply replaced with the highest spectral amplitude is inversely converted, there is a risk that an image that generates an image that does not exist in the original image is generated in a part of the image.
In addition, when image noise such as spike noise occurs in one of the images with a plurality of focal lengths changed, the highest spectral amplitude is not necessarily one-to-one with the in-focus because the higher hierarchical level coefficient of the multi-resolution expression is strongly affected. Does not correspond to. In the captured gray image, a phenomenon in which the shape of the object changes depending on the focal length or a phenomenon in which the brightness of the entire captured image increases as the blur increases. For this reason, when the object shape changes depending on the focal length, the same position cannot be accurately determined in the multi-resolution expression area, and the influence of the increased luminance on the entire image cannot be eliminated only by the spectral amplitude.
Also, it is vulnerable to the phenomenon that the analysis result greatly fluctuates even if the coordinate position to be analyzed in the multi-resolution analysis is moved a little. Thus, it is difficult to obtain a desired synthesized image by a method of synthesizing a multi-resolution expression using only the spectral amplitude.

The present invention has been made to solve such a problem, and easily and accurately synthesize a plurality of images focused on subjects at different distances so that a clear movie of each subject at different distances is obtained. It is an object of the present invention to provide an image forming apparatus that forms an image.

The present invention includes a video input unit 100 that focuses and images a plurality of subjects at different distances, and a conversion unit 200 that generates expression value sets at a plurality of different resolution levels for each video from the pixel values of each video. And means 300 for calculating the texture feature amount of the expression value set at the plurality of different resolution levels, and determining the clearest region from each video based on the texture feature amount of the expression value set at the plurality of different resolution levels. It is characterized by comprising means 400 and means 500 for combining the regions to form a subject image. The range of the clearest region includes both a plurality of pixels and one pixel.

  Since the image obtained from the image forming apparatus is composed of the pixels of the image that is the clearest among the images at different distances, the subject at all distances is clearly displayed as a whole. In the apparatus according to the present invention, a texture feature amount of an expression value set at a plurality of different resolution levels is formed for each pixel value for each distance image, and each distance is based on the texture feature amount of the expression value set. Since each pixel of the image of the image is determined to be the clearest, disturbances such as noise occur due to the synergy of the action and effect of the expression value set at multiple different resolution levels and the action and effect of the texture feature amount. Even if it is included, an image with high definition can be formed accurately.

  When comparing the method in the apparatus of the present invention with the case of using the nature of frequency and the case of using the difference waveform that is the difference between the luminance value of each image and its moving average value, a number of blocks are obtained to obtain the frequency. However, there is no restriction on the block division size which is not necessary and is caused by frequency separation accuracy.

The method using the difference waveform between the frequency component and the luminance value of each video and its moving average value has only a single (or a small number) of feature information about the video. Can provide indices that can be used to evaluate and judge image characteristics such as image roughness, fineness, image clarity, and flatness in a complex or comprehensive manner, and images and noise of natural images containing various frequency components. It is possible to effectively provide images with high definition even for images including disturbances.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing the configuration of an image forming apparatus according to the present invention, which extracts a focused portion from images at various distances and forms a clear image focused on all subjects.
In the present invention, first, an object at a plurality of different distances as shown in FIG. 10 by the video input means 100 shown in FIG. 1, that is, a tree T at a short distance, a person P at a medium distance, and a mountain M at a long distance. , And outputs each image focused on each subject.

Next, conversion means 200 for generating expression value sets at a plurality of different resolution levels is executed for each image captured in the memory by the image input means 100, and texture feature values are calculated for the expression value sets. Means 300 to execute. The texture feature amount is calculated in units of pixels or blocks, and the sharpest region is determined using the means 400 for determining the clearest region from each video from the texture feature amount in units of pixels or blocks. Naturally, the definition of the clearest region includes not only a plurality of pixel regions but also one pixel. By means of combining the pixels or blocks thus determined to form an image of the subject, an image in which subjects at all distances are clearly projected as a whole is formed.

2 has a large information capacity of each video captured from the video input means 100 that focuses and images a plurality of subjects at different distances. FIG. 2 is a block diagram illustrating the configuration of an apparatus according to the present invention that divides and extracts a focused portion to form a clear image focused on all subjects.

FIG. 5 shows a block diagram of the video input means 100 according to the first embodiment of the present invention that focuses on subjects at three different distances, that is, a short distance, a medium distance, and a long distance in one photographing system. In this configuration, each subject at three different distances, short distance, medium distance, and long distance, is distributed to three paths via the beam splitter. The distributed image is captured by the lens with images focused on main subjects at three different distances, ie, a short distance, a medium distance, and a long distance. The focus adjustment mechanism 103 is responsible for three different focusing operations.

Three different focused images of short distance, medium distance, and long distance for each passing through the lens are converted into electrical signals by CCD or two-dimensional image sensor 104, and are focused on each distance through video processing 105. The near image 1, middle image 2, and far image 3 shown in FIG. 10 are stored in the memory 106.

FIG. 10 shows images of a short-distance tree object T, a medium-distance person, and a long-distance mountain object P and M simultaneously stored in the memory 106 of FIG. The near image 1, the middle image 2, and the far image 3 are shown.

FIG. 3 shows an embodiment 2 of the present invention of an imaging system focused at three different distances, short distance, medium distance, and long distance. In the second embodiment, the video input unit 100, the conversion unit 200 that generates a representation value set of pixel values of each video at a plurality of different resolution levels, and the unit 300 that calculates the texture feature amount of the representation value set are parallel. To work.

FIG. 4 shows a third embodiment of the present invention in which an imaging system focused at three different distances, a short distance, a medium distance, and a long distance, is operated in a time division manner. In the third embodiment, the image input means 100, the conversion means 200 for generating the expression value set at a plurality of different resolution levels from the pixel values of each image, and the means 300 for calculating the texture feature quantity of the expression value set include: It operates by switching at regular time intervals.

Although not shown in the drawings, prior to the execution of the conversion means 200 for generating expression value sets at a plurality of different resolution levels from the pixel values of each video, each focused video is shifted by the optical axis of the optical system. If there is a difference in tilt or magnification (size, distortion), the center, tilt, and size of each video may be different, so that the video may not be synthesized or incorrect synthesis may occur. If possible, it is desirable to perform these pretreatments.

Next, the process of extracting the sharpest area (pixel) from the pixel values of each image focused on three different distances, short distance, medium distance, and long distance, and combining them to obtain a clear image explain.
The conversion means 200 for generating the expression value sets at the plurality of different resolution levels of the present invention shown in FIG. 1 or FIG. 2 is a wavelet transform, a Her transform, or a Hadamard transform. When applying the wavelet transform, a discrete wavelet transform or a transform based on a Gabor wavelet function is appropriate.

In the method using Fourier transform, which is often used in the field of image processing, the frequency component of the local wave of the image is buried in the noise component, but the transformation based on the Gabor wavelet function is this kind of local wave. Suitable for detecting frequency components of. In addition, reducing the amount of calculation is an issue for video processing in real time. In such a case, if the accuracy is sacrificed a little, the real-time video can be obtained by adopting the Her transform or Hadamard transform. Processing is also possible.

FIG. 6 schematically shows a fourth embodiment in which multi-resolution analysis by wavelet transform is applied as the conversion means 200 for generating a set of expression values at a plurality of different resolution levels for each video captured in the memory. It is. The left diagram of FIG. 6 shows the one-stage wavelet transform, and the right diagram shows the subband components generated from the first-stage wavelet transform, which are further subjected to the two-stage and three-stage wavelet transform. The subband LL1 component is extracted from the low frequency component in the original video, the HL1 component is the high frequency component in the horizontal direction of the original video, the LH1 component is the high frequency component in the vertical direction of the original video, and the HH1 component is the original video. The high-frequency component in the diagonal direction is extracted. In other words, this corresponds to extracting a region (edge information) having a large variation in pixel value in the horizontal direction, the vertical direction, and the diagonal direction. The nature of this wavelet transform makes it possible to accurately extract discontinuous boundaries and greatly contributes to accurately determining clear pixels.

FIG. 11 shows an embodiment in which one-step multi-resolution analysis is applied as a conversion means 200 for generating a set of expression values at a plurality of different resolution levels for pixel values of three images at short distance, medium distance, and long distance. 4 is expressed in more detail. A texture feature amount for each of the LL1 component, HL1 component, LH1 component, and HH1 component, which are sub-band components in the first stage, is obtained, and the clearest pixel in each video is determined from the texture feature amount. In this figure, the texture feature values of all the subband components are calculated and determined. However, depending on the feature of the video, a sufficiently clear pixel may be determined with only the LL1 component. In this case, the amount of calculation is reduced to a quarter of the case of determining from the whole, and is suitable for real-time processing.

Next, means 300 for calculating the texture feature quantity of the expression value set will be described with reference to FIGS. 7 to 9 and FIG. To calculate the texture feature quantity of the expression value set 300, that is, to calculate the texture feature quantity of the subband components in the fourth embodiment, a co-occurrence matrix (or co-occurrence matrix) is required.

To this end, first, in step S1 of FIG. 12, the block size L of the subband component, the relative position d (vector quantity having direction and distance) necessary for the calculation of the co-occurrence matrix, and the subband component are gradationized. Quantization step amount for setting is set. FIG. 7 shows the one applied to the subband component HL1 of the code 201 obtained from the multi-resolution analysis by the wavelet transform of the near (medium, far) video stored in the memory, and the texture feature calculation target The size of the block 4 is 3 × 3.

In step S3 of FIG. 12, the subband component of the calculation target block 4 of the texture feature amount of each video is converted into a subband component amount of q gradation by a quantization operation. In this step S3, the sub-band component value equal to or less than the minimum quantization amount is zero (0) value, so that the effect of automatically reducing and removing the noise component works. In the next step S4, the gradation of the point whose relative position is d from the point of the gradation value i from the q gradation subband component amount of the calculation target block 4 of the texture feature amount as shown in FIG. A co-occurrence matrix having a ratio M _d (i, j), (i, j = 0, 1,..., Q−1) whose value is j as ij elements is obtained.

FIG. 9 is composed of gradation values 6 of a set of expression values (subband components) at a plurality of different quantized resolution levels having the three gradation values (0, 1, 2) shown in FIG. The element value M _{d of} the occurrence matrix created considering only the gradation value of the expression value having the relative distance d = 1 from the left to the right with respect to the texture feature calculation target block 4 An example of (i, j) is shown. Since only one pair of gradation values (2, 1) surrounded by a dotted line appears in FIG. 8, M of element value 7 of (i, j) = (2, 1) of the occurrence matrix of FIG. _d (2,1) is 1. Here, reference numeral 5 in FIG. 9 represents local coordinates of the calculation target block of the texture feature quantity, reference numeral 6 represents the quantized gradation value of the subband component of the video, and reference numeral 7 in FIG. 9 represents an element of the co-occurrence matrix The value M _d (i, j).

In the next step S5, a plurality of texture feature amounts are calculated from the co-occurrence matrix shown in FIG. As typical texture feature amounts, 13 types of selection including angular second moment, contrast, entropy, correlation, variance, and the like can be considered. However, it is not necessary to use all of these texture feature values for each pixel or block in order to obtain the clearest image area for each pixel or block based on the texture feature values. Just choose. In the case of a general image, an angular second moment, contrast, and entropy are selected.

コントラストｆ_２は式２から計算され、ｆ_２は局所的な変化が大きい場合に大きな値になる。

エントロピｆ_３は式３から計算され、表現値集合の複雑さの測度を表し、表現値集合が偏りなく現れる場合は大きな値になる。

The angular second moment f ₁ is calculated from Equation 1 and indicates a poor degree of change in the expression value set.
If the change in gradation value is poor, M _d (i, j) of the element value 7 of the co-occurrence matrix is biased and f ₁ becomes a large value.

The contrast f ₂ is calculated from Equation 2, and f ₂ takes a large value when the local change is large.

Entropy f ₃ is calculated from Equation 3 and represents a measure of the complexity of the expression value set, and becomes large if the expression value set appears without bias.

ここでそれぞれｉ、ｊ方向の平均階調値をμ_ｘ、μ_ｙで表し、数５及び数６から計算される。同様にｊ、ｊ方向の偏差をσ_ｘ、σ_ｙで表し、数７から計算される。

Correlation f ₄ is calculated from Equation 4, f ₄ takes a large value when there are many uniform parts in the expression value set, and f ₄ is small when random data is included in a wide band such as random noise. Become. Therefore, the influence of disturbances such as noise can be reduced by adding the correlation f ₄ to the determination item of step 6.

Here, the average gradation values in the i and j directions are represented by μ _x and μ _y , respectively, and are calculated from Equations 5 and 6. Similarly, deviations in the j and j directions are represented by σ _x and σ _y and are calculated from Equation 7.

その他に分散、総和平均、総和分散、総和エントロピ、差分分散、差分エントロピ、相関情報測度のテクスチャ特徴量も選択できる。Homojiniti f ₅ is calculated from the equation 8, a large value when poor locally varying.

In addition, texture feature quantities such as variance, sum average, sum variance, sum entropy, difference variance, difference entropy, and correlation information measure can be selected.

In step S6 of FIG. 12, the clearest video is determined from the pixel or block of each video from the texture feature amount obtained for each pixel or block of each video. This determination may be made from only one item of the texture feature amount, for example, contrast f ₂ , or a plurality of combinations such as three combinations of angular second moment f ₁ , contrast f ₂ and entropy f _3. You may evaluate and determine comprehensively from the item of.

Further, when the texture feature amount distribution of the contrast f ₂ of each image is taken and the distribution state is compared in the clear region and the unclear region of each image, the latter region has a flat distribution or a sparse distribution. The distribution state is clearly different. If this fact is utilized, not only the sharpest image among pixels or regions of each image is determined by the single or a plurality of combinations of the texture feature amounts in this step S6, but also the pixel and the pixel neighborhood or the region and the region. If the distribution of the texture feature amount in the vicinity is obtained and the distribution is also taken into consideration to determine the clearest image among the pixels or regions of each image, the accuracy is further improved.

In addition, the pixel or area data obtained in step S6 may store the pixel values of the clearest video or area directly in the display memory. Or, it is replaced with a number, and only the identification symbol or number corresponding to each extracted pixel or area is recorded in the memory, and when synthesizing, the corresponding video data is read with reference to the identification symbol or number. Good.

In the above description, the video data has been described as pixel values (in general, R, G, B values in the case of a color image), but the video data has luminance values Y or color difference values I and Q and lightness values V or hues. Even in the case of saturation, the basic process (process) for obtaining the clearest pixel or region based on the texture feature amount of the expression value set at the plurality of different resolution levels does not change.

また記載の色相、彩度、明度値ＶについてはＲＧＢ画像からＨＳＶ値への変換はやや複雑ではあるけれども変換の式が考案されている。When obtaining a clear pixel or region focused from the texture feature amount of the expression value set at a plurality of different resolution levels of the luminance value Y of the pixel of each video, the luminance value of the NTSC system from the RGB pixel value of the RGB color image Equation 23 is used to determine Y. Here, I and Q are color difference values.

For the hue, saturation, and lightness value V described, although conversion from an RGB image to an HSV value is somewhat complicated, a conversion formula has been devised.

The invention's effect

As described above, according to the present invention, each pixel is composed of the pixel of the image that is the clearest among the images of different distances, so that subjects at all distances are clearly displayed as a whole. . In the apparatus of the present invention, a texture feature amount of an expression value set at a plurality of different resolution levels is formed for each pixel value for each distance image, and each distance value is based on the texture feature amount of the expression value set. Since each pixel of the image is determined to be the clearest, disturbances such as noise are included due to the synergy of the effect of the expression value set at multiple different resolution levels and the effect of the texture feature. However, it is possible to form an image with high definition accurately. The global effect of the expression value set at a plurality of different resolution levels and the texture feature quantity of the expression value set at that resolution level is suitable for the effect of reducing the amount of calculation and real-time processing. In addition, the effect is robust against slight changes in object shape and phase.

When comparing the method and frequency characteristics of the apparatus of the present invention with the case of using the difference waveform, which is the difference between the luminance value of each image and its moving average value, a number of blocks are obtained to obtain the frequency. There is no need to divide and there is no restriction on the block division size resulting from the frequency separation accuracy.

In addition, the method of synthesizing using the difference waveform between the frequency component and the luminance value of each video and its moving average value has only a single (or a small number) of feature information about the video. It can provide indexes that can evaluate and judge image features such as image roughness and fineness, image clarity, and flatness in a complex or comprehensive manner, such as images and noise of natural images containing various frequency components. Therefore, it is possible to provide an image with high definition effectively even for an image including a disturbance.

FIG. 2 is a block diagram showing a configuration of an image forming apparatus that extracts a focused portion from images of each distance and forms a clear image focused on all subjects. FIG. 3 is a block diagram illustrating a configuration of a video forming apparatus that divides a video of each distance according to claim 2 and extracts a focused portion to form a clear video focused on all subjects. . The second embodiment of the photographing system focused at three different distances of short distance, medium distance, and long distance is shown. This is a third embodiment in which an imaging system focused at three different distances, a short distance, a medium distance, and a long distance, is operated in a time division manner. FIG. 2 is a configuration diagram of Embodiment 1 of video input means for focusing on a subject at three different distances of a short distance, a medium distance, and a long distance in one photographing system. 10 schematically shows Example 4 to which multiresolution analysis by wavelet transform of each video captured in a memory is applied. The block for which the texture feature-value of the subband component HL1 of Example 4 obtained from multiresolution analysis by wavelet transform is calculated is shown. It is the figure explaining element value 7M _d (i, j) of the co-occurrence matrix calculated | required about the horizontal direction from the gradation value of the subband component of Example 4. An example of an occurrence matrix element value M _d (i, j) created from sub-band component gradation values 6 of three gradation values (0, 1, 2) is shown. The near image 1, the middle image 2, and the far image 3 focused on a short-distance tree subject T, a medium-distance person P, and a distant mountain M stored in the memory 106 are shown. FIG. 10 shows a fourth embodiment in which wavelet decomposition, texture feature amount calculation, and determination of the clearest region are applied to three pixel values of a short distance, a medium distance, and a long distance. The most sharp pixel is determined using means for determining the clearest pixel or region from each image according to the texture feature amount of the expression value set, and a part of the flow that forms an image in which all subjects are clearly displayed It is the shown flowchart figure.

Explanation of symbols

1 Near image stored in the memory 106 2 Medium image stored in the memory 106 3 Far image stored in the memory 106 Operation block 5 of the expression value set at a plurality of different resolution levels 5 Local coordinates 6 Quantized Tone value 7 of expression value set at a plurality of different resolution levels Element value M _d (i, j) of co-occurrence matrix
201 Subband component HL1 obtained from multi-resolution analysis by wavelet transform

Claims (2)

Video input means 100 that focuses and images a plurality of subjects at different distances, conversion means 200 that generates expression value sets at a plurality of different resolution levels of each video from the pixel values of each video, and the plurality Means 300 for calculating the texture feature amount of the expression value set at different resolution levels, means 400 for determining the clearest region from each video based on the texture feature amounts of the expression value set at the plurality of different resolution levels, An image forming apparatus comprising means 500 for combining the areas to form an image of a subject. A video input unit 100 that focuses and images a plurality of subjects at different distances, a unit 600 that divides each video into a plurality of regions, and a pixel value of a block divided by the division unit for each block. Converting means 200 for generating expression value sets at a plurality of different resolution levels; means 300 for calculating texture feature quantities of expression value sets at the plurality of different resolution levels; and expression values at the plurality of different resolution levels. An image forming apparatus comprising: means 400 for determining the clearest area from each image based on texture features of the set; and means 500 for combining the areas to form a subject image.

Images