JP2001298657A - Image forming method and image forming device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To photograph the objects at different distances and to form a clear image of every object. SOLUTION: The images focused on the objects P, T and M set at the short, medium and long distances are formed by the CCD 8, 9 and 10 respectively and stored in a memory. Each of these focused and photographed images of different distances is fractionated into plural blocks and a difference waveform, i.e., the difference between the luminance value and its movement mean value is formed in each block to the image of every distance. On the basis of difference waveforms, the clearest one of image blocks of every distance is decided and the clearest image blocks of every distance are connected together into the image of each object. In such a constitution, the accurate definition can be decided even with a small object that is not main and has no conspicuous outline or even when the miscellaneous beams are included in the blocks.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image forming method and apparatus, and more particularly, to an image forming a plurality of objects at different distances and forming in-focus images. The present invention relates to a forming method and an apparatus.

[0002]

2. Description of the Related Art When a subject is photographed by a video camera and displayed on a monitor such as a television, the focused image is only the conjugate point of the photographing lens, and other object images are blurred. For example, when a lens having a relatively long focal length is used, a person in the foreground is photographed, and if it is focused on, the background image in the distant view obtained is blurred. In addition, if the background is focused, the person of the foreground subject becomes a blurred image.

In order to focus on all the objects in the near and distant views, it is conceivable to use a lens with a short focal length (wide angle). In this case, however, the angle of view or magnification may not be suitable for the application. Yes, and not a solution that can be used in every situation. It is also conceivable to increase the aperture value. However, in this case, it cannot be said that this is a solution that can be used in all situations because of the amount of light.

[0004]

For this reason, it has been proposed to sequentially shift the focus by the same optical system and input the result to an image memory, and then to process the image later to perform image synthesis so that all areas are in focus. Have been.

[0005] Alternatively, for example, each subject is photographed using two (or three) independent photographing systems focused on a short distance, a medium distance (and a long distance), and then a short distance,
A method of extracting and synthesizing an image focused on a medium distance (and a long distance) has also been considered. For example, JP-A-3-
Japanese Patent No. 80676 discloses that each image focused at different distances is divided into a plurality of regions, and a region where the high-frequency component is maximum in each divided region is extracted from each image and synthesized. The in-focus image is obtained at each distance.

However, in the case of extracting and synthesizing an area having a large number of high-frequency components from each image as described above, for example, if a small subject having a conspicuous contour is located at a long distance in a divided area, or there is noise, In fact, although the divided area is in focus at a short distance, a video image is synthesized by extracting a long distance area, and a good focused image cannot be obtained over all distances. This can be solved to some extent, for example, by increasing the number of divisions of the video screen. However, if the number of divisions is increased, the image processing is also increased, and if the number of divisions is increased, the subject is subdivided into an unnecessary small area. That is, there is a problem that the effect is reversed.

Accordingly, the present invention has been made to solve such a problem, and a plurality of images focused on objects at different distances can be easily and accurately synthesized at different distances. An object of the present invention is to provide an image forming method and apparatus capable of obtaining a clear image of a certain subject.

[0008]

SUMMARY OF THE INVENTION The present invention relates to an image forming method and apparatus for imaging an object at a plurality of different distances to form an image of the object. The image at each distance captured in focus is divided into a plurality of blocks, and a difference waveform that is a difference between a luminance value and a moving average value is formed for each block at each distance image. Determining the sharpest video block among the video blocks at each distance based on the difference waveform of the video at each distance, and combining the sharpest video blocks determined for each block to obtain the subject. Is formed.

[0009] The image obtained in this way is composed of the images of the plurality of subdivided areas, each of which is the sharpest of the images at different distances. It is projected clearly. In that case, a difference waveform, which is a difference between the luminance value and the moving average value, is formed for each distance image for each block, and based on the difference waveform, which of the image blocks at each distance is the sharpest. Therefore, it is possible to accurately determine the sharpness even if the block includes a small subject other than the main having a distinctive contour or noise.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on an embodiment shown in the drawings.

The position at which a focused image or image formed by the optical system is obtained varies according to the distance between the subject and the optical system. For example, as shown in FIG. 1, a mountain M at a long distance (from about 10 m to ∞),
Tree T at medium distance (about 5-10m), short distance (about 1m)
Are obtained at specific positions given by the focal length of the photographing lens and the distance between the lens and the subject, respectively, and one (surface) CCD light receiving element is provided on the image side. By simply placing the surface, the film surface or the focus glass plate, each subject M in these distances,
Not all images of T and P can be seen clearly.

According to the present invention, images of two subjects (persons, trees) and three subjects (persons, trees, mountains) or more subjects having different image formation positions are synthesized by image processing, whereby these images are combined. Everything can be clearly projected on one plane (monitor, film, etc.).

The following describes a person P at a short distance (about 1 m).
Tree T at medium distance (approximately 5 to 10 m) and long distance (approximately 1
FIG. 2 shows an example of three subjects on a mountain M at 0 to ∞).
This will be described with reference to FIG.

In FIG. 2, reference numerals 1 to 7 denote an optical system for simultaneously photographing a subject P of a person at a short distance, a tree T at a medium distance, and a mountain object M at a long distance. , A relay system 2. A beam splitter 3 is disposed at the rear of the relay system 2, and one of the split light images is transmitted through a master lens 4 for focusing on a subject P via a CCD (or another type of CCD). (Dimensional image sensor) 8.

The light image split by the beam splitter 3 is further split by the beam splitter 5, and one of the split images is connected to another CCD 9 (or another system) via a master lens 6 for focusing on a subject T. 2
Image is formed on a three-dimensional image sensor, and the other is connected to another CCD through a master lens 7 for focusing on a subject M.
10 (or another type of two-dimensional image sensor).

As will be described later, the CCDs 8, 9, and 10 are image-processed by image processing devices 11, 12, and 13 for short-range, medium-range, and long-range, respectively, and then processed by a synthesis circuit 14. The images are synthesized and displayed on the monitor device 15. The image processing apparatuses 11, 12, and 13 are memories 11a, 12a, and 1 that store image-processed video data, respectively.
3a. Each of the image processing apparatuses 11, 12, 13 and the synthesizing circuit 14 can be constituted by an individual microprocessor, respectively, or can be realized by a common microprocessor for controlling the whole.

Although not shown in FIG. 2, each of the master lenses 4, 6, and 7 has a manual or automatic focusing mechanism for focusing on the main objects P, T, and M, respectively. .

When the automatic focus adjustment mechanism is employed, for example, the master lenses 4 and 6 are driven by auto focus control to always focus on the short-range and medium-range main subjects P and T which are difficult to focus. Keep it. Also, when the background object or the long-distance object M is focused, the depth of focus is naturally deep, so that even if the master lens 7 is fixed at infinity, almost no focus is practically impaired.

Hereinafter, a method of synthesizing images focused at different distances and forming an image focused on an object according to the present invention will be described. In the present invention, first, the preprocessing as shown in FIG. 3 is performed on each video.

As shown in step S1 of the flow chart of FIG. 3, the far, middle and short distance video signals output from the CCDs 8, 9 and 10 are converted into digital signals and then stored in the corresponding memories 11a, 12a, 13a.

In step S2, for example, if there is a difference in the displacement, inclination, or magnification (magnitude, distortion) of the optical axis of the optical system at each distance, the center, inclination, and size of each image are different, and image synthesis is performed. Therefore, mechanical error correction for the center, tilt, size, and distortion (distortion) is performed on the data stored in each memory. this is,
Since each shift is known depending on the specifications of the optical system to be used, it can be corrected by performing coordinate conversion of each image pixel (pixel). Among the errors, an error due to the distortion particularly appears in an image around the screen and is a kind of aberration generated due to the characteristics of the lens, and is a barrel-type or pincushion-type mechanical error. Since this correction value can be approximated by a power curve, it is corrected using a power equation.

Further, in step S3, for subsequent correction of centering, inclination, size and the like based on luminance and color tone,
Performs filtering such as brightness correction and color tone log correction. Thereby, the intensity value of each pixel is corrected.

Thus, each of the memories 11a, 12a,
13a, the coordinates (xn, y) of each pixel Pn
n) and the luminance values R of red (R), green (G) and blue (B)
(Pn), G (Pn), B (Pn) and their average luminance value L (Pn) are stored. For the coordinate values, the coordinate values in which the mechanical error has been corrected in step S2 are stored, and for the intensity values, the values filtered in step S3 are stored.

Even if the known mechanical error is corrected in this way, the center, the inclination, and the size of each image may not match. If such an error exists, it becomes difficult to accurately combine two or more pixels. Therefore, according to the present invention, the subject portion corresponding to each pixel is close, medium, and long distance by the following processing. Correct so that they match in the video.

Since the images at the near and middle (far) distances are considered to be approximately the same due to the correction of the mechanical error in step S2, a predetermined area of each image is selected, and this area is set horizontally. And a portion where the average brightness L of each pixel changes from the minimum value (maximum value) to the maximum value (minimum value) or a portion where the absolute value changes to the maximum, that is, a difference (differential) of the average brightness value of each pixel by vertical scanning. The coordinate value of the part where is maximized or the part where the absolute value is maximized is detected (step S4).
Then, the coordinate system of the middle distance image is translated in the horizontal and vertical directions based on the short distance image, and centering is performed so that the detected portions match within a predetermined allowable error (step S5).

On the other hand, in order to detect whether or not there is an error in the inclination and size (distortion) between the images, the coordinate values of other pixels corresponding to step S4 are detected in the area.
If the coordinate values also match within a predetermined allowable error, it is determined that there is no error in inclination or size, and the process moves to step S10.

On the other hand, if they do not match, step S7
In, inclination correction is performed. This is performed by performing coordinate rotation so that the other pixels also match within a predetermined tolerance. If an error still exists due to the coordinate rotation, it is determined in step S8 that size (distortion) correction is necessary, and size (distortion) correction is performed in step S9. This is, for example, a type of magnification correction, and is performed by multiplying one coordinate value by a predetermined coefficient.

The corrections performed in steps S5, S7 and S9 are collectively considered to be corrections by "coordinate conversion in a broad sense". As described above, by performing the coordinate conversion, the subject portion corresponding to each pixel is matched within a permissible error in the short distance image and the medium distance image.

Since the short-range image and the medium-range image match as described above, it is determined in step S10 whether or not it is necessary to correct another image, that is, a long-range image. If necessary, the process returns to step S4 to perform a coordinate conversion in a broad sense even on a long-distance image.

With the above processing, the centering, tilt correction and size correction of each image are completed, and the memories 11a, 12a,
13a stores a corrected coordinate value for each pixel, a luminance value for each color, and an average value thereof.

Next, a method for extracting a focused portion from each distance image and synthesizing a clear image will be described with reference to the flow of FIG.

According to the present invention, each image is divided into a plurality of blocks, and the sharpest block is extracted from each image for each block, and is synthesized to obtain a clear image. Therefore, the number of divisions for dividing the near, middle, and long distance images is set (step S1). FIG. 5A shows the near (middle, medium, and medium) stored in the memory 11a (12a, 13a).
A state in which the (far) image 20 is divided into a plurality of m blocks B _{1 to} B _m is shown. This block size is the pixel n
× n (for example, n = 16).

Subsequently, the variable i is set to 1 (step S).
21) Block B from near, medium and long distance video
₁ is read out (step S2).
2). In this case, the reading order of the pixels P _i (i = 1 to nn) does not return to the beginning of the next row when the end of each row (y) is reached, as shown in FIG. The corresponding pixel in the next row is read out as indicated by.

[0034] In step S23, to form a differential waveform as well as the mean difference waveform block B ₁ for each image. The difference waveform is obtained by subtracting the j-point moving average waveform from the original waveform of the video. As shown in FIG. 6A, an original waveform (indicated by a two-dot chain line) is, for example, a short-distance image (hereinafter referred to as a near image). ) Of the image waveform read out from the block B ₁ , the waveform value of which is an arithmetic mean value L ((R
+ G + B) / 3). The j-point moving average waveform (indicated by the dotted line) shows j at 3, 4,. . . . . Is a moving average of j luminance values (L). Therefore, the difference waveform (N) of the near image is as shown by the solid line. By forming the difference waveform in this manner, it is possible to reduce the influence of the luminance change when the contrast at the boundary between the noise and the subject is large on the image sharpness.

[0035] that described above is to form was the near-video difference waveform (N), similarly the difference waveform (M) of the middle picture for the block B ₁ and the far image difference waveform (F) Further, an arithmetic average waveform (hereinafter, referred to as an average difference waveform (A)) of a difference waveform of each of the near, middle, and far images can be formed. Each of these waveforms is illustrated in FIG.

[0036] Then, the near-in step S24, in, select the block of the most vivid images of each B ₁ block of the far image, which the image data of the block of B _1. Soon,
Among, the method determines whether any blocks are most vivid images among each B ₁ block of the far image, high frequency part detection method shall clear those rich in (1) the high frequency component, ( 2) There are a correlation coefficient method for determining sharpness from a correlation value of each difference waveform, and (3) a high amplitude method, and each method will be described below.

The high frequency part detection method, as shown in FIG. 7, each image block B ₁ of the difference waveform (e.g., the difference between the waveform of the near video) spectral components h _i performed fast Fourier transform (FFT) of The image block in which the center of gravity = Σx _i * h _i is maximized is set as a clear block, where x _i is the coordinates where the spectrum component appears and x _i is the coordinate where the spectrum component appears.

On the other hand, when judging a clear image from the correlation of each difference waveform, the following preparations are made. As shown in FIG. 6B, the difference waveform includes a difference waveform (N) for a near image, a difference waveform (M) for a middle image, and a difference waveform (F) for a far image.
Since there is an average difference waveform (A), the correlation coefficients between the difference waveforms are represented by R _AN , R _NM,. . . . . It is described. Here, a subscript sign indicates between the two difference waveforms. For example, R _AN is the correlation coefficient between the average difference waveform and the difference waveform of the near image, and R _NM is the correlation coefficient between the difference waveform of the near image and the difference waveform of the middle image.

[0039] FIG. 8 is illustrated the average difference waveform and near video difference waveform (N) (A), the luminance value k _i for each waveform difference at each pixel coordinate x _i, a _i and their difference value K _i (= k _i -
a _i ), and H _i = C (constant value) −K _i . As the correlation coefficient R _AN , the correlation of the waveform, for example, H
_i (vector components), ΣH _i, the reciprocal of _{K i, Σ (1 / K} i)
It can be either. Other correlation coefficients R _NM,. . . . Can be similarly defined, and can be any of them. As the correlation coefficient, a correlation coefficient obtained from the correlation between the frequency components and the correlation between the amplitude waveforms may be used instead of the correlation between the waveforms as described above. In the experiment, out of the correlation coefficient, so choose the correlation .SIGMA.H _i waveform good results have been obtained, using a correlation coefficient according to this definition, a clear image block R _AN, R _AM, from R _AF The determination method will be described.

First, it is determined whether the image of the average difference waveform (A) of the block is unclear (blurred). If the correlation coefficient between the difference waveforms of the respective images is small, it can be determined that at least two of the three images of near, medium, and far images are unclear. In this case, the average difference waveform (A) is also determined. It can be blurred (usually, when two images are blurred, the entire image, that is, the average difference waveform also becomes blurred).
On the other hand, if there is a significant correlation coefficient between the difference waveforms of the respective images, it can be determined that at least two of the three images of the near, middle, and far images are clear. In this case, It is assumed that the average difference waveform (A) is clear.

As described above, when the average difference waveform (A) is found to be sharp or unclear, if the average difference waveform (A) is unclear or close to it, the correlation coefficient of R _AN , R _AM , and R _AF is determined. If the image with the smallest value is sharp, while the average difference waveform (A) is sharp or close to it, R _AN , R _AM ,
The image having the largest correlation coefficient value in R _AF can be determined to be clear. That is, when all of the correlation coefficients between the near, middle, and far images are small, the image with the smallest correlation coefficient with the average difference waveform is a clear image, while the image with the larger correlation coefficient is If there is only one set, the video having the maximum correlation coefficient with the average difference waveform is clear, and therefore, a clear block can be extracted from each video block by this algorithm.

In the third high-amplitude method, as shown in FIG. 9A, the amplitude value (peak value) of the difference waveform of each image (near image in the figure) is shown in FIG. As described above, the image blocks in which the rearrangement and the moment of amplitude Σh _i * a _i are maximum are defined as clear blocks.

As described above, the near, middle, and far images B ₁
Since the sharpest block was extracted from among the blocks, the block is recorded in the memory.

Subsequently, it is determined whether the block has become the last block (block B _m ) (step S25). If not, the process proceeds to the next block (step S2).
6, S27), the data of the block is read, and the same processing is repeated. When the processing of the last block is completed, the clear blocks recorded in step S24 are combined into video data (step S27). Since the video data obtained in this way is composed of the sharpest video among the near, middle and distant video in each of the subdivided areas, the subjects at all distances as a whole can be clearly seen. It becomes the video data to be projected.

When a clear image is obtained by using the above-mentioned correlation coefficient, the correlation of the luminance value is calculated. Instead of the luminance value, frequency analysis is performed to obtain the spectrum intensity.
It is also possible to determine the sharpness by obtaining the correlation between the spectral intensities of the respective difference waveforms. That is, near (medium, far)
A spectrum as shown in FIG. 7 is obtained from the difference waveform of the image, the average spectrum is obtained, and a clear image can be determined from the correlation coefficient with the difference between the spectra.

The data of each block obtained in step S24 may be stored in an individual memory, or only the address corresponding to each extracted block in the memories 11a, 12a and 13a is recorded. When combining, the corresponding data may be read by referring to the address.

[0047]

As described above, according to the present invention,
Since each image in the plurality of subdivided regions is formed of the sharpest image among images at different distances, subjects at all distances as a whole are clearly displayed. In that case, a difference waveform, which is a difference between the luminance value and the moving average value, is formed for each distance image for each block, and based on the difference waveform, which of the image blocks at each distance is the sharpest. Therefore, it is possible to accurately determine the sharpness even if the block includes a small subject other than the main having a distinctive contour or noise.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram illustrating a state in which a subject at each distance is imaged.

FIG. 2 is a configuration diagram showing a configuration for synthesizing a clear image from images of a subject at a short distance, a medium distance, and a long distance.

FIG. 3 is a flowchart showing a flow for performing positioning of each image.

FIG. 4 is a flowchart showing a flow of subdividing each image into blocks and combining sharp blocks to form an image of a subject.

FIG. 5 is an explanatory diagram showing a state in which a video is divided into blocks.

FIG. 6 is a diagram showing a difference waveform and an average difference waveform of an image.

FIG. 7 is a diagram showing a frequency spectrum of a difference waveform.

FIG. 8 is a diagram illustrating a correlation between difference waveforms of an image.

FIG. 9 is a diagram illustrating a method of determining sharpness from an amplitude value of a difference waveform of an image.

[Explanation of symbols]

 8, 9, 10 CCD 11, 12, 13 Image processing device 14 Synthesis device 15 Monitor device

 ────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Seiichi Okochi 3-10-20 Chiyoda, Naka-ku, Nagoya City, Aichi Prefecture F-term (reference) 5C022 AA01 AA11 AB22 AC01 AC42 AC54 AC55 5C023 AA11 AA36 AA37 AA38 BA03 BA12 CA03 DA04 EA03 EA05

Claims (6)

[Claims] 1. An image forming method for imaging a subject at a plurality of different distances to form an image of the subject, wherein the subject at each distance is focused and imaged. The distance image is subdivided into a plurality of blocks, and a difference waveform that is a difference between a luminance value and a moving average value is formed for each distance image for each block, and the difference waveform of the distance image is formed for each block. A video forming method comprising: obtaining a sharpest video block among video blocks at respective distances for each block based on each block; and combining the sharpest video blocks obtained for each block to form a video of a subject. 2. The image forming apparatus according to claim 1, wherein a frequency component or an amplitude value of a difference waveform of the image at each distance is obtained to determine which distance of the image block is the sharpest. Method. 3. The method according to claim 1, wherein an average difference waveform is obtained as an average value of the difference waveforms of the images at the respective distances, and it is determined which image block at which distance is the sharpest based on the correlation between the difference waveforms. 2. The image forming method according to claim 1, wherein: 4. An image forming apparatus for imaging an object at a plurality of different distances to form an image of the object, wherein the image forming apparatus focuses and images the object at each distance. Means for subdividing the video at each distance into a plurality of blocks, and forming a difference waveform, which is the difference between the luminance value and the moving average value, for each video at each distance, and the video at each distance. Means for obtaining the sharpest video block among the video blocks at each distance based on the difference waveform of each block, means for combining the sharpest video blocks obtained for each block to form an image of the subject, An image forming apparatus comprising: 5. The image forming apparatus according to claim 4, wherein a frequency component or an amplitude value of a difference waveform of the image at each distance is obtained to determine which distance of the image block is the sharpest. apparatus. 6. An average value difference waveform which is an average value of difference waveforms of the images at respective distances is determined, and it is determined which image block at which distance is the sharpest based on a correlation between the difference waveforms. The image forming apparatus according to claim 4, wherein