JP3191928B2 - Image input / output device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to an image input / output device for inputting / outputting image information.

[Prior Art] Generally, in order to optically capture an image having excellent resolution and a large magnification and brightness, an imaging optical system using an optical element having a large aperture is required. However, an imaging optical element represented by a lens generally has a small depth of focus when the aperture is large. However, in the field of using image equipment such as a microscope, a camera, and an endoscope, it is desirable that the obtained image has excellent resolution and brightness, but at the same time, the image has a deep depth of focus. Is strongly required.

As a conventional technique for obtaining an image with a large depth of focus, for example, the document “WT Welford, Journal of Optical Society of Amer
As shown in “ica.Vol.50 p.749 (1960)”, there is a method using an imaging optical system provided with a special opening such as an annular opening. According to this means, there is an advantage that an image having a large depth of focus can be easily obtained, but there is a disadvantage that the resolution and the amount of light are significantly impaired.

Further, as another technical means, for example, a document “SASugimo
to and Y. Ichioka, Applied Optics Vol. 24.p. 2076 (198
5), or the method using image processing technology as introduced in “Tohoshi Ota, Atsuyoshi Sugihara, Noboru Sugie, IEICE Transactions (D), J66-D, 1245, (1983)”. is there.
In this method, a plurality of images at different in-focus positions (hereinafter, referred to as in-focus positions) in the object space are input,
This is a means for obtaining local dispersion information from these and synthesizing a focused part based on the information. According to this means,
An image with a deep depth of focus can be obtained without losing resolution or brightness. Therefore, it can be said that this is an extremely useful means. However, when the object is an object having a smooth surface or an object whose structure is greatly different from one place to another, application to the object is difficult. In addition, there are many practical inconveniences such as the necessity of large-scale hardware for realizing the apparatus, the complexity of the algorithm, and the like.

[Problems to be Solved by the Invention] As described above, the conventional technical means have the following disadvantages.

(1) Means for increasing the depth of focus using an imaging optical system provided with a special opening such as an annular opening has the disadvantage that the resolution and the amount of light are significantly impaired. In other words, although it is possible to obtain an image with a large depth of focus, the resolution and brightness are lost, and there is a drawback that the optical system cannot satisfy all of the depth of focus, resolution and brightness. Was.

(2) A method of inputting a plurality of images having different focal positions and synthesizing a focused partial image from these pieces of local dispersion information may be difficult to apply depending on the condition of the target object. There was a disadvantage that it often caused inconvenience.

Therefore, an object of the present invention is to provide an image input / output device that can reproduce an image with a large depth of focus without losing resolution or brightness, and that is practical, has a wide application range to a target, and can be easily realized. It is in.

[Means for Solving the Problems] In order to solve the above problems and achieve the object, an image input / output device of the present invention is configured as follows. In addition, characteristic configurations of the present invention other than the following will be clarified in the embodiments.

An image input / output device according to the present invention is a device for inputting / outputting image information, comprising: a focusing unit for focusing on a plurality of different object surfaces; Image input means, a range selecting means for selecting a range of a focal plane position of a plurality of images input by the image input means, and an image adding means for adding the plurality of images selected by the surrounding selecting means When,
Image processing means for performing recovery processing by spatial frequency filtering on the image added by the image adding means.

Embodiment (First Embodiment) FIG. 1 is a diagram showing a configuration of a first embodiment of the present invention.
An image of the object is formed by a lens 1 on a light receiving portion of an image sensor 2 including a charge-coupled device (hereinafter abbreviated as CCD), an image pickup tube, and the like. Note that, in a normal case, the input optical system is composed of a combination of a plurality of lenses, but only one is shown in the figure for simplification. An output signal from the image sensor 2 is supplied to an analog / digital converter (hereinafter, A / D converter).
The signal is converted into a digital signal by a converter 3. The digital signal is added by the adder 4 to the image signal recorded in the memory 5. The result of the addition is recorded in the memory 5 again. The above operation is repeatedly performed while moving the lens 1 by the in-focus position controller 6. That is, the above operation is performed while discretely changing the in-focus position of the optical system at appropriately set distance intervals and distance ranges. The addition is performed for each image thus input, and the result is stored in the memory 5.

Next, the added image is subjected to appropriate recovery processing by the recovery processing device 7. For example, high-pass filtering or band-pass filtering for the spatial frequency is performed. The above processing result is stored in the memory 5 again. The image signal subjected to the restoration processing stored in the memory 5 is converted into an analog signal by a digital / analog converter (hereinafter, abbreviated as a D / A converter) 8 and displayed on a display monitor 9. Is displayed. Control of the timing, signal flow, and the like in the above operation is performed by the controller 10.

As a means for changing the focal point position of the optical system, the image pickup element 2 may be moved by the focal point position controller 6, and the lens system may be fixed. Alternatively, the recovery processing device 7 may employ a pipeline processing method so that the result of the recovery processing is directly output to the D / A converter 8. Further, the added image recorded in the memory 5 is recorded on another recording medium such as a floppy disk, a magnetic tape, or an optical memory,
At the time of image reproduction, an image signal recorded from the recording medium may be read, and the read image signal may be subjected to recovery processing and displayed. That is, the image recording unit and the reproduction unit may be connected off-line via the recording medium.

According to the first embodiment having the above configuration, the following operation and effect can be obtained. That is, according to the present embodiment, since images having different focal positions are discretely input and added at the same time as the input, the processing can be performed by a device having a relatively simple configuration. The addition can be performed at a high speed.

Second Embodiment FIG. 2 is a diagram showing a configuration of a second embodiment of the present invention.
In this embodiment, the focus position is continuously changed over a predetermined distance range by the focus position controller 6. At the same time, it is configured to accumulate images formed on the light receiving section of the image sensor 2 (specifically, the photo sensor section of the solid-state image sensor, the light receiving surface of the image pickup tube, etc.). The imaging device 2
Are stored in the memory 5 after being converted into digital signals by the A / D converter 3. Next, after an appropriate recovery filtering process is performed by the recovery processing device 7, the data is recorded in the memory 5 again. The image that has been subjected to the recovery processing and recorded in the memory 5 is a D / A converter 8
Is converted to an analog signal by the display monitor 9
Will be displayed. The timing, data flow, and the like in the above operations are controlled by the controller 10.

According to the second embodiment having the above configuration, the following operation and effect can be obtained. That is, in the present embodiment, by utilizing the integration effect of the light energy of the image pickup device 2 itself, an image with a continuously changed focus is input and accumulated at the same time. Therefore, image input and addition are performed simultaneously by the image pickup device 2 itself, so that the configuration is extremely simplified and high-speed processing can be performed. In addition, since the focus position may be changed continuously over an appropriate distance range, control of the focus position is also simplified.

(Third Embodiment) FIG. 3 is a diagram showing a configuration of a third embodiment of the present invention.
In the present embodiment, the image formed by the lens 1 is
The image signal is converted into an image signal by the image sensor 2, and further converted to a digital signal by the A / D converter 3. The digital signal is then recorded by the selector 11 in one of the memories 5-1 to 5-m. The above operation is repeatedly performed while moving the lens 1 by the in-focus position controller 6. That is, the above operation is performed while the focus position of the optical system is discretely changed at appropriately set distance intervals and distance ranges. Thus, the input n sheets (n ≦
The image m) is recorded in each of the n memories 5-1 to 5-m. Next, the address correction device 12 performs magnification correction, positional deviation correction, and the like on the images recorded in the memories 5-1 to 5-m. The result is recorded again in the memories 5-1 to 5-m. The memories 5-1 to 5-1
The address-corrected image signal recorded in 5-m is selected by the selector 13. That is, among the n images, k images (k ≦ n) satisfying a predetermined setting condition are selected. The selected k images are added
It is added by 14. The image signal added by the adder 14 is recorded in the memory 15. The image signal recorded in the memory 15 is subjected to an appropriate restoration filtering process by the restoration processing device 7 and then recorded in the memory 15 again. The image after the recovery processing recorded in the memory 15 is D /
After being converted into an analog signal by the A converter 8, it is displayed on the display monitor 9. The controller 10 controls the above operation timing, data flow, setting of the selectors 11 and 13, and the like.

The original image signals recorded in the memories 5-1 to 5-m are recorded on another recording medium, and the image signals recorded from the recording medium are read at the time of image reproduction, and the read image signals are read. The image signal may be displayed after performing address correction, addition, recovery processing, and the like. That is, the image recording unit and the reproduction unit may be connected off-line via the recording medium.

According to the third embodiment having the above configuration, the following operation and effect can be obtained. That is, in this embodiment, a plurality of images having different focal positions are all recorded and stored, and a desired image is selected from these images and synthesized, so that the focus is arbitrarily set in the image. There is an advantage that a combined image can be reproduced. In addition, the address correction device 12 corrects a difference in magnification due to a difference in the degree of blur, or a positional shift due to blurring during image input or movement of a subject. . The address correction device 12 performs a positioning operation by performing, for example, local matching between images. Therefore, according to the present embodiment, it is possible to variously change the depth of focus and the setting of the focused portion in the image. This is also useful when it is necessary to change the magnification in addition processing. Furthermore, even when a device or a target moves during image input, a position shift occurs between the images, and blurring or the like occurs in a reproduced image, the adverse effect can be corrected. Therefore, the applicable range of the present device can be expanded.

(Fourth Embodiment) FIG. 4 is a diagram showing a configuration of a fourth embodiment of the present invention.
In this embodiment, the lens 1 in the image input optical system
Half mirrors 16-1 and 16-2 are provided at the rear of the camera, and, for example, three image pickup devices 2-1, 2-2 and 2-3 are arranged so that the distances from the lens 1 are different from each other. . The image signals from the respective image pickup devices 2-1, 2-2, and 2-3 correspond to the corresponding A-
The signals are converted into digital signals by the D converters 3-1, 3-2, 3-3, respectively. Except for the points described above, the configuration is the same as that of the first and third embodiments, so that illustration and description are omitted.

Note that the positions of the imaging elements 2-1, 2-2, 2-3 may be made variable so that the positions can be appropriately variably set in accordance with an object.

The fourth embodiment having the above configuration has the following operation and effects. In the present embodiment, by inputting images formed on a plurality of different surfaces, it becomes possible to simultaneously input a plurality of images having different focus positions. Therefore, it is not necessary to perform the mechanical operation of changing the focal point position, and
Due to its configuration, a device with a wider application range can be obtained.

(Fifth Embodiment) FIG. 5 is a diagram showing a configuration of a fifth embodiment of the present invention.
In the present embodiment, a Fresnel zone plate 17 set to focus on a plurality of positions is provided in the image input optical system, and a plurality of images focused on a plurality of object planes are simultaneously input to the image sensor 2. . Although an actual input optical system is configured using a plurality of lenses in addition to the Fresnel zone plate 17, only the Fresnel zone plate 17 is shown in the figure for simplification. An image signal from the image sensor 2 is recorded in a memory 5 converted into a digital signal by an A / D converter 3. The image recorded in the memory 5 is subjected to an appropriate restoration filtering process by the restoration processing device 7 and then recorded in the memory 5 again. The image that has been subjected to the recovery processing and recorded in the memory 5 is converted into an analog signal by the D / A converter 8 and displayed on the display monitor 9. The timing, data flow, and the like in the above operations are controlled by the controller 10.

It should be noted that the above-described in-focus position controller 6 is added to the configuration of FIG. 5 so that the in-focus positions are set so as to complement each other, and the same processing as in the first to third embodiments is performed on the input image. You may do so.

According to the fifth embodiment having the above configuration, the following operation and effect can be obtained. In the present embodiment, by using a Fresnel zone plate 17 designed to focus on a plurality of positions together with a lens (not shown), an in-focus image is formed on a plurality of object surfaces at once, This is input to the image sensor 2 at the same time, and the addition is substantially performed.

FIG. 6 is a front view of a Fresnel zone plate 17 designed to focus on a plurality of positions. In general, the Fresnel zone plate 17 is designed such that the circular openings are divided into annular zones (Fresnel annular zones) having the same area, and are opaque every other one. Such a Fresnel zone plate 17 has the same focal length as ± f ₀ = r ² / λ (where r ₁ is the radius of the center circle of the Fresnel zone plate 17 and λ is the wavelength of light). Works. Therefore, when the Fresnel zone plate 17 is combined with an imaging lens, the combined focal length becomes plural, and it is possible to focus on a plurality of object planes having different distances with the lens fixed. Theoretically, one Fresnel zone plate 17 has ± f ₀ in addition to ± f ₀
Although with _{f 0/3, f 0/} 5 ... becomes the focal length actually stronger intensity of light focused at the focal point of ± f ₀ is the small effect of other focus. In addition, the original focal length of the lens due to undiffracted light is preserved. However, instead of making every other zone opaque, if the phase of the transmitted light is delayed by π,
Although the position of ± f ₀ is the same, the intensity of light condensed there increases. Also, the original focal position of the lens disappears. Furthermore, if the phase difference is appropriately selected, the focus will be generated at three places of ± f ₀ and the original focal position of the lens. Although this Fresnel zone plate originally works strictly only at a specific wavelength, if the wavelength is selected near the central wavelength of visible light (λ = 550 nm), the effect of chromatic aberration is not so large in practical use. Note annular with a predetermined phase difference is created by vacuum deposition of transparent thin films, such as for example, MgF _2.

Thus, according to the present embodiment, it is possible to input and add images focused on a plurality of object surfaces only by using one optical element without performing a mechanical operation of changing the focal position. Therefore, the configuration can be extremely simplified, and the camera is more resistant to blurring caused by movement of the object or the like.

(Sixth Embodiment) FIG. 7 is a diagram showing a configuration of a sixth embodiment of the present invention.
In this embodiment, an optical element designed to intentionally produce chromatic aberration in the image input optical system, in which a lens 18 is provided, is provided. As the imaging device 2, a monochrome imaging tube or a solid-state imaging device having sensitivity to the entire width of the spectrum of visible light is used. With the above configuration, images focused on different positions depending on the wavelength of light are formed at once, and these are input and added by the image sensor 2. The following configuration is the same as in the fifth embodiment.

By providing a bandpass color filter in front of the image sensor 2 and inputting and recording images in a plurality of different wavelength regions, processing such as address correction and depth of focus operation can be performed in the same configuration as in the third embodiment. May be performed.

According to the sixth embodiment having the above configuration, the following operation and effect can be obtained. The present embodiment is an embodiment utilizing the property that the reflection spectrum of a general object is distributed over a wide range over almost the entire visible light range, and that the images at the respective wavelengths have a strong correlation with each other. That is, by using an optical element that artificially has chromatic aberration, an in-focus image is formed at a different position for each wavelength, and the addition is substantially performed by inputting the image to the image sensor 2. Like that. Therefore, it is possible to input and add images focused on a plurality of positions only by using an optical element having a large chromatic aberration without performing a mechanical operation of changing the focal point position. Thus, the configuration can be greatly simplified, and the configuration is more resistant to blurring caused by movement of the object or the like.

Here, a specific example of the recovery processing device 7 used in the first to sixth embodiments will be described. The restoration processing device 7 is a device that performs a process for applying an appropriate high-pass filter or band-pass filter to a spatial frequency to an image obtained by adding images having different focal positions as described above.

FIG. 8 is a block diagram showing one specific configuration example of the recovery processing device 7. As shown in FIG. Two-dimensional Fourier transform is executed by the FFT calculator 20 in the recovery processing device 7 on the image stored in the memory 5 and added for the images having different focal points, and the result is recorded in the memory 21. On the other hand, in the memory 22, the coefficients of the filter appropriately designed on the spatial frequency plane are recorded. The multiplication of the spatial frequency spectrum of the image recorded in the memory 21 and the filter coefficient recorded in the memory 22 is performed in the multiplier 23. The result is recorded in the memory 21 again. The spatial frequency image subjected to filtering recorded in the memory 21 is subjected to a two-dimensional inverse Fourier transform by the FFT calculator 20, and the result is recorded in the memory 5. According to this configuration, the filter can be designed into an arbitrary shape on the spatial frequency plane.

FIG. 9 is a block diagram showing another specific configuration example of the recovery processing device 7. The pixel component value designated by the address generator 30 provided inside the recovery processing device 7 is input to the multiplier 32 among the images added to the images having different focal positions stored in the memory 5. You. At the same time, the memory specified by the address generator 30
The coefficient recorded in 31 is input to the multiplier 32,
Multiplication between the two is performed. The calculation result in the multiplier 32 is added to the value recorded in the memory 34 in the adder 33, and the result is recorded in the memory 34 again. With the above configuration, the “convolution operation” is performed in a local area such as 3 × 3 pixels or 5 × 5 pixels in the image, and the result is recorded in the memory 5 again.

In this configuration example, instead of performing the filtering on the spatial frequency plane, the restoration processing is performed by performing a “convolution operation” with an appropriately designed mask on the image plane. Therefore, according to this configuration example, processing can be realized with a simple circuit configuration. In particular, when an effective filter can be designed with a small mask size, the amount of calculation is reduced, which is advantageous.

In addition, as a configuration example of performing mask processing on a screen, a pipeline-type processor can be used to execute processing at high speed.

Here, a method of designing a recovery filter will be described. First, a method of designing a recovery filter by simulation will be described. In general, the spatial frequency characteristic of an incoherent imaging optical system is represented by an autocorrelation of a pupil function.
It can be expressed by cal Transfer Function (abbreviated below as OTF). Assuming a circular aperture, the in-focus OTF
Can be expressed by the autocorrelation (expression) of the pupil function shown by the expression.

Here, (x, y) is a coordinate axis when the pupil plane is represented by rectangular coordinates, and (r, θ) are a radial component and an angular component when represented by cylindrical coordinates.

a ₀ represents the size of the pupil, and for example, the radius of the aperture of the lens may be set. In the case of a circular aperture, θ can be omitted because it has nothing to do with the angular direction. The out-of-focus OTF can then be represented by the autocorrelation of the generalized pupil function shown in the equation.

P (x, y) = P (r, θ) = P (r) exp [jkW (r; z)] where k = 2π / λ is a wave number, and W (r; z) is a wavefront aberration. It is expressed by the difference on the pupil plane between the wavefront W1 of light that is focused on a certain object plane and the wavefront W2 that is out of focus.
z is a coordinate on the optical axis, and is a quantity indicating how far from the focal position the focal point is, where z = 0. The wavefront aberration W (r; z) is approximately represented by W (r; z) = r ² · z (2 · f ² ) if approximating the paraxial region, and W (r; z) when the aperture of the lens is large. r; z) = r ² · z / [2 (f ² + r ² )]. Where f is the focal length of the lens and f≫z
Was assumed.

FIG. 10 is a diagram showing the above geometric relationship. In this manner, the OTF for a certain defocus amount z can be obtained.

FIGS. 11 (a) to 11 (d) show the OTF obtained as described above.
FIG. 6 is a diagram showing an operation procedure for obtaining a recovery filter based on the operation of the first embodiment. First, by changing z based on the setting conditions,
11 Obtain the OTF as shown in FIG. Then these OTFs
Are added to form a composite as shown in FIG. 11 (b).
Ask for OTF. Then, a restoration filter is designed so that the added combined OTF is restored to the OTF when the focus is achieved as shown in FIG. 11 (c). This restoration filter converts the synthesized OTF into H (u, v) = H (μ, φ) = H
(Μ) and the OTF in focus is H ₀ (μ), the recovery filter V (μ) is represented by V (μ) = H ₀ (μ) / H (μ). FIG. 11 (d) shows the above V (μ). Where (u, v) is the spatial frequency coordinate expressed in the orthogonal system,
(Μ, φ) is a spatial frequency coordinate expressed in a cylindrical system. Since the circular aperture does not depend on the angular direction, it is expressed only by the spatial frequency μ in the radial direction.

If the object is limited to some extent, the statistical properties of the image can be predicted, and the properties of the nozzle are known, the following Wiener filter can be used as the recovery filter. By using this Wiener filter, it is possible to reduce the influence of noise.

W (μ) = [H ₀ (μ) · | H (μ) | ² ] / [H (μ) ｛| H (μ) + | ² + Snn (μ) / Sgg (μ)｝] where Snn ( μ) is the power spectrum of the noise, Sgg
(Μ) is the power spectrum of the image. As a pseudo Wiener filter, a filter defined as shown in the following equation may be set, and the parameter P may be set appropriately.

W ′ (μ) = [H ₀ (μ) · | H (μ) | ² ] / [H (μ) ｛| H (μ) + | ² + P｝] Next, a method of experimentally obtaining a recovery filter. Will be described.
A test chart having a sufficiently flat surface is placed at a predetermined position, and the image is input and added while changing the focal point under the set conditions. Next, an image focused on the surface of the test chart is input. Then, an image obtained by inputting while changing the in-focus position and added together is subjected to an appropriate restoration filter, and is compared with the in-focus image. By this comparison, the recovery filter is adjusted so that the two images look equivalent, and the comparison is performed again. By repeating such an operation, a required recovery filter is obtained.
This method is practically effective.

The method of obtaining the recovery filter by the simulation and the experiment has been described above. Note that the restoration filter does not always have to restore the frequency characteristics at the time of focusing. For example, a high-frequency region may be further emphasized to make the image sharper, or conversely, the image may be completely restored. An image with a soft focus effect may be used instead.

Seventh Embodiment FIG. 12 is a diagram showing a configuration of a seventh embodiment of the present invention, and is a diagram showing an application example in which the present invention (particularly, a second embodiment) is applied to a reflection microscope. The light source 100 as shown in FIG.
The light emitted from the light source is guided by the epi-illumination device 101, and is finally irradiated on the surface of the material through the objective lens 102. A reflected light image from the sample is formed by the objective lens 102, and a TV camera 104 set on a lens barrel 103.
Is imaged. At this time, the focal plane for the material is continuously changed by the focus level driving device 105 within a set time. The image input during this time is
It is stored in the light receiving element of the TV camera 104. The material is X
The moving operation in the XY direction is performed by the -Y stage control device 106. The image stored within the set time is read by a reading device in the TV camera 104 and transferred to the camera driver 107 as an electric signal. The camera driver 107 also controls power supply to the TV camera 104 and the like. The image signal transferred to the camera driver 107 is then sent to the processor 108. The processor 108 includes an A / D converter, an image memory, a recovery processing device, a D / A converter, etc., performs an appropriate recovery process on the image signal, and gives the result to the TV monitor 109. Thus, the result is displayed by the TV monitor 109. The above operation is repeatedly performed for different portions of the material under the control of the XY stage control device 106 and displayed.
The microscope according to the present embodiment is comprehensively controlled by the controller 110, and conditions are set by an observer through the man-machine interface 111.

According to this embodiment, synthesis of an image with a large depth of focus can be realized with a relatively simple apparatus configuration while maintaining resolution and brightness with respect to the microscope. In the case of a microscope, a high-power objective lens must be used to observe a very fine structure. However, in general, a higher-power objective lens has a large NA and a shallow depth of focus. In such a case, the configuration according to the present embodiment is effective as a method of displaying an image with a large depth of focus. In particular, if the focus level driving device 105 and the processor 108 can be operated at high speed, real-time display is possible, which is more practical. Thus, the configuration of this embodiment includes the inspection of IC, LSI, minerals, paper, fiber,
It is useful in observing various objects in a wide range of industrial fields such as biological tissues.

In this embodiment, an example in which the present invention is applied to a reflection microscope is shown. However, the present invention can also be applied to other types of microscopes such as a transmission microscope and a fluorescence microscope. As for the image input and addition method, an adder is provided in the processor 108 and the first embodiment of the present invention
You may make it comprise similarly to an Example.

(Eighth Embodiment) FIG. 13 is a diagram showing a configuration of an eighth embodiment of the present invention, and is a diagram showing an application example in which the present invention is applied to a field sequential type electronic endoscope. As shown in FIG. 13, this device is roughly divided into an endoscope probe 200, an image input device 201, a color shift correcting device 202, a color information recording device 203, a focal depth increasing device 204, an image display device 205, and a controller 206. Become.

The endoscope probe 200 includes a monochrome solid-state imaging device 210 such as a CCD at the tip, and captures an image formed by the objective lens in the focal point position controller 211. The illuminating light in this case is Xe installed in the image input device 201.
Light from a white light source 212 such as a lamp is
After that, the light is guided into the endoscope probe 200 by a light guide 214 composed of an optical fiber or the like, and irradiated from the tip of the probe.

FIG. 14 is a plan view showing the structure of the rotary color filter 213. As shown in the figure, red (R), green (G), blue (B)
Are arranged intermittently in the rotation angle direction.

Returning to FIG. The rotating color filter 213 is driven to rotate by a motor 213M in the image input device 201, so that the irradiation light is sequentially changed in the order of R, G, and B. For this reason, the image pickup device 210 picks up an object illuminated with each color light as a monochrome image. The output image signal from the image sensor 210 is converted into a digital signal by an A / D converter 215 in the image input device 201, and then the selector 2
The data is allocated to areas for each of R, G, and B colors in the frame memory 217 by the 16 and stored in the corresponding areas. The above operation is controlled by the image input controller 218. Then, for each primary color image sequentially stored in each area of the frame memory 217, R and G or G and B are selected from the R, G, and B primary color images by another selector 219. The output from the selector 219 is guided into the color misregistration correction device 202,
The corresponding region between the primary color images is input to the corresponding region detecting device 220. Then, the shift amount of the R image or the B image with respect to the G image is locally detected by the detection device 220. The address generator 221 generates an address signal for correcting the R image and the B image based on the shift amount obtained by the corresponding area detection device 220 between the primary color images, and sends it to the frame memory 217. The frame memory 217 corrects the R image and the B image using the address signal and the buffer memory.

One color image thus obtained is guided to the color information recording device 203 via the selector 219.
The color image guided to the color information recording device 203 is added to the adder 2
22, the respective color components are added and the brightness Y = R + G + B
Is calculated. Further, the dividers 223-1 to 223-3 execute division for dividing the values of the component values R, G, and B by the brightness Y. As a result, the values of R / Y, G / Y, and B / Y are stored in the memories 224-1 to 224-3, respectively.

On the other hand, the R, G, B images stored in the frame memory 217 in the image input device 201 are guided into the depth of focus increasing device 204 and added by the adder 225. The result of the addition is stored in the frame memory 226. Frame memory 226
The restoration filtering process by the restoration processing device 227 is performed on the image stored in the storage device. The result is recorded in the frame memory 226 again. This frame memory 226
The image signal after the recovery process recorded in the color information device 203 is guided into the color information device 203.

The image signal guided into the color information device 203 and the color information stored in the memories 224-1 to 224-3 are multiplied by multipliers 227-1 to 228-3 for each color component. Done. The result is led into the image display device 205.

The multipliers 228-1 to 228 guided to the image display device 205.
The signal from 8-3 is converted into an analog signal by the D / A converters 229-1 to 229-3, and then displayed on the TV monitor 230. The above image processing and display unit are controlled by the controller 206.

As described above, the present embodiment generally has the property that there is a strong correlation between the three primary color images of R, G, and B, and the property that most of the blurring of the image depends on the brightness of the color information. When applying the present invention to a field sequential type electronic endoscope, an image at a different focal position is input for each primary color image, and a recovery process is performed by adding these images. .

Hereinafter, the operation of this embodiment will be described. First, the image input device 20
The primary color images of R, G, and B are sequentially input by 1. At this time, if the object or the endoscope probe 200 itself moves rapidly,
The relative position of each primary color image is different, so-called color shift occurs. Therefore, the color misregistration correction device 202 obtains the misregistration amount between the R image and the B image based on the G image by calculating matching between local partial images.
The R image and the B image are corrected based on the obtained color shift amount. When a color image at a certain in-focus position is input by such a method, the color information recording device 203 sets the values of R, G, and B to lightness information Y = R + G +
B normalizes each. Then, the obtained color information R / Y, G / Y, and B / Y are recorded. On the other hand, when a plurality of primary color images having different focal positions are input, these images are subjected to correction processing in the same manner as described above, and then added to perform recovery processing. By this operation, an image having a lightness Y 'with a large depth of focus is synthesized. And finally Y ′
By multiplying the image by the color information R / Y, G / Y, B / Y,
A color image with a large depth of focus is synthesized.

Therefore, according to the present embodiment, in the endoscope image,
There is an advantage that an image having a large depth of focus can be synthesized with a relatively simple configuration. In addition, since it is possible to use an objective lens having a small depth of focus and a large aperture, the power of the light source can be reduced. Moreover, when the objective lens as described above is used, the light accumulation time of the image sensor can be shortened, and the influence of color shift and the like can be reduced.

The following example can be considered as a modification of the present embodiment. The input R, G, B images are further added to the added image recorded in the frame memory 226. By doing so, an image having a deep depth of focus may be synthesized from images at three or more focal positions. In the above case, the positional shift between the image recorded in the frame memory 226 and the newly input G image may be corrected by using the color shift correcting device 202. Further, if image input is performed at a high speed and color misregistration is hardly a problem, the color misregistration correction device is used.
202 is not necessary. Further, when each primary color image is input, similarly to the first or second embodiment of the present invention, input and addition are performed, and an image having a deep depth of focus is synthesized for each primary color image. Good.

(Ninth Embodiment) FIG. 15 is a view showing a ninth embodiment of the present invention, in which the present invention is applied to an electronic endoscope using a color single-plate image sensor. As shown in the figure, a solid-state imaging device 302 whose image receiving surface is covered with a color mosaic filter 301 composed of R, G, and B filters is installed at the tip of the endoscope probe 300. The imaging element 302 captures an image formed by the objective lens in the focal position controller 303. In this case, the illumination light is emitted from the probe tip by guiding the light emitted from the white light source 305 in the apparatus main body into the endoscope probe 300 by the light guide 304. An output signal from the image sensor 302 is guided into the apparatus main body, and R, G, B
Separated into signals. In this embodiment, the color mosaic filter 301 is assumed to be composed of R, G, B filters, but may be composed of another color filter, for example, a complementary color filter of cyan, yellow or the like. In any case, in the color separation circuit 306, the color mosaic filter 301
Are separated. This isolated
The R, G, B signals are input to a matrix circuit 307, and Y, R−Y, B−
It is converted to a Y signal. The Y, RY and BY signals are A / D respectively.
The signals are converted into digital signals in converters 308-1 to 308-3. This digital signal is stored in the frame memory 310-1.
To 310-3, at which time the adders 309-1 to 30-3 are recorded.
9-3, the signal is added to the signal already recorded in the frame memories 310-1 to 310-3 and recorded. That is, cumulative addition recording is performed. With the above configuration, input and addition of images at different focal positions set by being controlled by the focal position controller 303 in the endoscope probe 300 are performed.

Next, Y recorded in the frame memory 310-1
The signal image is subjected to recovery processing by the recovery processing device 311 and is again recorded in the frame memory 310-1. Next, the signals recorded in the frame memories 310-1 to 310-3 are respectively converted into analog signals by the D / A converters 312-1 to 312-3, and then the signals are converted to NTSC encoders 313.
Is converted to an NTSC signal. And TV monitor 314
Displayed by The above processing is controlled by the controller 315.

As described above, in the present embodiment, the color mosaic filter 301
Is used to process the endoscope image input from the single-panel color imaging device 302 to extract the brightness component Y, and to increase the depth of focus by performing the processing of the present invention. In addition, this is an example of reproducing and displaying a color image. That is, the brightness components Y of the images at the different focal positions controlled and set by the focal position controller 303 in the endoscope probe 300 are cumulatively added and recorded by the adder 309-1 and the frame memory 310-1, and recovered. By performing the restoration filtering process by the processing device 311, the depth of focus is increased. On the other hand, the RY and BY signals, which are the color components, are added to the adder 309-2, the frame memory 310-2, the adder 309-3, and the frame memory 31.
Average color information between images having different focal positions is obtained by cumulative addition recording according to 0-3. These are combined and output as an NTSC signal. Thus, a color image with a large depth of focus is obtained. The blurred feeling due to the focus shift is almost dependent on the brightness component Y. Even if the color component is slightly blurred, it has almost no effect on human observation. Therefore, the intended purpose can be sufficiently achieved by performing the above processing. Thus, according to the present embodiment, the same effect as that of the eighth embodiment can be obtained for the electronic endoscope using the single-plate type color image sensor.

(Tenth Embodiment) FIG. 16 is a diagram showing the configuration of a tenth embodiment of the present invention, and the present invention is applied to an electronic endoscope using a single-plate type color image sensor as in the ninth embodiment. This is an example. As shown
The internal configuration of the endoscope probe 300 and the configuration of the white light source 305 are the same as those of the ninth embodiment shown in FIG. A signal from the image sensor 302 is separated into R, G, and B color signals by a color separation circuit 306, and then Y, R−
It is converted to a Y, BY signal. The Y, R-Y and BY signals are NTS
After being converted to NTSC signal by C encoder 313, A / D
It is converted into a digital signal by the converter 308. The digital signal is added to a signal previously recorded in the frame memory 318 by an adder 316, and then recorded again in the frame memory 318 via a selector 317. With the above configuration, the focal position controller 3 in the endoscope probe 300
Images at different focal positions set by 03 are input and added. The added image recorded in the frame memory 318 is provided to a comb filter 319. The comb filter 319 separates and extracts only the Y signal from the image signal, that is, the digital NTSC signal. The separated and extracted Y signal is subjected to a recovery processing device 320 for filtering a spatial frequency by a pipeline method.
, An appropriate recovery process is performed. The Y signal that has been subjected to the recovery processing is added to the original NTSC signal whose time difference has been adjusted by the delay circuit 322 in the adder 321 to become an NTSC signal having a deep focal depth. Then, the data is recorded in the frame memory 318 via the selector 317. The processed image recorded in the frame memory 318 is stored in the D / A converter 312.
After being converted into an analog signal by the TV monitor 314. The above operation is controlled by the controller 315.

As described above, in the present embodiment, the color image signal converted into the NTSC signal is input and added by using one frame memory, so that the same effect as that of the ninth embodiment can be obtained, and Size can be reduced.

(Eleventh Embodiment) FIG. 17 is a view showing a configuration of an eleventh embodiment of the present invention, and is an example in which the present invention is applied to an endoscope (fiberscope) using an optical fiber bundle. As shown in FIG. 17, the endoscope probe 400 has a focusing position controller including an objective lens and a driving device that continuously changes the focusing position.
Image guide 4 consisting of 401 and optical fiber bundle
02 is provided, and the image is optically transmitted as it is. In this case, the illumination light is emitted from the tip of the endoscope probe 400 by guiding the white light emitted from the light source device 404 into the endoscope probe 400 by the light guide 403. At the upper end of the endoscope probe 400, a camera 405 is installed. The camera 405 records an image transmitted by the image guide 402 on a silver halide film. In the above configuration, when the focus position is continuously changed by the focus position controller 401 while the shutter of the camera 405 is opened, an image is added to the silver halide film in the camera 405, Be recorded.

FIG. 18 is a diagram showing a configuration example of a means for optically performing a recovery process on an image recorded on the silver halide film. The multi-spectral light source 500 is a white light source having a uniform spectrum over the entire visible light range, or R, G, B
Are light sources that mix the three primary colors or emit light in different wavelength regions with time. Light from the multispectral light source 500 is condensed on the slit 502 by the collimator lens 501, and becomes an approximate point light source. The light that has passed through the slit 502 is converted into parallel light by a lens 503, and is applied to the color fism 504 on which an image is recorded by the camera 405 shown in FIG. The light that has passed through the color film 504 is collected by a lens 505.
This condensed light is appropriately filtered by a color spatial frequency filter 506 provided in the focal plane of the lens 505. The filtered light is converted into parallel light by the lens 507, and an image is reproduced. The reproduced image is recorded on a silver halide film by the camera 508. The reproduced image may be converted into an electric signal and stored in a memory.

As described above, in this embodiment, an image input in the same manner as in the second embodiment of the present invention is stored and recorded on a silver halide film. In addition, recovery filtering processing is performed on the reproduced image recorded on the color silver halide film by the optical system shown in FIG.

Next, the operation of the optical recovery processing shown in FIG. 18 will be described in detail. The endoscope image recorded on the color film 504 is optically Fourier-transformed by the lens 505, and an appropriate spatial frequency is filtered by the color spatial frequency filter 506. In general, an image obtained by a fiberscope is configured by using individual optical fibers as pixels. Therefore, it has a network structure 510 as shown in FIG. Therefore, the color spatial frequency filter 506 is designed so that the recovery filtering processing operation and the network structure 510 removal operation are simultaneously performed. The Fourier spectrum of the fiberscope image appearing on the focal plane of the lens 505 is as shown in FIG. That is, a zero-order spectrum 511 of the image itself appears at the center, and a higher-order spectrum 512 derived from the arrangement structure of the optical fibers appears at the periphery. Therefore, as shown in FIG. 21, a recovery process can be performed on the zero-order spectrum 511 in the central portion 521, and a filter 520 having an amplitude transmittance such that the higher-order spectrum 512 is cut off in the peripheral portion 522. This may be used as the filter 506. In this way, an image having no network structure 510 and a large depth of focus is reproduced. Since the maximum amplitude transmittance cannot exceed “1”, the recovery filter is designed to suppress the amplitude transmittance in a relatively low spatial frequency region. In FIG. 18, since a multispectral light source is used as the light source, the spectrum of the focal plane of the lens 505 appears at different positions depending on the wavelength. Therefore, the color spatial frequency filter 506 is formed of a color reversal film or the like, and is designed to have a spectral characteristic so that appropriate filtering can be performed for each wavelength spectrum.

The method of the optical recovery processing in the present embodiment is described in detail in Japanese Patent Application No. 61-227454.

Thus, according to the present embodiment, a fiberscope image having a large depth of focus can be reproduced with a relatively simple configuration. Therefore, as in the eighth embodiment, the power of the light source can be reduced. In this embodiment, the camera 405 is used.
Described the case of recording a fiberscope image on a silver halide film, but instead of the camera 405, T
The signal may be electronically processed and recorded using a V camera. That is, the image captured by the TV camera is converted by an A / D converter into a digital image whose sampling pitch is sufficiently small compared to the mesh structure of the fiberscope. You may do it.

(Twelfth Embodiment) FIG. 22 is a diagram showing a configuration of a twelfth embodiment of the present invention, and is a diagram showing an example in which the present invention is applied to an electronic camera.
This embodiment corresponds to the third embodiment in which the image input unit and the image processing display unit are connected off-line via a recording medium. As shown in FIG. 22, a plurality of images focused on different positions captured by the electronic camera 600 are input into the camera 600 as electric signals. All the input image signals are recorded on the recording medium 601 in the camera 600. The recording medium 601 includes a floppy disk, a magnetic tape, an optical memory, a semiconductor IC, a solid-state memory using a ferroelectric thin film, and the like. The image signal recorded on the recording medium 601 is read into the reading device 602. When the image signal recorded on the recording medium 601 is an analog signal, an A / D converter is provided in the reading device 602 to convert the image signal into a digital signal. The processor 603 includes the memories 5-1 to 5-m, the address corrector 12, the selector 13, the adder 14, the memory 15, the recovery processor 7, and the D / A converter 8 in the third embodiment shown in FIG. And a controller 10 and the like. Therefore, the same processing as that described in the third embodiment is executed by the processor 603 on the digital image signal transferred from the reading device 602. The processed image signal is transferred to the TV monitor 604 and displayed.
The setting of various conditions for processing in the processor 603 is as follows.
A man-machine interface 605 coupled to a controller in the processor 603 allows the observer to make settings.

According to the present embodiment, by recording a plurality of images having different focal points by the electronic camera 600, the same operation and effect as in the third embodiment can be obtained. In this embodiment, the third embodiment is applied to the electronic camera 600.
The first, second, fourth, fifth, and sixth embodiments can also be applied. That is, input and addition of in-focus images at different positions in the camera are performed, and this is recorded on a recording medium.
Only the recovery processing may be performed in the image processing unit connected off-line. If you do this,
The configuration of the device is further simplified.

In the case where the present invention is applied to a camera that records on a silver halide film, images having different focuses can be successively recorded by, for example, a continuous shooting method, and processing can be performed with the same configuration as in the twelfth embodiment. Further, as shown in the eleventh embodiment, it is possible to input and add images having different focuses on the silver halide film, and to perform optical or electrical recovery processing.

(Thirteenth Embodiment) FIG. 23 is a view showing the configuration of a thirteenth embodiment of the present invention, and is an application example in which the present invention is applied to a reflection microscope similarly to the seventh embodiment shown in FIG. It is. As shown in FIG.
Light emitted from the light source 700 is guided by the epi-illumination device 701, and is finally irradiated on the surface of the document via the objective lens 702. The reflected light image from the material is
TV camera 7 imaged by
Imaged by 04. At this time, the material is driven by a vibration actuator 706 installed on the stage 705 so as to vibrate at a specific frequency in the optical axis direction of the microscope. The vibration actuator 706 is driven by the vibration actuator driver 707 at an appropriate cycle and amplitude. An image signal captured by the TV camera 704 is transferred to a camera driver 708. The camera driver 708 also supplies power to the TV camera 704 and the like. The image signal transferred to the camera driver 708 is then sent to the processor 709. The processor 709 includes an A / D converter, an image memory, a restoration processing device, a D / A converter, etc., and performs an appropriate restoration process on the image signal by digital processing, or a band-pass filter or an analog circuit. It is configured to perform a recovery process with a high-pass filter. The processor 70
The image signal subjected to the restoration processing by 9 is transferred to the TV monitor 710 and displayed.

The processor 709 is provided with means for monitoring the power in a specific spatial frequency region of the input image signal using a bandpass filter. Thus, the amplitude of the vibration actuator 706 is automatically determined, and the vibration actuator 707 is operated according to the determined content.

According to the thirteenth embodiment having the above configuration, the following operation and effect can be obtained. In this embodiment, while the TV camera 704 is configured to vibrate the object at a specific amplitude one or more times while the image of one frame or one field is input, This has the function of integrating images having different focal planes. The vibration actuator
To initialize the amplitude of 706, the following method is used. First, input an image while changing the amplitude, and
At 9, the power of a specific spatial frequency region of the input image is monitored. The change of the power of the specific spatial frequency with respect to the amplitude is approximately as shown in FIG. In other words, when the amplitude becomes larger than the structure of the object in the depth direction, the image contains many blur components.
For this reason, the power decreases. Therefore, the threshold value TH is appropriately set in advance, and the amplitude value A when the power falls below the threshold value is used for actual processing. In this way, an image that is clearer and has an appropriate depth of focus with respect to the structure of the object can be obtained. The method of determining the integration range according to this method can be applied to the seventh embodiment.

Thus, according to the present embodiment, a sufficient effect can be obtained with a simple configuration. Further, if the recovery processing can be executed at a video rate, there is a great practical advantage that a processed image can be obtained in real time.

In the above embodiment, the object side is configured to vibrate, but the optical system side, that is, the relay lens or the image sensor provided in the objective lens 702 or the lens barrel 703 may be vibrated. Further, the present embodiment may be applied to an optical device other than a microscope, for example, an electronic camera or an endoscope, and may be configured to vibrate an optical system or an image sensor.

Fourteenth Embodiment FIG. 25 is a diagram showing a configuration of a fourteenth embodiment of the present invention, and is an application example in which the present invention is applied to a reflection microscope as in the thirteenth embodiment. In order to simplify the description, the reflection type microscope device is a light source 800, an epi-illumination device 801, and an objective lens.
Only 802 and lens barrel 803 are shown. A color TV camera 804 is installed on the lens barrel 803. This color TV camera 8
The dynamic range of the image sensor in 04 is, for example, 40 dB
Suppose that The epi-illumination device 801 is provided with a rotary optical shutter 805. The optical shutter 805 transmits and blocks the illumination light emitted from the light source 800 at a specific timing. In other words, this rotary optical shutter 805
26, as shown in FIG. 26, has regions 805A to 805C equally divided in the rotation direction of the disk 805D. And each area 805A ~
The 805C is provided with windows (hatched portions) a, b, and c, each having an area ratio of 10,000: 100: 1. Thus,
When the disk 805D is rotated by a motor 807 driven and controlled by a motor driver 806 so as to make one rotation per 1/10 second, a predetermined exposure is performed in 1/30 second (scan time of one frame by a TV camera). The amount will be given to the material. In this case, the illumination light is emitted via the half mirror 808 and the objective lens 802. And the reflected light image from the material is the color T controlled by the camera driver 809.
The image is captured by the V camera 804. Note that a Koehler illumination system is configured in the epi-illumination device 801 by a plurality of lenses and a diaphragm, but is not shown. The stage driving device 810 includes a vibration actuator as shown in the thirteenth embodiment, and vibrates the stage 811 at a predetermined amplitude at least once every 1/30 second. Thus, images having different focal planes are integrated on the light receiving surface of the color TV camera 804. In this way, three frames of color images are input with different exposure amounts. These color images are transferred to the processor 820 as R, G, and B primary color signals, respectively.

The image signal transferred to the processor 820 is converted into a digital signal by the A / D converter 821. The image signal of the first frame of the converted digital signal, that is, the image signal that is exposed through the window a of the rotary optical shutter 805 and is input simply passes through the adder 822 and is directly passed through the frame memory 823. Will be recorded. Next, the image signals of the second and third frames, that is, the windows b and c of the rotary optical shutter, are exposed, and the input image signals are added by the adder 822 to the frame memory 823.
Is added to the already recorded image signal. The result of the addition is recorded in the frame memory 823 again. In this way, R, G, and B have the logarithmic characteristics of the polygonal line approximation.
Primary color image data is stored in the frame memory 823. The image data logR and log stored in the frame memory 823
G and logB are converted to logY (Y = 0.3R + 0.5) by a logY conversion circuit 824.
9G + 0.11B) and recorded in another frame memory 825. The image signal recorded in the frame memory 825 is transferred to the recovery processing circuit 826. Recovery processing circuit 8
The image signal transferred to 26 is subjected to an appropriate filtering process with respect to a spatial frequency while maintaining a logarithmic characteristic. The image signal that has been subjected to the filtering process is sent to a dynamic range and gain adjustment circuit 827. Then, in this circuit 827, a gain adjustment value log b is added to the post-processed image signal logY ', and a dynamic range adjustment value a is multiplied to output a signal a log bY'. The image signal a log bY ′ is logarithmically compressed by a logarithmic converter 828, and log (a log bY ′)
Is output as This output value is input to a subtractor 829, where the subtraction is performed with the image signal logY recorded in the frame memory 825, and the signal log (a log bY '/
Y) is output. Output signal from the subtractor 829
log (a log bY '/ Y) is added to the logarithmically compressed three primary color signals logR, logG, logB recorded in the frame memory 823 in adders 830R, 830G, 830B. Each of the added outputs is input to a reciprocal converter (exponential converter) 831, where an antilogarithmic conversion is performed, and the signal a log bY ′ / Y
R, a log bY '/ YG, a log bY' / YB are calculated. These calculated values are converted into R, G, B video signals by the D / A converter 832 and displayed by the TV monitor 833.
The configuration of the processor 820 is based on the image processing apparatus for color logarithmic imaging described in Japanese Patent Application No. 62-234133. The operation of the apparatus having the above configuration is controlled by the controller 834, and the condition setting and the like are performed by the man-machine /
An interface 835 allows the observer to do so.

The fourteenth embodiment having the above configuration has the following operation and effect. The present embodiment adopts a color logarithmic imaging method, so that an image having a dynamic range exceeding the performance of the image pickup device of the TV camera 804 can be input without destroying the color balance, and the logarithmic filtering is more effective. It has the effect of performing recovery processing. Therefore, the operation of the color logarithmic imaging method will be described first. Here, the TV
The dynamic range of the image sensor of camera 804 is 40dB
It will be described as. If the subject has enough light and dark information that it can not be imaged with a dynamic range of 40 dB,
In this embodiment, the substantial dynamic range is expanded as follows. First, image with a sufficiently large exposure,
An image having information on dark parts is input. In this image, the bright parts are saturated. Next, an image is input with an exposure amount of 1/100 of the previously input image. Then, from the darker part of the saturated part in the input image,
The information for dB is obtained. Similarly, when an image is input with an exposure amount of 1/00, an image having information corresponding to a lighter area of 40 dB is input. By adding these, an image signal having a logarithmic characteristic of a polygonal line approximation is obtained, and can be handled as an image signal having a substantially wide dynamic range. Further, in order to keep the balance of the processed image, logarithmic compression is performed only on the luminance signal Y. In order not to affect the hue and saturation of the color image, the compression degree logY / Y of the luminance signal Y is set to R, G, B
And (logY / Y) R, (logY / Y) G, (logY
/ Y) B is obtained and output as three primary color signals. Further, in this embodiment, the gain and dynamic range at the time of display are adjusted, and (a logY / Y) R, (a logY / Y) G, (a log bY /
Y) Three primary color signals B are output. The control of the gain dynamic range may be performed manually, or may be automatically adjusted. The detailed operation of the above color logarithmic compression system is as described in the above-mentioned Japanese Patent Application No. 62-234133.

Next, a description will be given of an effect caused by performing a spatial frequency filtering process, that is, a so-called logarithmic filtering, on the luminance signal logY having a logarithmic characteristic. The distribution function of the reflected light is defined as follows.

And When both sides of the equation (a) are logarithmically transformed, Becomes Of these, illumination light or optical equipment without an illumination device corresponds to the distribution of natural light or external illumination. Have mainly low spatial frequency components. On the other hand, the reflectance distribution function of the object Have mainly low to mid-range spatial frequency components. So the
By creating a filter with the characteristics shown in Fig. 27, the unevenness of the illumination light of low spatial frequency components and the high spatial frequency region mainly dominated by noise components are cut, and the region showing the structure of the object is emphasized. can do. Therefore, if this logarithmic filtering is used, more effective recovery processing can be performed. Note that the spatial frequency values fl and fh in FIG. 27 may be set arbitrarily while the observer views the processing result by the man-machine interface 835, or when the type of image is limited and the component is known. May be set in advance.

Thus, according to the present embodiment, even if an imaging camera with a narrow dynamic range is used, it is possible to process an image having a substantially wide dynamic range. In general, in a camera with a narrow dynamic range, integration on the light-receiving surface increases the average intensity of the screen (that is, the 0th-order frequency intensity), and information in a high spatial frequency region cannot be recovered even if a recovery process is performed later. There is a problem that it is buried in noise to some extent. According to the present embodiment, the above-described problem can be solved, and it becomes possible to input image components in a high spatial frequency region while integrating them with a desired S / N. Further, by performing logarithmic filtering, more effective recovery processing is possible, and a sharp image can be displayed.

Although the present embodiment has been described as an application example of the reflection microscope, it is needless to say that the present invention can be similarly applied to other optical apparatuses having an illumination device such as an electronic endoscope. The optical shutter 805 may be an optical shutter using a liquid crystal plate or the like, in addition to the mechanical shutter as described in this embodiment.

(Fifteenth Embodiment) FIG. 28 is a diagram showing a configuration of a fifteenth embodiment of the present invention, in which the method of color logarithmic imaging and logarithmic filtering shown in the fourteenth embodiment is applied to an optical system without an illumination device. It is a figure showing the example of application applied to the device. An image of the object is formed by a lens 901 on a light receiving unit of a solid-state imaging device 902 such as a CCD. This image sensor 902 is controlled by a logarithmic imaging driver 903.For example, when the image sensor 902 is a CCD solid-state image sensor,
By changing the height of the overflow drain gate or horizontal transfer gate within the exposure time, the output signal is controlled so as to have a logarithmic characteristic. The details of the logarithmic compression by the solid-state imaging device are as described in the above-mentioned Japanese Patent Application No. 62-234133. An output signal from the image sensor 902 is converted into three primary color signals of logR, logG, and logB by a video processor 904, and further converted to a digital signal by an A / D converter 905, and then transferred to an image processing unit 910. Is done. The lens 901 is driven by a vibration actuator driver 906 to control the vibration actuator 907.
Accordingly, the image sensor 902 is driven so as to vibrate in the optical axis direction at a cycle sufficiently faster than the exposure time. Thus, images having different focal points are integrated on the light receiving surface of the image sensor 902 and input.

The image processing unit 910 includes a logY converter 911, a recovery processing circuit
912, a dynamic range and gain adjustment circuit 913, a logarithmic converter 914, a subtractor 915, adders 916R, 916G, 916B, and an antilogarithmic converter 917. In the image processing unit 910, as in the fourteenth embodiment shown in FIG.
Are subjected to logarithmic filtering, and the dynamic range and gain are appropriately adjusted for the three primary color signals (a log
bY '/ Y) R, (a log bY' / Y) G, and (a log bY '/ Y) B are output. The output signal from the image processing unit 910 is converted into a color analog signal by the D / A converter
Displayed by 21. The above operation is performed by the controller 930.
Is controlled by

The fifteenth embodiment having the above configuration has the following operation and effects. In the present embodiment, the solid-state imaging device 902 itself has a logarithmic compression input characteristic. Therefore, similarly to the fourteenth embodiment, the action of expanding the dynamic range without breaking the color balance and the recovery processing by logarithmic filtering are performed. And the advantage that the configuration is simplified.

Thus, according to the present embodiment, the same function and effect as those of the fourteenth embodiment can be exerted even on an optical device such as an electronic camera which does not normally have a lighting device. Also, the image sensor 902
Since the images having different in-focus positions are integrated themselves, the apparatus can be simplified.

Instead of having the image sensor itself have logarithmic input characteristics, several images were input with different exposure times, and by adding them, an image having logarithmic characteristics of a polygonal line approximation was synthesized. The processing may be performed with the same configuration as the processor 820 in the example.

(16th Embodiment) FIG. 29 is a diagram showing the configuration of a 16th embodiment of the present invention, and is an example in which the present invention is applied to a reflection microscope as in the 14th embodiment shown in FIG. . Light emitted from the light source 1001 is guided to the epi-illumination device 1002, and is irradiated on the surface of the document via the half mirror 1003 and the objective lens 1004. The reflected light image from the material is enlarged by the objective lens 1004 and the imaging lens 1006 in the lens barrel 1005, It is imaged. The SLM 1007 is driven and controlled by the SLM driver 1008 and functions as an incoherent = coherent converter. Note that the stage driving device 1009 is
Within the scan time set in consideration of the conversion response speed, the material mounting stage 1010 is driven within a set distance range in the optical axis direction of the microscope. The laser beam emitted by the laser 1011 is expanded in beam diameter by a laser beam expander 1012, and the optical path is changed by a half mirror 1013. Irradiated. On the reflection surface of the SLM1007, a microscope image is displayed as a refractive index distribution. Therefore, the reflection spatial pattern of the laser beam is propagated as a microscope image. The microscope image, that is, the reflected laser light is spatially Fourier-transformed by the lens 1014, and the filter 1015 disposed on the rear focal plane of the lens 1014 filters the spatial frequency. Thereafter, the reflected laser light is subjected to inverse Fourier transform by the lens 1016, and the image is input to the TV camera 1017. The filter 1015 is formed so that the amplitude transmittance increases from the center to the periphery, and is designed to be a high-pass filter for spatial frequencies. The above
The power supply and timing control of the TV camera 1017 are performed by the camera driver 1018. The image thus input is transferred to the processor 1020 via the camera driver 1018. The image signal input to the processor 1020 is converted into a digital signal by the A / D converter 1021 and recorded in the frame memory 1022. The digital image recorded in the frame memory 1022 is subjected to appropriate recovery processing by the circuit processing device 1023, and is recorded again in the frame memory 1022. The image signal thus subjected to the restoration processing is converted into an analog video signal by the D / A converter 1024 and displayed on the TV monitor 1025. The above operation is controlled by the controller 1026, and the condition setting is performed by the observer through the man-machine interface 1027 connected to the controller 1026.

Here, the SLM1007 will be described in some detail.
SLM1007 used here is an optical input type spatial modulator,
It is an optical functional element capable of writing and recording a two-dimensional optical signal, reading out with light, and the like. As the optical functional element, Bi ₁₂ SiO ₂₀ PROM using an optical Den properties and electro-optical effect of (BSO) (Pockls Read-out Opt
ical modulator) or LCLV combining photoconductive material and liquid crystal
(Liquid Crystal Light Value). Both have the effect of converting an input optical signal into an electric field distribution and controlling the refractive index distribution of the reflection surface. Therefore, the SLM1007 changes the polarization state of the laser light incident on the reflection surface side thereof according to the refractive index distribution of the reflection surface of the SLM1007, and allows the laser light to be read as a coherent light image by passing through the analyzer. It is configured. In this embodiment, an SLM having a particularly large dynamic range and a high response speed is used.

The sixteenth embodiment having the above structure has the following operation. In the present embodiment, high-pass filtering is performed optically as preprocessing when integrating images on the light receiving surface of the TV camera 1017. That is, in the present embodiment, the image input by the microscope is configured to propagate to the laser light via the SLM1007. Therefore, the spatial frequency filtering by the coherent optical system is performed, and the function of integrating images having different focal planes on the light receiving surface of the TV camera 1017 while suppressing the low spatial frequency is provided. By doing so, the limitation of the integration effect due to the limitation of the dynamic range of the TV camera 1017 can be solved. As a result, the integration input can be performed at a desired S / N also for the spatial frequency component.

Thus, according to the present embodiment, the dynamic range of the TV camera 1017 can be used effectively, and a clearer image can be displayed by the recovery processing in the processor.

(Seventeenth Embodiment) FIG. 30 is a diagram showing a configuration of a seventeenth embodiment of the present invention, and is an example in which the present invention is applied to a reflection microscope similarly to the sixteenth embodiment. In the configuration of this embodiment, the processor 1020 is omitted from the configuration of the 16th embodiment shown in FIG. 29, and the image signal from the TV camera 1017 is transmitted to the TV monitor 1025 via the camera driver 1018.
Is directly input to. Therefore the 29th
The same parts as those in the drawings are denoted by the same reference numerals, and detailed description will be omitted.

The sixteenth embodiment having the above configuration has the following operation and effect. The stage 1010 is driven by the stage driving device 1009 over the distance range in the optical axis direction of the microscope, and during the driving time, the microscope image is configured to be accumulated on the light receiving surface of the SLM 1007, so that the focal plane is different. A plurality of images will be integrated. When the above operation is completed,
The integrated image displayed on the reflection surface of the SLM 1007 is read by a laser beam, and a recovery process is performed by spatial frequency filtering using a coherent optical system. The result is captured by the TV camera 1017 and displayed on the TV monitor 1025. In other words, the present embodiment has an effect of performing a recovery process by optical filtering using the integration effect of images on the light receiving surface of the SLM1007.

Thus, according to the present embodiment, since the recovery processing can be performed in a complete real time, there is an advantage that the overall processing speed can be reduced.

Here, a method of processing and displaying a color image in the above-described sixteenth and seventeenth embodiments will be described. First, as shown in FIG.
A rotating color filter 1030 is provided in 1002. This rotary color filter 1030 is a device in which three primary color filters of R, G, and B are arranged in the rotation direction of the disk, similarly to the rotary optical filter 213 in the eighth embodiment shown in FIG. When the color filter 1030 is driven to rotate by the motor 1031, light emitted from the white light source 1001 is
, The color of the light can be temporally changed in the order of R, G, B. Each primary color illumination light obtained in this way is
When guided to 1005, each primary color illumination
Operations based on the configurations described in the seventeenth and seventeenth embodiments are sequentially performed.

FIG. 32 is a block diagram showing an example in which the processor 1020 in the sixteenth embodiment shown in FIG. 29 is configured for a color image. The output signal of each primary color illumination light input to the processor 1100 is converted into a digital signal by the A / D converter 1101, and then converted to a frame memory 1103 by the selector 1102.
R, 1103G, and 1103B are stored in a memory of a predetermined color. When all of the processing results by the three primary color illuminations are recorded in the frame memories 1103R, 1103G, and 1103B, they become (R,
G, B) → Y converter 1104 where the luminance signal Y =
(0.3R + 0.59G + 011B) is calculated. The above (R, G, B) →
The luminance signal Y output from the Y converter 1104 is supplied to the dividers 1105R, 1105G, and 1105B as signals for dividing the three primary color signals output by the frame memories 1103R, 1103G, and 1103B. Therefore, each of the dividers performs a division to calculate R / Y, G / Y, and B / Y. These calculation results are sent to multipliers 1106R, 1106G, 1106B.

On the other hand, the luminance signal Y is subjected to appropriate recovery processing by a recovery processing circuit 1107. The result Y 'is calculated by the multiplier 1106
R, 1106G, and 1106B, where the multiplication with the R / Y, G / Y, and B / Y signals is performed. Result of multiplication (R / Y)
Y ′, (G / Y) Y ′ and (B / Y) Y ′ are D / A converters 1
The signals are converted into analog video signals and output by 108R, 1108G, and 1108B. The operation in the above configuration is controlled by the controller 1109.

With such a configuration, the recovery processing is performed only on the luminance component of each of the sequentially input primary color signals so as not to lose the color balance. The results of the recovery processing are output simultaneously for the three primary colors, and can be displayed as a color image.

When applied to the seventeenth embodiment, the (R, G, B) → Y converter 1104 and the dividers 1105R, 1105 from the configuration shown in FIG.
G, 1105B, recovery processing circuit 1107, multipliers 1106R, 1106G, 1106B,
Remove the frame memories 1103R, 1103G, 1103B and D / A
A processor directly connected to the converters 1108R, 1108G, and 1108B is provided, and the processor is connected to the camera driver 1018 and the TV shown in FIG.
What is necessary is just to interpose between the monitor 1025. In this way, the three primary color image signals that have already been subjected to the recovery processing and are sequentially input are temporarily stored in a frame memory in the processor, and are simultaneously output to be displayed as a color image. be able to.

(Eighteenth Embodiment) FIG. 33 is a schematic diagram showing a configuration of an eighteenth embodiment of the present invention, and is a partial configuration diagram of optical high-pass filtering used when applying the present invention to a transmission microscope. . As shown in the figure, the transmitted illumination optical system is a Koehler illumination device.
Consists of 1200. Illumination light emitted from the light source 1201 is
The light is condensed by the collector lens 1202, passes through the field stop 1203 and the condenser stop 1204, and is irradiated on the document 1206 by the condenser lens 1205. At this time, the condenser aperture 1204 is made sufficiently small so that light substantially parallel to the document 1206 is irradiated. The light that has passed through the document 1206 is guided to the objective lens 1207, and is filtered by the filter 1208 installed on the rear focal plane of the objective lens 1207.
Filtering is performed on the spatial frequency of the 1206 image. The filter 1208 is configured such that the light absorption rate is large at the center and gradually decreases toward the periphery. That is, high-pass filtering is performed by suppressing low spatial frequency components. The image on which the high-pass filtering has been performed is formed on an image input system by an image forming lens 1209.

The configuration of the other parts conforms to the configuration of the seventh, thirteenth, fourteenth, and seventeenth embodiments arranged so as to be applied to a transmission microscope.

The eighteenth embodiment having the above configuration has the following operation and effects. This example was carried out using a transmission microscope.
By illuminating 06 in parallel, a coherent optical system is realized and has the effect of optically performing high-pass filtering of an image.

Thus, according to the present embodiment, as in the case of the sixteenth embodiment shown in FIG. 29, the dynamic range of the image input Can be used effectively. Therefore, an image having a large depth of focus can be displayed more clearly.

Here, a method of selecting the number of images to be added or the range to be integrated will be described.

FIGS. 34 (a) and 34 (b) are conceptual diagrams of the above selection method. Consider an object 1210 having a stepped surface structure as shown in the left part of FIG. 34a. When such an object 1210 is imaged by an optical system having a small depth of focus, the Fourier spectrum of an image focused somewhere on the object also has a relatively high spatial frequency component, but focuses on any part at all. A missing image has only low frequency components. This is shown in the right part of FIG. 34 (a). That is, it shows the Fourier spectrum F (u) of the image when the focal plane is set at the position indicated by the broken line with respect to the object 1210. In this Fourier spectrum, a value F (u) obtained by integrating the spectrum for a certain spatial frequency region (u1, u2)
Let's focus on (1, u2) (the area of the shaded area 1220). FIG. 34 (b) shows how the integral value F changes when the focal plane position is changed. From this figure, F (u1, u2)
If the region (z1, z2) in which the value of .gamma. As a result, a clearer long focal depth image can be synthesized. Therefore, as shown in the third and twelfth embodiments, when recording images with different focal planes, a band-pass filter is applied to the recorded image signal, and the resulting value is examined. By doing so, the relationship as shown in FIG. 34 (b) is obtained, and based on this, the image to be used for addition may be determined. In the case where the present invention is applied to the reflection type microscopes shown in the seventh, fourteenth, sixteenth, and seventeenth embodiments, the stage (or the objective lens) is pre-processed by a predetermined small interval. To ascend or descend. Then, a band-pass filter is applied to the input image signal at each focus level. By doing so, the relationship as shown in FIG. 34 (b) is obtained, and the integration range is determined. In particular,
When used for inspection of ICs, LSIs, and the like, similar patterns are repeated. Therefore, once the integration range is determined, optimal processing can be performed on any part of the object under the same conditions. It should be noted that a plurality of bandpass filters having different frequency bands may be used, and any one or all of them may be used to determine the range comprehensively.

(Nineteenth Embodiment) FIGS. 35 (a) and 35 (b) are views showing a nineteenth embodiment of the present invention, in which the present invention is applied to an electronic camera. The configuration of this embodiment is roughly divided into a camera body 1300, a recording medium 1400, a processor 1500, a man-machine interface 1600, and a TV monitor 1700. The camera body 1300 shown in FIG. 35 (a) includes a lens 1301, a shutter 1302, a focal point position controller 1303, an image sensor 1304, an A / D converter 1305, a memory 1306, and a matching circuit 130.
7, a memory 1308, a distance calculation circuit 1309, a controller 1310, and a writing device 13 for writing to the recording medium 1400
It consists of eleven. The camera body 1300 has the following operation. The camera 1300 performs two operations of pre-imaging and main imaging. First, in pre-imaging, the shutter 1302 hides a half surface of the lens 1301. Then, an image formed using the half surface on the non-hidden side is input to the image sensor 1304. The output signal of the image sensor 1304 at this time is converted into a digital signal by the A / D converter 1305,
It is recorded in the memory 1306. Next, the shutter 13
02 hides the half surface on the opposite side of the lens 1301 from the case described above. Thus, the image input as before is recorded in the memory 1306. In this way, the lens 13
The two matching images 01 are alternately used, and stereo matching is performed by a matching circuit 1307 on two images input with parallax. By doing so, the distance to the object is measured. In the above matching, for example, as shown in FIGS. 36 (a) and 36 (b), a plurality of detection points 1315 are provided in an image, and a local area 1316 set around each of the detection points 1315 is set. And 1317. No. 36
FIG. 36A shows the left image plane, and FIG. 36B shows the right image plane.

FIG. 37 shows an example of the configuration of the matching circuit 1307 in the memory 13.
It is a figure shown with 06 and 1308. The memory 1306 stores a left image memory 1306 for recording two images having left and right fields of view.
a and a right image memory 1306b. An address generator 1320 is provided in the matching circuit 1307.
This address generator 1320 is shown in FIGS. 36 (a) and 36 (b).
As shown in the figure, for the left image, a setting area 1316 centered at a certain detection point 1315, and for the right image, a certain detection point 1315
An address of a setting area 1317 (a setting area having the same size as the setting area of the left image) centering on a point shifted by several pixels in the parallax direction from 1315 (the number of shifted pixels is assumed to be S) is generated. This generated address is stored in the memory 13
Send to 06. From the memory 1306, the set area 131
The 6,1317 left and right image signals are sent to a subtractor 1321 in the matching circuit 1307. In the subtractor 1321, both image signals are subtracted for each pixel. The absolute value of this subtraction result is calculated by the absolute value calculator 1322. In this way,
The absolute value of the difference between the corresponding pixel signals of the left and right setting areas 1316 and 1317 is calculated. These values are then added to the accumulator 1323
And the result is sent to the decision unit 1324. Next, the address generator 1320 generates the same address as the first for the left image in the memory 1306. For the right image, the address of the setting area 1317 is shifted by one or two pixels in the parallax direction from the address of the right image that was initially set (that is, the center point of the new setting area is shifted S-1 or S1 from the detection point). -2 in the parallax direction). Then, the same calculation result is sent to the determiner 1324. Such an operation is performed for one detection point in a certain set range (for example, the center point of the setting area 1317 of the right image is shifted S to-
It repeats for the range moved to S). The judgment unit 1324
Compares the calculation result for each shift amount, detects the shift amount when the shift amount becomes the minimum, and sends it to the memory 1308. Such an operation is performed for each detection point. The above operation is controlled by the matching circuit controller 1325.

Next, the operation of the above-described matching circuit 1307 will be described. For stereo matching, it is necessary to find corresponding points between left and right images having parallax. The circuit performs the following operation for this purpose.

ρ (t) = Σ _R | fr (x + t, y) −fl (x, y) |... (c) where x: disparity direction, t: shift amount, fr (x, y): right image, fl ( x, y): left image, sigma _R: is an operator, representing the full adder in the setting area. For a certain detection point, t is set in a certain range (for example, t
= S to −S), and calculate the expression (c),
Find the minimum t. By doing so, it is possible to obtain the corresponding point of the right image with respect to the detection point of the left image. As a method of detecting the corresponding point, the matching circuit 1307 may be configured to perform a correlation operation as shown in the following equation instead of the equation (c).

φ (t) = Σ _R fr (x + t, y) · fl (x, y) (d) Also, the bias and the gain of the left and right amount images are corrected by the normalized cross-correlation as shown in the following equation, and the accuracy is improved. May be performed.

Here, r: average value in the setting area of the right image l: average value in the setting area of the left image σr: standard deviation in the setting area of the right image σr: standard deviation in the setting area of the left image Although the position of the setting area 1316 in the image is fixed and the corresponding point is searched in the right image, the relationship between the two may be reversed. Alternatively, the corresponding point may be detected by moving the position of the setting area in both images about the detection point as the center. Further, by applying the correlation theorem in Fourier transform, an FFT calculator may be provided in the matching circuit 1307 to perform a correlation calculation as shown in the following equation.

φ _F (u, v) = F ⁻¹ [F {fr (x, y)} · F {fl (x, y)} ^* ] (F) where F: operator of Fourier transform F ⁻¹ : inverse In this case, there is no need to calculate the correlation value while shifting the relative position t between the two images as in the case of calculating the equations (c) to (e). In this case, the cross-correlation image φF ( The corresponding point can be obtained by detecting the position of the peak of (u, v). Further, when the influence of noise is small, the corresponding point may be detected with high accuracy by the phase correlation method represented by the following equation.

On the other hand, the memory 1306 and the matching circuit 1307
38 together with the selector 1330, the image is separated and recorded in the direction perpendicular to the parallax direction, and the matching processing for each partial image is performed in parallel, thereby reducing the overall calculation time. May be.

Now, the amount of left and right displacement at each detection point recorded in the memory 1308 is sent to the distance calculation circuit 1309 as shown in FIG. 35 (a). The distance calculation circuit 1309 calculates the distance between the camera body 1300 and each detection point from the parallax caused by inputting an image using each half of the lens 1301 and the amount of displacement. The result is sent to the controller 1310. Note that after the pre-imaging, the focus position controller 1303 may be automatically controlled based on the distance data at the detection point at the center of the screen so that the viewfinder is focused on the center, and an autofocus operation may be performed.

Next, the main imaging in the camera 1300 will be described. The controller 1310 sets the range of movement of the focal point position on the object plane for performing the integration input based on the distance information sent from the distance calculation circuit 1309.
An example of how to set the range of the focal point will be described. From the distance data at each detection point, a standard deviation representing the degree of variation is obtained. Then, from the average value of the distances, ± kσ
(Where k is between 1 and 2) is determined as an integration range. However, when calculating the average value and standard deviation of distance,
Data calculated at infinity, such as the sky, is excluded. The integration range may be manually set by the photographer. In the actual imaging, the shutter
1302 are all released, and within the exposure time, the focal point position controller corresponding to the integration range set by the controller 1310
Driving 1303, the image sensor with different focus position images
Integrate on the light receiving surface of 1304. The output signal from the image sensor 1304 is converted into a digital signal by the A / D converter 1305 and recorded in the memory 1306. The integrated image recorded in the memory 1306 and the amount of left and right deviation at each detection point in the image recorded in the memory 1308 are recorded on the recording medium 1400 by the writing device 1311. Note that an adder may be provided in the camera main body 1300 to cumulatively add images having different focal points in the main imaging.

The recording medium 1400 includes a floppy disk, an optical memory, a card incorporating a semiconductor IC, a magnetic tape, a solid memory other than a magnetic material, a memory using an organic substance, and the like.
It has a role of transferring data obtained in the camera body 1300 to the processor 1500 offline.

Next, the configuration and operation of the processor 1500 shown in FIG. 35 (b) will be described. The data written in the recording medium 1400 is read by the reading device 1501. Then, the image data is recorded in the memory 1502, and the amount of left and right displacement at each detection point is recorded in the memory 1503. Then, the image data recorded in the image memory 1502 is sent to the recovery processing circuit 1504. Further, the data of the deviation amount recorded in the memory 1503 is sent to the coefficient generator 1505. The coefficient generator 1505 generates a coefficient value corresponding to the input shift amount, and sends it to the recovery circuit 1504. The image signal that has been subjected to the appropriate restoration processing by the restoration processing circuit 1504 is a D / A converter 1506
Is converted to an analog video signal and displayed on the TV monitor 1700. The operation of the processor 1500 is controlled by the controller 1507. Also, commands such as condition setting and image display can be performed by the observer through the man-machine interface 1600 connected to the controller 1507.

The operation of the recovery process in the processor 1500 will be described. In the recovery processing circuit 1504, spatial filtering is performed by the convolution of the local area, and the recovery processing is executed.

FIG. 39 is a conceptual diagram of the recovery process. In this figure, a “convolution operation” (convolution) is performed on a certain pixel in a neighboring 5 × 5 pixel area with a set mouse 1510. Then, the operation result is replaced with a target pixel. Is the pixel (however,
(Not performed for the surrounding two lines). By performing the restoration process in such a manner, different filtering can be performed according to the position in the image. Therefore, when the integrated input image in the camera body 1300 does not become a uniformly blurred image, by changing the coefficient of the mask 1510 according to the position of the image, an image in which any part is in focus can be obtained. Can be displayed. In order to perform such processing, in the pre-imaging, the distance between the camera body 1300 and the subject at some detection points 1315 in the image is measured, and which part in the integrated image is blurred and how much Is determined, and the coefficient of the mask 1510 is determined based on the information.

Thus, according to the present embodiment (the nineteenth embodiment), it is possible to determine the range of the focal point position to be integrated by the pre-imaging. Further, even when the manner of blur depends on the location in the image, appropriate recovery processing can be performed. Therefore, the range of application of the present invention to an electronic camera can be expanded. Note that the coefficient of the recovery processing mask 1510 in the processor 1500 of the present embodiment can be arbitrarily changed according to the location, so that this configuration is applied so that the observer can freely change the depth of focus as desired. May be.

Twentieth Embodiment FIGS. 40 (a) and 40 (b) are diagrams showing a configuration of a twentieth embodiment of the present invention. As in the nineteenth embodiment, the present invention is applied to an electronic camera. This is an application example. The configuration of this embodiment is roughly divided into a camera body 1800, a recording medium 1900, a processor 2000, a man-machine interface 2100, a TV monitor 2200, and the like. First, the camera body 1800 performs pre-imaging and main imaging operations in the same manner as described in the nineteenth embodiment. First, in pre-imaging, an image of an object input from the center of the lens 1801 is reflected by a mirror 1802 and input to an autofocus (hereinafter abbreviated as AF) sensor 1803.
The AF sensor 1803 is based on a phase difference AF method similar to the stereo matching shown in the nineteenth embodiment, and outputs data corresponding to the phase difference between two images formed with parallax by a distance measuring circuit 1804. Send to In the ranging circuit 1804, the AF
Based on the data sent from the sensor 1803, the camera body
Measure the distance from 1800 to the object. The measured distance data is recorded in the memory 1805. The above operation is performed similarly for several objects set by the photographer,
The results are recorded in the memory 1805. Next, the distance measurement results of a plurality of points recorded in the memory 1805 are sent to the controller 1806, and conditions for the main imaging are set.

Next, the operation in the main imaging will be described. The controller 1806 performs appropriate weighting on images at different focal positions based on the distance data from the camera body 1800 to the object measured in the pre-imaging, and controls to perform weighted addition input. That is, the focus position controller 1807 discretely changes the focus position over the range set by the pre-imaging. An image formed by the lens 1801 in each of the resulting states is input to the image sensor 1808. The image signal input to this image sensor 1808 is A / D
After being converted into a digital signal by the converter 1809, the digital signal is multiplied by a multiplier 1811 by a predetermined coefficient value recorded in the memory 1810. The multiplied value is added to an adder 1812.
Is added to the image signal recorded in the memory 1813, and the result is stored in the memory 1813 again. With the above configuration, a plurality of input images are weighted and added while discretely changing the in-focus position, and the result is recorded in the memory 1813. Note that the killer 1802 is optically designed from the beginning so that it is mechanically removed from the optical path when the main imaging is performed or does not disturb the imaging. Ranging data at several observation points recorded in the memory 1805, and the memory
The image data recorded in 1813 is
Is recorded on the recording medium 1900.

Next, the configuration and operation of the processor 2000 shown in FIG. 40 (b) will be described. The data recorded on the recording medium 1900 is read by the reading device 2001. Then, the image data is recorded in the memory 2002, and the distance measurement data is input to the controller 2003. The memory 2002
Is sent to the recovery processing device 2004. Then, based on the distance measurement data, the controller 2003
Appropriate recovery processing is performed according to the conditions set in. The image signal subjected to the restoration processing by the restoration processing device 2004 is converted into an analog video signal by the D / A converter 2005, and then displayed on the TV monitor 2200. Note that the recovery processing in the processor 2000 and various operations at the time of image output can be performed by the man-machine interface 2100 connected to the controller 2003.

The twentieth embodiment having the above configuration has the following operation and effects. First, in the pre-imaging by the camera 1800, the photographer aims at several subjects to be photographed at the center of the screen and presses the distance measurement button. By doing this,
The distance to each subject is recorded in the memory 1805 in the camera. The controller 1806, based on those distance measurement data,
The input conditions are set so that the image that has been added and added is almost the same as the subject selected by the photographer. The main imaging is performed in this state. Further, the processor 2000 has an operation of performing an appropriate recovery process based on the distance data measured by the pre-imaging.

A method of setting a weighted addition input condition from a plurality of distance measurement data will be described. Here, for the sake of simplicity, it is assumed that there are two subjects at different distances from the camera body 1800. According to the gist of the present invention, in order to display an image having a large depth of focus by applying a space-invariant recovery filter to an image input and added while changing the focal point position,
It is necessary that the blur of the input image is Space invariant. Therefore, first, the blur amount is shown by geometrical approximation due to defocus.

FIG. 41 is a diagram showing the geometric relationship. Camera body
It is assumed that the object point A1 is at a distance of l1 from the imaging surface 1820 of 1800, and the object point A2 is at a distance of l2. The distance between the object points A1 and A2 is d. The upper part of FIG. 41 shows the geometric relationship when the object point A1 is focused. Object point A2 at this time
The blurring amount δ2 is represented by the diameter of the projection of the light beam emitted from A2 on the imaging surface 1820. The lower part of FIG. 41 shows the geometric relationship when the object point A2 is focused. At this time, object point A1
The blur amount δ1 is represented by the diameter of the projection of the light beam emerging from the object point A1 on the imaging surface 1820. Let f be the focal length of the lens 1801,
Assuming that f≪l1, δ2, δ2 can be expressed by the following equation using approximation.

Here, F is the F number of the camera, and F = f / D (D: diameter of the exit pupil of the lens). Comparing Equations (h) and (i), it can be seen that the blur amount δ2 of the object point A2 when focusing on the object point A1, and the object point A1 when focusing on the object point A2.
The magnitude relationship with the blur amount δ1 is δ2> δ1. It can be seen that the difference becomes remarkable when l1 is small, that is, when the subject A1 in front is closer.
In such a case, the object point in the image added and input while changing the position of the focal plane from the object points A1 to A2 at regular intervals.
The blurring of the subjects A1 and A2 is not equal. That is, object point A
The subject in 2 is more blurred. Therefore, in the present embodiment, the weighted addition is performed to make the blur of the added input image close to Space invariant. A method of determining the weighting factor at that time will be described below.

As shown in FIG. 42, it is assumed that the in-focus position is set at m places at equal intervals in the range from point A1 to point A2. Then, each position is multiplied by a weight ωi (i = 1, 2,..., M) and added. A2 point and A at each focal point
The blur amounts δ2 (i) and δ1 (i) of the object at one point (i =
1,2, .., m) is represented by the following equation.

Therefore, the blur amount of the object at the object points A2 and A1 in the weighted image is Can be represented by These points In order to obtain ωi (i = 1, 2,... M) that is minimized under the constraint condition of, Lagrange's undetermined multiplier method is used. That is, The operation of the following expression is performed. From the equations (o) and (m), the following simultaneous system (p) holds for the unknown ωi (i = 1, 2,..., M). The expression (p) is displayed in a matrix for easy expression.

By solving this equation (p), the optimum ωi (i = 1,
2, .., m) is obtained.

When the distance (l1 and d) from the camera body 1800 to the subject is known by the above method, how to set the weighting coefficient is checked in advance. Then, these data are stored in the camera body 1800, and in an actual operation, an appropriate weighted addition input is performed based on the distance to each subject measured in the pre-imaging. In the case of a camera in which the imaging lens 1801 is replaceable, the data of the weighting factor is recorded in a read-only memory (ROM) installed in the taking lens 1801, and the camera body 1800 is used at the time of shooting.
Data may be transferred to the controller 1806 to perform the operation.

According to the present embodiment (twentieth embodiment), it is possible to control the depth of focus such that all the subjects arbitrarily selected by the photographer are in focus. Further, by performing the weighted addition, the blurring of the image becomes a state close to the Space invariant, so that the recovery processing can be easily performed.

(21st Embodiment) FIG. 43 is a diagram showing the configuration of the 21st embodiment of the present invention, and is an example in which the present invention is applied to an electronic camera as in the 19th and 20th embodiments. . This embodiment differs from the camera body 1800 of the twentieth embodiment shown in FIG.
And the multiplier 1811 are excluded. The camera body 2300 in this embodiment also performs two operations of pre-imaging and main imaging. Among them, the pre-imaging is the same as the content described in the twentieth embodiment. Therefore, the explanation is omitted. The operation of the main imaging in the present embodiment will be described. The controller 2306 controls the focal point position controller 2307 so that input and addition of images are performed according to the conditions determined in the pre-imaging.
The in-focus position controller 2307 discretely changes the in-focus position according to the above conditions. Lens 2301 in each state
Is input to the image sensor 2308.
The image signal input to the image sensor 2308 is converted by the A / D converter 2
After being converted into a digital signal by 309, adder 2312
, And is added to the image signal recorded in the memory 2313. The addition result is recorded in the memory 2313 again. According to the above configuration, a plurality of input images are cumulatively added while the focal point position is discretely changed. The result is recorded in the memory 2313. Then, the distance measurement data at some observation points recorded in the memory 2305 and the image data recorded in the memory 2313 are recorded on the recording medium 1900 by the writing device 2314. The added input image recorded on the recording medium 1900 is read by the processor in the same manner as in the twentieth embodiment, and an appropriate recovery process is performed on the basis of the distance measurement data also read from the recording medium 1900. Is displayed on the TV monitor. The configuration of the processor, the TV monitor, and the man-machine interface in this embodiment is the same as that of the twentieth embodiment.

In the twenty-first embodiment having the above structure, the following operation and effect can be obtained. As described in the twentieth embodiment, FIG.
The blur of the image added and input while changing the focal point position at equal intervals from point A1 to point A2 does not become Space invariant. However, in the present embodiment, since the following means is adopted, the method of blurring the added input image is determined by Space invarian
It will be close to t. Focus on object point A as shown in Fig. 44.
Set from 1 to A2. That is, the distance between the focal planes is sparse near the object point A, and denser as the object point A2 is approached. Let us focus on the subject at the object point A2 far from the camera body 2300 when the focal plane is set in this way. An image having a large amount of blur when the focus of the camera body 2300 is set to a closer point is smaller than when the focal plane is set at an equal interval. Therefore, the way of blurring of the subject at the object point A2 in the added image becomes smaller. On the other hand, with respect to the target at the object point A1, the blur of the added image is larger than when the focal plane is set at equal intervals. Therefore, when the entire added image is viewed, the blur can be made close to the Space invariant.
Note that the distance between the focal planes in this embodiment is the coefficient ωi (i = 1, 1) calculated by the method described in the twentieth embodiment.
2, .., m) can be easily set. Therefore,
Also in this embodiment, an appropriate addition input may be performed based on the data set in advance, based on the distance to each subject measured in the pre-imaging.

As described above, according to the present embodiment, with a simpler configuration,
The same effect as that of the 20th embodiment can be obtained.

(22nd Embodiment) FIG. 45 is a view showing the configuration of the 22nd embodiment of the present invention, in which the present invention is applied to an electronic camera as in the 19th, 20th and 21st embodiments. It is an example. Camera body 2400 of this embodiment
Is different from the configuration of the camera 2300 in the twentieth embodiment in that the adder 23 is used.
Equivalent to 12 In the present embodiment, the camera body 2400 also performs two operations of pre-imaging and main imaging. Among them, the pre-imaging is the same as the content described in the twentieth embodiment, and thus the description is omitted. In the main imaging, the controller 2406 controls the in-focus position controller 2407 so that the integrated input of the image is performed according to the condition determined by the pre-imaging. The image formed by the lens 2401 is integrated on the light receiving surface of the image sensor 2408 while changing the focal point at a predetermined speed within the exposure time. The input image is A / D converter 240
After being converted into a digital signal by 9, it is recorded in the memory 2413. The recorded image data is stored in the memory 2405
The data is recorded on the recording medium 1900 by the writing device 2414 together with the distance measurement data recorded on the recording medium 1900. The integrated input image recorded on the recording medium 1900 is read out by the same processor as that shown in the twentieth embodiment, and the same
Appropriate recovery processing is performed based on the distance measurement data read from 1900, and displayed on the TV monitor. The configuration of the processor, the TV monitor, and the man-machine interface for the observer to perform various operations in the present embodiment is described in the twentieth embodiment.
This is the same as the embodiment.

In the twenty-second embodiment having the above structure, the following operation and effect can be obtained. In the present embodiment, similarly to the twentieth and twenty-first embodiments, the function of controlling the focal plane so that the blur of the integrated image does not depend on the location from the distance measurement data of some subjects arbitrarily selected by the photographer. Having. In this embodiment, the position of the focal plane within the exposure time is controlled with a change characteristic as shown in FIG. 46, corresponding to the case where the geometric relationship as shown in FIG. 41 exists. That is, in the vicinity of the point A1 which is closer to the camera body 2400, the image is sparsely integrated by moving the focal point position quickly, and densely by moving slowly in the vicinity of the distant point A2. By doing in this way, the integrated image has a Space invariable manner in which the blur does not depend on the location, as in the operation shown in the twentieth embodiment.
It becomes something close to ant. The curve f (x) in FIG.
Is determined so that the coefficient ωi (i = 1, 2,..., M) obtained in the twentieth embodiment is a differential coefficient with respect to the coordinate axis x representing the in-focus position. That is, And Incidentally, as shown in FIG. 47, the integrated input may be performed while changing the focal point position in a step-like manner. In this case, the exposure time at each in-focus position is the coefficient ω obtained in the twentieth embodiment.
It is desirable to make the ratio i (i = 1, 2,..., m).

According to the present embodiment, the twentieth embodiment and the
The same effect as that of the 21st embodiment can be obtained.

Here, an example of an optical system for inputting images having different focal points used in the first, second, third, fifteenth, nineteenth, twentieth, twenty-first, and twenty-second embodiments and the like will be described.

FIG. 48 is a diagram showing a configuration example for a zoom lens. Lens system is focusing system 2501, variator system 250
It consists of a compensator system 2503 and a relay lens system 2504. Each lens system may be further composed of a plurality of lenses, but is omitted in the drawings. Of these, the relay lens system 2504 does not perform the mechanical movement associated with the zoom operation, and the image sent from the compensator system 2503 is
It has a role of forming an image on the light receiving surface of the image sensor 2506. Therefore, the relay lens system 2504 is connected to the relay lens driver 2505.
By driving in the optical axis direction, an image having a different focal plane can be input regardless of the focal length of the lens. The relay lens driver 2505 (corresponding to the focal point position controller in each of the above-described embodiments) includes an electromagnetic motor,
It is composed of an actuator using an ultrasonic motor, a piezo element or the like.

It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[Effects of the Invention] According to the present invention, an image with a large depth of focus can be reproduced without losing resolution or brightness, and the processing is simple. Therefore, there is a practical benefit such as easy implementation of the device, and furthermore, since the surface of the object is not affected by any property, the applicable range is wide, and in addition, the range of the focused portion is set. By variously selecting, it is possible to provide an image input / output device having advantages such as obtaining an image having an optimum depth of focus and S / N corresponding to the depth structure of the object, etc. An image input / output device having the advantage described above can be provided.

[Brief description of the drawings]

1 to 48 are diagrams showing an embodiment of the present invention. FIG. 1 is a diagram showing the configuration of the first embodiment, FIG. 2 is a diagram showing the configuration of the second embodiment, and FIG. FIG. 4 shows the configuration of the fourth embodiment, FIG. 4 shows the configuration of the fourth embodiment, FIG. 6 shows the configuration of the fifth embodiment, and FIG. 6 shows the Fresnel shown in FIG. FIG. 7 is a front view of the zone plate, FIG. 7 is a diagram showing the configuration of the sixth embodiment, and FIGS. 8 and 9 are block diagrams showing different specific examples of the recovery processing apparatus in the first to sixth embodiments. , FIG. 10 is a geometrical relation diagram relating to a method of designing a recovery filter in the recovery processing apparatus, and FIGS.
FIG. 12 (d) is a view showing an operation procedure relating to a method of designing a recovery filter, FIG. 12 is a view showing the configuration of the seventh embodiment, and FIG.
FIG. 14 is a diagram showing the structure of the eighth embodiment, FIG. 14 is a plan view showing the structure of the recovery color filter shown in FIG. 13, FIG. 15 is a diagram showing the structure of the ninth embodiment, and FIG. FIG. 17 shows the configuration of the tenth embodiment, FIG. 17 shows the configuration of the eleventh embodiment, and FIG.
A diagram showing the optical recovery processing of the film recorded image shown in the figure,
FIG. 19 is a diagram showing a network structure in a fiberscope image, FIG. 20 is a diagram showing a Fourier spectrum image of a fiberscope image appearing at the time of the optical recovery processing, and FIG.
The figure is a front view showing a filter used for the optical recovery processing, and FIG. 22 is a view showing the configuration of the twelfth embodiment. FIG. 23 is a diagram showing the configuration of the thirteenth embodiment, FIG. 24 is a diagram for explaining the operation of the thirteenth embodiment, and shows a power characteristic of amplitude versus a specific spatial frequency, and FIG. FIG. 26 is a diagram showing the configuration of the 14th embodiment, FIG. 26 is a front view of the rotary optical shutter of the 14th embodiment, FIG. 27 is a logarithmic filter characteristic diagram of the 14th embodiment,
28 is a diagram showing the configuration of the fifteenth embodiment, FIG. 29 is a diagram showing the configuration of the sixteenth embodiment, FIG. 30 is a diagram showing the configuration of the seventeenth embodiment,
FIG. 31 is a schematic diagram for explaining a method for processing and displaying a color image in the 16th embodiment and the 17th embodiment, and FIG.
FIG. 33 is a diagram showing a configuration of a color image processor in a diagram showing a partial modification of the 17th embodiment, FIG. 33 is a diagram showing a partial configuration of optical high-pass filtering in the 18th embodiment, FIG. 34 (a) and FIG. b) is a diagram for explaining the image selection method of the eighteenth embodiment, FIG. 35 (a) and FIG. 35 (b) are diagrams showing the configuration of the nineteenth embodiment, FIG. 36 (a) Fig. 36 (b)
Is a diagram showing the stereo matching means of the nineteenth embodiment, showing a left image plane and a right image plane, respectively.
FIG. 38 shows a configuration of the matching circuit of the nineteenth embodiment, FIG. 38 shows a modification of the configuration of the matching circuit and the memory of the nineteenth embodiment, and FIG. 39 explains the recovery processing means of the nineteenth embodiment. FIGS. 40 (a) and 40 (b) are views showing the structure of the twentieth embodiment, FIGS. 41 and 42 are operation explanatory diagrams of the twentieth embodiment, and FIGS. Diagram showing the configuration of the twenty-first embodiment,
44 is an explanatory view of the operation of the 21st embodiment, FIG. 45 is a view showing the configuration of the 22nd embodiment, FIGS. 46 and 47 are explanatory views of the operation of the 22nd embodiment, and FIG. FIG. 4 is a diagram illustrating a configuration example of an optical system in focus position control such as the first embodiment. 1 ... lens, 2 ... imaging device, 3 ... A / D converter, 4
... adder, 5 ... memory, 6 ... focal point position controller,
7 recovery processing device, 8 D / A converter, 9 display, 10 controller.

Continuation of the front page Judge Hideo Taguchi Judge Hidemi Kobayashi Judge Masahiko Koike Judges (56) References JP-A-59-34769 (JP, A) JP-A-63-8621 (JP, A) JP-A Sho 63- 298211 (JP, A)

Claims (12)

(57) [Claims] 1. An apparatus for inputting and outputting image information, comprising: focusing means for focusing on a plurality of different object surfaces; and image input means for inputting a plurality of images focused on different object surfaces by the focusing means. Range selection means for selecting a range of the focal plane position of a plurality of images input by the image input means; image addition means for adding the plurality of images selected by the range selection means; An image input / output device, comprising: image processing means for performing a restoration process by spatial frequency filtering on an image added by an addition means. 2. An apparatus for inputting and outputting image information, comprising: a focusing unit for focusing on a plurality of different object planes; and a plurality of images focused on different object planes by the focusing unit. An image input / output apparatus, comprising: an image addition input unit to be combined; and an image processing unit that performs a restoration process by spatial frequency filtering on an image added by the image addition input unit. 3. The image input means has a filter means for performing band-pass filtering, and the range selecting means determines a position of a focused object plane based on an output obtained by the filter means. 2. The image input / output device according to claim 1, wherein a range to be changed is selected. 4. An image according to claim 1, wherein said range selecting means has an object plane interval setting means for setting an interval between a plurality of object planes to be unequal within a selected range. I / O device. 5. The image input means comprises a plurality of image pickup means arranged respectively on a plurality of different image planes so as to simultaneously input a plurality of images focused on a plurality of different object planes. The image input / output device according to claim 1, wherein: 6. The image addition input means includes: an image pickup element for electrically converting a light image; a color separation means for separating an output of the image pickup element into primary colors or complementary colors; and an output of the color separation means for primary colors. Or a first logarithmic compression circuit for performing logarithmic compression for each complementary color, wherein the image processing means performs an antilogarithmic conversion circuit for performing an inverse logarithmic conversion on a signal from the first logarithmic compression circuit;
A matrix conversion circuit that performs a linear matrix conversion on the output of the antilogarithmic conversion circuit, a second logarithmic compression circuit that logarithmically compresses the output signal of the matrix conversion circuit, an output signal of the second logarithmic compression circuit, 3. The image input / output apparatus according to claim 2, further comprising: a color signal synthesizing unit that synthesizes a color signal using the output signal of the logarithmic compression circuit. 7. The image input / output apparatus according to claim 1, wherein said image adding means performs weighted addition. 8. The image addition means calculates a weighting coefficient for all of the plurality of input images based on characteristics of an optical imaging system and focusing conditions when a plurality of images are input by the image input means. The image input / output apparatus according to claim 7, further comprising: a weighting factor deriving unit configured to multiply all of the plurality of input images by the weighting factor and to add the images. 9. The image processing device according to claim 1, wherein said image processing means has coefficient storage means for preliminarily storing coefficients of a spatial frequency filter of an image in consideration of spatial frequency characteristics in said image input means. 3. The image input / output device according to 2. 10. The image input means for inputting a plurality of images for each of different wavelength regions, and the image adding means converts the plurality of images having different focuses input by the image input means into different images. The image processing means includes a luminance signal extracting means for extracting a luminance signal from the image added by the image adding means, and a luminance signal for performing a restoration process on the extracted luminance signal. Signal recovery means, and image synthesis means for synthesizing one image based on the luminance signal subjected to the recovery processing by the luminance signal recovery means and each image added for each wavelength region. The image input / output device according to claim 1. 11. The image input means has means for changing the position of the focused object plane and means for changing the wavelength range of the light, wherein the position of the focused object plane is determined by the focusing means. Means for inputting a plurality of images having different wavelength regions of light, and means for adding the images input by the image input means, and the image processing means comprises a plurality of images added by the image adding means. Image synthesizing means for multiplying each image obtained by dividing each image by the image subjected to the restoration processing to synthesize one image,
The image input / output device according to claim 1, comprising: 12. The focusing means according to claim 1, further comprising:
3. The image input / output apparatus according to claim 2, further comprising means for changing a moving speed of a focused object plane position.