JPH05183809A - Imaging-system for providing multiple simultaneous real-time image - Google Patents

Abstract

PURPOSE: To provide many simultaneous real time images from a single image sensor. CONSTITUTION: This imaging system 10 uses the image sensor 12 for providing the first set of electric signals in response to input received from a scene. The system 10 is provided with a processor 16 for processing the first set of the electric signals so as to provide the second set of the electric signals and a display device 18 for displaying the combination of the improved images of a part where the scene is not enlarged and the other part of the scene displayed by the second set of the electric signals.

Description

Detailed Description of the Invention

[0001]

[Industrial applications]

The present invention relates to imaging systems. More particularly, the present invention relates to a system for providing multiple simultaneous images.

It will be appreciated that the present invention is not so limited because it will be described herein with reference to specific embodiments for illustrative applications. One of ordinary skill in the art with ordinary skill in the art and use of the teachings provided herein will recognize additional modifications, applications and embodiments within the scope of the appended claims.

[Prior Art]

Many applications require providing wide-angle images and one or more high-resolution zoom images. This usually requires one sensor per image. It is advantageous to provide all of these images from a single sensor. For example, in forward-looking infrared (FLIR) systems, many situations arise in which simultaneous wide-angle and magnified viewing of a scene is desirable. Wide-angle images are used to find and specify targets, while magnified images of each of the targets can be used to identify each target and aim weapons if necessary. For example, a wide-angle view of the pilot is needed, and the shooter simultaneously requests a close-up view of a particular portion of the scene.

Previously,
For each desired image, the imaging system required a set of lenses and associated optics to achieve both a wide-angle field of view and one or more magnified images. As a feature of the boresighted optical field of view, these systems often include an optical field-of-view switch that can be activated whenever the operator desires an alternative image. Unfortunately, there are some limitations associated with simply providing optical devices in sequence to provide a continuous image of the target in real time.

To the extent that a system equipped with such a sequential optical device provides a sequential image of the scene,
The zoomed image is generally confined to the center of the wide-angle image. Due to this common aiming regulation,
It prevents another operator from zooming into a portion of the image near the edge of the wide-angle image.

[0006] Furthermore, a particularly important limitation with this sequential optics approach is that these systems do not provide images that are essentially simultaneous. That is,
Only one set of optics is selected at a time, so only one image is available at a time. This is an obvious limitation in situations where it is desirable to provide one image to more than one operator at a time, such as for military systems.

Another limitation of this sequential optical device approach is that the magnification of the lens is usually fixed and limited separately. Only one magnification is available at any one time.

Multiple simultaneous optics overcome some of the problems associated with sequential optics by providing multiple simultaneous images, but these systems are considerably more expensive and costly than imaging systems. Increase weight and weight. (This is particularly problematic for FLIR systems, which are generally expensive). These systems are generally mechanically complex and have precise manufacturing tolerances that require precise assembly and maintenance. In addition, multiple simultaneous optics generally increase the weight of the gimbal that holds the sensor, thereby limiting the performance of its parent system. Scanning efficiency is reduced if the optics share a common focal plane (since each image utilizes only a portion of the scan).

Finally, there are many systems in the field where the ability to display multiple simultaneous images is desirable. However, improvements to existing systems that provide this limited traditional functionality are too expensive to provide a practical choice.

Providing multiple images from a single optic solves many of the problems associated with multiple simultaneous optics. For example, on February 14, 1989, R. Zwi
US patent application Ser. No. 0731 filed by rn
1785 discloses techniques for achieving simultaneous images from a single optical device. However, this system requires one display for each image.

Accordingly, there is a need in the art for an inexpensive non-mechanical system or technique for providing multiple simultaneous real-time images with varying images on a single display.
Ideally, such a system would allow for easy (non-scalable) improvements to existing systems without compromising their performance.

[0012]

The need for the art is addressed by the system of the present invention which provides multiple simultaneous real-time images from a single image sensor. The system is adapted for use with an image sensor that provides a first set of electrical signals in response to electromagnetic energy received thereby from a scene. The system includes a processor for processing the first set of electrical signals to provide a second set of electrical signals, and an unzoom of the scene.
ed) portion and at least one display device for displaying an enhanced image of the other portion of the scene represented by the second set of electrical signals.

In this embodiment, the processor is a circuit for increasing the sampling rate of the image sensor, and the enhanced image of a portion of the scene is a high resolution magnified image of that portion of the scene. A deconvolver for processing a sampled signal at high speed such that
over).

[0014]

EXAMPLE FIG. 1 shows an imaging system 1 of the present invention.
The block diagram of the No. 0 example is shown. The system 10 includes a conventional television camera or video imager 12 which provides a first set of electrical output signals representative of the scene within its image. Any type of television camera or video image device can be used. In this embodiment, the television camera or video imager 12 is a non-integrated imager such as a forward looking infrared (FLIR) sensor.

In the embodiment of FIG. 1, the output of camera 12 is provided to image processor 16. The image processor 16 is, in this embodiment, a second set of electrical signals representing a combination of a wide-angle image from the camera 12 and one or more sharp, high-resolution magnified images of a portion of the wide-angle image. To generate. The output of this processor 16 is given to a display device 18 for simultaneously displaying a wide-angle image from the camera 12 and its magnified image to the operator. Those skilled in the art will appreciate the camera 1
2. It will be appreciated that the resolution of the image processor 16 and the display device 18 can be varied without departing from the scope of the invention. Moreover, the size of the wide-angle image and the magnification factor can be varied without departing from the scope of the invention.

The processor 16 includes a multiple image registration circuit 26, a frame memory 28, a deconvolver 30 and a point spread function circuit 36. The system 10 includes an operator controller 20 for selecting the area to be enlarged, and a format controller 22 for providing the necessary control signals to the processor 16 for controlling its zooming. Further included. In addition, system 10 includes a servo 24 that controls the aiming angle of camera 12. The operator controller 20 is implemented using a joystick or a light pen and the format controller 22 is implemented in a digital buffer. Those skilled in the art will recognize that other methods can be used to implement operator controller 20 and format controller 26 without departing from the scope of the present invention.

Describing the operation, the operator selects the image of the camera 12 by the operator control device 20. The controller 20 controls the aiming angle of the camera 12 by the servo 24. Those skilled in the art will recognize that the type of servo used may be changed without departing from the scope of the invention. The operator also controls the area of the wide-angle image to be magnified on the control device 2.
Select by 0. The number of magnified areas that can be selected is limited only by the number of magnified images displayed on the screen.
Controller 20 provides this information to format controller 22. This format controller 22 is
The processor 16 is controlled, and the processor 16 controls the servo 2
Control 4

In the format control device 22, where the processor 16 outputs a signal corresponding to a wide-angle image of the camera 12, and where the processor 16 outputs a signal corresponding to a magnified image of a part of the wide-angle image of the camera 12. The image processor 16 is controlled by providing a signal indicating If you need a magnified image,
The processor 16 raises the sampling rate of the camera 12 above the Nyquist criterion and provides a fast sampled signal to a selected portion of the display where a second set of electrical signals provides a sharp image of a portion of the wide-angle image of the camera 12. A deconvolving (deconvol) to correspond to a high resolution magnified image
ve)

On May 14, 1985, R. Zwirn
Other issued U.S. Pat. No. 4,517,599, the teachings of which are incorporated herein, describes a technique for processing the output of camera 12 to provide such an enhanced image. Is disclosed. Increased sampling of a portion of the scene results in multiple image registrations from a standard small video frame provided by camera 12, resulting in a multi-registered portion of the video frame consisting of multiple subpixels of the reduced area. Provided by circuit 26. Image motion or camera jitter between successive standard video frames determines the displacement of sub-pixels in the multiply registered portion of the video frame. In effect, this multiple image registration circuit 26 forms a mosaic of displaced subpixels of successive video frames produced by the camera 12.

Implementation of multiple image registration on existing system hardware is achieved using servo errors or camera platform stabilization gyro errors to compensate for correlation tracker image movement. The multiple image registration circuit 26 typically stores the correspondingly displaced pixel data in the frame
It includes means for displacing the address located in the memory 28. Those skilled in the art will recognize that other techniques can be used to increase the sampling rate of the camera, including the use of small detectors to achieve dense sampling in a single frame. ..

The multiplex image registration circuit 26 receives a control signal from the format control device 22, outputs a deliberate displacement to the servo 24, and outputs an address shift to the frame memory 28. This frame memory 28 stores the multiple registered portions of the video frame produced by the multiple image registration circuit 26, along with the remaining normal portion of the video frame produced by the camera 12. When the operator instructs the wide-angle image of the camera 12 to be displayed, the multiple image registration circuit 26 simply transfers this data to the frame memory 28 and stores it at its original address. If the operator indicates that a magnified image of a portion of the wide-angle image should be displayed, then the multiple image registration circuit 26 generates a mosaic of sub-pixels representing this magnified image, as outlined above. Operate.

Normal video is stored in frame memory 28 as a series of samples received from camera 12 via registration circuit 26. However, the multiple registered videos are stored in the frame memory 28 discontinuously. The magnification factor determines how to distribute the multiple registered pixels. For example, for double expansion, every other address and every other line is skipped when video data is being stored. A horizontal shift of half a pixel on the line of sight of camera 12 before the next frame will fill the alternate addresses skipped along these lines. Filling the lost row requires a half pixel shift vertically in the line of sight and one pixel shift in memory address to fill every other pixel in the skipped column. .. Finally, a half pixel shift in the horizontal direction before the next frame will fill in the missing pixels in the previously skipped row. Four video frames in total
Required to initially fill the memory 28.

In a similar manner, for every three times magnification, every second pixel and every second line are filled in each frame, and the line of sight of the camera 12 before each successive frame. It requires that there is a third pixel shift in. Therefore, triple the expansion is the video
Nine video frames are required to initially fill the memory 28. Format controller 22 provides coordinates to frame memory 28 and controls the storage of video data therein. The multiple image registration circuit 26 provides a signal to the servo 24 to control the required pixel offset and the direction of that offset prior to each subsequent video frame.

The data in the video memory 28, which represents the combined wide-angle image and the selected magnified image, is typically filtered before being displayed on the display device 18.
In the preferred embodiment, this filter varies according to the point spread function of the selected video system. Therefore, there is one filter for the portion of the display device that represents the wide-angle image and one filter for each portion of the display device that represents the particular magnification. Image processing device 1
The deconvolver 30 of 6 provides such a filter function. The point spread function circuit 36 provides the deconvolver 30 with the appropriate point spread function coefficient corresponding to the magnification factor for the portion of the display being filtered.

The format controller 22 provides a signal to the point spread function circuit 36 and the deconvolver 30 to determine which portion of the display device is being filtered, and
Indicate which is the appropriate filter to use. The output of deconvolver 30 is a second set of electrical output signals representing a wide-angle field of view containing the magnified image selected for display on display 18. Those skilled in the art will recognize that a variety of different filters can be used without departing from the scope of the invention.

The filtering technique of this embodiment is illustrated by FIGS. 2 (a) to 2 (c). Figure 2
(A) traverses the image plane of the camera 12 for an original object with a sharp edge (solid line) and a blurry image (dotted line) distorted from the original object by the point diffusion function of the camera 12. Shows the intensity of the image as a function of image position. FIG. 2B illustrates that high frequencies are reproduced by convolution processing. The result is the enhanced (non-blurred) image illustrated in FIG.

The deconvolver 30 effectively utilizes a convolution mask of the style shown in FIG. x
The axis represents position along a first direction transverse to the image plane of camera 12, and the y axis represents position along a second direction transverse to the image plane of camera 12 and perpendicular to the first direction. The z-axis represents intensity. As explained in the reference above, this mask has a negative number (the) of this point spread function.
Negative) and the positive impulse function of the center pixel of weight 2 are used to approximate the inverse function of the sensor degradation. The total weight of all masks is equivalent to 1.

The camera 12 by the image processing device 16
The enhancement of the original image from is due to the simultaneous display of both, instead of increased resolution and minimal blurring,
A wide-angle image including a selected portion of the display device 18 is capable of showing a magnified image. FIG. 4 (a) shows the original image as viewed by camera 12, with portions 40 and 50 of display device 18 selected for magnification. Figure 4
(B) shows the enhanced image seen on display 18, with selected portions 40 and 50 magnified and having a higher resolution than the original image from camera 12.

Although the present invention has been described herein with reference to embodiments and particular applications, the present invention is not so limited. Those having ordinary skill in the art and having access to the teachings of the present invention will recognize additional modifications and applications within its scope. For example, the invention is not limited to the type of camera or the type of display used with it. Moreover, the invention is not limited to a single display device. Any number of display devices and corresponding image processing devices can be used as integral to a particular application. Also, video enhancement is not limited to expansion. Any form of video enhancement is provided as needed for a particular application. Therefore, any and all such modifications, applications and embodiments are within the scope of the following claims.

[Brief description of drawings]

FIG. 1 is a block diagram of an illustrative embodiment of an imaging system of the present invention.

FIG. 2 (a) shows the intensity of the original object and its blurred image as a function of position across the image plane of the sensor used in the present invention. (B) is
The convolution of the present invention
FIG. 6 is a diagram depicting high frequencies reproduced by the process. (C) is a diagram showing a high quality (suppressed blurring) image obtained from the deconvolution processing adopted in the illustrative embodiment of the present invention.

FIG. 3 is a graphic perspective view of a surface corresponding to a convolution mask used by the image processing apparatus of the imaging system of the present invention.

4A and 4B are diagrams showing an original image and an image respectively generated on a display device by an image processing device used in the imaging system of the present invention.

Claims (17)

[Claims] 1. An imaging system for providing first and second simultaneous real-time images from a sensor for viewing on a display device or for subsequent processing, comprising:
An image sensor that provides a first set of electrical signals in response to inputs received from a scene, the first set of electrical signals being processed to represent the first and second simultaneous real-time images. A processing means for providing a second set of electrical signals, wherein the first simultaneous real-time image is an image of a portion of the scene represented by the first set of electrical signals. The two simultaneous real-time images are enhanced in the representation of the first simultaneous real-time image in response to the processing means being an enhanced image of a portion of the scene, and in response to the second set of electrical signals. A second simultaneous real-time image display means for displaying the imaging system. 2. The imaging system of claim 1, wherein the processing means controls the resolution enhancing means and the enhancing means for providing the enhanced image of at least one portion of the scene. Imaging system including format controller means for performing. 3. The imaging system of claim 2, wherein the resolution enhancing means includes sampling rate means for selectively increasing the sampling rate of the image sensor. 4. The imaging system of claim 3, wherein the sampling rate means is the image
From the sequential video frames generated by the sensor,
An imaging system including multiple image registration means for forming at least one multiple registered video portion of a frame of displaced subpixel mosaics. 5. The imaging system according to claim 4, wherein the enhancing means is controlled by frame memory means for selectively storing the multiple registered video frames and the format controller. An imaging system including the video frame generated by the image sensor. 6. The imaging system according to claim 5, wherein said enhancing means is controlled by said format controller to provide said multiple registered video frame and a set of deconvolved signals. An imaging system including deconvolver means for selectively deconvolving the video frames generated by the image sensor. 7. The imaging system of claim 6, wherein said deconvolver means is said multi-registered video with an inverse of a point spread function associated with degradation of said image sensor at said increased sampling rate. An imaging system that includes means for deconvolving the above portion of the frame. 8. The imaging system of claim 6, wherein the deconvolver means is inverse of the point spread function associated with the image sensor degradation and the video produced by the image sensor. An imaging system that includes a means for deconvolving a frame. 9. The imaging system of claim 1, wherein the display device is connected to the processing means to display a combination of the scene and the enhanced image of a portion of the scene. An imaging system including at least one display device made. 10. The imaging system of claim 9, wherein the enhanced image of the scene is a portion of a magnified, high resolution image of the scene. 11. The imaging system of claim 9, wherein the enhanced image of the scene is a portion of a magnified high resolution image of portions of the scene. 12. An imaging system for providing first and second simultaneous real-time images from an image sensor, for providing a first set of electrical signals in response to inputs received from a scene. Sensor means; processing a first set of electrical signals to provide a second set of electrical signals representative of said first and second simultaneous real-time images, said first simultaneous real-time images of said scene A processing means that is an image of a part,
The enhancing means including enhancing means for providing an enhanced image of the portion of the scene to provide the second simultaneous real-time image and format controller means for controlling the enhancing means. Is a processing means including means for effectively increasing the sampling rate of the sensor means, and an inverse function of a point spread function associated with the degradation of the sensor means, obtained from the increased sampling rate of the sensor means. Means for deconvolving the video being provided to provide a deconvolved set of signals; and the second simultaneous real time within the display of the first simultaneous real time image in response to the second electrical signal. An imaging system comprising display means for displaying an image. 13. The imaging system of claim 12, wherein the means for effectively increasing the sampling rate of the sensor means comprises: from successive video frames produced by the sensor means; An imaging system including multiple image registration means for forming a mosaic of displaced subpixels. 14. The imaging system of claim 13, wherein the means for effectively increasing the sampling rate of the sensor means comprises the mosaic of subpixels displaced from the multiple image registration means. To selectively store,
And an imaging system including frame memory means for selectively storing a portion of the scene represented by the first set of electrical signals. 15. The imaging system according to claim 12, wherein the display means is connected to the processing means to combine the portion of the scene with the enhanced image of the portion of the scene. An imaging system including at least one display device adapted to display. 16. The imaging system of claim 15, wherein the enhanced image is a high resolution magnified image of a portion of the scene. 17. A first and second image sensor.
A method for providing a simultaneous real-time image of the image for viewing on a display device; providing a first set of electrical signals in response to input received from a scene; Processing to provide a second set of electrical signals representative of the first and second simultaneous real-time images, the first simultaneous real-time image being an image of at least a portion of the scene. And wherein the second simultaneous real-time image is a resolution-enhanced image of at least one portion of the scene; and of the first simultaneous real-time image in response to the second set of electrical signals. Displaying the second simultaneous real-time image within the display.