JPH0358234B2 - - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of the Invention] The present invention relates to an image data reduction method and an image data reduction apparatus used in carrying out the method, and in particular to an image data reduction apparatus that can operate automatically and/or semi-automatically. It concerns a foveated electronic camera.

[Background of the invention]

Complex automated control equipment, such as surveillance television cameras, robotic automation, and target tracking equipment, often requires signal processing of visual information image samples. The total number of image pixels that are processed depends on both the field of view dimensions of the image and the spatial resolution of the image. Obtaining a high degree of spatial resolution over a large field of view requires a very large number of pixels. However, processing such a large number of image pixels is impractical.

One solution to this problem in the human eye is to provide relatively high spatial resolution in one region of the imager's visual field (the fovea located in the center of the eye's retina),
Other areas of the imager's field of vision (the periphery of the eye's retina) provide a relatively low resolution, so that the spatial portion of the image that was initially in the imager's low-resolution area is now in the imager's high-resolution area. Adjust and move the imager so that the Therefore,
Humans move their eyes and head so that the image of the object initially observed in the low-resolution area near the edge of their visual field is viewed in the high-resolution area of the fovea.

[Purpose of the invention]

The purpose of this invention was to maintain the ability to view objects with high spatial resolution while still processing objects imaged at arbitrary positions within a relatively wide field of view, most of which had low resolution. The goal is to significantly reduce the number of image pixels. However, the image reduction method implemented by the present invention is quite different from that used by the human eye.

[Summary of the invention]

In practicing the invention, the spatial frequency spectrum of an image represented by an input video signal is analyzed to generate a plurality of image spatial frequency spectra representing groups of sequentially arranged consecutive subspectral bands. Take out each separate output video signal. The image represented by the input video signal is a relatively high resolution, relatively wide field of view image that is comprised of a first predetermined number of pixels. However, the first band of the first ordered group of each separate output video signal represents a relatively high resolution of the image represented by the input video signal. Furthermore, only this first band consists of a first predetermined number of pixels. Each other band in the group of output video signals exhibits lower resolution and has fewer pixels than the band in the group immediately before it. Thus, the last group of bands consists of a second predetermined number of pixels that is the smallest number of pixels of all bands in the group. Such analyzes are known from the prior art.

According to the invention, in conjunction with the analysis as described above, the dimensions of the field of view represented by at least one band of the group are determined by spatially measuring the pixels of that one band through a spatial window that is preferably movable. It is reduced by passing through the ordered subset. This subset consists of no more than a second predetermined number of pixels.

[Explanation of Examples]

Hereinafter, the present invention will be explained in detail with reference to the drawings.

FIG. 1 shows an apparatus that includes as essential elements an automatic electronic camera 100 (foveated) and a computer 102. The apparatus of FIG. 1 may also optionally include an operator stage 104. The camera 100 has high resolution as an essential element.
It includes wide field imager means 106 and image data reduction means 108. Imager means 106 and data reduction means 108 may be integrated into a single housing of camera 100 as shown in FIG. 1, or they may be separate modules of camera 100.

The imager means 106 comprises a black and white or color television camera for viewing objects within a relatively large viewing spatial area, and from which a video signal is taken as input to the data reduction means 108. This video signal represents, all in real time, each pixel of each successive image frame of relatively high resolution captured by imager means 106. For example, each binary image frame from imager means 106 consists of 512 x 512 = 262,144 pixels. 30 consecutive image frames per second
Occurs at a frame rate. In this case, a continuous stream of pixels is provided by the video signal output of the imager means 106 to the data reduction means 108 at a rate of about 8 million pixels per second. However, in some cases, robotic automation or surveillance camera equipment may
providing a resolution of 512 x 512 pixels or more and/or a frame rate of 30 frames per second or more (thus increasing the pixel rate of the video signal supplied to the data reduction means 108 to 8 million or more per second). It is preferable.

Analysis of images by a computer typically requires that the image pixels be in digital rather than analog form. In order to provide a sufficiently high resolution gray scale,
Typically each image pixel level is digitized with 8 bits per pixel. Therefore, in the absence of image data reduction measures, in a real-time situation, a computer can
Thousands or more bits need to be processed in real time. There are very few image analysis computers that can operate at such a rate.
Computers that do this are extremely expensive.

The data reduction means to which this invention is primarily directed significantly reduce the amount of data that must be processed by computer 102 without sacrificing either the high resolution or wide field capabilities of imager 106. I can do it.

The reduced image data output from the data reduction means 108, which constitutes the output from the recessed automatic electronic camera 100, is provided as input to the computer 102. Computer 102 analyzes the reduced image data provided thereto according to its program. The program will, of course, depend on the particular purpose of the apparatus shown in FIG. For example, in the case of a monitoring device, computer 1
02 is programmed to identify significant changes in the scene viewed by the imager means 106, such as the presence of moving objects, objects with one or more particular formations, etc. Computer 102 may also include output to some utilization means (not shown), such as an alarm in the case of a monitoring device. Another example of the device shown in FIG. 1 is a robotic automation device. In this case, the computer 102 is programmed to provide the desired "eye-hand" interaction between the mechanical hand application and the imager 106. More particularly, the computer 102 performs the above operations according to both the information contained in the reduced data supplied to the computer 102 from the imager means 106 and the return signal received from the mechanical manual utilization means. A command signal is provided as an output to the mechanical hand.

The imager 10 of the camera 100 depends on the usage condition.
6 may be stationary or movable. For example, in the case of robotic automation, it is generally desirable to provide a transportation means 110 for the imager. The output from the computer 102 controls the movement means 110 in accordance with object information contained in the reduced image data provided to the computer with respect to the spatial region within the field of view of the imager means 106. In this case, the moving means 110 returns a return signal to the computer 102 to display the actual position of the imager means 106.

As mentioned above, the camera 100 and the computer 1
In combination with 02, it is possible to provide a fully automated device that does not require a human operator. However, if desired, the apparatus of FIG. 1 may be provided with an optional operator stage 104. As shown in FIG. 1, the stage 104 consists of a display section 112 and a manual control section 114. The display unit 112 displays image information retrieved by the computer 102 to the operator, and the manual control unit 114 transmits manual command signals to the computer 102.
02. By way of example, the purpose of these manual command signals is to select image information to be displayed on display 112 and/or to select image information to be displayed on display 112 and/or to
02 to data reduction means 108, movement means 11
0, or to manually control each output sent to any or all of the utilization means (not shown).

Referring to FIG. 2, there is shown a first embodiment of a data reduction means incorporating the principles of the present invention. A video signal input from the imager means 106 to the data reduction means 108, which may be a sampled signal from a solid state imager such as a CCD imager, or a continuous signal from a television tube imager. ) is applied to an analog-to-digital (A/D) converter 200, which converts the level of each successively occurring pixel of the video signal into a multi-bit (e.g., 8-bit) digital number. Each successive binary image frame represented by the video signal consists of horizontal P _X pixels and vertical P _Y pixels. Imager means 1
Since 06 is a high resolution imager, the values of each of P _X and P _Y are relatively large (eg, 512).
The video signal itself is a temporal signal derived by scanning during each frame period, and the binary spatial image is viewed by the imager of the imaging means 106. The digital output from A/D converter 200 is provided as an input to spatial frequency spectrum analyzer 202. Another embodiment of the spectrum analyzer 202 is described below in FIG.
Shown in Figure a.

The spatial frequency spectrum analyzer 202 is
In response to a digitized video signal representing each successive imager frame provided as input thereto, an ordered group of N+1 (N is a plurality of integers) separate video output signals L ₀ . L _N-1 ,
Take out G _N. Each video output signal L ₀ ...L _N-1 , G _N
constitute successive subspectral bands of the spatial frequency spectrum of the image specified by the pixels of each successive image frame of the digitized input video signal to analyzer 202. Each video output signal L ₀ ...L _N-1 identifies the bandpass band of the spatial frequency spectrum of the image, L ₀ identifies the highest spatial frequency bandpass band of the image spectrum, and L _N-1 identifies the bandpass band of the spatial frequency spectrum of the image. Identify the lowest spatial frequency bandpass band in the spectrum. G _N identifies the low-pass residual band. This lowpass residual band contains all spatial frequencies of the image's spatial frequency spectrum that are below the spatial frequencies of the L _N-1 bandpass band. _Each of the bandpass bands L ₀ . The highest spatial frequency is ₀ , the original L ₀ bandpass band has a center frequency of _30/4 , the original L ₁ bandpass band has a center frequency of 30/8, and the original L ₂ bandpass band has a center frequency of 30/8. The bandpass band has a center frequency of 3 ₀ /16, and so on). Thus, the first band L ₀ of the group of output video signals has a relatively high resolution, comparable to the input video signal to spectrum analyzer 202 . Furthermore, this first band L ₀ of the group consists of a similar number of pixels per frame (P _X ·P _Y ) as the input video signal to the analyzer 200. However, each of the other bands in the group exhibits lower spatial resolution and consists of fewer pixels than the immediately previous band in the group. Therefore, the final band G _N (residual band) of the group is the second predetermined number of pixels ( _{P '} Consisting of

In FIG. 2, each band pass band of L ₀ ...L _N-1 is _{P '}
04-(N-1) as input. Also, each gate 204-0...204-
(N-1) has individual window center control signals provided as control inputs from computer 102 as shown in FIG. Each gate 204-0...
...204-(N-1) of each frame.
The localized binary space portion consisting of P′ _X and P′ _Y is passed through _the gate as each output L′ ₀ . Each gate therefore acts as a spatial window for this binary spatial portion through which it passes. Each gate 2
04-0...204-(N-1) determines the relative position of this localized spatial portion for each frame. In Fig. 2, each output from gate 204-0...204-(N-1)
L _{' 0 .}

Image data reduction means 108 for camera 100 shown in FIG. 2 requires relatively few components to provide image data for computer 100 reduced in accordance with the principles of the present invention. . In this case, the computer 102
suitable memory means for at least temporarily storing the reduced image data provided by the image data reduction means 108 and this stored reduced image data control signal returned to the data reduction means 108. and selection means for drawing out the . However, in some cases it may be desirable to provide the memory means and selection means as part of the data reduction means 108 of the camera 100 rather than in the computer 102.
FIG. 2a shows this alternative embodiment of the image data reduction means 108.

As shown in FIG. 2a, each output L' ₀ . . . L' _N-1 , G' _N from the embodiment of FIG. 2 is not sent from camera 100 to computer 102. Alternatively, an alternative embodiment of the data reduction means 108 shown in FIG. 2a includes a group of memories 206-0... each associated with one output L' ₀ . . . L' _N-1 , G _N . ...206-N is included. During each successive frame period, L′ ₀ …
...L _{' N-1 , each P '}
-0... written to one of 206-N. After a time delay provided by each of the memories 206-0...206-N, each of these memories is read;
The output signals therefrom are provided as separate inputs to selection switch 208. In response to a switch control signal provided by computer 102, selection switch 208 selects memories 206-0, .
6-P′ _X・P′ _Y read from any one of the group N
Data reduction means 1 for storing pixels in camera 100
It is supplied to the computer 102 as an output from 08.

The spatial frequency analyzer 202 of FIG. 2 simply consists of a plurality of band-pass filters and a low-pass filter, each band-pass filter producing a respective band-pass signal L ₀ ...L _N-1 as an output, and a low-pass filter. The pass filter generates a residual signal G _N. If there is,
One or more bandpass filters may be used instead of a lowpass filter. However, it is preferable that the configuration is shown in either FIG. 3 or the alternative FIG. 3a. Examples of the spatial frequency spectrum analyzer shown in FIGS. 3 and 3a were described in 1984.
Patent Application No. 134032 (1982) filed under the name "Signal Processing Device" on June 27,
No. 37811) is shown in detail in the specification.

More specifically, the embodiment shown in FIG.
The hierarchical pyramid signal processing algorithm (referred to as the Burt Pyramid) published by Dr. Peter J. Burt can be executed in real time. FIG. 3a is another type of real-time hierarchical pyramid signal processor known as an FSD (filter subtract decimate) pyramid.

As shown in FIG.
The pipeline consists of: Each stage has a digital clock signal CL ₁ ,
CL ₂ ...operates at a sample rate determined by the frequency of CL _N. The frequency of the clock signal supplied to a particular one of those stages is lower than the frequency of the clock supplied to any stage preceding that stage. Preferably, the stages 300-2...300-
The frequency of each of the N clocks is one-half the frequency of the previous stage's clock. In the following discussion, it is assumed that this preferred relationship between the clock signals CL ₁ . . . CL _N is the case.

As shown in FIG. 3, stage 300-1 includes a convolution filter and decimation means 3.
02, delay means 304, subtraction means 306,
expansion and interpolation filter means 308. An input stream of digitized pixels G ₀ with a sampling rate equal to the frequency of clock CL ₁ is fed through a convolution filter and decimation means 302 to clock CL ₂ .
Generate an output stream of pixel G ₁ with a sample rate equal to the frequency of . G ₀ is the digitized video signal input to analyzer 202. The convolution filter has a low-pass function that reduces the original center frequency of each image represented by G ₁ to 1/2 of the corresponding original center frequency represented by G ₀ . . At the same time, decimation reduces the sample density in each element by 1/2.

Each pixel of G ₀ is provided via delay means 304 to subtraction means 306 as a first output. At the same time, the reduced density pixels of G ₁ are provided to a magnification and interpolation filter 308 which increases the sample density of the G ₁ pixels back to the density of G ₀ . The interpolated G ₁ pixel with expanded density is then provided as a second input to subtraction means 306 . Due to the presence of the delay means 304, each pair of samples G ₀ and G ₁ corresponding to each other in spatial position is delivered to the first subtraction means 306 at coincident times with each other.
and a second input. Subtraction means 306
The output stream of successive samples _L0 from identifies the highest spatial frequency of each element of the scanned image.

The configuration of each stage 300-2...300-N is essentially the same as the configuration of stage 300-1. However, the higher ordinal stage 300-2...300-
Each of N operates with a lower spatial frequency signal occurring at a lower sample density than the previous stage. More specifically, the output stream of successive samples L ₁ represents the next highest spatial frequency in each image source, so that the convolution of stage 300-N, as shown in FIG. stages 300-1...30 with the low frequency residual signal G _N derived from the output of the filter and decimation means;
It consists of _a stream of octave samples L ₀ .

As shown in detail in the above-mentioned Japanese Patent Application No. 59-134,032, the first advantage of the Bad Pyramid is that it eliminates noticeable artifacts introduced during image processing. Each analyzed output L ₀ is derived in such a manner as to minimize the amount of undesired visual effects introduced)...
A reconstructed image is synthesized from L _N-1 and G _N. A disadvantage of the Burt pyramid is that it requires a convolution filter and a decimation plus expansion and interpolation filter per analyzer stage.

The FSD pyramid analyzer shown in Figure 3a is similar to the Burt pyramid analyzer in several respects. First, FSD analyzers also generally use similar sampled signal conversion means 30-
1, 300-2...300-N pipelines. Second, each stage has a digital clock signal CL ₁ , CL ₂ . . . CL _N supplied to it individually.
operates at a sample rate determined by the frequency of . Third, the frequency of the clock signal applied to any particular stage is preferably one-half the clock frequency of the immediately preceding stage.

However, the particular configuration making up each stage of the FSD pyramid analyzer, shown schematically as 300-K in Figure 3a, is similar to the Bird Pyramid Analyzer.
Each stage of the analyzer, e.g. stage 300-1 in FIG.
The structure is somewhat different from that of the . More specifically, each stage 300-K (where K is any value between 1 and N) of the FSD pyramid analyzer shown in FIG. 3a is a convolution filter 302.
a, decimation means 302b, delay means 30
4, and subtraction means 306.

The output from convolution filter 302a is provided as an input to subtraction means 306 prior to decimation by decimation means 302b. This configuration allows the FSD pyramid
It makes it unnecessary to provide magnification and interpolation filters in each stage of the analyzer. By omitting the expansion and interpolation filters, the cost and inherent delay of each stage of the FSD pyramid analyzer shown in Figure 3a is reduced compared to the cost and inherent delay of each stage of the Burt pyramid analyzer shown in Figure 3. can be significantly reduced.

In order to explain the operation of the apparatus shown in FIGS. 1 and 2 (or FIGS. 1 and 2a),
Reference is made to FIG.

Rectangle 400 represents a relatively large dimension of the binary space region defined by the entire image frame of pixel samples. The G _N output from the analyzer 202, which is supplied to the computer 102 without passing through the window gate, is _P ′
Represent this entire spatial image frame spatial region with a low resolution given by P′ _Y only.
Thus, as shown in FIG. 5, this G _N represents a low-resolution, full-field image 500 of one or more objects (eg, a vase 502) within the spatial region viewed by the camera 100. For purposes of explanation, P′ _X and
It is assumed that the values of P′ _Y are both 6. Therefore, the total area of the full-field low-resolution spatial image region 400 shown in FIGS. 4 and 5 consists of only 36 pixels.

The L' _N-1 output from window gate 204-(N-1) represents localized sub-region 402. Sub-region 402, whose horizontal and vertical elements are only 1/2 of the elements of spatial region 400, is only 1/4 of the area of region 400. However, as shown in FIG. 5, subregion 402 also consists of 36 pixels, thus providing an intermediate view of vase 502 with a higher resolution than that provided by lower resolution full view 500.

Similarly, each window gate 204-(N-2) and 204-(N-3) not shown in FIG.
Each localized space sub-region 404 and 40 is represented by L′ _N-2 and L′ _N-3 outputs from
6 also consists of 36 pixels as shown in FIG. Therefore, the area represented by spatial sub-region 404 is only 1/4 of spatial sub-region 402 (or 1/16 of the total viewing region 400 area). Therefore, the resolution of intermediate field 506 of vase 502 is higher than the resolution of intermediate field 504 (intermediate field 50
The resolution of 4 is higher than that of the lower resolution full field of view 500). Similarly, the area of the spatial sub-region 406 is only 1/4 of the area of the spatial sub-region 404 (or 1/64 of the area of the full field of view spatial region 400). Therefore, field of view 508 of vase 502 is represented at full resolution.

For convenience of explanation, the value of N was assumed to be only 3 in describing the operation of the present invention. Therefore, in this case L' ₀ represents the spatial sub-region 406, and the spatial sub-regions 404, 402, 400 are defined by L' ₁ , L' ₂ (L' _N-1 ) and G ₃ (G _N ), respectively. It is expressed as In reality, N is a value of 3 or more (usually
at least 5 or 6). Furthermore, in reality each value of P _X and P _Y is greater than 6 (for example, 32
Also 16). In this case, 512/512 pixels (or sometimes 1024/1024 pixels)
The spatial image region represented by the high-resolution, wide-field video signal from imager 106 identified by is reduced by data reduction means 108 to
It is reduced to 5 or 6 different resolution viewing conditions of either 16.16 or 32.32.

As shown in FIG. 2, computer 102 provides individual window center control signals to each of window gates 204-0...204-(N-1). This allows each spatial sub-region (e.g. spatial sub-region 40
2,404 and 406) on the computer 102
Move according to command signals from. For example, as shown schematically by the arrows in FIG. It can be moved to its current position within region 400 (indicated by a solid line). In this way, any portion of the total field of view of the spatial region 400 can be shown at any resolution with each higher resolution.

In the case of a surveillance camera, the computer 102 first analyzes a low-resolution full field of view and identifies objects of interest (e.g., moving objects, objects with a particular shape, etc.) within this full field of view. Determine whether it appears to exist in any subregion. If it appears, the computer examines the sub-region at a higher resolution to see if the object of interest is actually present. Even in automatic equipment using robots,
In order to perform “eye-hand” adjustment, computer 1
A somewhat similar test is available by 02.

A great advantage of this invention is that it greatly reduces the amount of data that needs to be processed by the computer at the same time without reducing either the resolution or viewing capabilities of the imager means.

In the case of FIG. 2a, the computer 102 generates a switch control signal for the selection switch 208, and the computer ₁₀₂ generates a switch control signal for the selection switch ₂₀₈ .
Selectively test only the data stored in any one of 6-N at any time. This may further reduce the amount of data that needs to be processed by computer 102.

It will be clear that in some cases it may be desirable to pass even the last band of the bands through the movable window which sends even fewer pixels than said second predetermined number of pixels to the output of the image data reduction means 108. . Furthermore, if the imager means 106 is provided with moving means 110, it is possible to maintain each window in a predetermined constant spatial relationship to each other and to move the object of interest under computer control to the highest resolution. It is possible to move the imager means so as to bring it into the window. Further, instead of the selection switch 208, it is possible to simultaneously select the outputs of any two or more of the memories 206-0 to 206-N, and display the outputs of the selected memories on the display unit 112 at the same time. In some cases, it may be desirable to use a selection switch. It goes without saying that the present invention also includes structures modified in this way.

[Brief explanation of drawings]

FIG. 1 is a basic block diagram of an apparatus using a concave automatic electronic camera equipped with image data reduction means of the present invention, and FIG.
2a is a basic block diagram of a first embodiment of the data reduction means; FIG. 2a is a basic block diagram of a second embodiment of the image data reduction means of FIG. 1; and FIG.
FIG. 2 is a basic block diagram of a first preferred embodiment of the spatial frequency spectrum analyzer of FIG.
3a is a basic block diagram of a second preferred embodiment of the spatial frequency spectrum analyzer of FIG. 2; FIG. 4 is a schematic illustration of the operation of the spatially movable window of the invention; and FIG. The figure schematically illustrates the relative resolution and field of view for each of the subspectral band images derived from the output of FIG. 2. 100...Concave electronic camera, 106...Imager means, 108...Image data reduction means, 200...A/D converter, 202...Spatial frequency spectrum analyzer, 204-0 to 20
4-(N-1)... Window gate, L ₀ ~ _{L N-1} , G _N ...
Second video signal group, _P'X・_P'Y ...pixels.

Claims (1)

Claims: 1. A method for use with an input video signal representing a first predetermined number of pixels taken from an image of a relatively wide predetermined field of view and defining said image at a predetermined spatial resolution, comprising: The spatial frequency spectrum of the image represented by the input video signal is analyzed to produce a plurality of separate output video signals representing a group of sequentially arranged consecutive subspectral bands of the image spatial frequency spectrum. (a) a first band of said group exhibits relatively high spatial resolution and comprises said first predetermined number of pixels; and (b) a first band of said group each local band exhibits a lower spatial resolution and consists of a smaller number of pixels than the band immediately preceding it in the group;
the last band in said group is comprised of a second predetermined number of pixels that is the minimum number of pixels in all bands in said group; and further, at least one band in said group reducing the size of the field of view represented by the at least one band by passing a spatially localized subset of the pixels of the field of view through a spatial window within the predetermined field of view; A data reduction method, wherein the subset consists of a number of pixels not greater than said second predetermined number. 2. All pixels in each successive frame of an image of a spatial region within a relatively wide predetermined field of view of the imager means, all pixels in each of said frames specifying said image with a predetermined spatial resolution. said imager means for retrieving in real time a first video signal representing pixels of said first video signal; means for retrieving a plurality of respective separate second video signals representing a group of successive subspectral bands arranged in the order of: (a) a first of said group; (b) each band of said group has a lower spatial resolution than the immediately preceding band of said group; and consists of a smaller number of pixels,
The last band in said group is comprised of a second predetermined number of pixels that is the minimum number of pixels in all bands in said group; means for reducing the size of the field of view represented by the at least one band of the group by passing a spatially localized subset of the bands through a spatial window within the predetermined field of view; , said subset comprising a number of pixels not greater than said second predetermined number, thereby obtaining a reduced video output of data from the camera.