EP1181816A4

EP1181816A4 - Method and apparatus for dual resolution foveating display output

Info

Publication number: EP1181816A4
Application number: EP00915857A
Authority: EP
Inventors: Nathaniel Joseph Mccaffrey; Lambert Ernest Wixson; Der Wal Gooitzen Siemen Van
Original assignee: Dialog Semiconductor GmbH
Current assignee: Digital Imaging Systems GmbH
Priority date: 1999-02-24
Filing date: 2000-02-24
Publication date: 2006-11-02
Also published as: EP1181816A1; JP2003526231A; WO2000051345A8; WO2000051345A1; WO2000051345A9

Abstract

A novel method and apparatus for providing dual resolution read out from a CMOS APS imager (215) with both concurrent high-resolution and low-resolution area display outputs. The imager (215) uses dual read out paths (230, 110) to increase the frame rate of the wide field of view while simultaneously providing high-resolution images for object identification. The first imager output (230) consists of a wide field of view (WFOV) output that uses a lower resolution that may be used to determine location of objects in the field. The second imager output (110) consists of a narrow field of view (NFOV) that contains at least one sub-frame of the total image area. The second image(s) may be used to replace or overwrite segments of the WFOV image. Thus, the dual-resolution imager (215) can provide concurrent high-resolution imager tracking while maintaining a low-resolution wide area image.

Description

METHOD AND APPARATUS

FOR DUAL RESOLUTION FOVEATING

DISPLAY OUTPUT

CROSS REFERENCE TO RELATED APPLICATIONS

This application relates to Provisional Application Serial Number 60/120,268, entitled, "CMOS APS FOVEATING ARRAY WITH RANDOM X-Y FOVEAL AREA" filed on February 24, 1999, which is assigned to the same assignee and is incorporated by reference herein. Applicant claims the benefit of the priority filing date of February 24, 1999 pursuant to 35 U.S.C. § 119(e)(1).

FIELD OF THE INVENTION

The present invention is related to the field of solid state imagers. More specifically, the present invention is related to providing concurrent dual resolution outputs of imaging devices.

BACKGROUND OF THE INVENTION

Traditionally, an Active Pixel Sensor (APS) imager is formed by arranging photoelectric or photosensitive, solid state devices in a rectangular or square array. Each photoelectric solid state device, termed a "pixel," converts photonic energy into an electrical charge and outputs the

electric charge when the pixel is selected. Imager arrays of 512 x 512, 1024 x 1024 and 2048 x 2048 pixels are well known in the art. Typically, an image is created by serially reading out the electric charge of each pixel cell in the imager array by selecting a row and then sequentially reading the electrical charge of each pixel cell in the selected row. After the last pixel cell in the selected row is read, a next row is selected and the pixel cell read out process repeats, row by row, until each pixel cell has been read. In one method of sequentially reading the image array rows, entitled "progressive" scanning, after each row is completely scanned, the pixel cells in the next adjacent row are read or scanned. Thus, the array rows are scanned in sequential order, i.e., row 1, 2, 3 through M. In another scanning method, entitled "interlaced" scanning, after a selected row is scanned, the scan proceeds to a next non-contiguous row. In one example of interlaced scanning, alternate rows are scanned in the order 1, 3, 5, through M-1 and then 2, 4, 6, through M, where M is an even number of rows.

To track small targets with good resolution over the imaging area, a high-resolution imager containing a large number of pixel cells is necessary. For imager arrays of the same overall dimension, the number of pixel cells contained in the imager array determines the resolution of the imager. For example, an array of 2048 x2048 pixels has a resolution sixteen times greater than an array of 512 x 512. High-resolution imagery is typically desired to obtain a clear and sharp image. As the number of pixel cells increases however, the time necessary to read the entire image significantly increases. This time to read the image, known as the image Frame time, is determined by the pixel read out speed and the total number of pixels in the imager array. As the pixel read out speed is typically fixed by the hardware used, the Frame time is determined by the number of pixel cells that must be read. Accordingly, as the resolution increases, the Frame time also increases. Increased Frame time causes moving images to appear to change in a sporadic and irregular manner. For example, objects moving faster that the Frame time may appear to jump from one position to another rather than smoothly transitioning between these positions. Hence, there is a need to provide a high-resolution imager with a Frame time of a short enough duration to capture transitions smoothly while maintaining an acceptable image quality field of view.

SUMMARY OF THE INVENTION

A method and apparatus for providing dual-resolution read out from a solid state CMOS Active Pixel Sensor (APS) imager is presented. The present invention provides for a first output that generates a low resolution, wide field of view (WFOV) image, and a second output that generates a higher resolution, narrow field of view (NFOV) image. The high-resolution output is generated using the outputs of individual pixel cells in the image array, while the low-resolution output is generated by combining a plurality of pixel cells into a single output. Using a WFOV and an NFOV image read out processor, an entire image may be quickly scanned and displayed using a low resolution image, while at least one designated area of the imager array may be displayed using a high resolution image

In one embodiment of the invention, the high-resolution output may overwrite or replace a low-resolution image. In a second embodiment of the invention, the high-resolution image may be directed to a second device to display high-resolution images independently of a lower resolution image.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages, nature and various additional features of the invention will appear more fully upon consideration of the illustrative embodiments to be described in detail in connection with the accompanying drawings. In the drawings:

Figure 1 illustrates a prior art pixel cell and associated control logic used in APS pixel arrays;

Figure 2 illustrates a block diagram implementation of a dual-resolution foveating array in accordance with the principles of the invention;

Figure 3a illustrates an exemplary embodiment of dual-resolution circuitry according to the principles of the invention; Figure 3b illustrates a second exemplary embodiment of the dual-resolution circuitry according to the principles of the invention;

Figure 4 illustrates an exemplary embodiment of the dual-resolution circuit logic in a foveating array in accordance with the principles of the invention;

Figure 5a depicts a scene to be digitally captured; Figure 5b depicts a low-resolution digital image of the scene illustrated in Figure 5a;

Figure 5c illustrates an exemplary high-resolution region within the low-resolution image illustrated in Figure 5b;

Figure 5d illustrates a high-resolution image of the high -resolution region of the scene illustrated in Figure 5a; Figure 5e illustrates a foveating image created in accordance with an exemplary operation of the dual-resolution array in accordance with the principles of the invention; and

Figure 6 illustrates an exemplary system using the dual-resolution foveating array in accordance with the principles of the invention. DETAILED DESCRIPTION OF THE INVENTION

A CMOS APS imager includes a pixel area and a read out/processing logic. The pixel area consists of a photosensitive area for photoelectron generation in a matrix configuration, addressing circuitry and amplification circuits that operate to address and access individual pixel cells. The read out and processing area provides for signal conditioning of input and output data. Figure 1 illustrates a typical CMOS APS pixel cell and associated processing circuitry and logic. As illustrated, solid state photoelectric device 100 is used to collect photonic energy over a designated time period and convert the collected energy into an electrical charge. The electrical charge is then amplified by amplifier circuit 106, and by the appropriate selection of Row Select Switch 104 and Column Select Switch 102, the amplified electrical charge is transferred to video output line 110. After the read out of photocell 100 occurs, reset switch 108 is activated to return the pixel cell charge to a nominal value. This reset prevents electrical charge collected before the read out from influencing electrical charge collected during the next period of photon collection.

Figure 2 illustrates a typical row and column arrangement of pixel cells 100 and corresponding logic circuitry used to sequentially read out the electrical data from the pixel cells 100. In this embodiment, a plurality of pixel cells 100 are arranged in an M x N array of twelve pixel cells 100 per row in eight rows —i.e., 12 x 8 imager array. For clarity only a sample of the pixel cells 100 within the imager array are specifically labeled. Thus, pixel cells 240, 241, 242 and 243 represent a limited number of individual pixel cells 100 that are included in a first row of twelve pixel cells. Similarly, pixel cells 250, 251, 252 and 253 represent a limited number of individual pixel cells 100 that are included in a second row of twelve pixel cells. Using a matrix format, an imager array of any size can thus be created by increasing the number of rows and columns. Further, Row Select Lines 200, 202, 204 and 206 actually extend to address each of the pixel cells in rows 1, 2, 3 and 4 respectively and have been limited in Figure 2 to addressing the middle set of pixel cells only to provide clarity to the illustration.

The electrical charges collected in the first row of pixel cells, of which pixel cells 240, 241, 242 and 243 are illustrative examples, may be serially read out on output line 110 by selecting Row Select Line 200 and selecting each individual Column Line 216, 214, 212 and 210 used to address pixel cells 240, 241, 242 and 243. In one scheme, column lines 216, 214, 212 and 210 are selected in sequential order. At the completion of reading the twelve pixel cells in this illustrated first row, the processing continues to serially collect electrical charges in the next row (row 2) by selecting Row Select Line 202 and selecting the individual column lines 216, 214, 212 and 210. The electrical charges collected at the pixel cells of the second row, of which pixel cells 250, 251, 252 and 253 are illustrative examples, may then be read out, sequentially for example, on output line 110. This pixel cell read out process continues, row by row, until the last row has been reached. At the conclusion of the pixel cell read out each pixel cell is reset. This reset may occur after an individual pixel cell is read, after a row of pixel cells have been read or after the entire array imager has been read. The timing of the reset is dependent on the specific implementation. Figure 2 further illustrates an exemplary grouping of individual pixel cells in a P x Q array 215, in accordance with the principle of the invention. In this illustrative embodiment array 215 is composed of sixteen pixel cells 100 arranged as a 4 x 4 array. In this configuration the array 215 is deemed a single entity with a single output 230, which is formed as a composite of the outputs of the pixel cells within the array (in this case, sixteen pixel cells). Array 215 can be composed of any number of pixel cells 100 that may be addressed as a single entity.

In accordance with the principle of the invention, output of array 215 is selected by activating Group Select line 220, which causes an output to be generated, concurrently, from each pixel cell in the array. The concurrently generated outputs are then merged together to form a single composite output 230. Similar groups of P x Q arrays having similar merged outputs may be created throughout the entire imager array. These P x Q arrays may then be superimposed upon the individual addressing lines, represented by Row Select Lines 200, 202, 204, and 206 and Column Select Lines 216, 214, 212 and 210. In this overlapping of addressing lines, outputs of the individual pixel cells, 240-243, 250-253, 260-263 and 270-273 and the composite output 230 may thus be obtained. These outputs may be obtained substantially concurrently or sequentially.

Figure 3a illustrates one exemplary embodiment of logic circuitry used to achieve pixel cell dual read out in accordance with the principles of the invention. As illustrated a dual output pixel 101 has a first output that is connected to corresponding amplifiers 106 and row and column switches 104 and 102, respectively, and second output that is grouped together to form a single composite output through amplifier 320.

As illustrated in this exemplary embodiment, six dual output pixel cells 101 are arranged^" in a 2 x 3 array wherein the two rows are labeled row m and row m+1, respectively, and the three columns are labeled n, n+1 and n+2, respectively. It should be understood that it is not intended to limit the array size to that illustrated. The number of pixel cells 101 in each row and in each column may be expanded to include any number of pixel cells. Hence P x Q arrays of any rectangular or square arrangement can be created in accordance with the principle of the invention.

In this illustrated configuration, a pixel cell output may be transmitted to output line 110 by activating, or closing the appropriate row and column switches. The output of pixel cell 101a is directed to output line 110 by selecting Row Select line 200 and Column Select line 216. The selection of Row Select Line 200 causes switches 104a through 104c to close, and the selection of Column Select line 216 causes switch 102a to close. Similarly, the output of pixel cell 101c is directed to Output line 110 by selecting Row Select line 200 and Column Select line 210. The selection of Column Select line 210 causes switch 102c to close. In this manner of addressing, each pixel cell within Row n may be read out on output line 110 by selecting line 210 and by selecting, in a sequential order, each Column Select line in the dual resolution imager (Column Select lines 216, 214 and 210).

Further, in accordance with the principle of the invention, the composite output of array 215 can be output to Group output 230 by selecting Group Row Select 300 and Group Column Select 360. The selection of Group Row Select 300 and Group Column Select 360 may be performed concurrently, or sequentially, with the selection of Row Select lines and Column Select lines. In either concurrent or sequential mode, selecting Group Row Select 300 causes switches 310a through 310f to close, and the second outputs of the corresponding pixel cells (101a through lOlf) are combined in amplifier 320. When Group Column Select 360 is selected, switch 330 closes and the combined output is made available as Group Output line 400 through amplifier 340.

In the illustrated embodiment, amplifier 320 is a summing amplifier that sums the values on the corresponding input lines. In another embodiment, the output of amplifier 320 may be an average of the values on the corresponding input lines. In still another embodiment, as illustrated in Figure 3b, each second output of a dual output pixel cell can be directed to an amplifier that adjusts the pixel cell output value by a pre-determined amount. Illustrated in Figure 3b are amplifiers 380a through 380f that adjust a corresponding pixel cell second output. When the gains of amplifiers 380a through 380f are set substantially equal, the pixel cell outputs provided to summing amplifier 320 correspond to the level of energy collected in the cells. In another embodiment, each pixel cell second output may be adjusted based on an amplifier gain value that is determined from the position of the pixel cell within the P x Q array. In this embodiment, pixel cells closer to the center of the P x Q array have a greater weight than pixel cells toward the edges of the array. For example, the amplifier gains may be Gaussian distributed based on the position of the pixel cell in the array.

Figure 4 illustrates one exemplary embodiment of an imager array architecture in accordance with the principle of the invention. Illustrated is a 6 x 4-pixel imager array that is divided into four pixel cell groups using a 3 x 2 pixel cell group, similar to pixel cell group 215 of Figure 3a. As should be understood, the illustrated array is merely representative of an imager in accordance with the principles of the invention and that array imagers may be constructed of any number of rows and columns of pixel cells.

In the illustrated embodiment, individual dual output pixel cells 101a through lOlf constitute a first pixel group that is addressable by selecting Group Row Select line 300 and Group Column Select line 360. The selection of Group Row Select 300 and Group Column

Select 360 cause switches 301 and 330 to close and output 230 is thus made available on Group Output 400. Similarly, pixel cells lOlg through 1011 constitute a second pixel group that is addressable by selecting Group Row Select line 300 and Group Column Select line 361. The selection of Group Row Select line 300 and Group Column Select line 361 cause switches 301 and 331, respectively, to close and output 233 is made available on Group Output 400. The two remaining pixel groups, similarly, are addressed by Group Row Select 221 and Group Column Select 360 and 361, respectively. These two groups generate outputs 232 and 231, respectively, which are made available on Group Output 400. Thus, by reading the plurality of pixel cell groups contained within the pixel array an image may be created that is of lower resolution than the capability of the imager array.

According to the principles of the invention, a dual-output pixel cell is used as a basic pixel cell in the imager array. A high-resolution image of a selected region may then be obtained by reading one output of each pixel cell in the selected region, while a low resolution image may be obtained by reading the combined outputs of each pixel cell groups. Thus, in Figure 4, a high resolution output can be achieved in a region containing only pixel cells lOla-lOlf and 101m- lOlr by reading the outputs of these individual cells while a low-resolution image can be concurrently obtained by reading the composite outputs of these same pixel cells. In this embodiment illustrated, the high-resolution image has a resolution approximately six times better than the corresponding low-resolution image.

Also illustrated in Figure 4 is Group Reset line 410 that may be used to reset each pixel cell within at least one pixel group. The single Group Reset line 410 resets all the pixel cells in the entire 6 x 4-imager array. However, it should be understood that in another embodiment, a plurality of Group Reset lines, one for each pixel cell group may also be included. When a plurality of Group Reset lines are used, a pixel cell group may be reset after a pixel cell group is read out rather than waiting for the entire image to be read. For the same reasons as previously discussed, only a single Reset line 108 has been included in the embodiment illustrated in Figure 4. It would be understood that means for resetting individual pixel cells, such as 101a, and 101b are typically included in imager arrays and the invention may use a single pixel reset switch or a plurality of such switches to reset individual pixels after a high resolution output read is performed. Figure 5a represents an exemplary image 500 that is desired for display. As discussed, each pixel cell in the imager array captures a portion of the image 500 and each pixel must be read out. Using a high resolution output, the time to read every pixel cell may become excessively long and the image may appear to move in an irregular manner. Thus, a lower resolution image with a shorter Frame time may produce adequate results. Figure 5b illustrates a digital representation of the image 500 using the low-resolution output. The low-resolution output is obtained by processing the composite output of a plurality of pixel cell groups. The low-resolution image is advantageous as the Frame time is held to a minimal value. However, the image does not display the clarity that is desired.

Figure 5c illustrates a selected region 530 where a high-resolution image is desired. In region 530, a high-resolution output is obtained by reading the output of each individual pixel cell. The selection of region 530 may be based on object recognition algorithms, user selection and intervention, or motion analysis algorithms. For example, a user may use a mouse or joystick or keyboard inputs to outline at least one desired area of high-resolution display. In the embodiment of a user interface, a joystick is the preferred mode. Figure 5d illustrates the high-resolution image 540 obtained by reading the first output of each pixel cell within region 530. Image 540 utilizes the high-resolution capability of the imager array while using only a portion of the time that is required to read the entire image in high- resolution mode. In one embodiment Figures 5b and 5d may be displayed as two disjoint images on the same imaging devices. They can also be displayed on different imaging devices. In this embodiment, high-resolution details of a selected region of the image may be viewed while the low-resolution image 510, as illustrated in Figure 5b, remains available to a user for viewing. Accordingly, each pixel cell group in the array is read to obtain a complete low-resolution image. In another embodiment, the image 540 of Figure 5d may be merged with the image of

Figure 5b. As illustrated in Figure 5e, the background image 510 remains in low-resolution mode while the high-resolution image 540 is concurrently displayed in high-resolution mode. Thus, a foveated image 550 is created. This foveated image 550 is advantageous as only those regions that are of interest are displayed with high resolution while the Frame time is not substantially increased over the time to create the entire low-resolution image 510. As the low- resolution image 510 and high-resolution image 540 may be created concurrently, the time to create the foveated image 550 is not substantially increased over the time to create the low- resolution image 510.

In still another embodiment, when foveated image 550 is desired, those pixel cell groups within the selected regions of high-resolution imagery need not be read. In this embodiment, the scanning logic can prevent the low-resolution pixel groups from being read, as it is known that this data will be overwritten. In this mode, the time to read the imager array is reduced as the number of pixel groups read is reduced by the size of the selected foveating regions. For example, in a 2048 x 2048 CMOS APS imager having arrays composed of 4 x 4 dual output pixel cells, a high-resolution image would be a full 2048 x 2048 pixels and a low-resolution image would be reduced to 512 x 512. The high-resolution image has a frame rate that is sixteen times lower than the low-resolution image when equal output clocking frequencies are used. Selecting, for example, a high-resolution image area of 400 x 400 pixels, the low-resolution image would be reduced to 412 x 412 ((2048 - 400)/4). Thus, the number of pixel group outputs is reduced from 512 to 412.

Figure 6 illustrates a system incorporating the dual-resolution imager of the present invention. An imaging device, in this example, a camera 600, outputs a WFOV image 602 and a NFOV image 604 to processor 610. Processor 610 then may output the two images to separate displays 620, 630 or combine the images into a composite image that is displayable on a single display 640. Processor 610 also provides inputs 608 to imager 600 to define the number and size of the NOFV images.

The examples given herein are presented to enable those skilled in the art to more clearly understand and practice the instant invention. The examples should not be considered as limitations upon the scope of the invention, but as merely being illustrative and representative of the use of the invention. Numerous modifications and alternative embodiments of the invention will be apparent to those skilled in the art in view of the foregoing description. Accordingly, this description is to be construed as illustrative only and is for the purpose of teaching those skilled in the art the best mode of carrying out the invention and is not intended to illustrate all possible forms thereof. It is also understood that the words used are words of description, rather than limitation, and that details of the structure may be varied substantially without departing from the spirit of the invention and the exclusive use of all modifications which come within the scope of the appended claims is reserved.

Claims

WHAT IS CLAIMED

1. A method for producing a dual resolution readout in an imaging device containing a plurality of photo-sensitive electrical devices arranged in a matrix, each photo-sensitive electrical device having a first output and a second output comprising the steps of: dividing said plurality of photo-sensitive electrical devices into at least one first group of photosensitive electrical devices generating a corresponding first signal for each of said at least one first group; selecting at least one second group of said plurality of photo-sensitive electrical devices, each of said photo-sensitive electrical devices within said at least one second group generating a corresponding second signal; and displaying said first group signals and said second signals

2. The method as recited in claim 1 wherein said at least one second group is randomly positioned with respect to said at least one first group.

3. The method as recited in claim 2, wherein said at least one second group is designated by from the group of object recognition algorithms, user interactive inputs and motion analysis.

4. The method as recited in claim 1 wherein said at least one first group signal is generated at a first rate and said corresponding second outputs are scanned at a second rate.

5. The method as recited in claim 1 wherein said first signal is generated by the weighted accumulation of said photosensitive electrical device first outputs.

6. An dual resolution imager comprising

A plurality of photo-sensitive electrical devices each having a first output and a second output; processing circuits responsive to said first outputs; processing circuits responsive to said second outputs, wherein said first outputs produce a high-resolution image and said second outputs produce a lower resolution image.

7. The imager as recited in claim 6 wherein said second output processing circuit further comprises an addressing scheme to concurrently address a plurality of photo-sensitive electrical devices.

8. The imager as recited in claim 7 wherein said second output processing circuit further comprises a plurality of amplifiers, one of said amplifiers connected to each of said addressable plurality of photo-sensitive electrical devices second outputs.

9. The imager as recited in claim 6 further comprising a weighting amplifier connected between each of said photo-sensitive electrical device second output and said corresponding amplifier.

0. A system to display images using a dual-resolution format comprised of; a dual-resolution imager array having a plurality of photosensitive electrical devices each having a first output and a second output processing circuits for scanning a plurality of first regions at a first rate each of said first regions having a plurality of said photosensitive electrical devices, each of said first regions generating a first region output; processing circuits for selecting at least one second region, processing circuits for scanning said at least one second region at a second rate, said second region generating a plurality of second region outputs, one said second region output for each photo-sensitive electrical device in said at least one second region; processing circuits to display said first region outputs and second region outputs.