CN1183743C - Image sensor with cropping - Google Patents

Abstract

An image sensor (102) which can output only desired portions of an image. Thus, the image is essentially cropped at the time it is produced, eliminating the generation of data from the image sensor (102) corresponding to undesired portions of the data. This reduces the storage requirements for data being digitally processed, reducing or eliminating the need for a frame buffer memory. In a preferred embodiment, a single silicon chip (100) including a CCD array (102) includes a number of registers (112) which can be programmed for each frame to indicate which lines should be read out. Within each line, the chip (100) can be controlled by timing pulses to quickly read out data before or after the desired cropped portion of the image, with the quickly read data simply being discarded. The actual data desired can then be read out at a slower clock rate to allow processing on the fly.

Description

Imageing sensor and method of operation thereof with clipping function

Technical field

This invention is applicable to imageing sensor, especially is connected to the video camera that resembles on the USB (USB).

Background technology

The ability that obtains and store still life and digital video photo become come more important.Digital picture can be operated on computers, can also transmit by the Internet.Digital video camcorder has limited being extensive use of of it because of its high cost, and the function of taking pictures reduces cost and taken like a shot by the people.

Fig. 1 is the high-level block diagram of a typical video camera, and light receives by a camera lens 10 and sends one group of sensor array 12 to, as charge (CCD) array.The signal that is detected is sent to the front-end circuit 14 of a simulation then, amplifies and processing signals, makes it to change into digital form.Signal through digitized processing is stored in 16 li of frame buffer memories then.

Treatment circuit 18 is handled image with digital form, perhaps is a program processor or an ASIC(Application Specific Integrated Circuit) (ASIC).Treatment circuit is sent to bus 20 to view data so that be sent on the computer.

A bus standard that is used to be connected to the peripheral hardware of computer is USB.Because USB is a universal serial bus, and it and a plurality of peripheral hardware be the fact of sharing the time, and the data bandwidth that therefore sends computer to can be restricted.Like this, treatment circuit 18 will compress to reduce the quantity of the digit order number that will transmit the data that will transmit, and also can carry out some processing, resembles convergent-divergent, prunes and filters.A typical picture format is common inner transformation form CIF, it provides 352 pixels * 288 row, each pixel is represented by 24 bits that again wherein red, green, blue look (RGB) respectively accounts for 8, and the data that send in bus are considerable quantities.Say that generally compress technique need reduce to about every pixel 3 or 4 from 24 with data.

Fig. 2 is typical C CD array 12 block diagrams.As seen from the figure, it comprises one group of photodiode 22 array of lining up row, column, some vertical shift registers 24, and in order to receiving numerical value from photodiode, each photodiode is all used an image charge (chargeup).These vertical registers are downloaded the into shift register 26 of a level, continuously data are shifted away by output loop 28.

Fig. 3 has represented the senior sequencing control of ccd array 12 visually.The horizontal transmission pulse 30 of some regular spacings is arrived horizontal shifting register 26 with these row synchronous (clock).With single horizontal feed pulse 35 is that the example amplification describes.For each such pulse, operate time pulse 37 full line data are taken out integrated circuit block continuously synchronously, and total data is all taken out synchronously, next line will be transferred and go out vertical transfer register by another horizontal transmission impulsive synchronization and go forward side by side into the horizontal shift register, behind Data Transfer Done, new one group clock signal 37 again can be continuously with new data sync trip 28, and this process is repeated to the entire image conveying automatically and finishes.Simultaneously, new image can charge to photodiode capacitance again, is stored on the vertical shift register by vertical transfer pulse 32 immediately.Process repeats then, and the horizontal feed pulse is with on every circuit shift-in horizontal shifting register 26, so that subsequently bits all in the circuit is shifted out.

In case one circuit is loaded into horizontal shifting register 26, the whole piece circuit must be unloaded before next bar horizontal feed pulse.Otherwise the data in the shift register will be capped, and the image of this line also can disappear.Simultaneously, the horizontal feed pulse needs accurately timing to safeguard to guarantee output row before the read next image.Otherwise,, also can lose image if all row are read out in the time of distributing.Therefore, see Fig. 1 back again, buffer storage 16 needs to store at least one complete frame that comes from ccd array (all row of image).Say that generally frame buffer memory can be by treatment circuit 18 asynchronous conducting interviews so that carry out compression, pruning etc.In addition, treatment circuit 18 also can its access bus 20 backs from buffering number of memory accesses certificate.

Some application does not need frame buffer, resembles the NTSC standard, is bearing from the transmission of the TV of making a video recording.Because these data can transmit by a dedicated channel, do not need frame buffer, also shared bus needn't be arranged.

For USB, several bus access patterns are arranged here: control model, magnanimity transfer mode and isochronous mode are arranged.Isochronous mode is used to transmit video data, relates to peripheral hardware and bus and consults to obtain a bandwidth.Because bus and other peripheral hardwares are shared, bandwidth will be restricted.Maximum guarantees that bandwidth is 1k byte/ms, if bus is crowded, with the bandwidth that provides less than this bandwidth.In addition, because the bus shakiness, the sequential that when sends these bytes through bus can be different.So frame buffer memory also can provide a pooling feature, make the sequential of treatment circuit adaptation bus and obtain valuable data.

Summary of the invention

The object of the present invention is to provide a kind of imageing sensor and method of operation thereof, this imageing sensor can an output image needs part.Like this, image can be pruned when producing, and has eliminated the generation that does not need data that comes from map sensor.This has just reduced the call data storage of the digital processing that will carry out, and reduction or elimination are to the needs of buffer storage.

According to an aspect of the present invention, a kind of method that is operatively connected the imageing sensor to the digital camera of shared external bus is provided, comprise that step is as follows: adopt different reading speeds to begin to read described transducer at selected first row, this row is not first row of imageing sensor, and described sensing data is provided in the described image sensor in the described digital camera; Finish to read described sensing data at selected second row, this row is not last column of described imageing sensor; Described data transaction is become number format; And directly send the data of above-mentioned number format to a digital signal processing circuit, and needn't in frame buffer, carry out intermediate storage, described digital signal processing circuit is set in the described digital camera, wherein said image sensor comprises second data for the treatment of first data of being handled by described digital signal processing circuit and waiting to be pruned, described first data comprise the described data that are transmitted, and described second data comprise the data that are stored in the described selected first row delegation before.

According to a further aspect in the invention, provide a kind of imageing sensor that is arranged in the digital camera, having comprised: an image detecting element array; First register is used for storage information and begins to read row with what discern above-mentioned array, and this row is not first row; Second register is used to store the information of a plurality of row of the above-mentioned array that indication will read; A timing sequencer, be used to provide a clock signal to above-mentioned Image Detection cell array, give an encoder so that send from the view data of described image detecting element array with speed corresponding to data transfer rate on the shared external bus, encode so that need not be stored in the frame buffer memory and described pictorial data can be sent to described encoder temporarily, wherein said timing sequencer is configured to provides one or more clock signal, to reflect the availability of described shared external bus, wherein said encoder and described image sensor are set in the described digital camera those clock signals by non-uniform spacing.

In one embodiment, a silicon chip integrated circuit block that contains ccd array comprises the register of some, can be to each frame programming to confirm which row should read.First register is stored the quantity that reads the row of skipping before the beginning, and second register stored the quantity of the row that will read.In each row, integrated circuit block is controlled promptly to prune part front and back reading of data at the needs of image by time pulse.Valid data read under a slower clock rate and the while (on the fly) handles.

This invention can be used for not being with buffer storage video camera.The cancellation of buffer storage has reduced video camera cost and size.When the present invention provides the horizontal transmission pulse to finish this function by making bus control unit provide signal to control to clock generator, therefore, transport pulse except the regular spacing in CCD integrated circuit block explanation, mentioned, they also can non-regular spaces transport pulse, to be consistent with the USB sequential.Also consider the cushion space availability of a complete horizontal line in the bus control unit for the sequential of the transport pulse of CCD.

The influence of also being subjected to frame rate and compression speed etc. for the horizontal feed pulse sequence of CCD, their influences are stored in the data in the buffer of bus control unit, and are stored in the data volume in the buffering area before bus access next time.Delegation's storage capacity in the bus control unit buffer can guarantee that full line can be transmitted out this CCD before next horizontal transmission pulse.

By eliminating frame buffer, this invention makes CCD and analog front circuit be combined into single silicon integrated circuit piece.In one embodiment, timing sequencer also is integrated on the same integrated circuit block.

If think to understand in more detail the essence and the advantage of this invention, also will be described in conjunction with following accompanying drawing.

Description of drawings

Fig. 1 is the square frame diagram of the video camera system of prior art;

Fig. 2 is the square frame diagram of the ccd array of prior art;

Fig. 3 is the chronological diagram of the CCD integrated circuit block of prior art;

Fig. 4 is the block diagram according to the video camera system of one embodiment of the present of invention;

Fig. 5 is according to the sequential chart that the invention provides to the ccd array pulse;

Fig. 6 is the diagram of an embodiment of the DSP of Fig. 4;

Fig. 7 is the diagram of explanation according to an image sensor array of pruning of the present invention;

Fig. 8 is the chronological diagram that reads different sequential incidents in the process at frame;

Fig. 9 be explanation provide two different, corresponding in the background by the pipeline (pipeline) of the part of being pruned or the diagram of view data channel;

Figure 10 is the square frame diagram according to diagram sensor IC piece of the present invention.

Embodiment

Fig. 4 is a block diagram of using the video camera of this invention.This system comprises a camera lens 10 and one group of sensor array, resembles CCD assembly 12, as shown in Figure 1, is connected on the USB20.

The signal that comes from the CCD integrated circuit block offers AFE (analog front end) 40, comprise differential amplifier or relevant double sampling device (CDS) 42, they offer an automatic gain control circuit AGC44 with analog signal, and these data offer an analog-digital converter ADC46 again then.

These digitized signals needn't be stored in the frame buffer memory, but directly offer digital processing circuit 48.They are transported to earlier that video digital signal processor 50 compresses, pruning, convergent-divergent and other functions be as filtering etc.One is treated, and these numerical datas are transported to a line buffer 52.Line buffer 52 is being stored the single file data from the horizontal shifting register of CCD integrated circuit block 12.

Be provided for a bus interface that comprises a bus controller buffer 56 and a bus control unit 58 from the data of line buffer 52.Preferablely be, bus controller buffer 56 can be stored two USB frames with maximum possible speed or 2k byte, depend on frame rate, compression speed etc., and it can change to 10 row even 15 to 20 row, if phenomenon takes place to damage.

Bus control unit 58 provides on circuit 60 and closes a control signal to timing sequencer 62, and timing sequencer 62 provides clock signal to CCD integrated circuit block 12 on circuit 64.Clock signal 64 is included in level and the vertical transfer pulse described in Fig. 3.The vertical transport pulse provides as prior art, and periodically loads charge value from ccd array.The horizontal feed impulse wave will be along with the processing and the data of column circuits are adjusted so that matched data reading from ccd array to the transmission of USB down.

In one embodiment, AFE (analog front end) 40 is integrated into (the same with ccd array 12) identical semiconductor integrated circuit piece substrate 411 with timing sequencer 62.The cancellation of frame buffer make single integrated circuit directly and the digital processing circuit link up.So just make more compact structure, the less video camera that needs costliness.

The sequential of the horizontal transmission pulse that Fig. 5 explanation provides on circuit 64.Bit is come out in groups synchronously by information-processing circuit as required as shown in the figure, rather than a consecutive pulses stream.After sending first group triplex row (line) 66, have a time to pause, handle these pixels simultaneously and flow to bus, provide next to organize two row 68 then again.This perhaps can be according to the availability and the intermittence of timing sequence generating that is transported to the data of USB of frame buffer.As shown in the figure, next what transmit is single file 69, next by a group of some pulses 70 being made up of four lines of carrying.After the entire image conveying finishes, provide a vertical transport pulse 72.The example of Fig. 5 is simple an elaboration, please notes because horizontal pulse is to provide between two vertical pulses.In order to show the slit, pulse more recently provides.Like this, news little group pulse is provided fastly in case with this circuit treatable fast as far as possible speed pixel data is sent.In one embodiment, a complete image is that very near impulsive synchronization is come out at interval by one, produces pre-treatment in next vertical transport pulse and finishes.If processing not exclusively, can produce intermittently before next pulse, this intermittently meets the availability of handling sequential and USB.Like this, horizontal pulse can all provide continuously, and is just faster than the speed of prior art, and the sequential of the later one-tenth group pulse of vertical pulse can be adjusted to just in time before or after next vertical pulse, to adapt to processing procedure and bus timing.

The primary clustering of a concrete Video DSP 50 of Fig. 6 presentation graphs 4.First module 74 is circuit, extracts YUV information from the photomosaic that CCD provides.Be adjuster 76 then, its converted image is to adapt to X and Y.At last, compressor circuit 78 packed datas.Each element comprises that local memory is to store the data of handling.Usually, 2 line buffers are served circuit 74, and the single file buffer is served adjuster 76, and another single file buffer is to serve compressor circuit 76.This invention also comprises the buffer of these inner appointments, has also eliminated the needs to the DSP external frame buffer.Inner DSP line buffer is enough little so that be integrated in the same integrated circuit according to its logic.

In experiment, when video camera is worked, bus control unit 58 can and USB consult so that on bandwidth, guarantee the data of every millisecond of acquisition q.s one and determined the quantity of bandwidth, frame rate, size and compression speed enroll program to meet available bandwidth by treatment circuit 48.This will determine the quantity of the row read from bus controller buffer 56 on each transducer of whole USB, and read capable speed from ccd array 12.

More particularly, timing sequencer 62 produces the horizontal feed pulse automatically, unless forbidden by bus control unit.Timing sequencer produces pulse with one than prior art faster rate, compression subsequently but suppressor pulse not, and expand pulse by the pulse that suppresses right quantity.

Fig. 7 illustrates that has an image sensor array 80 of pruning zone 82, and in prior art, entire image 80 is sent to signal processing circuit, and it can propose and prune the pixel of district's 82 correspondences so that handle in cut operation.In this invention, be not to round image, but begin to read that skip those top row, the process of reading 86 places of being expert at stop from delegation 84 from image reading sensor.Obviously, reduce handled data volume so widely, thereby also reduced essential storage space.

In addition, for every row, it is most economical reading full line simply, needn't adjust original position, can need other circuit like this.Use this method, all pixels before pixel 88 all can read with high speed and abandon.Pixel in the target area (pixel 88 is to pixel 90) goes out to handle subsequently synchronously.Pixel between pixel 90 and the image end goes out synchronously with rapid rate again, and is dropped again.

Fig. 8 is the sequential block diagram of a typical frame duration, divides the different time period of t1～t6 (t1 is that frame begins the place).Timing sequencer will produce time sequence information and give this array, then they or break away from (locate off) integrated circuit block or be integrated into image sensor circuit itself, integrated circuit block has register, stores the parameter of some control timing.

Storing in the register has 5 main parameters, the continuity that the control transducer reads.

1, TimeToFirstLine (to first line time): from end exposure (transmission from the photovoltaic regions to the array) to the clock periodicity that begins to read first row, [t2].

2, SkippedLines (skipping line number): activating the line number that to skip before capable the reading from these.It can be used for dual mode: as at original position of row or as counter.

3, TargetLines (target line number): for further handling the line number that reads, the combination of skipping line number and target line number has defined object tape (area-of-interest), has eliminated at the time overhead of pruning the zone.

4, TimeToNextLine (to time of next line): the clock periodicity from the last pixel of a target line to the first black pixel of next target line, this parameter is used to expand reading target line in time.

5, TimeToNextFrame (to the time of next frame): the clock periodicity [t6] of the beginning of (photovoltaic regions transmission) from last capable sequence to next frame.

Following preset parameter is a sensor parameters:

PixelsPerLine: all pixel quantities in the delegation (black, activate) (t4)

LinesPerFrame: the quantity of all row in the frame

SkipTime: skip (t3) * of delegation's required time (should be as much as possible little)

TransferTime: the charging view data transmits into the time overhead of vertical register (t1) (*) from photovoltaic regions

PixelClock: transducer (pixel) clock frequency, preferably 12MHz.

Frame rate is represented with following formula:

1/ frame rate=PixelClock*[TransferTime+TimeToFirstLine+

(LinesPerFrame-AvtiveLines)*SkipTime+TargetLines*

(PixelsPerLines+TimeToNextLine)+TimeToNextFrame]

In one embodiment, t2 and t6 have the resolution in PixelClock cycle to guarantee maximum flexibility.Minimum, perhaps t2 or t6 have this resolution, make the entire frame time is had the resolution of a clock cycle, then can have coarse relatively resolution to other variable.Can save some figure places to counter like this.

For frame rate is reached between 10fps and the 60fps, and in order to satisfy the needs of some processing/software rear end, program parameters is generally in following scope in one embodiment:

1, TimeToFirstLine: high to 5ms (*)

2、SkippedLines：0～LinesPerFrame-1

3、TargetLines：1～LinesPerFrame

4, TimeToNextLine: high to 5ms (*)

5, TimetoNextFrame: high to 5ms (*) (bigger) for low frame rate

(*): minimum value is so that make frame rate high as much as possible.

DLL (dynamic link library):

Register is by an I2C interface routineization, and register is double buffering, and the register of an activation is only changed at the frame boundaries place, keeps the continuity of register value on the time in entire frame.

In one embodiment, the I2C interface characteristics are to move when 1MHz, to guarantee in time to transmit all parameters in frame duration.

In one embodiment, be not the pruning district 82 that only sends among Fig. 7, zone 82 can send more continually than whole sensor array 80.For example, prune the district and include mobile image-region, these local remainders than the image that background is provided upgrade more continually.For example, zone 82 can send 100 frames by per second, and whole zone may send 25 frames by per second, and the result is that zone 82 every conveyings just send entire frame for four times one time.

Fig. 9 is the block diagram of system that utilizes an aspect of this invention.Imageing sensor 92 provides data for digital signal processor (DSP) circuit 94, and it is handled background respectively and prunes the zone.Background can moving area 82 can be carried by second pipeline 98 by carrying off and on as a conveyance conduit 96 of first channel.They can be used as two independently information channel on a USB, send.Like this, the present invention makes frame size switch between two kinds of different forms with frame rate.

Figure 10 is an embodiment of a semiconductor integrated circuit piece 100.A sensor array 102 provides data for a shift register 104, and this register is come out continuously synchronously by an output circuit 106.This circuit is set up in the ccd array of standard.This invention has also added a timing sequencer 108 on same integrated circuit block, rather than by the extraneous sequential that produces.Also have a sensor array arrange control circuit 110 simultaneously, be not only control signal to sensor array is provided, signal is provided also for other elements of integrated circuit block 100.(control circuit does not show to avoid making image complicated).One group of register 12 is stored and is used for the parameter that the serializing transducer is read continuously, as above-described.Each register is two registers, and double buffering is provided.Like this, data just can be read at first register, and the previous data in second register are used to handle current frame.Finish in case current frame reads, the data of first register are sent to second register, or are directly used in next frame, and second half of double buffer register can be loaded the frame sequential that is used for subsequently simultaneously.The most handy I2C bus 114 is used the standard I 2C standard of calling data and a control line.

This invention on the other hand, integrated circuit block 100 is provided with other analog circuit of process sensor data.Especially, a relevant double sampling device (CDS) 116 is connected output 106.Automatic gain control (AGC) circuit 118 links to each other with CDS116.At last, analog-digital converter (ADC) 170 converts pixel data to digital form, carries by an output line 122, carries out digital processing, such as the DSP94 by as shown in Figure 9.

People in the present technique field should understand that the present invention can and can not break away from substantive characteristics with other specific forms realization.For example, not to read undesired pixel data in the delegation apace, a plurality of outputs of the horizontal shifting register of CCD can be set.Can increase circuit and circuit route though have output in each pixel position, can increase the output coarse resolution of for example about every 10-20 pixel, this meticulous resolution can obtain by changing the clock read-out speed.Therefore the description of front is illustrative, and nonrestrictive, scope of the present invention is limited by following claim.

Claims (10)

1, a kind of method that is operatively connected the imageing sensor to the digital camera of shared external bus comprises that step is as follows:

Adopt different reading speeds to begin to read described transducer at selected first row, this row is not first row of imageing sensor, and described sensing data is provided in the described image sensor in the described digital camera;

Finish to read described sensing data at selected second row, this row is not last column of described imageing sensor;

Described data transaction is become number format; And

Directly send the data of above-mentioned number format to a digital signal processing circuit, and needn't carry out intermediate storage in frame buffer, described digital signal processing circuit is set in the described digital camera,

Wherein said image sensor comprises second data for the treatment of first data of being handled by described digital signal processing circuit and waiting to be pruned, described first data comprise the described data that are transmitted, and described second data comprise the data that are stored in the described selected first row delegation before.

2, method as claimed in claim 1 also comprises step:

Select the position of starting and ending at selected first row;

Read data between the described starting and ending position with the speed of the reading rate that is different from the data outside the described starting and ending position.

3, method as claimed in claim 2 is to read when reading between the starting and ending position data in the data outside the described starting and ending position wherein at every turn, but they read the clock frequency difference.

4, method as claimed in claim 2 wherein is not read when the data outside the described starting and ending position are reading some data between the starting and ending position.

5, method as claimed in claim 1 also comprises step:

Handle the view data that reads from described imageing sensor in a video digital signal processor (DSP), described processing comprises to be compressed described image;

The view data of handling is offered a bus; And

A clock signal is offered described imageing sensor, so that send view data, so that do not use the frame buffer memory of the entire frame that is used to store described view data from described image sensor array with speed corresponding to data transfer rate on the described bus;

Wherein one or more described clock signal is consistent with the sequential with described bus brokenly at interval.

6, method as claimed in claim 1, wherein said selected first row comprises the first of described first data; Described selected second row comprises a decline of described first data; Described digital signal processing circuit is set in the described digital camera.

7, method as claimed in claim 1 is wherein used a condition that was provided with before described first and second data of storage on the described image sensor, define described selected first and second row.

8, a kind of imageing sensor that is arranged in the digital camera comprises:

An image detecting element array;

First register is used for storage information and begins to read row with what discern above-mentioned array, and this row is not first row;

Second register is used to store the information of a plurality of row of the above-mentioned array that indication will read;

A timing sequencer, be used to provide a clock signal to above-mentioned Image Detection cell array, give an encoder so that send from the view data of described image detecting element array with speed corresponding to data transfer rate on the shared external bus, encode so that need not be stored in the frame buffer memory and described pictorial data can be sent to described encoder temporarily

Wherein said timing sequencer is configured to provides one or more clock signal, those clock signals by non-uniform spacing reflecting the availability of described shared external bus,

Wherein said encoder and described image sensor are set in the described digital camera.

9, imageing sensor as claimed in claim 8 also comprises:

The 3rd register was used to store from above-mentioned image detecting element end exposure to a plurality of clock cycle that begin to read first row;

The 4th register is used to store from one and reads last capable pixel reads first capable pixel to the next one a plurality of clock cycle;

The 5th register is used to store last column of reading since a frame a plurality of clock cycle to next frame, and

A control circuit is used for reading in the described capable and described capable data that add between the above a plurality of row that begin to read that begin to read with the speed of described register specifications.

10, imageing sensor as claimed in claim 7, comprise to go up a digital signal processor of the speed of data transfer rate corresponding to shared outside USB (USB), so that can being transmitted to described digital signal processor, described pictorial data compresses

Wherein said digital signal processor and described image sensor are set in the same image-processing system.

Images