AU2011205092A1 - Change detection sampling for video sensor - Google Patents

Abstract

-21 Abstract CHANGE DETECTION SAMPLING FOR VIDEO SENSOR Disclosed in an on-chip method (1000) for adaptively sampling a pixel value captured using an image sensor on the chip, comprising reading (1001) a first sample value for a pixel of the sensor, the first sample value representing an amount of light captured by the pixel cell over a first time period, determining (1003) a second sample value over a second time period, 0 determining (1005) a threshold based on a photon shot noise value associated with one of the first and the second sample values, determining (1007) a difference between the first and the second sample values, and outputting (1009), from the image sensor, a value captured from the pixel cell and resetting (1011) the pixel cell if the determined difference exceeds the determined threshold. P000013 /5478844_1 270711 -1/10 Fig. 1 P0000 13 /5478845_1 270711

Description

S&F Ref: P000013 AUSTRALIA PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3 of Applicant: chome, Ohta-ku, Tokyo, 146, Japan Actual Inventor(s): Andrew James Dorrell Nagita Mehrseresht Address for Service: Spruson & Ferguson St Martins Tower Level 35 31 Market Street Sydney NSW 2000 (CCN 3710000177) Invention Title: Change detection sampling for video sensor The following statement is a full description of this invention, including the best method of performing it known to me/us: 5845c(5483359_1) -1 CHANGE DETECTION SAMPLING FOR VIDEO SENSOR TECHNICAL FIELD OF THE INVENTION The current invention relates to the field of moving image capture, and in particular, to capture at high frame rates. 5 BACKGROUND In the field of still image and video capture, there is increasing interest in high frame rates. In the context of video, high frame rates allow the slowing down of motion when the captured data is played at standard rates. In the context of still image capture, high speed burst capture allows for fast bracketing of focus or other parameters that permit image improvement in post 0 processing. Several problems arise in high speed image capture. For example, due to reduced exposure times, the amount of measured image signal is reduced, leading to noisier images and reduced dynamic range. In addition, due to the high speed required, the sensor itself may suffer from heat related problems. Furthermore, the available bandwidth for transferring pixel data from 5 the sensor chip and through subsequent processing stages and onto storage can be a limiting factor. The most common solution to the bandwidth problem is to run many hardware circuits in parallel. This provides the required bandwidth, but leads to increased cost and power consumption during operation, which in turn may result in heat problems. ?0 Commonly, high speed sensors provide a windowed mode of operation, in which a reduced area of image pixels is read. By reducing the number of pixels read per frame the required bandwidth is reduced. However this method reduces the field of view, creating an undesirable user experience. An alternative approach is to reduce the spatial resolution of each frame by sub-sampling or pixel binning. In this case, temporal resolution is improved at the expense of 25 reduced spatial detail in each frame. One technique attempts to solve these problems by acquiring frames by alternately using windowing and sub-sampling methods, and combining the resulting frames to produce an image. This approach is a generalisation of the concept of interlacing, which is one of the oldest bandwidth reduction techniques used in video. This approach leads to improved performance, but still leads to a loss of detail outside the 30 windowed region. A widely acknowledged problem associated with the use of regular sub-sampling patterns is that they lead to structured artefacts that are visible during playback. Due to the large image area involved, and the sensitivity of human vision to structure, these artefacts can often be P000013 / 5478844_1 270711 -2 highly objectionable. To avoid structured artefacts in cases where full progressive frame capture is not possible, unstructured patterns for interlace sampling have been proposed. These approaches result in sub-sampling artefacts that appear to be noise-like, and are therefore less objectionable. In computer graphics, under-sampling provides an important 5 performance optimisation and randomised sampling patterns have also been employed as a means of reducing the visibility of the resulting aliasing artefacts. The approach is generally avoided in image capture however due to the trivial cost of intensity measurement at each sensor pixel and the reduced quality can result from under-sampling on the sensor's fixed grid. 0 Later work in stochastic sampling for computer graphics has indicated that adapting the sample density to intensity gradients in the image data can reduce the artefact caused by this form of sub-sampling. In general terms the idea is related to non-uniform sampling theory, which advocates a maximally random sampling lattice with sample density determined by generalisation of the Nyquist criteria. It should be noted however that the theory of non 5 uniform sampling is not explicitly considered in the stochastic sampling formulations provided in the field of computer graphics. Instead, a key problem has become one of determining how to adapt the sample density locally to the data. Content adaption is also relevant to image sensor design, where it provides a method for bandwidth reduction. One approach uses a high speed imaging apparatus that utilises input !0 from an auxiliary motion sensor to determine read rates for each windowed area of the sensor. Compared to an ideal stochastic sampling arrangement, the approach has a disadvantage in that reading of entire blocks is required even when there is change in only a few pixels. It has a further disadvantage in having abrupt boundaries to sampling regions which may become visible in the output unless extremely conservative change detection is implemented. Thus 25 the scheme cannot achieve a significant bandwidth reduction unless the scene motion is highly localised, occurring in only a small number of blocks. To overcome these limitations a per pixel sampling decision can be made. Another approach describes a sensor that uses a number of criteria, including change detection, to independently determine the read rate for each pixel in the imaging array. 30 However, this sensor also has a problem in that its change detection algorithm generates a sampling pattern that varies as a function of both motion and incident light intensity. As a result it fails to achieve a good approximation of a data adapted stochastic sampling of the scene data. As a direct result, setting a detection threshold for artefact free acquisition leads to P000013/5478844_1 270711 -3 unnecessarily high sample rates. By comparison, lower thresholds, and sample rates, lead to undetected motion in darker regions and structured artefacts. It should be noted that it is not an objective of sensor level adaption technology to achieve stochastic sampling, but rather it is to provide an accurate selection of pixels that contain new 5 information. It has therefore been considered an important problem to minimise the noise component in this change detection. Finally work has been conducted in the field of so called "event sensors". These sensors achieve a change detection which is more independent of incident light level by using a pixel which has a logarithmic response. While this also has the advantage of extending the dynamic 0 range of the device, the approach is not suitable for high quality photographic imaging where a linear pixel response is required for colour and interpolation processing. The approach also makes control of other photographic parameters such as shutter speed problematic or impractical. SUMMARY 5 It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements. The inventors have determined from consideration of current approaches that it would be desirable to have a sensor with a substantially linear response that can achieve the goal of having a noise-like sample distribution, the sampling density of which is determined by the ?0 sampling needs of the data, and in particular, where change detection is substantially independent of the level of incident light intensity at each pixel. The term "sampling density" is a measure of the number of pixels being read out per unit area of the pixel array (eg see 311 in Fig. 3). Accordingly, in a scene being imaged, more pixels are read out in regions which exhibit greater changes in movement and/or light intensity 25 than in regions which exhibit smaller changes in movement and/or light intensity. Disclosed are arrangements, referred to as Shot Noise Based Threshold (SNBT) arrangements, which seek to address the above problems by making a decision on whether or not to sample a pixel value based upon whether a rate of change of pixel signal intensity at the pixel in question exceeds a threshold that is based upon the signal to noise ratio of the 30 received pixel signal. Change detection is based on a threshold that is an approximation of the square root of the light intensity measured at the pixel. When the sensor is operated in a Poisson noise dominated mode, this ensures a change detection threshold that is substantially P0000 13 / 5478844 1 270711 -4 independent of the level of light intensity. The photon shot noise contributed by the incident light provides a source of randomness that ensures a noise-like sampling pattern is achieved. According to a first aspect of the present invention, there is provided an on-chip method for adaptively sampling a pixel value captured using an image sensor on the chip, the method 5 comprising the steps of: reading a first sample value for a pixel of the sensor and maintaining the first sample value at the pixel, the first sample representing an amount of light captured by the pixel over a first time period; determining a second sample value for the pixel over a second time period, 0 incorporating the first time period, wherein the second sample value includes the amount of light captured over the first time period; determining a threshold based on a photon shot noise value associated with one of the first and the second samples; determining a difference between the first and the second samples; and 5 outputting, from the image sensor, a value captured from the pixel and resetting the pixel if the determined difference exceeds the determined threshold. According to another aspect of the present invention, there is provided an on-chip method for adaptively sampling a pixel value captured using an image sensor on the chip, the image sensor comprising a plurality of pixel cells, the method comprising the steps of: !0 reading a first sample value for a pixel cell of the sensor and maintaining the first sample value at the pixel cell, the first sample value representing an amount of photo electrons integrated by the pixel cell over a first time period; determining a second sample value for the pixel cell over a second time period , wherein the second sample value represents an amount of photo-electrons integrated by the 25 pixel cell over the second time period of equal or shorter duration to said first time period; determining a light intensity for each of the first and second time periods; determining a threshold based on a photon shot noise estimate associated with the light intensity of either the first or second time periods; determining a difference between the light intensity for the first and the second time 30 periods; and outputting, from the image sensor, a value captured from the pixel cell and resetting the pixel cell if the determined difference exceeds the determined threshold. P000013 / 5478844_1 270711 -5 According to another aspect of the present invention, there is provided a chip-based image sensor comprising an array of pixels, at least some of which comprise: a photo detector for converting incident light over a first time period to a first sample value, and for converting incident light over a second time period incorporating the first time 5 period to a second sample value, wherein the second sample value includes the sample value generated over the first time period; an analogue memory element; a transfer mechanism for transferring charge from the photo detector to the memory element to thereby establish respective sample values at the photo detector and the memory 0 element; and a change detection module comprising: means for determining from the sample values at the photo detector and the memory element if a change of incident light intensity has occurred at the photo detector during the second time period; and 5 means for, if a change of intensity is detected, controlling outputting of a value from the pixel. Other aspects of the invention are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS One or more embodiments of the invention will now be described with reference to the 0 following drawings, in which: Fig. 1 is a schematic block diagram of the main functional components of an image sensor chip that implements an SNBT arrangement; Fig. 2 is a schematic block diagram of a pixel within the active imaging area of the image sensor chip of Fig. 1; 25 Fig. 3 is a schematic block diagram illustrating control logic to determine the read status of a pixel according to one SNBT arrangement; Fig. 4 a and b are schematic block diagrams illustrating two embodiments of control logic to determine a scene change status of a pixel; Fig. 5 is a circuit diagram for a pixel according to one SNBT arrangement; 30 Fig. 6 is a circuit diagram for a threshold calculation according to one SNBT arrangement; Fig. 7 is a flow diagram for reading a pixel based on detected scene change according to one SNBT arrangement; P000013 / 5478844_1 270711 -6 Fig. 8 is a flow diagram for reading a pixel based on detected scene change according to a further SNBT arrangement; Fig. 9 depicts a spatial arrangement of pixels in an image sensor according to one SNBT arrangement; and 5 Fig. 10 is a flow diagram showing a process for practicing the disclosed SNBT arrangements. DETAILED DESCRIPTION INCLUDING BEST MODE Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the 0 purposes of this description the same function(s) or operation(s), unless the contrary intention appears. It is to be noted that the discussions contained in the "Background" section and the section above, relating to prior art arrangements, relate to discussions of documents or devices which may form public knowledge through their respective publication and/or use. Such discussions 5 should not be interpreted as a representation by the present inventor(s) or the patent applicant that such documents or devices in any way form part of the common general knowledge in the art. Fig. I is a schematic block diagram of the main functional components of an image sensor chip 100 that implements, on chip, an SNBT arrangement. Fig. I shows an imaging area 1 10, !0 comprising an array of pixels such as 115, a memory 120 for storing the integration time for each pixel in the imaging area, readout logic 130 for performing the operations required to determine when a pixel in the imaging area should be read, and for performing the read and outputting of a digitised value for the pixel when appropriate. The terms "pixel" and "pixel cell" are used interchangeably in this description, unless the 25 context indicates otherwise. The operations of the imaging area 110, the memory area 120 and the readout logic 130 are coordinated by control signals generated by a control unit 140 and communicated by a plurality of control lines 142, 143 and 144. Fig. 2 is a schematic block diagram of a pixel 200 within the active imaging area 110 of the 30 image sensor chip 100 of Fig. 1. The pixel (ie pixel cell) 200 comprises a photo-diode 201 (or an equivalent photo-detector) for converting incident light into photo-charge (also referred to as photo-electrons), an analogue memory element 202 for storing a charge value, and a transfer mechanism 203 for transferring charge from the photodiode to the memory element. P0000 13 / 5478844_1 270711 -7 The pixel 200 is controlled by timing signals generated by a control unit and communicated by control lines 211. The pixel 200 outputs voltages, corresponding to the voltage developed across the photodiode and memory elements, on data lines 212 and 213 respectively. Different embodiments for elements 201 through 203 are possible according to whether the 5 pixel is reset after regular polling events. The operation of the pixel will be explained in greater detail subsequently. Fig. 3 is a schematic block diagram illustrating control logic to determine the binary read status of a pixel according to one SNBT arrangement. A pixel is only read out (ie output) if a read status 324 (ie "Rsel") is asserted for the pixel. This value 324 is used by the control unit 0 312 to generate appropriate timing signals for performing the readout operation if required. Readout of a pixel is also associated with a reset of the pixel, in which the pixel and storage are set to an "empty" state and the corresponding memory element storing the pixel's exposure time in poll cycles is set to zero. Readout of a pixel constitutes the end of a measurement cycle and the commencement of the next measurement cycle. 15 In a first SNBT arrangement, the value on the photo-diode is the value which is output whereas in a second SNBT arrangement the value on the storage element is output. In both cases the output value corresponds to the charge accumulated between pixel reset and the determination of an asserted read status, ie to the charge accumulated during the immediately expired measurement cycle. 20 During pixel polling, each row of pixels of a pixel array 311 is polled and processed separately. For each pixel in the polled row at a specified column position, data from the pixel corresponding to the charge on the storage element and the charge on the photo-diode are applied to the two data lines 321 and 322 respectively (these data lines being equivalent to the lines 213 and 212 in Fig. 2 respectively). At the same time, a corresponding exposure time Te 25 is read from a memory array 313 and placed on the data line 323. In an SNBT arrangement detailed herein, the integration time Te is converted from a digital value to an analogue voltage on the data line 323 to facilitate its use in analogue decision logic. The decision to read a pixel is determined if the output from any one of three decision 30 blocks 331, 332 or 333 is asserted. A decision block 333 compares the voltage on the data line 323, corresponding to the exposure time Te, to a predetermined value Tmax provided on data line 326 and asserts if the P0000 13 / 5478844_1 270711 -8 integration time Te exceeds the predetermined maximum Tmax, thus ensuring a minimum read rate for every pixel in the sensor. A decision block 332 compares the voltage 321 corresponding to the charge accumulated at the storage element 202 of the pixel during integration to a predetermined value Vmax 5 (having a reference numeral 327), and asserts if the charge there exceeds the predetermined maximum Vmax, thus minimising the likelihood of pixel saturation. In an alternative arrangement, decision block 332 compares the voltage 322 corresponding to the charge accumulated at the photodiode 201 of the pixel during integration to a predetermined value Vmax (having a reference numeral 327), and asserts if the charge there exceeds the 0 predetermined maximum Vmax, thus minimising the likelihood of pixel saturation. The device chosen for this comparison is the device most prone to saturation. So for example, in the arrangement subsequently described with reference to Fig. 4a, the voltage 321 corresponding to the charge accumulated at the storage element 202 of the pixel would be employed, whereas in the arrangement subsequently described with reference to Fig. 4b, the 15 voltage 322 corresponding to the charge accumulated at the photodiode 201 of the pixel would be employed. A decision block 331 uses the values 322, 321 on the photo-diode and storage element respectively, in combination with the exposure time Te, to determine whether a change of light intensity has occurred at the pixel, thus ensuring that pixels that are receiving light from 20 moving or changing parts of the scene being imaged are read with greater frequency than pixels which are receiving light from more static parts of the scene. The operation of the decision block 331 is modified by a voltage signal 325 (referred to as "Sen") from the control unit which permits adjustment control of a motion detection sensitivity. 25 The presented change detection system in Fig. 3 is able to provide the control signal 324 (ie "Rsel") which results in a stochastic readout pattern, the density of which is proportional to the change rate of moving or changing parts of the scene. It is this functionality that the present SNBT arrangement addresses through the design of the decision block 331 which is now described in more detail in Figs. 4a and 4b. 30 Figs. 4a and 4b are schematic block diagrams illustrating two arrangements of control logic 400a and 400b for determining a scene change status (also referred to as a readout status) of a pixel. P000013 / 5478844_1 270711 -9 For the arrangement of Fig. 4a, it is assumed that the photo-charge collected at the pixel is accumulated at the storage element by regular transfer of all the charge from the photo-diode (eg 201) to the storage element (eg 202), this photo-charge resulting in a voltage V,_A, (ie 401) at the storage element. As a result, the voltage on the photo-diode, this voltage being referred 5 to as VA, (ie 402), corresponds to the photo-charge collected at the photo-diode since the previous polling event. At a scale block 411, implemented as a voltage controlled amplifier, V,_,, (ie 401) is scaled by the inverse of a current exposure time Te (ie 407). The scaled result, denoted by a voltage V, (ie 405), is subsequently input to a differential amplifier 413 along with the voltage 0 VA, (ie 402). An output voltage 416 of this amplifier 413 represents a rate of change of incident light intensity at the pixel since the previous polling event. This voltage 416 is compared by a block 414 to a threshold V, (ie 406) to determine a change status, Csel (ie 404) according to Csel= 1 if '" V, >V. (EQN I) 0 otherwise 5 The threshold V (ie 406) is determined by a block 412 based on V, (ie 405) and a sensitivity control signal Sen (i.e. 403). The detail of the threshold determination performed by the block 412 is described in more detail with reference to Fig. 6. Returning to Figs. 4a and 4b, in an alternative arrangement, the threshold calculation performed by the block 412 uses VA, (ie 402) rather than V, (ie 405). In such an arrangement, 20 the rate of change of incident light intensity at the pixel is determined in a similar manner to that depicted in Fig 4a, ie inputting VA, (ie 402) and V, (ie 405) into the differential amplifier 413, and comparing the output of the amplifier 413 to a threshold Vh (ie 406) in the block 414, the threshold being determined based on VA, (ie 402) rather than V, (ie 405). For the arrangement of Fig. 4b, it is assumed that the photo-charge collected at the pixel is 25 accumulated at the photo-diode (ie 201), and the storage element (ie 202) receives a "snapshot" of the photodiode voltage at the end of each polling interval (ie a snapshot of the photodiode voltage is provided to the storage element, leaving the charge undisturbed on the photodiode). As a consequence, the voltage on the photo-diode V, (ie 409) is equal to the P000013 / 5478844_1 270711 -10 voltage on the storage element V,, (ie 401) plus any additional charge V, accumulated since the last polling event. At a block 411, implemented as a voltage controlled amplifier, V,_, (ie 401) is scaled by the inverse of a current exposure time Te (ie 407). The voltages V,-A, and V, are input to a 5 differential amplifier 415 in order to calculate VA, (ie 402) which corresponds to the photo charge collected at the photodiode (ie 201) since the previous polling event. The scaled result, denoted V, (ie 405) is subsequently input to a differential amplifier 413 along with V, (ie 402). The output 416 of this amplifier 413 represents a rate of change of incident light intensity at the pixel since the previous polling event. This voltage 416 is 0 compared at block 414 to a threshold V,, (ie 406) to determine a change status, Csel (ie 404). The threshold V,,, (ie 406) is determined by a block 412 based on V, (ie 405), or in an alternative arrangement based on VA, , and a sensitivity control signal Sen (ie 403) according to V,,, = Sen x = Sen x V'A 7 (EQN 2) 5 The threshold calculation carried out by the block 412 is described in more detail with reference to Fig. 6. Fig. 5 is a circuit diagram of a pixel according to one SNBT arrangement. Fig. 5 depicts a number of signals Vrst, Vrow and Vrow2. Fig. 5 shows several instances of the signals Vrow and Vrow2. In the following description, a reference to Vrow should be understood to be a 20 reference to all the instances of Vrow in Fig. 5. A similar comment applies to the signal Vrow2. Accordingly, a statement that the signal Vrow is asserted means that each instance of Vrow in Fig. 5 is asserted. A similar comment applies to the signal Vrow2. The circuit 500 comprises a photo-diode 511 and a capacitive storage element 512 and the circuit 500 is controlled by a voltage reset signal Vrst 501, which is used to reset the pixel 500 25 in combination with the signal Vrow2 (ie 503). Asserting the signal Vrow connects data lines 521 and 522 to buffer stages 513 and 514, the data lines 521 and 522 providing voltages corresponding to a photo-diode charge and a storage element charge respectively. Asserting the signal Vrow2 503 causes the photo-diode voltage to be replicated on the capacitive storage element 512, leaving the photo-diode voltage 30 intact. P000013 /5478844 1 270711 -11 Fig. 6 is a diagram of a circuit 600 for performing a threshold calculation such as that performed by the blocks 412 in Figs. 4a and 4b, according to one SNBT arrangement. The relationship between an input voltage V,, and an output voltage V, for this circuit 600 is given by the following relationship: 5 v = (EQN3) where k is a transconductance parameter determined by the fabrication process of the chip in question, and Rc is selected to ensure that the output V 0 , 602 is a good approximation of the square root of the input voltage V,, 601. The input and output voltages and correspond respectively to the voltages V, (ie 405) and Vth (ie 406) in Figs. 4a and 4b. 0 This output value V, (being equivalent to Vth in Figs. 4a and 4b) is proportional to a Signal To Noise Ratio (SNR) of the input voltage V,, (being equivalent to V, in Figs. 4a and 4b) due to photon shot noise. If V,, was noise free, then it would yield a perfect change detection threshold for the pixel. However V,, contains noise. The spatial distribution of this noise is statistically independent (ie the noise associated with each pixel in the pixel array in 5 question, such as 311, is substantially statistically independent of the spatial location of the pixel in the array 311) which leads to a change detection threshold Vth (ie 406) with random variation about its ideal (ie noise-free) value. By comparing the noise contaminated threshold Vth (ie 406) thus determined to the change rate (ie 416 and 417 in Figs. 4a and 4b respectively) at the pixel, using the block 414 in Figs. 20 4a and b, a spatially random sampling pattern is thus generated. The density of the pattern is proportional to the probability of the measured intensity change corresponding to a scene change (as opposed to noise). Fig. 7 is a flow diagram for a pixel reading process 700 based on a detected scene change according to an SNBT arrangement as shown in Fig 4a. 25 The process 700 starts at a step 710 in which voltages V,_,, (ie 401) and Va, (ie 402) are polled from the analogue memory element (ie 213) and the photodiode (ie 212) respectively. In a following step 720 an exposure time T, is polled from the memory array 313. A voltage V, is then determined in a following step 730 by scaling the voltage V,_,, (ie 40 1) by the inverse of the current exposure time T, (ie 407). In an equivalent alternative arrangement, the P000013 / 5478844_1 270711 -12 scaling operation 411 may be conducted at a later stage. In this circumstance rather than scaling the voltage V,_,, V,, and V, are scaled by the current exposure time T, (ie 407). The purpose of the equivalent alternative arrangement and the steps of process 700 is to normalise the threshold Vth and intensity estimates to the same exposure time Te. A change 5 detection threshold V,, (ie 406) is determined in a following step 740, the threshold V,, being proportional to the square root of the voltage V (see EQN 2). A following step 750 compares the absolute difference between the voltage V, and the voltage V., to the threshold V, which was determined in the step 740. If the absolute difference between the voltage V and the voltage VA, is larger than the threshold V, the [0 process 700 follows a YES arrow 751, a "TRUE" read status is asserted for the pixel in question, and the value V, is output from the pixel in a step 780. Output of a pixel value involves conversion of the analogue voltage to a digital value and making that digital value available to a circuit external to the image sensor 100. Different arrangements are possible for the output process, in particular the value V, may be digitised and output directly. 15 Alternatively, the value for V,_, may be digitised and digitally scaled by T using digital logic. In yet another possible arrangement, V,_,, and VA, are summed and this summed value scaled by the inverse of the exposure time T after increasing it by one poll cycle. Output arrangements may include steps such as correlated double sampling (CDS) that are widely practiced to reduce read and other forms of noise. 20 Output of the pixel is followed by reset of the pixel in a step 790 in which the analogue memory element and photodiode are set to an "empty" state and the corresponding memory element storing the pixel's exposure time in poll cycles is set to zero. Returning to the decision step 750, if the absolute difference between the voltage V, and the voltage VA, is not larger than the threshold, V,, the process 700 follows a NO arrow 752, and 25 the integration at the pixel in question continues without reading out the integrated voltage. In a following step 760 the charge in the photodiode is added to the analogue memory element so that the memory element now reads as a voltage V, = V,_6, + VA, and then the photodiode is reset such that voltage VA, at the photodiode is notionally reset to zero in a following step 765. P000013 /5478844 1 270711 -13 A following step 770 is the start of next integration cycle (ie ,measurement cycle) during which the photodiode accumulates photo-charge generated since the previous polling cycle and the exposure time 7, is increased by one poll cycle. The process 700 may then be repeated for a new polling cycle by repeating the aforementioned steps from the step 710. 5 Fig. 8 is a flow diagram for a pixel reading process 800 based on detected scene change according to an SNBT arrangement as shown in Fig. 4b. The process starts at a step 810 in which the voltages V,-,, (ie 401) and V, (ie 409) are polled from the analogue memory element and the photodiode respectively. In a following step 820 the voltage VA, is determined by determining a difference between the voltage V, and 0 the voltage V,_,,. In a subsequent step 830 a exposure time T is polled from a memory array such as 313. The voltage V is then determined in a following step 840 by scaling the voltage V, (ie 401) by the inverse of the exposure time T (ie 407). In an equivalent alternative arrangement, rather than scaling the voltage V,-,, VA, and V,, are scaled by the current exposure time T, (ie 407). The purpose of the equivalent alternative arrangement and the 5 steps of process 700 is to normalise the threshold Vth and intensity estimates to the same exposure time Te. A change detection threshold voltage V,,, (ie 406) is determined in a following step 850, the threshold being proportional to the square root of the voltage V, (see EQN 2). A following decision step 860 compares the absolute difference between the voltage V, and the voltage V, 20 to the threshold voltage V,,, which was determined in the step 850. If the absolute difference between the voltage V and the voltage VA, is larger than the threshold voltage V,,, the process 800 follows a YES arrow 861, a "TRUE" read status is asserted for the pixel and the pixel value output in a following step 870. Output of a pixel value involves conversion of the analogue voltage to a digital value and making that digital 25 value available to a circuit external to the image sensor 100. Different arrangements are possible for the output process, in particular the value V, may be digitised and output directly. Alternatively, the value for V,-, may be digitised and digitally scaled by YT using digital P0000 13 / 5478844_1 270711 -14 logic. The arrangement of Fig. 4b further permits output as the value V, scaled by T, + I Output arrangements may include steps such as correlated double sampling (CDS) that are widely practiced to reduce read and other forms of noise. Readout of a pixel is followed by reset of the pixel in a step 880 in which the analogue 5 memory element and photodiode are set to an "empty" state and the corresponding memory element storing the pixel's exposure time in poll cycles is set to zero. Returning to the decision step 860, if the absolute difference between the voltage V and the voltage VA, is not larger than the threshold V, Ithen the process 800 follows a NO arrow 862, and the integration at the pixel in question continues without reading out the integrated 0 voltage. In a following step 890 a snapshot of the voltage V, on the photo-diode is stored, for future use, as V,_,, at the analogue memory element. The photodiode then keeps accumulating photo-charge (in a following step 895) until the next polling cycle starts. The exposure time T is also increased by one poll cycle in the step 895. The next polling cycle then starts by repeating the aforementioned steps from the step 810. 5 In both the methods outlined in Figs. 7 and 8, integration of photo-electrons is extended until a termination condition is met. The termination condition is one of a maximum exposure time being reached, a saturation level being reached, or a change in intensity exceeding some multiple of the standard deviation of the photon shot noise is detected. The change detection process described makes use of analogue processing suitable for 20 implementation in a sensor chip. It would equally be possible, and may be advantageous in some circumstances, to implement much of this process using equivalent digital circuitry. In this case, the memory array 313 sores both the exposure time and the accumulated pixel signal corresponding to V,, and at each cycle of the process 700, the value VA, is read from the photo-diode and digitised prior to performing the remaining calculations. In this arrangement 25 the photo-diode is reset after each read. Fig. 9 depicts a spatial arrangement 900 of pixels in an image sensor according to one SNBT arrangement. Rather than integrate dedicated processing according to the SNBT arrangement into every pixel of a sensor, it may be preferable to have a sub-set of pixels dedicated to the detection of motion. This approach, often referred to as spatial multiplexing, P000013 / 5478844 1 270711 -15 is used for colour capture but has also been proposed for High Dynamic Range (HDR) imaging and most recently, for achieving phase based auto-focus using a single image sensor. In this SNBT arrangement some blue pixels (903) in a conventional Bayer pattern sensor are substituted with "change detection" (M) pixels 905 according to the SNBT arrangement, 5 whilst retaining full resolution for red 901 and green 902. This pixel selection is made because human vision has its lowest acuity in blue. Because dedicated motion pixels do not require a colour filter, the reduction in optical efficiency that would result from the greater number of active components at the pixel will not be as significant. The result for motion detection at each M pixel results in the reading of coloured pixels for the surrounding neighbourhood. 0 Spatial multiplexing provides an SNBT arrangement whereby more of the change detection logic could be built into the pixels, potentially giving a net performance advantage. Fig. 10 is a flow diagram showing a process 1000 for practicing the disclosed SNBT arrangements. The process 1000 commences with a step 1001 in which the photo detector 201 reads, in regard to incident light impinging upon the detector 201, a first sample value over a 5 first time period, where the first sample value may be a charge representing the incident light which has been detected over the first period. The process 1000 then follows an arrow 1002 to a step 1003. In the step 1003 the photo detector 201 reads, in regard to incident light impinging upon the detector 201, a second sample value over a second time period which incorporates the first time period, where the second sample value may be a charge 0 representing the incident light which has been detected over the second time period which incorporates the first time period. The process 1000 then follows an arrow 1004 to a step 1005. The step 1005 determines a threshold based upon shot noise associated with one of the first and the second sample values. The process 1000 then follows an arrow 1006 to a step 1007 which determines a difference between the first and the second sample values. The 25 process 1000 then follows an arrow 1008 to a step 1009 which, if the determined difference exceeds the threshold, outputs, from the image sensor, a value captured from the pixel cell and resets the pixel cell. INDUSTRIAL APPLICABILITY The arrangements described are applicable to the computer and data processing 30 industries and particularly to the image capture industry. The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. P000013 / 5478844 1 270711 -16 In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings. P000013 /5478844_1 270711

Claims (17)

1. An on-chip method for adaptively sampling a pixel value captured using an image sensor on the chip, the image sensor comprising a plurality of pixel cells, the method 5 comprising the steps of: reading a first sample value for a pixel cell of the sensor and maintaining the first sample value at the pixel cell, the first sample value representing an amount of light captured by the pixel cell over a first time period; determining a second sample value for the pixel cell over a second time period 0 incorporating the first time period, wherein the second sample value represents an amount of light captured by the pixel cell over the second time period which includes the amount of light captured over the first time period; determining a threshold based on a photon shot noise value associated with one of the first and the second sample values; 5 determining a difference between the first and the second sample values; and outputting, from the image sensor, a value captured from the pixel cell and resetting the pixel cell if the determined difference exceeds the determined threshold.

2. A method according to claim 1, wherein the pixel cell comprises a photo detector and !0 an analogue memory element, and the step of determining a difference between the first and the second samples comprises determining a difference between a value at the photo detector and a value at the memory element.

3. A method according to claim 2, wherein the step of outputting a value from the pixel 25 cell comprises outputting the value at the photo detector or outputting the value at the memory element.

4. A method according to claim 2, wherein either: the first sample value is collected by the photo detector and transferred from the photo 30 detector to the memory element; or the first sample value is collected by the photo detector and a snapshot of the first sample value is transferred from the photo detector to the memory element leaving the first sample value at the photo detector intact. P0000 13 / 5478844 1 270711 -18

5. A method according to claim 1, wherein the outputting step can be adjusted to control motion detection sensitivity. 5

6. A method according to claim 1, wherein the threshold is proportional to the signal to noise ratio of one of the first and the second sample values.

7. A chip-based image sensor comprising an array of pixel cells, at least some of which comprise: ) a photo detector for converting incident light over a first time period to a first sample value, and for converting incident light over a second time period incorporating the first time period to a second sample value, wherein the second sample value includes the sample value generated over the first time period; an analogue memory element; 5 a transfer mechanism for transferring charge from the photo detector to the memory element to thereby establish respective sample values at the photo detector and the memory element; and a change detection module comprising: means for determining from the sample values at the photo detector and the J memory element if a change of incident light intensity has occurred at the photo detector during the second time period; and means for, if a change of intensity is detected, controlling outputting of a value from the pixel cell. 5

8. An image sensor according to claim 7, wherein the means for determining if a change of incident light intensity has occurred comprises: means for determining a difference between the first sample value and the second sample value; and means for determining a threshold based on a photon shot noise associated with one of 0 the first sample value and the second sample value; wherein a change of incident light intensity has occurred if the difference exceeds the threshold. P000013/5478844_1 270711 -19

9. An on-chip method for adaptively sampling a pixel value captured using an image sensor on the chip, the image sensor comprising a plurality of pixel cells, the method comprising the steps of: reading a first sample value for a pixel cell of the sensor and maintaining the first sample value at the pixel cell, the first sample value representing an amount of photo electrons integrated by the pixel cell over a first time period; determining a second sample value for the pixel cell over a second time period, wherein the second sample value represents an amount of photo-electrons integrated by the pixel cell over the second time period of equal or shorter duration to said first time period; determining a light intensity for each of the first and second time periods; determining a threshold based on a photon shot noise estimate associated with the light intensity of either the first or second time periods; determining a difference between the light intensity for the first and the second time periods; and 5 outputting, from the image sensor, a value captured from the pixel cell and resetting the pixel cell if the determined difference exceeds the determined threshold.

10. A method according to claim 9 where the first and second time period have a common end point.

11. A method according to claim 9 where the second time period starts at an end of the ) first time period.

12. A method according to claim 9, wherein the pixel cell comprises a photo detector and an analogue memory element, said method further comprising the step of transferring a value from the photo-diode to the analogue memory element if the determined difference does not exceed the determined threshold. 5

13. A method according to claim 12 where the value transferred to the analogue memory element is added to an existing value at the analogue memory element. P000013 / 5478844 1 270711 -20

14. A method according to claim 13 wherein the step of determining a second sample value comprises determining a difference between a value at the photo detector and a value at the memory element.

15. A method according to claim 9 wherein the first sample value is stored digitally in a 5 memory array on the sensor and the second value is determined by reading a value on a photodiode in a pixel array, said method further comprising the step of adding a digitised value from the photo-diode to the digital memory element and resetting the photo-diode if the determined difference does not exceed the determined threshold, D

16. An on-chip method for adaptively sampling a pixel value, substantially as described herein with reference to any one of the embodiments, as that embodiment is shown in the accompanying drawings.

17. A chip-based image sensor comprising an array of pixel cells, substantially as 5 described herein with reference to any one of the embodiments, as that embodiment is shown in the accompanying drawings. DATED this twenty-eighth Day of July, 2011 CANON KABUSHIKI KAISHA 0 Patent Attorneys for the Applicant SPRUSON&FERGUSON P000013 / 5478844_1 270711