JP2018042139A - Imaging element, operation method of imaging element, imaging apparatus, and electronic equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accomplish arithmetic other than processing essential for imaging without adding another memory or arithmetic circuit to an imaging element.SOLUTION: A part of a plurality of analog-digital converters carries out analog-digital conversion of a pixel signal constituting a low resolution image of a resolution lower than that of a pixel array, and another part other than the part of the analog-digital converters carries out arithmetic processing using the pixel signal constituting the low resolution image. This disclosure can be applied to an imaging element.SELECTED DRAWING: Figure 5

Description

  The present disclosure relates to an image sensor and an operation method of the image sensor, an image pickup apparatus, and an electronic apparatus, and in particular, can perform computations other than processing essential for imaging without newly adding a memory or an arithmetic circuit. The present invention relates to an imaging element, an imaging element, an operation method of the imaging element, an imaging apparatus, and an electronic device.

  Among captured images, an object region detection method called an inter-frame difference method that detects an object region by taking a difference between a past image and an input image is widely used as a general technique.

  In view of this, a method has been proposed in which this technique is used to have a memory for each pixel in order to realize an inter-frame difference in the image sensor and to perform an analog value difference process (see Non-Patent Document 1).

  In addition, a technique called a background difference method for detecting an object region by taking a difference between a background image and an input image has been proposed (see Non-Patent Document 2).

  Furthermore, a technique for removing noise by taking a weighted average of a plurality of images in the time direction has been proposed.

S.Ma, J, Chen JSSCC1999, A Singl-Chip CMOS APS Camera with Direct Frame Difference Chris Stauffer, W.E.L Grimson, Adaptive background mixture models for real-time tracking

  However, in the technique described in Non-Patent Document 1, the sensor is increased in size by having a memory for each pixel. Moreover, there is a possibility that sufficient detection performance cannot be exhibited with a simple difference between frames.

  Further, in order to realize the technique described in Non-Patent Document 2, it is necessary to secure a frame memory that can be read back and written back after background update processing, and the circuit becomes complicated if the function is realized by an analog circuit. Therefore, it is generally held in a digital frame memory, but the area increases and power consumption increases.

  Furthermore, since the above-described noise removal method similarly needs to be written back to the frame memory, it is also complicated when implemented by an analog circuit, and it is necessary to secure a digital frame memory separately, which increases power consumption and area. .

  The present disclosure has been made in view of such a situation. In particular, the present disclosure can realize operations other than processing essential for imaging without newly adding a memory or an arithmetic circuit.

  An imaging device according to one aspect of the present disclosure includes a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array, and each column in which the plurality of pixels are arranged A plurality of analog-to-digital converters that convert the pixel signals from analog signals to digital signals, and a portion of the plurality of analog-to-digital converters has a lower resolution than the resolution of the pixel array The pixel signals constituting the image are analog-to-digital converted, and at least the analog-digital conversion unit other than the part is an image sensor that performs arithmetic processing using the pixel signals constituting the low-resolution image.

  The analog-to-digital conversion unit includes a storage unit that stores the result of the arithmetic processing, and at least the analog-to-digital conversion unit other than the part has a low resolution for a plurality of frames having different timings. Pixel signals constituting the image can be stored.

  When a pixel signal of a new low-resolution image is supplied to at least the analog-digital conversion unit other than the part, the pixel signal stored in the storage unit of the adjacent analog-digital conversion unit is read, The pixel signals constituting the low-resolution images for a plurality of frames having different timings can be stored by shifting and storing them in the storage unit.

  Pixels stored in its own storage unit every predetermined number of frames when a pixel signal of a new low-resolution image is supplied to at least the analog-digital conversion unit other than the part By sequentially shifting and storing the signals in the storage units of the analog-digital conversion units adjacent to each other, it is possible to store the pixel signals constituting the low-resolution images for a plurality of frames having different timings.

  At least the analog-digital conversion unit other than the part executes arithmetic processing for obtaining inter-frame difference images of pixel signals constituting the low-resolution image corresponding to a plurality of frames having different timings, which are stored in the storage unit. You can make it.

  At least the analog-to-digital conversion unit other than the part reads out pixel signals constituting the low-resolution image for one frame at a predetermined timing stored in the storage unit, The inter-frame difference image can be obtained and overwritten in the storage unit.

  When a pixel signal of a new low-resolution image is supplied to at least the analog-digital conversion unit other than the part, the adjacent analog-digital conversion unit stores the pixel signal stored in its storage unit The ring buffer can be configured by storing the pixel signals constituting the low-resolution images of a predetermined number of a plurality of frames having different timings by shifting and storing them in the storage unit.

  A calculation unit that calculates a difference image between a pixel signal constituting the low-resolution image of a predetermined number of frames having different timings stored in the ring buffer and a pixel signal of the new low-resolution image; You can make it.

  The arithmetic unit includes an average value in each pixel of pixel signals constituting the low resolution image from the oldest frame to a predetermined oldest frame stored in the ring buffer, and pixels of the new low resolution image The absolute difference value with respect to the signal is calculated, a pixel having the absolute difference value larger than a predetermined threshold is defined as a first pixel value, and a pixel having the absolute difference value smaller than the predetermined threshold is defined as a second pixel value. A difference image composed of binary images can be calculated.

  The arithmetic unit includes a weighted average value corresponding to the timing of the pixel signal constituting the low-resolution image from the oldest frame stored in the ring buffer to the predetermined oldest frame, and the new A difference absolute value with a pixel signal of a low-resolution image is calculated, a pixel having the difference absolute value larger than a predetermined threshold is set as a first pixel value, and a pixel having the difference absolute value smaller than the predetermined threshold is set as a second pixel. It is possible to calculate a difference image consisting of a binary image having a pixel value of.

  The arithmetic unit includes a difference between a pixel signal constituting the low-resolution image from the oldest frame stored in the ring buffer to a predetermined oldest frame and a pixel signal of the new low-resolution image. An absolute value is calculated, and a pixel whose each difference absolute value is larger than a predetermined threshold is set as a first pixel value, and a pixel whose one of the difference absolute values is smaller than a predetermined threshold is set as a second pixel. It is possible to calculate a difference image consisting of a binary image having a pixel value of.

  A calculation unit that calculates an average value of pixel values of pixels constituting the low-resolution image of a predetermined number of frames at different timings stored in the ring buffer may be further included.

  The calculation unit is configured to calculate a weighted average value corresponding to the timing of pixel values of pixels constituting the low-resolution image of a plurality of frames at a predetermined number of times stored in the ring buffer. Can be.

  The low-resolution image may be an image composed of any one of an average value, a representative value, and a median for each of a plurality of pixel groups of the pixels constituting the pixel array.

  An operation method of an imaging element according to one aspect of the present disclosure includes a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array, and the plurality of pixels are arranged. A plurality of analog-to-digital converters for converting the pixel signals from analog signals to digital signals, and a part of the plurality of analog-to-digital converters, An analog-digital conversion is performed on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-digital conversion unit other than the part uses the pixel signal constituting the low-resolution image. This is an operation method of the image sensor that executes

  An imaging apparatus according to an aspect of the present disclosure includes a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array, and each column in which the plurality of pixels are arranged A plurality of analog-to-digital converters that convert the pixel signals from analog signals to digital signals, and a portion of the plurality of analog-to-digital converters has a lower resolution than the resolution of the pixel array The pixel signal constituting the image is analog-to-digital converted, and at least the analog-to-digital conversion unit other than the part is an imaging device that performs arithmetic processing using the pixel signal constituting the low-resolution image.

  An electronic apparatus according to an aspect of the present disclosure includes a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array, and each column in which the plurality of pixels are arranged A plurality of analog-to-digital converters that convert the pixel signals from analog signals to digital signals, and a portion of the plurality of analog-to-digital converters has a lower resolution than the resolution of the pixel array The pixel signals constituting the image are analog-to-digital converted, and at least the analog-digital conversion unit other than the part is an electronic device that executes arithmetic processing using the pixel signals constituting the low-resolution image.

  In one aspect of the present disclosure, a pixel signal corresponding to the amount of incident light is generated by a pixel array in which a plurality of pixels are arranged in a two-dimensional array, and each column in which the plurality of pixels are arranged The plurality of analog-to-digital converters convert the pixel signal from an analog signal to a digital signal, and a part of the plurality of analog-to-digital converters generates a low-resolution image having a resolution lower than that of the pixel array. The constituent pixel signals are subjected to analog-to-digital conversion, and arithmetic processing using the pixel signals constituting the low-resolution image is executed by at least the analog-to-digital conversion unit other than the part.

  According to one aspect of the present disclosure, by using a column AD circuit that is not used for image processing, computation other than processing that is essential for imaging is realized without newly adding a memory or an arithmetic circuit. It becomes possible.

It is a figure explaining the structural example of the conventional image pick-up element. It is a figure explaining the structural example of the conventional image pick-up element. It is a figure explaining the example of composition of the image sensor to which the art of this indication is applied. It is a figure explaining the detailed structural example of the column AD circuit of FIG. It is a figure explaining the detailed operation example at the time of imaging of the column AD circuit of FIG. It is a figure explaining the detailed operation example at the time of the calculation of the column AD circuit of FIG. It is a figure explaining the operation example of the column AD circuit of FIG. It is a flowchart explaining the difference calculation process between frames by the image pick-up element of FIG. It is a figure explaining the shift process of the pixel signal per frame using the memory in a column AD circuit. It is a figure explaining the shift process of the pixel signal for every other frame using the memory in a column AD circuit. It is a flowchart explaining the shift process of the pixel signal every N frames using the memory in a column AD circuit. It is a block diagram explaining the example of a structure of the arithmetic circuit which implement | achieves a background difference calculation when comprising a ring buffer using the memory in a column AD circuit. It is a block diagram explaining the background difference calculation when comprising a ring buffer using the memory in a column AD circuit. It is a block diagram explaining the advantage compared with the conventional structure of the background difference calculation when comprising a ring buffer using the memory in a column AD circuit. It is a flowchart explaining the background difference calculation process when comprising a ring buffer using the memory in a column AD circuit. It is a block diagram explaining the 1st application example of the arithmetic circuit which implement | achieves a background difference calculation when comprising a ring buffer using the memory in a column AD circuit. FIG. 17 is a flowchart for explaining background difference calculation processing when a ring buffer is configured using a memory in a column AD circuit by the image sensor of FIG. 16. FIG. 11 is a block diagram illustrating a configuration example of an arithmetic circuit that implements noise removal processing as a second application example when a ring buffer is configured using a memory in a column AD circuit. FIG. 19 is a flowchart for describing noise removal processing when a ring buffer is configured using a memory in a column AD circuit by the image sensor of FIG. 18. FIG. It is a figure explaining the modification in a column AD circuit. It is a figure explaining the example of the arithmetic circuit realizable by the column AD circuit of FIG. It is a figure explaining the arithmetic processing implement | achieved by combining the arithmetic circuit realizable by the column AD circuit of FIG. FIG. 21 is a diagram illustrating a configuration example in the case of connecting a horizontal transfer line in the column AD circuit of FIG. 20. It is a block diagram which shows the structural example of the imaging device as an electronic device to which the camera module of this indication is applied. It is a figure explaining the usage example of the camera module to which the technique of this indication is applied.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

<Configuration example of a general imaging device>
(Configuration example of image sensor for extracting foreground image and background image)
There is a conventional technique (interframe difference method) in which a moving object region is detected by taking a difference between a past image and an input image. Therefore, for example, as shown in Non-Patent Document 1 described above, a method has been proposed in which a memory is provided for each pixel in an image sensor (image sensor) and a difference process is performed by an analog circuit. ing.

  Further, as shown in Non-Patent Document 2, a technique has been proposed in which a technique called a background difference method for detecting an object region by taking a difference between a background image and an input image is realized in an image sensor. In this case, since the configuration of the memory for storing the background image is essential, the device configuration becomes large, and every time the input image is supplied, the background image is read out to be updated, and the updated background image Is overwritten, the operation becomes complicated and the apparatus configuration becomes complicated.

  More specifically, for example, as shown in FIG. 1, the imaging device 1 described above includes an AD (Analog Digital) conversion unit 11, an image reduction unit 12, a digital frame memory 13, a background update algorithm processing unit 14, a subtraction. The unit 15, the absolute value conversion unit 16, and the threshold processing unit 17 are configured.

  The AD conversion unit 11 converts a pixel signal composed of an analog signal supplied from a pixel array (not shown) into a digital signal and outputs the digital signal to the image reduction unit 12.

  The image reduction unit 12 reduces the image so as to reduce the resolution of the pixel signal converted into the digital signal, stores the image in the digital frame memory 13, and outputs the image to the subtraction unit 15.

  The digital frame memory 13 stores the reduced image signal supplied from the image reduction unit 12, supplies the reduced image signal to the background update algorithm processing unit 14, and overwrites the processing result of the background update algorithm processing unit 14. .

  The background update algorithm processing unit 14 updates the background image using the reduced image that is the input image supplied from the image reducing unit 12 and writes it back to the digital frame memory 13. That is, the background image is sequentially updated and held in the digital frame memory 13 by this processing.

  The subtraction unit 15 obtains a difference between the reduced image supplied from the image reduction unit 12 and the background image stored in the digital frame memory 13 and subjected to the background update algorithm processing by the background update algorithm processing unit 14. The foreground image is supplied to the absolute value converting unit 16.

  The absolute value converting unit 16 obtains the absolute value of the difference value between the reduced image that is the input image and the background image, and configures the difference image.

  The threshold processing unit 17 is composed of a moving image by binarization processing in which the pixel value of pixels larger than the threshold is set to 1 and the other pixels are set to 0 by comparing the difference value val and the threshold th in the difference image. Estimate the foreground area.

  Next, the operation of the image sensor of FIG. 1 will be described.

  First, the AD converter 11 converts an analog image signal into a digital signal. Next, the image reduction unit 12 reduces the resolution of the image signal converted into the digital signal and stores it in the digital frame memory 13.

  Next, the background update algorithm processing unit 14 repeats the process of updating and writing back the background image using the image signals sequentially stored in the digital frame memory 13.

  The subtraction unit 15 obtains a difference between the reduced image and the background image stored in the digital frame memory 13 and outputs the difference to the absolute value conversion unit 16.

  The absolute value converting unit 16 supplies the threshold image processing unit 17 with a difference image composed of the absolute difference between the reduced image that is the input image and the background image read from the digital frame memory 13.

  The threshold value processing unit 17 compares the pixel value val of each pixel of this difference image with the threshold value th to obtain a binary image in which the pixel value of a large pixel is 1 and the pixel values of other pixels are 0. As a result, an image for estimating the foreground area composed of the moving image is output.

  In the case of the image pickup device 1 of FIG. 1, the digital frame memory 13 is indispensable for sequentially updating the background image, so that the apparatus becomes large.

  Further, the background update algorithm processing unit 14 needs to read the reduced image stored in the digital frame memory 13, update the background image, and then write it back to the digital frame memory 13, which increases power consumption. End up.

(Configuration example of image sensor that improves image quality)
Furthermore, an image sensor that improves image quality by weighted averaging of a plurality of images in the time direction has been proposed, such as the image sensor shown in FIG. 2 that have the same functions as those of the image sensor 1 in FIG. 1 are denoted by the same reference numerals and names, and description thereof will be omitted as appropriate.

  2 differs from the image sensor 1 in FIG. 1 in that a weighted average is used instead of the background update algorithm processing unit 14, the subtraction unit 15, the absolute value conversion unit 16, and the threshold value processing unit 17. This is the point that the conversion unit 41 is provided.

  The weighted averaging unit 41 assigns a large weight to the input image, which is the image processed by the weighted averaging unit 41 stored so far in the digital frame memory 13 and the reduced image that is the input image. By performing weighted averaging as described above, noise is removed from the image and output. Further, the weighted averaging unit 41 stores the weighted averaged image in the digital frame memory 13 and repeats the same processing to gradually remove noise.

  However, in this case as well, the weighted averaging unit 41 reads the image that is the previous processing result from the digital frame memory 13, performs the weighted averaging process using the input image, and writes it again in the digital frame memory 13. It is necessary to return, and power consumption becomes large. Further, since the digital frame memory 13 is an essential configuration, it is necessary to secure an installation area, and the apparatus becomes large.

<Example of Configuration of Imaging Device of Present Disclosure>
Therefore, the imaging device according to the present disclosure has a function of generating pixel signals using all column AD circuits provided for each column for all pixels when imaging with high resolution. In addition, the imaging device of the present disclosure is not intended to capture an image. For example, when functioning as a sensor that detects each region of a background image and a foreground image, the resolution is reduced. Using the processed image, other arithmetic processing is executed using a column AD circuit that is not used for image processing caused by lowering the resolution. By realizing such an operation, a function for capturing a high-resolution image and a function as a sensor can be switched without adding a new configuration to the configuration of the image sensor.

  More specifically, FIG. 3 illustrates a configuration example of the imaging element of the present disclosure. 3 includes a pixel array 71 having pixels arranged in two dimensions of m rows × n columns, and column AD circuits 72-0 to 72- () provided for each of n columns of vertical transfer lines. n-1) and an arithmetic circuit 73. In the following description, when it is not necessary to particularly distinguish each of the column AD circuits 72-0 to 72- (n-1), they are simply referred to as the column AD circuit 72, and the other configurations are also referred to similarly. To do.

  The pixel array 71 includes a photodiode for each pixel, generates a pixel signal corresponding to the amount of incident light, and supplies the pixel signal to the column AD circuit 72 of each column via a vertical transfer line.

  A column AD (Analog Digital) circuit 72 converts a pixel signal composed of an analog signal supplied via a vertical transfer line into a digital signal and supplies the digital signal to the arithmetic circuit 73.

  The arithmetic circuit 73 performs various arithmetic processes on the pixel signal composed of the digital signals supplied from the column AD circuits 72-0 to 72- (n-1), and outputs an arithmetic result.

  In addition, the pixel array 71 captures a high-resolution image signal using all the pixels, and further operates differently depending on the operation mode. The operation modes include an image capturing mode for capturing a high resolution image using all pixels and a sensor mode using information on a low resolution image. For example, in the sensor mode, a difference image that identifies a foreground region in which a foreground image made up of moving images in the image is captured and a background region in which a background image is captured is output.

  More specifically, in the case of the sensor mode, the pixel array 71 generates a low-resolution image with respect to the resolution of all pixels by pixel addition or thinning, and sends the low-resolution pixel signal to the corresponding column AD circuit 72. Supply. FIG. 3 shows a configuration when a low-resolution image of p pixels including a predetermined number of pixel groups 71a-1 to 71a-p among the pixels constituting the pixel array 71 is captured.

  Therefore, in the sensor mode, a column AD circuit 72 that is not used for image processing is generated due to the use of a low-resolution image. Therefore, a difference is obtained by using the column AD circuit 72 that is not used for image processing. Arithmetic processing other than imaging processing, such as generating an image, is executed.

  More specifically, as shown in FIG. 3, for example, pixel signals that treat the pixel array 71 as a pixel in a pixel group 71 a-1 including a predetermined number of pixels are output from the column AD circuits 72-0 to 72-. 2 and the pixel signals that treat the pixel group 71a-2 as one pixel are processed by the column AD circuits 72-3 to 72-5. A pixel signal that treats the pixel group 71a- (p-1) as one pixel is processed by the column AD circuits 72- (n-6) to 72- (n-4), and the pixel group 71a-p is treated as one pixel. Are processed in each of the column AD circuits 72- (n-3) to 72- (n-1).

  The column AD circuits 72-0 to 72-n include calculators 72a-0 to 72a-n, respectively, but sequentially shift the calculation results to the adjacent column AD circuits 72 in units of one frame.

  That is, for example, a pixel signal of the 0th frame is supplied to the column AD circuit 72-0 as a pixel signal with the pixel group 71a-1 as one pixel, and a pixel signal with the pixel group 71a-2 as one pixel is A pixel signal that is supplied to the column AD circuit 72-3, and the pixel group 71a- (p-1) as one pixel is supplied to the column AD circuit 72- (n-6), and the pixel group 71a-p Is supplied to the column AD circuit 72- (n-3), each pixel signal is converted from an analog signal to a digital signal and held.

  Before the pixel signal of the next first frame is supplied, the column AD circuits 72-0, 72-3,... 72- (n-6) and 72- (n-3) are held respectively. Are shifted to the column AD circuits 72-1, 72-4,... 72- (n-5), 72- (n-2) on the right side.

  When a new pixel signal of the first frame is supplied, a pixel signal with the pixel group 71a-1 as one pixel is supplied to the column AD circuit 72-0, and the pixel group 71a-2 is set as one pixel. The pixel signal is supplied to the column AD circuit 72-3,..., And the pixel signal having the pixel group 71a- (p-1) as one pixel is supplied to the column AD circuit 72- (n-6), and the pixel A pixel signal having the group 71a-p as one pixel is supplied to the column AD circuit 72- (n-3).

  That is, generally, the column AD circuit 72 converts a pixel signal constituting an image from an analog signal to a digital signal in units of rows and outputs the converted signal. However, in the pixel array of the image sensor 51 of the present disclosure, in the sensor mode, the pixel group 71a is regarded as one pixel, so that a low-resolution image is configured and a memory including a latch circuit included in the plurality of column AD circuits 72 and the like. In addition, a plurality of frames of the low-resolution image are stored in place of the frame memory and used for buffering and computation.

  In addition, as a process of regarding the pixel group 71a as one pixel, for example, the pixel value of each pixel constituting the pixel group 71a may be added to obtain an average, or any of the representative pixels The pixel value may be used, or the median of the pixel values of the pixels constituting the pixel group 71a may be used.

  In the image sensor 51 of FIG. 3, an example is shown in which low resolution image signals are accumulated for three frames by repeating similar processing. When a pixel signal for one frame with a low resolution is newly supplied, the oldest pixel signal is discarded and the latest three frames of pixel signals are accumulated.

(Detailed configuration example of column AD circuit)
Here, a detailed configuration example of the column AD circuit 72 will be described with reference to FIG.

  As shown in FIG. 3, each of the column AD circuits 72 includes an arithmetic unit 72a. More specifically, the arithmetic unit 72a includes, for example, a comparator 91, a combinational circuit 92, and The memory 93 is used.

  More specifically, as shown in FIG. 4, the column AD circuits 72-0 to 72-2 include comparators 91-0 to 91-2, combinational circuits 92-0 to 92-2, and a memory 93-, respectively. 0 to 93-2. The comparator 91 compares the pixel signal supplied from the vertical transfer line with the reference voltage, and supplies the comparison result to the combinational circuit 92. The combinational circuits 92-0 to 92-2 realize AD conversion processing in the image capturing mode. In the sensor mode, various circuits combined to function as an AD circuit are appropriately used. Used to perform various operations and supply them to the memories 93-0 to 93-2. The memories 93-0 to 93-2 are composed of, for example, a latch circuit and temporarily store the calculation results.

  The combinational circuits 92-0 to 92-2 include, for example, selectors 101-0 to 101-2, 102-0 to 102-2, sign inversion units 103-0 to 103-2, and selectors 104-0 to 104-2. , Adders 105-0 to 105-2, absolute value conversion units 106-0 to 106-2, and selectors 107-0 to 107-2.

  The selectors 101 and 102, the sign inversion unit 103, the selector 104, the adder 105, the absolute value conversion unit 106, and the selector 107 of the combinational circuit 92 are a so-called counter or the like together with the comparator 91 and the memory 93 in the image capturing mode. It realizes equivalent functions and realizes AD conversion. In the sensor mode, the selectors 101 and 102, the sign inverting unit 103, the selector 104, the adder 105, the absolute value converting unit 106, and the selector 107 of the combinational circuit 92 are adjacent to the column AD circuit on the left side in the drawing. 72 pixel signals are sequentially acquired and stored in its own memory 93, or a predetermined calculation is performed together with a value stored in the memory 93 and written back to the memory 93. Further, the value stored in the memory 93 is supplied to the column AD circuit 72 adjacent on the right side.

  In FIG. 4, the selector 101 outputs 0 or 1 based on the comparison result of the comparator 91 and supplies it to the selector 102. The selector 102 selectively outputs either the value in the memory 93 of the column AD circuit 72 on the left side or the value supplied from the selector 101 to the adder 105. The sign inversion unit 103 inverts the sign of the value stored in the memory 93 and supplies it to the selector 104. The selector 104 selectively outputs the value of its own memory 93, the value of its own memory 93 from the sign inverting unit 103, the sign of which is inverted, or 0 to the adder 105. Adder 105 adds the value output from selector 102 and the value output from selector 104 and outputs the result to absolute value converting section 106 and selector 107. The absolute value converting unit 106 obtains the absolute value of the value supplied from the adder 105 and supplies it to the selector 107. The selector 107 selectively stores either the value supplied from the adder 105 or the value supplied from the absolute value conversion unit 101 in the memory 93.

  With such a configuration, the column AD circuit 72 transfers the value stored in the AD conversion circuit or the memory 93 of the column AD 72 on the left side, or is stored in the memory 93 with the transferred value. It functions as an arithmetic circuit that adds a certain value.

  That is, when functioning as an AD conversion circuit, the selector 101 as shown in the left part of FIG. 5 based on the comparison result of the comparator 91 as shown by the thick lines in the left part and the right part of FIG. 1 selectively outputs 1 or the selector 101 selectively outputs 0 as shown in the right part of FIG. Then, the selector 102 supplies the output value of the selector 101 to the adder 105. The selector 104 reads out the value stored in the memory 93 and outputs it to the adder 105. The adder 105 adds the values supplied from the selectors 102 and 104, respectively, and outputs the result to the selector 107 and the absolute value converting unit 106. The selector 107 stores the value output from the adder 105 in the memory 93. Thereafter, the same processing is repeated.

  Further, when functioning as an arithmetic circuit, when reading the value of the memory 93 of the column AD circuit 72 on the left side to its own memory 93, as shown in the left part of FIG. 72 is selected and supplied to the adder 105, and the selector 104 supplies 0 to the adder 105. Thereby, the adder 105 supplies the value of the memory 93 of the column AD circuit 72 on the left side to the absolute value conversion unit 106 and the selector 107 substantially. The selector 107 stores the value stored in the memory 93 of the left column AD circuit 72 supplied from the adder 105 in its own memory 93.

  Furthermore, when functioning as another arithmetic circuit, when the value stored in its own memory 93 and the value of the memory 93 of the column AD circuit 72 on the left are added and read back to its own memory 93, FIG. 6, the selector 102 selects the value of the memory 93 of the column AD circuit 72 on the left side and supplies it to the adder 105, and the sign inversion unit 103 is stored in its own memory 93. The selector 104 inverts the sign of the existing value, and supplies the adder 105 with the value whose sign has been inverted by the sign inversion unit 103. As a result, the adder 105 adds the value of the memory 93 of the column AD circuit 72 on the left to the value stored in its own memory 93 and supplies the sum to the absolute value conversion unit 106 and the selector 107. The absolute value converting unit 106 obtains the absolute value of the addition result by the adder 105 and supplies it to the selector 107. The selector 107 writes the addition result supplied from the absolute value converting unit 106 back into its own memory 93 and stores it.

(Inter-frame difference calculation processing)
Next, referring to the operation explanatory diagram of FIG. 7 and the flowchart of FIG. 8, the pixel signal is sequentially calculated, the calculation result is transferred to the column AD circuit on the right, and the difference consisting of the absolute value of the difference between frames. A process for realizing the inter-frame difference process for outputting an image will be described.

  In FIG. 7, the processing in the column AD circuit 72-0 in the 0th column (column0) and the column AD circuit 72-1 in the 1st column (column1) will be described, but the same applies to the subsequent columns. Are processed. Here, each combinational circuit 92 in the column AD circuit 72 performs AD conversion processing, calculates and outputs the absolute difference value of the sequentially supplied pixel values, and stores it in the memory 93. And

  That is, in step S11 (FIG. 8), as shown in the upper left part of FIG. 7, the combinational circuit 92-0 of the column AD circuit 72-0 receives the 0th frame supplied via the vertical transfer line. The pixel signal is AD converted and stored in the memory 93-0 as a pixel value val0 composed of a digital signal.

  In step S12, as shown in the middle left part of FIG. 7, the combinational circuit 92-1 of the column AD circuit 72-1 is connected to the memory of the column AD circuit 72-0 adjacent to the left side via the wiring L-0. The pixel value val0 is read from 93-0 and stored in the memory 93-1. This operation corresponds to the operation on the left side of FIG. 6 described above.

  In step S13, as shown in the lower left part of FIG. 7, the combinational circuit 92-0 of the column AD circuit 72-0 AD-converts the pixel signal of the first frame supplied via the vertical transfer line. Then, it is stored in the memory 93-0 as a pixel value val1 composed of a digital signal.

  In step S14, as shown in the upper right section of FIG. 7, the combinational circuit 92-1 of the column AD circuit 72-1 is connected to the memory of the column AD circuit 72-0 adjacent to the left side via the wiring L-0. The pixel value val1 is read from 93-0, the pixel value val0 stored in the memory 93-1 is read, the difference absolute value is calculated, and the difference absolute value ad01 (= (abs-diff01) = | val1-val0 |) is written back to the memory 93-1. This operation corresponds to the operation on the right side of FIG. 6 described above.

  In step S15, as shown in the middle right part of FIG. 7, the calculation result ad01, which is the absolute difference value stored in the memory 93-1, is output as the interframe difference calculation result.

  In step S16, as shown in the lower right part of FIG. 7, the combinational circuit 92-1 of the column AD circuit 72-1 is connected to the memory of the column AD circuit 72-0 adjacent to the left side via the wiring L-0. The pixel value val1 is read from 93-0 and stored in the memory 93-1. That is, the pixel value val1 is shifted by one column. This operation corresponds to the operation on the left side of FIG. 6 described above.

  Thereafter, when the pixel signals are sequentially supplied, the processes after step S13 are sequentially repeated, and an image composed of the pixel signals of the inter-frame differences is sequentially output.

  By repeating the above processing, it is possible to sequentially output a difference image composed of absolute difference values between pixels of preceding and following frames.

  That is, since it is possible to obtain the inter-frame difference image without newly adding the configuration of the image sensor while the image is a low-resolution image with respect to the resolution of the pixel array 71, for example, a moving image region with a low resolution It is possible to generate a difference image that distinguishes a foreground region that is not a moving image, that is, a background region that does not move. Also, until the intruder is detected as a foreground area in a monitoring image or the like by detecting the foreground area that is a moving image area, the sensor operates in the sensor mode, and only when the intruder is detected, the image capturing mode is set. It is possible to switch and capture a high-resolution image. At this time, in the sensor mode, only a part of the column AD circuit 72 is used, and only when the intruder is detected, the entire column AD circuit 72 is used with high resolution. Until the person is imaged, the sensor mode with relatively low power consumption can be used, and the intruder can be reliably imaged with high resolution. Therefore, the necessary intruder image can be obtained with high resolution while reducing power consumption. Imaging can be performed. In addition, since it is not necessary to add a frame memory or the like, the apparatus configuration can be reduced in size.

(Ring buffer)
In the above description, the example in which the inter-frame difference is sequentially obtained using the pixel signal whose resolution has been reduced has been described. However, the pixel signal is sequentially stored in the memory 93 in the column AD circuit 72, and the pixel signal of the next frame is further stored. A ring buffer between a plurality of frames may be configured by performing a process of only shifting to an adjacent column AD circuit 72 each time.

  The basic operation can be realized by simply repeating the operation of shifting the pixel signal to the adjacent column AD circuit 72 without executing the above-described difference calculation.

  That is, as shown in the upper left part of FIG. 9, first, the pixel signal “1” of the frame F1, which is the first frame, is AD-converted by the column AD circuit 72-0 and stored in the memory 93-0.

  Next, when the pixel signal of the frame F2, which is the second frame, is supplied, the pixel signal “1” stored in the memory 93-0 is adjacent to the column AD as shown in the upper center of FIG. The pixel signal “2” of the new frame F2 is stored in the memory 93-0 of the column AD circuit 72-0 while being shifted to the memory 93-1 of the circuit 72-1.

  Further, when the pixel signal of the frame F3 which is the third frame is supplied, the column AD circuit in which the pixel signal “1” stored in the memory 93-1 is adjacent as shown in the upper right part of FIG. The pixel signal “2” that has been shifted to the memory 93-2 of 72-2 and stored in the memory 93-0 is shifted to the memory 93-1 of the adjacent column AD circuit 72-1. Then, the pixel signal “3” of the new frame F3 is stored in the memory 93-0 of the column AD circuit 72-0.

  When the pixel signal of the frame F4 that is the fourth frame is supplied, the pixel signal “1” stored in the memory 93-2 of the column AD circuit 72-2 is displayed as shown in the lower left part of FIG. ”Is discarded, and the pixel signal“ 2 ”stored in the memory 93-1 is shifted to the memory 93-2 of the adjacent column AD circuit 72-2, and the pixel signal“ 2 ”stored in the memory 93-0 is stored. 3 "is shifted to the memory 93-1 of the adjacent column AD circuit 72-1. Then, the pixel signal “4” of the new frame F4 is stored in the memory 93-0 of the column AD circuit 72-0.

  Further, when the pixel signal of the frame F5 which is the fifth frame is supplied, the pixel signal “2” stored in the memory 93-2 of the column AD circuit 72-2 as shown in the lower center of FIG. ”Is discarded, and the pixel signal“ 3 ”stored in the memory 93-1 is shifted to the memory 93-2 of the adjacent column AD circuit 72-2, and the pixel signal“ 3 ”stored in the memory 93-0 is stored. 4 "is shifted to the memory 93-1 of the adjacent column AD circuit 72-1. Then, the pixel signal “5” of the new frame F5 is stored in the memory 93-0 of the column AD circuit 72-0.

  When the pixel signal of the frame F6 that is the sixth frame is supplied, the pixel signal “3” stored in the memory 93-2 of the column AD circuit 72-2 is displayed as shown in the lower right part of FIG. ”Is discarded, and the pixel signal“ 4 ”stored in the memory 93-1 is shifted to the memory 93-2 of the adjacent column AD circuit 72-2, and the pixel signal“ 4 ”stored in the memory 93-0 is stored. 5 "is shifted to the memory 93-1 of the adjacent column AD circuit 72-1. Then, the pixel signal “6” of the new frame F6 is stored in the memory 93-0 of the column AD circuit 72-0.

  Thereafter, when a pixel signal is sequentially supplied for each frame, it is sequentially shifted to the memory 93 of the column AD circuit 72 adjacent in the right direction in the figure, and the pixel signal of the oldest frame is discarded. The pixel signals for the latest three frames are always accumulated, and as a result, a ring buffer can be realized.

(Ring buffer that buffers pixel signals every N frames)
In the above description, an example in which the ring buffer is configured by shifting the pixel signal to the memory 73 of the adjacent column AD circuit 72 every frame has been described. However, the pixel signal is shifted every N frames and the ring buffer is configured. May be configured.

  That is, as shown in the upper left part of FIG. 10, first, the pixel signal “1” of the frame F1, which is the first frame, is AD converted by the column AD circuit 72-0 and stored in the memory 93-0.

  Next, when the pixel signal of the frame F2, which is the second frame, is supplied, the pixel signal “1” stored in the memory 93-0 is adjacent to the column AD as shown in the upper center portion of FIG. The pixel signal “2” of the new frame F2 is stored in the memory 93-0 of the column AD circuit 72-0 while being shifted to the memory 93-1 of the circuit 72-1.

  Further, when the pixel signal of the frame F3 that is the third frame is supplied, the pixel signal “2” stored in the memory 93-0 is discarded as shown in the upper right part of FIG. Signal “3” is overwritten.

  When the pixel signal of the frame F4 that is the fourth frame is supplied, the column AD circuit in which the pixel signal “1” stored in the memory 93-1 is adjacent as shown in the lower left part of FIG. The pixel signal “3” that has been shifted to the memory 93-2 of 72-2 and stored in the memory 93-0 is shifted to the memory 93-1 of the adjacent column AD circuit 72-1. Then, the pixel signal “4” of the new frame F4 is overwritten and stored in the memory 93-0 of the column AD circuit 72-0.

  Further, when the pixel signal of the frame F5 which is the fifth frame is supplied, the pixel signal “4” stored in the memory 93-0 is discarded as shown in the lower center part of FIG. The pixel signal “5” of the correct frame F5 is stored in the memory 93-0 of the column AD circuit 72-0.

  When the pixel signal of the frame F6 that is the sixth frame is supplied, the pixel signal “5” stored in the memory 93-0 is discarded as shown in the lower right part of FIG. Is overwritten by the pixel signal “6” of the correct frame F6 and stored in the memory 93-0 of the column AD circuit 72-0.

  Subsequently, when pixel signals are sequentially supplied, the pixel signals of the oldest frame are shifted to the memory 93 of the column AD circuit 72 adjacent in the right direction in the figure every N frames (every other frame). Discarded, the pixel signals for the latest three frames are always accumulated every other frame, and as a result, a ring buffer can be realized.

(Shift processing for shifting the pixel signal to the column AD circuit adjacent to the right side every N frames)
Next, with reference to the flowchart of FIG. 11, a shift process for shifting the pixel signal to the column AD circuit adjacent on the right side every N frames will be described.

  In step S31, a counter n is initialized to 1 by a control unit (not shown).

  In step S32, a control unit (not shown) determines whether a pixel signal of a new frame has been supplied, and the same processing is repeated until it is transmitted.

  If a pixel signal of a new frame has been supplied in step S32, the process proceeds to step S33.

  In step S33, a control unit (not shown) determines whether or not the counter n is N + 1. For example, in the case of every N = 1 frame described with reference to FIG. 10, since the counter n = 1 in the first process and not 2 (= N + 1), the process proceeds to step S34.

  In step S34, the combinational circuit 92-0 performs AD conversion on the pixel signal and discards the pixel signal stored before AD conversion to the memory 93-0, and then overwrites and stores the pixel signal. Advances to step S35. In the first process, since there is no value, it is stored as it is.

  In step S35, the control unit (not shown) increments the counter n by 1, and the process returns to step S32.

  That is, until the counter n reaches N + 1, even if the processing of steps S32 to S35 is repeated and a pixel signal of a new frame is supplied, the pixel signal is AD-converted and the memory of the column AD circuit 72-0 The value of 93-0 continues to be overwritten.

  If the counter n becomes N + 1 in step S33, the process proceeds to step S36.

  In step S36, the combinational circuit 92-1 reads out the pixel signal from the memory 93-0 of the column AD circuit 72-0 adjacent on the left side in FIG. 4, and stores the pixel signal in the memory 93-1. Shift by minutes.

  In step S37, the combinational circuit 92-0 AD-converts the pixel signal and stores it in the memory 93-0, and the process proceeds to step S38.

  In step S38, the combinational circuit 92-0 determines whether or not the process is completed. If the process is not completed, the process returns to step S31, and the subsequent processes are repeated. Then, when it is determined in step S38 that the process is to be terminated, the process is terminated.

  That is, by such processing, the pixel signals are sequentially shifted to the memory 93 of the column AD circuit 72 on the right side every arbitrary N frames, and a ring buffer can be configured.

<Configuration example of arithmetic circuit for calculating background image using ring buffer>
In the above, an example in which a ring buffer is configured using the memories 93-0 to 93- (n-1) of the column AD circuits 72-0 to 72- (n-1) has been described. A configuration example of the arithmetic circuit 73 that uses this to calculate the difference image between the background image and the foreground image will be described.

  FIG. 12 shows a configuration example of an arithmetic circuit 73 that calculates a background image using the ring buffer including the memory 93 described above. In FIG. 12, the memory 93-0 has a buffer 110a for storing the pixel signal of the latest frame, and a buffer comprising memories 93-1 to 93- (n-1) for storing the pixel signals of the past frame. 110b, and the buffers 110a and 110b constitute a ring buffer 110.

  The arithmetic circuit 73 in FIG. 12 includes an addition unit 111, a gain amplifier 112, a subtraction unit 113, an absolute value conversion unit 114, and a threshold processing unit 115.

  The adder 111 stores the memory 93- (n−) which holds the pixel signals of the oldest frame from the pixel signal of the oldest frame out of the memories 93-0 to 93-N constituting the ring buffer 110, respectively. 1) to 93- (n-4) pixel signals are added and supplied to the gain amplifier 112.

  The gain amplifier 112 holds the pixel signal of the oldest frame from the pixel signal of the oldest frame to the memory 93- (n−1) among the memories 93-0 to 93-N constituting the ring buffer 110. ) Through 93- (n−4) are set to a gain of ¼ to obtain an average value of the pixel signals and output the result to the subtractor 113.

  The subtracting unit 113 averages the pixel signal of the memory 93-0 in which the pixel signal of the newest frame is held and the pixel signal of the oldest frame from the oldest frame pixel signal supplied from the gain amplifier 112 to the fourth frame. Is obtained and supplied to the absolute value conversion unit 114.

  The absolute value converting unit 114 obtains the absolute value of the pixel value that is the difference output from the subtracting unit 113 as the difference absolute value, and outputs it to the threshold processing unit 115.

  The threshold processing unit 115 compares the pixel value val, which is the difference absolute value supplied from the absolute value converting unit 114, with the predetermined threshold th and sets the pixel value of a pixel larger than the predetermined threshold th to 1, A binarized difference image in which the pixel values of the other pixels are 0 is output. From the binarized difference image, an image including a foreground region having a pixel value “1” and a background region having a pixel value “0” is generated, and the foreground region and the background region are estimated. It becomes possible to do. The pixel values 0 and 1 may be interchanged, or may be expressed by other binary values.

  That is, when the memories 93-0 to 93- (n-1) sequentially shift the pixel signal every other frame to store the pixel signal and constitute the ring buffer 110, the pixel value is set every N (= 1) frames. Assume that frames 0 (frame 0) to 10 (frame 10) are sequentially stored in the memory 93-0 as shown in the uppermost row of FIG.

  In this case, the pixel signal of frame 0 (frame 0) is transferred to the ring buffer 110 formed by the memories 93-1 to 93- (n-1), and the frame 0 (frame 0) is transferred to the memory 93- (n-5). Is stored at the timing of supply, and frame 1 (frame 1) is skipped. Then, at the timing of frame 2 (frame 2), the pixel signal of frame 0 (frame 0) is shifted to the memory 93- (n-4) and at the same time, the frame 93 (frame 2) is transferred to the memory 93- (n-5). Is supplied and stored, and this state is maintained even at the timing of frame 3 (frame 3).

  At the timing of frame 4 (frame 4), the pixel signal of frame 0 (frame 0) is shifted to the memory 93- (n-3), and the pixel signal of frame 2 (frame 2) is shifted to the memory 93- (n 4), the frame 4 (frame 4) is supplied and stored in the memory 93- (n-5), and this state is maintained even at the timing of the frame 5 (frame 5).

  Further, at the timing of frame 6 (frame 6), the pixel signal of frame 0 (frame 0) is shifted to the memory 93- (n-2), and the pixel signal of frame 2 (frame 2) is shifted to the memory 93- (n -3), the pixel signal of frame 4 (frame4) is shifted to the memory 93- (n-4), and the frame 6 (frame6) is supplied to the memory 93- (n-5). This state is also maintained at the timing of frame 7 (frame 7).

  At the timing of frame 8 (frame 8), the pixel signal of frame 0 (frame 0) is shifted to the memory 93- (n-1), and the pixel signal of frame 2 (frame 2) is shifted to the memory 93- (n -2), the pixel signal of frame 4 (frame4) is shifted to the memory 93- (n-3), and the pixel signal of frame 6 (frame6) is transferred to the memory 93- (n-3). After the shift, frame 8 (frame 8) is supplied and stored in the memory 93- (n-5), and this state is maintained even at the timing of frame 9 (frame 9).

  By this process, for example, at the timing when the frame 8 (frame 8) is supplied to the memory 93-0, the average value of the frames 0, 2, 4, 6 (frame 0, 2, 4, 6) and the frame 8 ( The absolute value of the difference from frame 8) is obtained as the pixel value of the difference image.

  That is, conventionally, as shown in the upper part of FIG. 14, each time a pixel signal of each frame is supplied, the immediately preceding background image (background image (t−1)) is read and the input image ( The weighted average using the weight α with the input image (t)) is obtained as a background image (= background image (t−1) × α + input image (t) × (1−α)) and written back. It was repeated. For this reason, the configuration of the arithmetic circuit 73 is complicated by reading, calculating, and writing back, and the power consumption is increased.

  However, according to the configuration of the arithmetic circuit 73 of FIG. 12, as shown in the lower part of FIG. 14, for example, at the timing when the pixel signal of frame 8 (frame 8) is supplied, the stored frames 0 and 2 are stored. , 4, 6 (frame 0, 2, 4, 6) are read out, the average value is obtained, and the absolute value of the difference from the pixel signal of frame 8 (frame 8) only has to be calculated. It is possible to simplify the process, and further, it is possible to reduce the power consumption because the writing back process is unnecessary.

(Background difference calculation processing)
Next, background difference calculation processing by the calculation circuit 73 of FIG. 12 will be described with reference to the flowchart of FIG. Here, it is assumed that an average value of pixel signals for four frames is obtained from the oldest one.

  In step S51, the gain amplifiers 111-0 to 111-3 are the pixels of the oldest frame from the pixel signal of the oldest frame to the Nth of the memories 93-0 to 93- (n-1) constituting the ring buffer, respectively. The pixel signals of the memories 93- (n-1) to 93- (n-4) that hold the signals are read out.

  In step S <b> 52, the addition unit 111 adds the pixel signals of the oldest frame pixel from the oldest frame pixel signal held in the ring buffer 110 and supplies the result to the gain amplifier 112.

  In step S <b> 53, the gain amplifier 112 sets the gain of the added pixel signal to ¼, obtains the average value of the pixel signals of the oldest frame from the oldest frame pixel signal to the subtracting unit 113.

  In step S54, the subtractor 113 subtracts the pixel signals of the oldest frame from the pixel signal of the memory 93-0 in which the pixel signal of the newest frame is held and the pixel signal of the oldest frame supplied from the adder 112. A difference from the average value of the signal is obtained and supplied to the absolute value converting unit 114.

  In step S55, the absolute value converting unit 114 obtains the absolute value of the pixel value that is the difference output from the subtracting unit 113, and outputs the absolute value to the threshold processing unit 115 as the difference absolute value.

  In step S56, the threshold processing unit 115 compares the pixel value val, which is the difference absolute value supplied from the absolute value converting unit 114, with the predetermined threshold th and the pixel value of the pixel larger than the predetermined threshold th. 1 is output, and a binary difference image with the pixel values of the other pixels being 0 is output.

  By using the binarized difference image, it is possible to estimate the foreground region and the background region. In the above example, the foreground region and the background region are estimated using the absolute difference between the average value of the oldest four frames and the pixel signal of the latest frame among the pixel signals stored in the ring buffer 110. However, for the average value of the oldest plurality of frames, the average value of the number of frames other than 4 frames may be obtained.

<First application example>
In the above example, the ring buffer 110 using the memories 93-0 to 93- (n-1) uses the absolute difference value between the pixel signal in the latest frame and the average value of the pixel signals for the four oldest frames. However, only the pixels with the absolute difference when the absolute difference between the pixel signal in the latest frame and each of the four oldest frames is larger than the threshold are regarded as the foreground region, and the other pixels It may be regarded as a background area.

  FIG. 16 shows the case where the absolute value of the difference between the pixel signal in the latest frame and each of the four oldest frames is larger than the threshold by the ring buffer 110 using the memories 93-0 to 93- (n-1). This is a first application example of the arithmetic circuit 73 in which only the pixels having the difference absolute value are regarded as the foreground region and the other pixels are regarded as the background region.

  The arithmetic circuit 73 in FIG. 16 includes subtracters 131-0 to 131-3, absolute value conversion units 132-0 to 132-3, threshold value processing units 133-0 to 133-3, and a determination unit 134.

  The subtracters 131-0 to 131-3 read out the oldest four frames of pixel signals stored in the memories 93-N to 93- (N-3), and the latest ones stored in the memory 93-0. The difference from the pixel signal of the input image is obtained and supplied to the absolute value converting sections 132-0 to 132-3, respectively.

  The absolute value converting units 132-0 to 132-3 supply the supplied difference values to the threshold processing units 133-0 to 133-3 as absolute difference values.

  Each of the threshold processing units 133-0 to 133-3 compares the absolute difference value with a predetermined threshold value, sets the pixel value to 1 for pixels larger than the threshold value, and sets the pixel value to 0 for other pixels. A converted image is generated and supplied to the determination unit 134.

  The determination unit 134 sets the pixel value of only the pixels whose pixel values are all 1 to 1 for each pixel of the four binarized images supplied from the threshold processing units 133-0 to 133-3. The pixel values of the other pixels are set to 0.

  With such a configuration, for example, in the case of an average value obtained by adding pixel values in time series for a range that includes leaves of trees in the background and is not regarded as a stable background area due to fluctuations in the wind, etc. In some cases, the values vary, and although it is a background image, it may be regarded as a foreground image and may not be stably regarded as a background image. However, if all of the pixel values at the same position at each timing are regarded as foreground images, the moving object is detected only by shaking the leaves of the tree. Therefore, only the range in which the subject has moved can be reliably regarded as the foreground area.

(Background difference calculation processing by the calculation circuit of FIG. 16)
Next, background difference calculation processing by the calculation circuit of FIG. 16 will be described with reference to the flowchart of FIG.

  In step S71, the subtracters 131-0 to 131-3 have the oldest four frame pixel signals (N pixel signals) stored in the memories 93- (n-1) to 93- (n-4). ).

  In step S72, the subtracters 131-0 to 131-3 obtain the difference between the read out pixel signals for the four oldest frames and the pixel signal of the latest input image stored in the memory 93-0. , And supplied to the absolute value conversion units 132-0 to 132-3, respectively.

  In step S <b> 73, the absolute value converting units 132-0 to 132-3 obtain absolute values of the supplied difference values, and supply them to the threshold processing units 133-0 to 133-3 as absolute difference values.

  In step S74, each of the threshold processing units 133-0 to 133-3 compares the absolute difference value with a predetermined threshold, sets the pixel value to 1 for pixels larger than the threshold, and sets the pixel value to 0 for other pixels. Is generated and supplied to the determination unit 134.

  In step S <b> 75, the determination unit 134 calculates pixel values of only pixels whose pixel values are all 1 for each pixel of the four binarized images supplied from the threshold processing units 133-0 to 133-3. 1 and the pixel values of the other pixels are 0.

  With such a configuration, for example, a range in which leaves are included in the background and the area is not stably regarded as the foreground area due to fluctuations in the wind, for example, is reliably regarded as the background area, and the subject has moved stably. Only the range can be regarded as the foreground region.

<Second application example>
In the above description, the ring buffer 110 using the memories 93-0 to 93- (n-1) uses the ring buffer 110 to compare the difference absolute value between the pixel signal in the latest frame and the pixel signals for the four oldest frames. Although the example of using has been described, noise removal may be realized by obtaining an average of pixel signals in all frames stored in the ring buffer 110.

  FIG. 18 shows an operation of the arithmetic circuit 73 that realizes noise removal by obtaining an average of pixel signals in all frames by the ring buffer 110 using the memories 93-0 to 93- (n-1). A configuration example is shown.

  The arithmetic circuit 73 in FIG. 18 includes an adder 151 and a gain amplifier 152. The adder 151 adds the pixel signals of all frames for n frames stored in the memories 93-0 to 93-(n−1) and supplies the sum to the gain amplifier 152. The gain amplifier 152 includes a gain amplifier 152 that obtains an average value in the ring buffer by multiplying the addition result of the adder 151 by a gain of 1 / n.

  Such a configuration makes it possible to remove noise.

(Noise removal processing by the arithmetic circuit of FIG. 18)
Next, the noise removal processing by the arithmetic circuit 73 in FIG. 18 will be described with reference to the flowchart in FIG.

  In step S <b> 91, the adder 151 reads and adds all the n frames of pixel signals stored in the memories 93-0 to 93-(n−1), and supplies the addition result to the gain amplifier 152.

  In step S92, the gain amplifier 152 multiplies the addition result by a gain of 1 / n to obtain an average value of the pixel signals.

  In step S93, the gain amplifier 152 outputs an average value for all frames of the pixel signal.

  As a result, by averaging the pixel signals for all frames stored in the ring buffer 110 configured by the memories 93-0 to 93- (n-1), an image from which noise is removed is output. Is possible. In the above description, an example in which all images are uniformly added and averaged has been described, but a new image may be given a greater weight to obtain a weighted average.

<Third application example>
In the above, the example in which the value of the memory 93 of the column AD circuit 72 is sequentially shifted to the memory 93 of the right column AD circuit 72 has been described. For example, as shown in FIG. The AD circuit 72 may be shifted, the vertical transfer line connection may be switched on or off, and the connection between adjacent vertical transfer lines may be switched.

  That is, in FIG. 20, a horizontal transfer line 191 that connects the vertical transfer lines 192 is provided, and switches 181-0 to 181-2 that turn on or off the connection between the vertical transfer lines 192 are provided. Yes. Further, switches 182-0 to 182-2 for switching on / off the connection of the vertical transfer line are provided.

  Further, the value of the memory 93 provided between the column AD circuits 72-0 and 72-1, and between the column AD circuits 72-1 and 72-2 is transferred to the adjacent left column AD circuit 72. In addition to the wirings L1-0 and L1-1, wirings L2-1 and L2-2 for transferring to the right column AD circuit 72 are provided.

  In addition, in each structure of FIG. 20, about the structure provided with the same function as FIG. 4, the same code | symbol is attached | subjected and the description shall be abbreviate | omitted suitably. That is, FIG. 20 is different from FIG. 4 in that a combinational circuit 92 ′ is provided instead of the combinational circuit 92. The combinational circuit 92 ′ has the same basic function as the combinational circuit 92, but differs in that a selector 171 is further provided. The selector 171 selects either the pixel signal of the memory 93 of the column AD circuit 72 on the right side or 0 from the right according to the operation mode, and supplies the selected signal to the adder 105. Therefore, in the combinational circuit 92 ′ in FIG. 20, the adder 105 adds the three values supplied from the selectors 102, 104, and 171 and supplies the sum to the absolute value conversion unit 106 and the selector 107.

  With such a configuration, various calculations can be realized.

  More specifically, as indicated by the column AD circuit 72-11 in the upper left part of FIG. 21, the comparator 91-11 compares the pixel signal of the analog signal supplied from the vertical transfer line with the reference. The combinational circuit 92′-11 is supplied to the combinational circuit 92′-11 so that the combinational circuit 92′-11 can function as a count-up circuit or a count-down circuit used for AD conversion. In FIG. 21, the thick line on each wiring indicates the movement path of the pixel signal.

  Further, as indicated by the column AD circuits 72-21 and 72-22 at the upper center of FIG. 21, the combination circuit 92′-22 of the column AD circuit 72-22 is connected to the column AD on the left side via the wiring L1. The pixel signal stored in the memory 93-21 of the circuit 72-21 is read out and stored in the memory 93-22, whereby the pixel signal stored in the memory 93-21 of the column AD circuit 72-21 is It becomes possible to function as a right shift processing circuit to be moved to the memory 93-22 of the column AD circuit 72-22.

  Further, as shown by the column AD circuits 72-31 and 72-32 in the upper right part of FIG. 21, the combination circuit 92′-31 of the column AD circuit 72-31 is connected to the column AD on the right side via the wiring L2. The pixel signal stored in the memory 93-32 of the circuit 72-32 is read out and stored in the memory 93-31, whereby the pixel signal stored in the memory 93-32 of the column AD circuit 72-32 is It becomes possible to function as a left shift processing circuit to be moved to the memory 93-31 of the column AD circuit 72-31.

  Further, as shown in the lower left part of FIG. 21, as shown by the column AD circuit 72-51, an initial value is set by storing a pixel signal as an initial value in the memory 93-51 of the column AD circuit 72-51. It becomes possible to function as a circuit.

  Further, as shown in the lower center portion of FIG. 21, as shown by column AD circuits 72-71 and 72-72, the combinational circuit 92′-72 of the column AD circuit 72-72 is connected via the wiring L1. Functions as a right shift arithmetic circuit that reads out pixel signals stored in the memory 93-71 of the column AD circuit 72-71 on the left, calculates the values together with the values stored in the memory 93-72, and stores them in the memory 93-72 It becomes possible to make it.

  Further, as shown in the lower right part of FIG. 21, as shown by the column AD circuits 72-91 and 72-92, the combinational circuit 92′-91 of the column AD circuit 72-91 is connected via the wiring L2. The pixel signal stored in the memory 93-92 of the column AD circuit 72-92 on the left is read out, calculated using the value stored in the memory 93-91, and stored in the memory 93-91 to shift left. It can function as an arithmetic circuit.

  By combining such arithmetic circuits, various arithmetic processes can be realized.

  For example, column AD circuits 72-0 to 72-5 are provided for 6 columns from the 0th column (column0) to the 5th column (column5), and the column AD circuits 72-0 and 72-4 are used for AD conversion. The column AD circuits 72-1 to 72-3 function as a right shift processing circuit, the column AD circuit 72-5 functions as an initial value setting circuit, and the column AD circuit 72-4 Furthermore, by making it function as a right shift arithmetic circuit as shown in FIG. 5, it becomes possible to realize the processing as shown in FIG.

  In FIG. 22, the processes from the 0th frame (frame 0) to the 3rd frame (frame 3) in the column AD circuits 72-0 to 72-5 of each column (column 0 to 5) are performed at timings T1 to T13, respectively. It is shown.

  In addition, it is assumed that the column AD circuit 72-5 of column 5 functions in advance as an initial setting circuit, and a threshold value (thresh) is set in the memory 93-5.

  When the pixel value (val0) of the 0th frame (frame0) is input to the column AD circuit 72-0 of column0 at timing T1, the combinational circuit 92′-0 of the column AD circuit 72-0 performs AD conversion. At timing T2, the data is stored in the memory 93-0 in the column AD circuit 72-0 (column0).

  At timing T3, the combinational circuit 92'-1 of the column AD circuit 72-1 shifts the pixel value (val0) stored in the memory 93-0 in column0 to the memory 93-1 in column1.

  When the pixel value (val1) of the first frame (frame1) is input to the combination circuit 92′-0 of the column AD circuit 72-0 of column0 at the timing T4, the combination circuit 92′− of the column AD circuit 72-0 is input. 0 is AD converted and stored in the memory 93-0 in the column 0 at the timing T5.

  At timing T6, the pixel value (val0) is shifted from the memory 93-1 in the column 1 to the memory 93-2 in the column 2, and similarly, the pixel value (val1) is changed from the memory 93-0 in the column 0 to the memory 93- in the column 1. Shifted to 1.

  At timing T7, AD conversion is also sequentially performed on the pixel value (val2) of the second frame (frame2), and is stored in the memory 93-0 of column0 at timing T8.

  At timing T9, the pixel value (val0) of column2 is shifted to the memory 93-3 of column3, the pixel value (val1) of column1 is shifted to the memory 93-2 of column2, and the pixel value (val2) of column0 is changed. It is shifted to the memory 93-1 of column1.

  When the pixel value (val3) of the third frame (frame3) is input at timing T10, as shown in FIG. 23, the switches 181-0 to 181-3 of the horizontal transfer line 191 are turned on and connected, The column AD circuit 72-0 of column0 and the column AD circuit 72-4 of column4 simultaneously perform AD conversion, and the pixel value (val3) is stored in the memories 93-0 and 93-4 at timing T11. .

  At timing T12, the pixel values val0 to val3 of column0 to 3 are shifted to the memories 93-3 to 93-1 of the column AD circuits 72-3 to 72-1, which are adjacent to the right side, respectively. At the same time, in the column 4 column AD circuit 72-4, the combinational circuit 92-4 causes the pixel value (val0) stored in the memory 93-3 of the column 3 column AD circuit 72-3 and the column 4 column AD. The difference absolute value is obtained by calculation with the pixel value (val3) AD-converted by the circuit 72-4, and the calculation result (abs_diff03) is stored in the memory 93-4 of the column AD circuit 72-4 of column4.

  At timing T13, the threshold value (thresh) set in the column AD circuit 72-5 of column5 is shifted to the column AD circuit 72-4 of column4, and is subtracted from the difference absolute value (abs_diff03), and the subtraction result (diff03) Save.

  When the subtraction result (diff03) is larger than a predetermined threshold, the pixel is regarded as a foreground region including a moving object.

  By such an operation, it is possible to construct a difference image composed of a binarized image composed of a foreground region and a background region.

  In the above description, an example in which a difference image is calculated using the column AD circuit 72 that is not used for AD conversion by using a low-resolution image has been described. However, the configuration of the combinational circuit 92 that constitutes the column AD circuit 72 is described. By devising it, it can be applied to other arithmetic processing. An example in which the ring buffer 110 is configured using the column AD circuit 72 that is not used for AD conversion, the difference image is obtained by the configuration of the arithmetic circuit 73 using the ring buffer 110, and noise is removed. As described above, the configuration of the arithmetic circuit 73 may be changed to be used for other arithmetic operations.

  In the above description, an example in which the exposure time is the same for all pixels has been described. However, low-resolution images with different exposure times are stored separately in groups of the column AD circuit 72 to remove whiteout and blackout. You may do it.

  As described above, according to the imaging device of the present disclosure, it is not necessary to add a digital frame memory by performing arithmetic processing using the memory 93 in the column AD circuit 72, so that the mounting area can be reduced accordingly. Therefore, it is possible to reduce the size of the apparatus. In addition, since there is no need to provide a digital frame memory, it is possible to reduce the leakage current and to realize power saving.

  In addition, since the pixel can be made smaller than in the method having a memory in the pixel, it is possible to realize a larger number of pixels or a smaller area.

  In addition, by adopting a configuration that detects the moving object in the column AD circuit, it is possible to detect motion while the subsequent configuration of the column AD circuit is completely stopped at normal times, thus realizing power saving. It becomes possible.

<Application examples to electronic devices>
The above-described imaging element 51 can be applied to various electronic devices such as an imaging device such as a digital still camera or a digital video camera, a mobile phone having an imaging function, or other devices having an imaging function. .

  FIG. 24 is a block diagram illustrating a configuration example of an imaging device as an electronic apparatus to which the present technology is applied.

  An imaging apparatus 501 shown in FIG. 24 includes an optical system 502, a shutter apparatus 503, a solid-state imaging element 504, a driving circuit 505, a signal processing circuit 506, a monitor 507, and a memory 508. The imaging apparatus 501 shown in FIG. Imaging is possible.

  The optical system 502 includes one or more lenses, guides light (incident light) from the subject to the solid-state image sensor 504, and forms an image on the light receiving surface of the solid-state image sensor 504.

  The shutter device 503 is disposed between the optical system 502 and the solid-state imaging element 504, and controls the light irradiation period and the light-shielding period to the solid-state imaging element 504 according to the control of the drive circuit 1005.

  The solid-state image sensor 504 is configured by a package including the above-described solid-state image sensor. The solid-state imaging device 504 accumulates signal charges for a certain period in accordance with light imaged on the light receiving surface via the optical system 502 and the shutter device 503. The signal charge accumulated in the solid-state image sensor 504 is transferred according to a drive signal (timing signal) supplied from the drive circuit 505.

  The drive circuit 505 drives the solid-state image sensor 504 and the shutter device 503 by outputting a drive signal that controls the transfer operation of the solid-state image sensor 504 and the shutter operation of the shutter device 503.

  The signal processing circuit 506 performs various types of signal processing on the signal charge output from the solid-state imaging device 504. An image (image data) obtained by the signal processing by the signal processing circuit 506 is supplied to the monitor 507 and displayed, or supplied to the memory 508 and stored (recorded).

Also in the imaging apparatus 501 configured as described above, a circuit configuration required for calculation is obtained by applying the imaging element 51 instead of the optical system 502, the shutter apparatus 503, and the solid-state imaging element 504 described above. Imaging can be performed by switching between the image capturing mode and the sensor mode without increasing the number, and the apparatus can be reduced in size and power can be saved.
<Usage example of solid-state image sensor>

  FIG. 25 is a diagram illustrating a usage example in which the above-described imaging element 51 is used.

  The camera module described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays as follows.

・ Devices for taking images for viewing, such as digital cameras and mobile devices with camera functions ・ For safe driving such as automatic stop and recognition of the driver's condition, Devices used for traffic, such as in-vehicle sensors that capture the back, surroundings, and interiors of vehicles, surveillance cameras that monitor traveling vehicles and roads, and ranging sensors that measure distances between vehicles, etc. Equipment used for home appliances such as TVs, refrigerators, air conditioners, etc. to take pictures and operate the equipment according to the gestures ・ Endoscopes, equipment that performs blood vessel photography by receiving infrared light, etc. Equipment used for medical and health care ・ Security equipment such as security surveillance cameras and personal authentication cameras ・ Skin measuring instrument for photographing skin and scalp photography Such as a microscope to do beauty Equipment used for sports-Equipment used for sports such as action cameras and wearable cameras for sports applications-Used for agriculture such as cameras for monitoring the condition of fields and crops apparatus

In addition, this indication can also take the following structures.
<1> a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than that of the pixel array, and at least the part of the analog-to-digital converters An image sensor that performs arithmetic processing using pixel signals that form a resolution image.
<2> The analog-digital conversion unit includes a storage unit that stores a result of the arithmetic processing,
The image sensor according to <1>, in which at least the analog-to-digital conversion unit other than the part stores pixel signals constituting a low-resolution image for a plurality of frames having different timings.
<3> At least the analog-to-digital conversion unit other than the part receives the pixel signal stored in the storage unit of the adjacent analog-to-digital conversion unit when a pixel signal of a new low-resolution image is supplied. The image pickup device according to <2>, wherein pixel signals constituting low-resolution images for a plurality of frames having different timings are stored by reading and shifting to the storage unit.
<4> At least the analog-to-digital conversion unit other than the part is when a pixel signal of a new low-resolution image is supplied, and is stored in the storage unit for each predetermined number of frames. The pixel signals constituting the low-resolution images for a plurality of frames with different timings are stored by sequentially shifting the stored pixel signals to the storage unit of the analog-digital conversion unit adjacent to each other, and storing the pixel signals. Image sensor.
<5> Arithmetic processing for obtaining an inter-frame difference image of pixel signals constituting the low-resolution image for a plurality of frames having different timings stored in the storage unit, at least the analog-digital conversion unit other than the part The imaging device according to <2>.
<6> At least the analog-to-digital conversion unit other than the part reads out a pixel signal constituting the low-resolution image for one frame at a predetermined timing, which is stored in the storage unit, and creates a new frame pixel The image sensor according to <5>, wherein an inter-frame difference image with a signal is obtained and overwritten in the storage unit.
<7> At least the analog-digital conversion unit other than the part, when a pixel signal of a new low-resolution image is supplied, the pixel signal stored in the storage unit of the analog-digital conversion unit adjacent to the analog-digital conversion unit The ring buffer is configured by storing pixel signals constituting low-resolution images of a predetermined number of a plurality of frames having different timings by shifting and storing the data in the storage unit of the conversion unit. <2> Image sensor.
<8> A calculation unit that calculates a difference image between a pixel signal constituting the low-resolution image of a plurality of frames of a predetermined number of different timings stored in the ring buffer and a pixel signal of the new low-resolution image The image sensor according to <7>, further including:
<9> The calculation unit includes an average value in each pixel of pixel signals constituting the low-resolution image from the oldest frame stored in the ring buffer to a predetermined oldest frame, and the new low-resolution image. The absolute value of the difference from the pixel signal is calculated, a pixel having the absolute difference greater than a predetermined threshold is defined as a first pixel value, and a pixel having the absolute difference smaller than the predetermined threshold is defined as a second pixel value. The imaging device according to <8>, wherein a difference image including binary images is calculated.
<10> The calculation unit includes a weighted average value corresponding to the timing of pixel signals constituting the low-resolution image from the oldest frame stored in the ring buffer to a predetermined oldest frame, and A difference absolute value with a pixel signal of a new low-resolution image is calculated, a pixel having the difference absolute value larger than a predetermined threshold is defined as a first pixel value, and a pixel having the difference absolute value smaller than the predetermined threshold is determined. The image sensor according to <9>, wherein a difference image including a binary image serving as a second pixel value is calculated.
<11> The calculation unit includes each of a pixel signal constituting the low-resolution image from the oldest frame stored in the ring buffer to a predetermined oldest frame, and a pixel signal of the new low-resolution image The difference absolute value of each of the difference absolute values is calculated, a pixel in which each of the difference absolute values is larger than a predetermined threshold is set as a first pixel value, and a pixel in which any of the difference absolute values is smaller than the predetermined threshold The image sensor according to <9>, wherein a difference image including a binary image serving as a second pixel value is calculated.
<12> The imaging apparatus according to <7>, further including a calculation unit that calculates an average value of pixel values of pixels constituting the low-resolution image of a plurality of frames at a predetermined number of times stored in the ring buffer. element.
<13> The computing unit computes a weighted average value corresponding to the timing of pixel values of pixels constituting the low-resolution image of a predetermined number of frames at different timings stored in the ring buffer. The image pickup device according to <12>.
<14> The imaging device according to <1>, wherein the low-resolution image is an image including any one of an average value, a representative value, and a median for each of a plurality of pixel groups of pixels that form the pixel array.
<15> a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
An image pickup method for an image pickup device including a plurality of analog-digital conversion units that are provided for each column in which the plurality of pixels are arranged and convert the pixel signal from an analog signal to a digital signal,
A part of the plurality of analog-digital conversion units performs analog-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-digital conversion unit other than the part includes the analog-digital conversion unit, An operation method of an image sensor that executes arithmetic processing using pixel signals constituting a low-resolution image.
<16> a pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-to-digital converters other than the part are An imaging device that performs arithmetic processing using pixel signals that form a low-resolution image.
<17> A pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-to-digital converters other than the part are An electronic device that performs arithmetic processing using pixel signals that make up a low-resolution image.

  51 Image sensor, 71 pixel array, 71a, 71a-1 to 71a-p pixel group, 72, 72-0 to 72- (n-1) column AD circuit, 73 arithmetic circuit, 91 comparator, 92, 92-0 to 92- (n-1) combinational circuit, 93, 93-0 to 93- (n-1) memory, 101, 101-0 to 101- (n-1) circuit, 102, 102-0 to 102- (n -1) Adder, 103, 103-0 to 103- (n-1) circuit, 110 ring buffer, 111, 111-0 to 111-4 gain amplifier, 112 adder, 113 subtraction air, 114 absolute value conversion unit , 115 threshold processing unit, 131, 131-0 to 131-3 subtraction, 132, 132-0 to 132-3 absolute value conversion unit, 133, 133-0 To 133- (n-1) threshold determination unit, 134 determination unit, 151 adder, 152 gain amplifier, 181, 181-0 to 181-2 switch, 182, 182-0 to 182-2 switch, 191 horizontal transfer line , 192, 192-1 to 192-2 Vertical transfer

Claims (17)

A pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-to-digital converters other than the part are An image sensor that performs arithmetic processing using pixel signals that form a low-resolution image. The analog-digital conversion unit includes a storage unit that stores a result of the arithmetic processing,
The imaging device according to claim 1, wherein at least the analog-to-digital conversion units other than the part store pixel signals constituting low-resolution images for a plurality of frames having different timings. When at least a part of the analog-digital conversion unit is supplied with a pixel signal of a new low-resolution image, the analog-digital conversion unit reads the pixel signal stored in the storage unit of the adjacent analog-digital conversion unit, The imaging device according to claim 2, wherein pixel signals constituting low-resolution images for a plurality of frames having different timings are stored by being shifted and stored in the storage unit. At least the analog-digital conversion unit other than the part is a pixel signal stored in its storage unit every predetermined number of frames when a pixel signal of a new low-resolution image is supplied. The image sensor according to claim 2, wherein pixel signals constituting low resolution images for a plurality of frames having different timings are stored by sequentially shifting and storing them in the storage units of the adjacent analog-digital conversion units. . The analog-to-digital conversion unit other than at least the part executes a calculation process for obtaining an inter-frame difference image of pixel signals constituting the low-resolution image for a plurality of frames having different timings stored in the storage unit. The imaging device according to claim 2. The analog-to-digital conversion unit other than at least the part reads out pixel signals constituting the low-resolution image for one frame at a predetermined timing stored in the storage unit, and outputs the pixel signals of a new frame. The image sensor according to claim 5, wherein an inter-frame difference image is obtained and overwritten on the storage unit. When at least a part of the analog-digital conversion unit receives a pixel signal of a new low-resolution image, the analog-digital conversion unit stores the pixel signal stored in the storage unit of the adjacent analog-digital conversion unit. The imaging device according to claim 2, wherein the ring buffer is configured by storing pixel signals constituting low-resolution images of a predetermined number of frames having different timings by being shifted and stored in the storage unit. . And a calculation unit for calculating a difference image between a pixel signal constituting the low-resolution image of a predetermined number of frames stored in the ring buffer and having a different timing, and a pixel signal of the new low-resolution image. The imaging device according to claim 7. The arithmetic unit includes an average value of each pixel signal constituting the low resolution image from the oldest frame to a predetermined oldest frame stored in the ring buffer, and a pixel signal of the new low resolution image. The absolute value of the difference is calculated, a pixel having the absolute difference value larger than a predetermined threshold is set as a first pixel value, and a pixel having the absolute difference value smaller than the predetermined threshold is set as a second pixel value 2 The image sensor according to claim 8, wherein a difference image composed of value images is calculated. The calculation unit includes a weighted average value corresponding to the timing of the pixel signal constituting the low-resolution image from the oldest frame stored in the ring buffer to a predetermined oldest frame, and the new low-level image. A difference absolute value from the pixel signal of the resolution image is calculated, a pixel having the difference absolute value larger than a predetermined threshold is set as a first pixel value, and a pixel having the difference absolute value smaller than the predetermined threshold is set as a second pixel value. The imaging device according to claim 9, wherein a difference image composed of a binary image as a pixel value is calculated. The arithmetic unit calculates absolute differences between pixel signals constituting the low-resolution image from the oldest frame to the predetermined oldest frame stored in the ring buffer and the pixel signal of the new low-resolution image. A value is calculated, a pixel whose absolute value of each difference is larger than a predetermined threshold value is set as a first pixel value, and a pixel whose either absolute value of the difference is smaller than a predetermined threshold value is set as a second pixel value. The imaging device according to claim 9, wherein a difference image composed of a binary image as a pixel value is calculated. The imaging device according to claim 7, further comprising a calculation unit that calculates an average value of the pixel values of the pixels constituting the low-resolution image of a predetermined number of frames at different timings stored in the ring buffer. The calculation unit calculates a weighted average value corresponding to the timing of pixel values of pixels constituting the low-resolution image of a plurality of frames of a predetermined number of different timings stored in the ring buffer. The imaging device according to 12. The imaging device according to claim 1, wherein the low-resolution image is an image including any one of an average value, a representative value, and a median for each of a plurality of pixel groups of pixels constituting the pixel array. A pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
An image pickup method for an image pickup device including a plurality of analog-digital conversion units that are provided for each column in which the plurality of pixels are arranged and convert the pixel signal from an analog signal to a digital signal,
A part of the plurality of analog-digital conversion units performs analog-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-digital conversion unit other than the part includes the analog-digital conversion unit, An operation method of an image sensor that executes arithmetic processing using pixel signals constituting a low-resolution image. A pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-to-digital converters other than the part are An imaging device that performs arithmetic processing using pixel signals that form a low-resolution image. A pixel array in which a plurality of pixels that generate pixel signals according to the amount of incident light are arranged in a two-dimensional array;
A plurality of analog-digital converters provided for each column in which the plurality of pixels are arranged, and converting the pixel signal from an analog signal to a digital signal;
A part of the plurality of analog-to-digital converters performs analog-to-digital conversion on a pixel signal constituting a low-resolution image having a resolution lower than the resolution of the pixel array, and at least the analog-to-digital converters other than the part are An electronic device that performs arithmetic processing using pixel signals that make up a low-resolution image.

Images