KR20210135358A - Image sensor architecture - Google Patents

Abstract

The image sensor comprises: a first integrated circuit layer comprising pixel sensors grouped into pixel sensor groups based on position; a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer comprising image processing circuitry groups each configured to receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups comprising: further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the sensor; a third integrated circuit layer in electrical communication with the second integrated circuit layer, the third integrated circuit layer each receiving processed pixel information from a corresponding image processing network group and object detection for the processed pixel information during operation of the image sensor containing groups of neural networks configured to perform analysis for

Description

Image sensor architecture

<Cross-Reference to Related Applications>

This application claims priority to U.S. Patent Application Serial No. 16/376,634, filed April 5, 2019, which is incorporated herein by reference in its entirety.

<Technology>

BACKGROUND This disclosure relates to image sensors and systems including image sensors.

This specification describes techniques for an image sensor that captures images and performs processing on the image sensor to detect objects. This specification further describes data transmission protocols for transmitting data from an image sensor to a remote processor.

Generally, one innovative aspect of the subject matter described herein includes: a first integrated circuit layer comprising pixel sensors grouped into pixel sensor groups based on position; a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer comprising image processing circuitry groups each configured to receive pixel information from a corresponding pixel sensor group; processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor; a third integrated circuit layer in electrical communication with the second integrated circuit layer, the third integrated circuit layer comprising neural network groups each configured to receive processed pixel information from a corresponding image processing network group, the neural network groups comprising: further configured to perform analysis for object detection on the pixel information processed during operation of the image sensor; and a network for outputting information representing results of analysis for object detection by neural network groups, wherein a first integrated circuit layer is stacked on a second integrated circuit layer, and the second integrated circuit layer is a third integrated circuit layer It can be embodied in an image sensor, which is laminated on a circuit layer.

Other embodiments of this aspect include corresponding methods of performing actions corresponding to an image sensor and performing actions to form an image sensor.

Each of these and other embodiments may optionally include one or more of the following features. In some aspects, the results of the analysis for object detection include selected regions of interest for pixels representing the detections; metadata including time and geometric location information; intermediate computational results before object detection; statistical information regarding network certainty level; and at least one of a group consisting of classifications of detected objects. In certain aspects, the neural network network groups each include circuitry configured to implement a convolutional neural network.

In some aspects, convolutional neural networks detect objects sensed by a pixel sensor group corresponding to an image processing network group, each corresponding to a neural network network group. In some implementations, the third integrated circuit layer includes circuitry configured to implement a recurrent neural network. In certain aspects, the recurrent neural network receives information about objects detected by all of the neural network groups, and detects objects sensed across multiple of the pixel sensor groups.

In some aspects, each of the neural network groups is located directly below an image processing network group that provides processed pixel information to the neural network group. In certain aspects, each of the image processing circuitry groups is located directly below or in the immediate vicinity of a pixel sensor group that provides pixel information to the image processing circuitry group. In some implementations, the first integrated circuit layer, the second integrated circuit layer, and the third integrated circuit layer are integrated into a single integrated chip.

In some aspects, each of the pixel sensor groups includes the same number of pixel sensors. In certain aspects, image processing operations performed on pixel information include high dynamic range fusion prior to processing. In some aspects, each of the image processing circuitry groups includes an analog-to-digital converter.

Generally, one innovative aspect of the subject matter described herein is an action of acquiring a plurality of images captured by pixel sensors of an image sensor; analyzing the plurality of images for object detection using neural network circuitry integrated in the image sensor; generating, for each of the plurality of images using neural network circuitry integrated in the image sensor, neural network output data related to results of analysis of the plurality of images for object detection; and transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data of each of the plurality of images.

Other embodiments of this aspect include corresponding image sensors configured to perform the actions of the method.

Each of these and other embodiments may optionally include one or more of the following features. In some aspects, the action of transmitting the neural network output data for each of the plurality of images and image data for the subset of the plurality of images instead of image data of each of the plurality of images comprises an image for a first image of the plurality of images. data and neural network output data for the first image, and neural network output data for the second image, before and without sending the image data for the second image, for the third image of the plurality of images. and sending the image data and neural network output data for the third image.

In certain aspects, the actions include analyzing the second image of the plurality of images for object detection, while transmitting image data for the first image of the plurality of images and neural network output data for the first image. In some implementations, the actions include generating neural network output data related to results of analysis of the plurality of images for object detection, while capturing the third image. In some aspects, the action of transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for the subset of the plurality of images instead of image data of each of the plurality of images, is the subset of the plurality of images. sending the image data of , at a certain number of frames per second, and sending neural network output data for the images while image data for the subset of the plurality of images is not being transmitted.

In certain aspects, the subset of the plurality of images includes all Nth images captured by the image sensor, where N is an integer greater than one. In some implementations, the image data for a particular image of the images includes data representing values for each pixel in the image. In some aspects, the neural network output data for a particular image of images includes selected regions of interest for pixels representing detections; metadata including time and geometric location information; intermediate calculation results before object detection; statistical information about the level of network certainty; and one or more of the classifications of the detected objects.

In certain aspects, the neural network output data for a particular image of the images represents partially processed data provided to a processor for further processing. In some implementations, the action of sending the neural network output data and the image data includes sending the neural network output data and the image data to the central processing unit.

Certain embodiments of the subject matter described herein may be implemented to realize one or more of the following advantages. An advantage of this technology is that inclusion of multiple stacked integrated circuit layers in an image sensor allows processing to be distributed among the layers, as opposed to exclusively using processors remote from the image sensor to process the image data. It may be that parallel processing by layers and/or more processing may occur on the image sensor.

Performing additional processing in layers of the image sensor instead of outside the image sensor may eliminate the need to process specific data outside the image sensor, which may reduce the amount of bandwidth used to output information from the image sensor. For example, since some processing may have already been performed on the image sensor for the frames, only a subset of the frames captured by the image sensor may need to be output. Another advantage may be that multiple layers may be arranged such that information may be transmitted over short distances between the layers, thereby allowing for faster transmission of data.

The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims.

1 is a block diagram of an exemplary image sensor having three integrated circuit layers.
2 is a flow diagram of an exemplary process for detection of an object using three integrated circuit layers of an image sensor.
3 is a block diagram of an exemplary integrated circuit layer having pixel sensor groups.
4 is a block diagram of an exemplary integrated circuit layer having groups of image processing circuitry.
5 is a block diagram of an exemplary integrated circuit layer having groups of neural networks.
6 is a diagram of timing for an exemplary image sensor without neural network circuitry.
7 is a diagram of timing for an exemplary image sensor having neural network circuitry.
8 is a flow diagram of an exemplary process for transmitting neural network output data and image data from an image sensor having neural network circuitry.
9 is a block diagram of an exemplary system including an image sensor used by an autonomous vehicle.
10 is a block diagram of an exemplary system including groups of pixel sensors and neural network circuitry on separate chips.
Like reference numbers and designations in the various drawings indicate like elements.

1 is a block diagram of an exemplary image sensor 100 having three integrated circuit layers. The image sensor 100 may detect objects using three integrated circuit layers. For example, the image sensor 100 may capture an image including a person and output an indication of "detected person". As another example, the image sensor 100 may capture an image and output a part of the image including the vehicle detected by the image sensor 100 .

The three integrated circuit layers include a first integrated circuit layer 110 , a second integrated circuit layer 120 , and a third integrated circuit layer 130 . The first integrated circuit layer 110 is laminated on the second integrated circuit layer 120 , and the second integrated circuit layer 120 is laminated on the third integrated circuit layer 130 . For example, the first integrated circuit layer 110 is in direct contact with the top of the second integrated circuit layer 120 , and the third integrated circuit layer 130 is in direct contact with the bottom of the second integrated circuit layer 120 . do.

The first integrated circuit layer 110 may be in electrical communication with the second integrated circuit layer 120 . For example, the first integrated circuit layer 110 and the second integrated circuit layer 120 may be physically connected to each other by interconnects. The second integrated circuit layer 120 may be in electrical communication with the third integrated circuit layer 130 . For example, the second integrated circuit layer 120 and the third integrated circuit layer 130 may be physically connected to each other by interconnects.

The first integrated circuit layer 110 may have the same area as the second integrated circuit layer 120 . For example, the length and width of the first integrated circuit layer 110 and the second integrated circuit layer 120 may be the same, but the heights may be different. The third integrated circuit layer 130 may have a larger area than the first and second integrated circuit layers 110 and 120 . For example, the third integrated circuit layer 130 may have a length and width that are both 20% greater than the length and width of the first and second integrated circuit layers 110 , 120 .

The first integrated circuit layer 110 groups by position into pixel sensor groups (each pixel sensor group is referred to as a “pixel group” in FIG. 1 ) 112A- 112C (collectively referred to as 112 ). and an array of pixel sensors. For example, the first integrated circuit layer 110 may include a 6400x4800 array of pixel sensors grouped into 320x240 pixel sensor groups, each pixel sensor group containing an array of 20x20 pixel sensors.

Each of the pixel sensor groups 112 may include 2x2 pixel sensor sub-groups. For example, each of the pixel sensor groups of 20x20 pixel sensors may include 10x10 pixel sensor sub-groups, each pixel sensor sub-group having a red pixel sensor in the upper left, a green pixel sensor in the lower right, a first clear pixel sensor on the lower left and a second transparent pixel sensor on the upper right, each sub-group being Red-Clear-Clear-Green ( RCCG) sub-groups.

In some implementations, the size of the pixel sensor groups can be selected to increase silicon utilization. For example, the size of the pixel sensor groups may allow more silicon to be covered by pixel sensor groups having the same pattern of pixel sensors.

The second integrated circuit layer 120 may include image processing circuitry groups (each image processing circuitry group is referred to as a “process group” in FIG. 1) 122A-122C (collectively referred to as 122). can For example, the second integrated circuit layer 120 may include 320x240 image processing circuitry groups.

The image processing circuitry groups 122 may each be configured to receive pixel information from a corresponding pixel sensor group, and perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor 100 . may be further configured to perform

In some implementations, each image processing circuitry group 122 may receive pixel information from a single corresponding pixel sensor group 112 . For example, image processing circuitry group 122A may receive pixel information from pixel sensor group 112A and not any other pixel group, and image circuitry processing group 122B may receive pixel information from pixel sensor group 112B. ) and may not receive pixel information from any other group of pixels.

In some implementations, each image processing circuitry group 122 can receive pixel information from a number of corresponding pixel sensor groups 112 . For example, image processing circuitry group 122A may receive pixel pixel information from both pixel sensor groups 112A and 112B and not from other pixel groups, and image processing circuitry group 122B may It may receive pixel information from pixel group 112C and other pixel groups and not from other pixel groups.

Because the image processing circuitry groups 122 may be physically close to the corresponding pixel sensor groups 112 , allowing the image processing circuitry groups 122 to receive pixel information from the corresponding pixel groups causes the first integration Pixel information may be rapidly transmitted from the circuit layer 110 to the second layer 120 . The greater the distance over which information is transmitted, the longer it may take to transmit. For example, pixel sensor group 112A may be directly above image processing circuitry group 122A, and pixel sensor group 112A may not be directly above image processing circuitry group 122C, so pixel sensor group 112A ) and the image processing circuitry group 122C, sending pixel information from the pixel sensor group 112A to the image processing circuitry group 122A sends the pixel information from the pixel sensor group 112A to the image processing circuitry group 122C. It can be faster than sending pixel information to

The image processing circuitry groups 122 may be configured to perform image processing operations on pixel information that the image processing circuitry groups 122 receive from the pixel groups. For example, image processing circuitry grouping 122A may perform high dynamic range fusion on pixel information from pixel sensor group 112A, and image processing circuitry grouping 122B may include pixel sensors High dynamic range fusion can be performed on the pixel information from group 112B. Other image processing operations may include, for example, analog to digital signal conversion and de-mosaicing.

Having the image processing circuitry groups 122 perform image processing operations on pixel information from the corresponding pixel sensor groups 112 causes the image processing operations to be performed in parallel by the image processing circuitry groups 122 in a distributed fashion. can be done For example, the image processing circuitry group 122A may perform image processing operations on the pixel information from the pixel group 122B at the same time that the image processing circuitry group 122B performs image processing operations on the pixel information from the pixel sensor group 112A. image processing operations may be performed on the .

The third integrated circuit layer 130 comprises neural network groups 132A-132C (each neural network group is referred to as an “NN group” in FIG. 1 ) (132A-132C) (collectively referred to as 132). and full image neural network circuitry 134 . For example, the third integrated circuit layer 130 may include 320x240 neural network groups.

Neural network groups 132 may each be configured to receive processed pixel information from a corresponding image processing network group, and perform analysis for object detection on the processed pixel information during operation of image sensor 100 . may be further configured. In some implementations, the neural network groups 132 may each implement a convolutional neural network (CNN).

In some implementations, each neural network group 132 may receive processed pixel information from a single corresponding image processing network group 122 . For example, neural network group 132A may receive processed pixel information from image processing circuitry group 122A and not from any other image processing network group, and neural network group 132B may receive image processing circuitry group 122A. It may receive processed pixel information from circuitry group 122B and not from any other image processing circuitry group.

In some implementations, each neural network group 132 can receive processed pixel information from a number of corresponding image processing network groups 122 . For example, neural network group 132A may receive processed pixel information from both image processing network groups 122A and 122B and not from other image processing network groups, and neural network group 132B ) may receive processed pixel information from both image processing circuitry group 122C and other pixel groups and not processed pixel information from other pixel groups.

Because the neural network groups 132 may be physically close to the corresponding image processing network groups 122, allowing the neural network groups 132 to receive processed pixel information from the corresponding image processing network groups The processed pixel information may be rapidly transferred from the second integrated circuit layer 120 to the third integrated circuit layer 130 . The greater the distance over which information is transmitted, the longer it may take to transmit. For example, the image processing network group 122A may be directly above the neural network group 132A, so that if there are interconnects between the image processing network group 122A and the neural network group 132C, the image processing network group ( Sending processed pixel information from 122A to neural network group 132A may be faster than sending processed pixel information from image processing network group 122A to neural network group 132C.

Neural network groups 132 may be configured to detect objects from processed pixel information that neural network groups 132 receive from image processing network groups 122 . For example, neural network group 132A may detect objects from processed pixel information from image processing circuitry group 122A, and neural network group 132B may detect objects from processed pixel information from image processing network group 122B. Objects can be detected from pixel information.

Allowing the neural network groups 132 to detect objects from processed pixel information from the corresponding image processing network group 122 may cause the detection to be performed by each of the neural network groups 132 in a distributed manner in parallel. have. For example, neural network group 132A may detect processed pixels from image processing network group 122A at the same time neural network group 132B detects objects from processed pixel information from image processing network group 122B. It is possible to detect objects from information.

In some implementations, neural network groups 132 may perform intermediate processing. Accordingly, the image sensor 100 may perform some intermediate processing using the three integrated circuit layers 110 , 120 , and 130 and output only intermediate results. For example, the image sensor 100 may capture an image including a person, and output an indication of "a region of interest in a partial region of an image" without classifying an object of interest (person). Other processing performed outside the image sensor 100 may classify the region of interest as a person.

Accordingly, the output from the image sensor 100 may include some data representing the output of some convolutional neural network. This data itself may be difficult to decipher, but as it continues to be processed outside of image sensor 100, the data can be used to classify regions as containing people. This hybrid approach has the advantage of reducing the required bandwidth. Thus, the output from neural network groups 132 is based on selected regions of interest for pixels representing detections, metadata including temporal and geometric position information, intermediate computation results prior to object detection, and network certainty level. statistical information about the object, and one or more of classifications of the detected objects.

In some implementations, neural network groups 132 may be configured to implement CNNs with high recall and low precision. The neural network groups 132 may respectively output a list of detected objects, where the object was detected, and the detection timing of the object.

The full image neural network network 134 may be configured to receive, from each of the neural network groups 132 , data representing objects detected by the neural network groups 132 , and to detect objects from the data. For example, neural network groups 132 may not be able to detect objects captured by multiple pixel groups because each individual neural network group may only receive a portion of the processed pixel information corresponding to the object. However, the full image neural network network 134 may receive data from a plurality of neural network groups 132 to detect objects sensed by a plurality of pixel groups. In some implementations, the full image neural network circuitry 134 may implement a recurrent neural network (RNN). Neural networks may be configurable with respect to their architecture (number and type of layers, activation functions, etc.), as well as the actual values of neural network components (eg, weights, biases, etc.).

In some implementations, letting the image sensor 100 perform processing simplifies the processing pipeline architecture, provides higher bandwidth and lower latency, allows for selective frame rate operations, and allows for a stacked architecture. This may result in cost savings, higher system reliability as the integrated circuit may have fewer potential points of failure, and significant cost and power savings for computational resources.

In some implementations, the third integrated circuit layer 130 may be bonded to the second integrated circuit layer 120 using a silicon-to-silicon direct bonding method or a “flip chip” approach, wherein the substrate is between the layers. It is interposed and the connections are implemented using wire bonding from the silicon to the substrate.

2 is a flow diagram of an exemplary process 200 for detection of an object using three integrated circuit layers of an image sensor. Process 200 may be performed by image sensor 100 of FIG. 1 or some other image sensor.

The process 200 may include obtaining 210 pixel information by pixel sensors that are in the first integrated circuit layer and are grouped into pixel sensor groups based on position. For example, each of the pixel sensors of the pixel sensor groups 112 of the first integrated circuit layer 110 may generate pixel information that is analog signals representing the intensity of light sensed by the pixel sensor. In some implementations, each of the pixel sensor groups includes the same number of pixel sensors. For example, each of the pixel sensor groups 112 may include 400 pixel sensors forming 20x20 RCCG sub-groups.

The process 200 includes performing 220 image processing operations on pixel information from a corresponding pixel sensor group to provide pixel information processed by the image processing circuitry groups of the second integrated circuit layer. may include For example, image processing circuitry group 122A may receive pixel information from pixel sensor group 112A and perform image processing operations on pixel information from pixel sensor group 112A, while image processing circuitry group 122B may receive pixel information from pixel sensor group 112B and perform image processing operations on the pixel information from pixel sensor group 112B.

In some implementations, each of the image processing circuitry groups is located directly below a pixel sensor group that provides pixel information to the image processing circuitry group. For example, image processing circuitry group 122A may be located directly below pixel sensor group 112A, and image processing circuitry group 122B may be located directly below pixel sensor group 112B. In some implementations, each of the image processing circuitry groups can be located in the immediate vicinity of a pixel sensor group that provides pixel information to the image processing circuitry group. In some implementations, the second integrated circuit layer can be in electrical communication with the first integrated circuit layer. For example, the second integrated circuit layer 120 may be connected to the first integrated circuit layer 110 by interconnects formed of an electrically conductive material.

The process 200 may include, by the neural network groups of the third integrated circuit layer, performing 230 analysis for object detection on the processed pixel information from the corresponding image processing network group. . For example, neural network group 132A may receive processed pixel information from image processing network group 122A and detect objects from processed pixel information from image processing network group 122A, whereas neural network group 122A Group 132B may receive pixel information from image processing circuitry group 122B and detect objects from the processed pixel information from image processing circuitry group 122B. In another example, neural network groups 132 receive processed pixel information from corresponding image processing network groups 122 , and display selected regions of interest, temporal and geometric location information for pixels representing detections. At least one of metadata including included metadata, intermediate calculation results before object detection, statistical information about a network certainty level, and classifications of detected objects may be output.

In some implementations, the neural network network groups each include a circuitry configured to implement a CNN. For example, neural network group 132A may include circuitry for implementing a first CNN, and neural network network group 132B may include circuitry for implementing a second, different CNN.

The convolutional neural networks may each detect objects sensed by the pixel sensor group corresponding to the image processing network group corresponding to the neural network network group. For example, the CNN of the neural network group 132A may detect a first object captured by the pixel sensors of the pixel sensor group 112A, and the CNN of the neural network group 132B is the pixel sensor group 112B ) and detect the second object captured by the pixel sensors of

In some implementations, the third integrated circuit layer includes circuitry configured to implement the RNN. For example, the third integrated circuit layer 130 is configured for an RNN that receives information about objects detected by all of the neural network groups and detects objects detected across multiple of the pixel sensor groups. It may include a network that is

In some implementations, each of the neural network groups is located directly below an image processing network group that provides processed pixel information to the neural network group. For example, neural network group 132A may be located directly below image processing circuitry group 122A, and neural network group 132B may be located directly below image processing circuitry group 122B.

In some implementations, the third integrated circuit layer can be in electrical communication with the second integrated circuit layer. For example, the third integrated circuit layer 130 may be connected to the second integrated circuit layer 120 by interconnects formed of an electrically conductive material.

Process 200 may include outputting information 240 indicative of results of analysis for object detection by neural network groups. For example, the third integrated circuit layer 130 may include circuitry configured to output metadata from the image sensor 100 to a central processing unit external to the image sensor 100 , wherein the metadata is a group of neural network networks. Specifies the objects detected by the s 132 .

In some implementations, process 200 may be performed when a first integrated circuit layer is deposited on a second integrated circuit layer and a second integrated circuit layer is deposited on a third integrated circuit layer. For example, the process 200 may be such that the lower portion of the first integrated circuit layer 110 is in direct contact with the upper portion of the second integrated circuit layer 120 and the lower portion of the second integrated circuit layer 120 is the third integrated circuit layer. This may be performed by the image sensor 100 in direct contact with the upper portion of the 130 .

In some implementations, process 200 may be performed when the first integrated circuit layer, the second integrated circuit layer, and the third integrated circuit layer are integrated into a single integrated chip. For example, process 200 may be performed by image sensor 100 being a single integrated chip.

3 is a block diagram of an exemplary integrated circuit layer 300 having pixel sensor groups. In some implementations, the integrated circuit layer 300 may be the first integrated circuit layer 110 shown in FIG. 1 . The integrated circuit layer 300 includes pixel sensor groups 310 , row drivers, circuitry 320 for timing and automotive safety integrity level, and interconnects 330 . As shown in FIG. 3 , each of the pixel sensor groups 310 includes multiple 2×2 arrays of RCCG sub-groups, and each of the pixel sensor groups 310 includes the same number of 2×2 arrays of RCCG sub-groups. include However, other implementations of the integrated circuit layer 300 may include different pixel sensor groups. For example, each group of pixels may include 3x3 arrays of RCCG sub-groups.

4 is a block diagram of an exemplary integrated circuit layer 400 having image processing circuitry groups. In some implementations, the integrated circuit layer 400 can be the second integrated circuit layer 120 shown in FIG. 1 . The integrated circuit layer 400 may include image processing circuitry groups 410 , circuitry 420 for logic and automotive safety integrity levels, and interconnects 430 . As shown in Figure 3, each image processing circuitry group 410 includes high/low analog-to-digital converters (ADCS), circuitry for high dynamic range fusion, automotive safety integrity. circuitry for the level, circuitry for the pixel memory, and circuitry for the multiplexer. The multiplexer routes information between the layers as groups of pixels serialized on a single pixel (multiple bits per pixel) or on a single link (connection) towards an integrated circuit layer 500 with a group of neural networks. may allow for flexibility. Each of the image processing circuitry groups 410 may cover the same area as the pixel sensor group 310 that provides pixel information to the image processing circuitry group 410 .

5 is a block diagram of an exemplary integrated circuit layer 500 having groups of neural networks. In some implementations, the integrated circuit layer 500 may be the third integrated circuit layer 130 shown in FIG. 1 . The integrated circuit layer 400 may include neural network groups 510 , circuitry 520 for logic and automotive safety integrity levels, interconnects 530 and an RNN.

As shown in FIG. 5 , each of the neural network network groups 510 may include circuitry for a memory, a CNN, and a de-multiplexer. The de-multiplexer may deserialize the bits and pixels to recover the captured pixel configuration. Each of the neural network groups 510 may cover the same area as the image processing network group 410 that provides the processed pixel information to the neural network group 410 .

6 is a diagram 600 of timing for an exemplary image sensor without neural network circuitry. Diagram 600 shows a first row 610 representing the timing of transmission of frames, a second row 620 representing the timing of capture of frames, and a third row 630 representing exemplary time frames associated with the actions. ) may be included.

Diagram 600 shows how a frame, also referred to as an image, can be captured by the image sensor every 100 milliseconds for 10 milliseconds and then the transmitted frame is transmitted for 20 milliseconds. The image sensor may be idle for the remaining 80 milliseconds of the 100 milliseconds. Because the processing of the frames is done outside the image sensor and takes 80 milliseconds per frame, the image sensor can only send a single image every 100 milliseconds. For example, it may take 80 milliseconds to detect an object in the entire image by a central processing unit external to the image sensor. Accordingly, diagram 600 may show how only a single frame is used every 100 milliseconds.

7 is a diagram 700 of timing for an exemplary image sensor having neural network circuitry. For example, diagram 700 may depict timing for image sensor 100 of FIG. 1 .

The diagram 700 shows a first row 710 representing the transmission timing of frames, a second row 720 representing the timing of generating neural network output data related to objects detected within the frames, and capturing the frames. a third row 730 representing timing, and a fourth row 740 representing exemplary time frames associated with the actions.

Diagram 700 may show how frames are captured by the image sensor every 10 milliseconds, where every 100 milliseconds the image sensor transmits a frame for 20 milliseconds and the remaining 80 milliseconds captured by the image sensor. The neural network output data generated for 10 frames is transmitted. Thus, although in diagram 700 the image data for a single frame is still transmitted per 100 milliseconds, since the neural network output data for 10 frames is also provided every 100 milliseconds, the image sensor is 10 times larger than in diagram 600 . It can provide information about many frames.

Since the neural network output data from the image sensor can already indicate whether objects have been detected in the frame and other frames, having the image sensor send the frame for further processing along with the neural network output data reduces the amount of processing outside the image sensor. can be reduced

As shown in diagram 700, when a frame is captured, neural network output data for the frame is generated. For example, if frame A is captured, neural network output data may be generated from frame A. In another example, once frame B is captured, neural network output data may be generated from frame B.

Neural output data for a frame is generated while another frame is being captured. For example, frame B may be captured while neural network output data for frame A is being generated. Using the image sensor 100 , the first integrated circuit layer 110 captures frame B, while the third integrated circuit layer 130 detects objects and generates neural network output data representing the detected objects.

As shown in diagram 700, neural network output data for a frame is generated and then transmitted. For example, neural network output data from frame A may be generated and transmitted once frame A has finished transmitting. In some implementations, transmission of the neural network output data may begin once the group is finished processing. This can be effective for rolling shutter sensors.

In some implementations, with respect to transmission, image sensor 100 may also multiplex image grayscale or color data with neural network output data. For example, the entire image grayscale or color data with object and temporal information pre-processed may be multiplexed into a single data stream. The multiplexed data streams may have much lower output bandwidth requirements than the full image stream information, eg, four or more times less. Multiplexing with a single serial link of multiplexed information as a camera output instead of multiple links can greatly simplify the vehicle level architecture as the number of physical links can be reduced.

8 is a flow diagram of an exemplary process 800 for transmitting neural network output data and image data from an image sensor having neural network circuitry. Process 800 may be performed by image sensor 100 of FIG. 1 or some other image sensor.

Process 800 may include acquiring 810 a number of images captured by pixel sensors of the image sensor. For example, the pixel sensors of the pixel groups 112C of the first integrated circuit layer 110 of the image sensor 100 may capture a different frame every 10 milliseconds.

Process 800 may include analyzing 820 the plurality of images for object detection using neural network circuitry integrated into the image sensor. For example, the neural network groups 132 of the third integrated circuit layer 130 of the image sensor 100 may detect objects within each of the portions of a frame in which the neural network group has received processed pixel information. have.

Process 800 includes generating, for each of the plurality of images, using neural network circuitry integrated in the image sensor, neural network output data related to results of analysis of the plurality of images for object detection ( 830 ) can do. For example, the first neural network group 132A may generate metadata indicating that a first object has been detected in a portion of frame A, and the second neural network group 132B may generate metadata indicating that an object is detected in another portion of frame A. It can generate metadata indicating that it was not detected.

Process 800 may include, from the image sensor, transmitting 840 neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data of each of the plurality of images. can For example, the image sensor 100 may include metadata generated by each of the neural network groups indicating whether the neural network group has detected objects in respective portions of each of the frames captured by the image sensor 100 . Data corresponding to , and image data of only all tenth frames captured by the image sensor may be transmitted.

In some implementations, transmitting the neural network output data for each of the plurality of images and image data for the subset of the plurality of images instead of image data of each of the plurality of images comprises: Send the image data and neural network output data for the first image, send the neural network output data for the second image, before and without sending the image data for the second image to the third image of the plurality of images and transmitting the neural network output data for the image data and the third image. For example, the image sensor 100 transmits image data of frame A, transmits neural network output data for frames AI, transmits image data for frame K, and transmits neural network output data for frames JT. can send

In some implementations, transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data of each of the plurality of images, comprises: transmitting the set of image data at a specific number of frames per second, and transmitting neural network output data for the images while image data for the subset of the plurality of images is not being transmitted. For example, the image sensor 100 may transmit image data at 10 frames per second when image data for each frame is transmitted for 10 milliseconds, and the remaining 900 milliseconds per second while image data is not being transmitted while sending neural network output data for other images.

In some implementations, transmitting the neural network output data and image data includes sending the neural network output data and image data to the central processing unit. For example, the image sensor 100 sends image data for frame A and neural network output data for frames AI to the computer, so that the computer detects additional objects using the neural network output data for frame A and frames AI can make you do

In some implementations, process 800 includes analyzing the second image of the plurality of images for object detection, while transmitting image data for the first image of the plurality of images and neural network output data for the first image. include For example, the neural network groups 132 of the third integrated circuit layer 130 may detect objects of the frame B while the neural network output data generated for the frame A is being output from the image sensor 100 . .

In some implementations, process 800 includes generating neural network output data related to results of analysis of the plurality of images for object detection, while capturing the third image. For example, neural network groups 132 of third integrated circuit layer 130 are neural networks associated with detecting objects in frame B while pixel groups of first integrated circuit layer 110 are capturing frame C. You can create output data.

In some implementations, in process 800 , the subset of the plurality of images for which image data is transmitted includes all Nth images captured by the image sensor, where N is an integer greater than one. For example, the subset of multiple images may be all tenth images captured by image sensor 100 . In some implementations, in process 800 , image data for a particular image of images includes data representing values for each pixel in the image. For example, the image data may be red-green-blue (RGB) intensity values for each pixel of the image.

In some implementations, in process 800 , the neural network output data for a particular image of images is selected regions of interest for pixels representing the detections, metadata including temporal and geometric position information, intermediate prior to object detection, and in some implementations. It includes one or more of calculation results, statistical information about the network certainty level, and classifications of detected objects. For example, neural network output data for frame A may indicate that a person has been detected at specific coordinates of frame A. In some implementations, at process 800 , neural network output data for a particular image of images represents partially processed data that is provided to a processor for further processing. For example, the neural network output data may represent partially processed data that must be further processed outside the image sensor 100 to detect the objects, instead of representing the detected objects.

9 is a block diagram of an example system 900 that includes an image sensor 100 used by an autonomous vehicle 910 . An autonomous vehicle 910 is a self-driving vehicle that captures images, detects objects in the images, and then drives based on the objects that the autonomous vehicle 910 detects. can For example, the autonomous vehicle 910 may detect another vehicle moving in front of the autonomous vehicle 910 and maintain a pace immediately behind the vehicle. In another example, the autonomous vehicle 910 may detect and stop a person in front of the autonomous vehicle 910 .

The autonomous vehicle 910 may include camera optics 920 , an image sensor 100 and a computer processing module 940 . The camera optical system 920 may include a lens that corrects light. For example, the camera optical system 920 may include a panoramic lens. The image sensor 100 may receive the light corrected by the camera optical system 920 , and capture images based on the light reflected on the image sensor 100 .

The image sensor 100 may transmit data to the computer processing module 940 through a data transmission cable electrically coupling the image sensor 100 and the computer processing module 940 . For example, the image sensor 100 captures a frame for 10 milliseconds every 100 milliseconds, and sends the image data for every tenth frame captured and the image sensor ( 100) can transmit neural network output data for all frames captured by . The computer processing module 940 receives image data and neural network output data from the image sensor 100 , optionally performs additional object detection using the image data and neural network output data, and autonomously drives based on the detected objects. It may determine how the vehicle 910 should move.

Using the image sensor 100 that performs at least some processing for object detection on the image sensor 100, the computer processing module 940 can determine how to move based on images acquired every 10 milliseconds. On the other hand, using another image sensor, the computer processing module 940 can determine how to move based on images acquired only every 100 milliseconds. Thus, by using the image sensor 100 instead of other image sensors, the autonomous vehicle 940 can detect objects more quickly and move in response to the objects.

10 is a block diagram of an example system 1000 that includes groups of pixel sensors and neural network circuitry on separate chips. System 1000 may be a camera comprising a first chip 1110 comprising pixel sensor groups and a second chip 1120 comprising neural network circuitry. The pixel sensor groups and neural network circuitry may be similar to those described above, except that they are not included in different layers within a single integrated chip, but instead are in first chip 1110 and second chip 1120 . For example, a first chip 1110 and a second chip 1120 may be positioned on top of each other in the system 1000 and connected by an electrically conductive material, so that the data can correspond to the pixel sensor groups and the neural network. It can be transmitted between networks.

Embodiments of the objects and operations described herein, including the structures disclosed herein and structural equivalents thereof, may be implemented in digital electronic circuitry, or in computer software, firmware or hardware, or combinations of one or more of these. can be implemented. Embodiments of the subject matter described herein may be implemented as one or more computer programs, ie, one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus. have.

A computer storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Further, while a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term "data processing apparatus" refers to all kinds of apparatus, devices for processing data, including, for example, a programmable processor, a computer, a system on a chip, or many of the foregoing or combinations thereof. and machines. The device may include special purpose logic circuitry, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The device may also include, in addition to hardware, code that creates an execution environment for the computer program in question, eg, processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or one or more of these. It may contain code that constitutes a combination of things. The device and execution environment may realize a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

A computer program (also referred to as a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, which alone program), or in any form, including modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be a part of a file that holds other programs or data (eg, one or more scripts stored in a markup language document), as a single file dedicated to that program, or as multiple coordinated files ( for example, one or more modules, sub-programs, or files that store portions of code). The computer program may be deployed to run on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communication network.

The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may be performed by a special purpose logic circuitry, for example, a field programmable gate array (FPGA) or application-specific integrated circuit (ASIC), and an apparatus may be implemented as such.

Processors suitable for the execution of a computer program include, by way of example, general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from either read-only memory or random access memory or both. The essential elements of a computer are a processor for performing actions according to the instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, eg, magnetic, magneto-optical disks, or optical disks to receive data from, or to transfer data thereto. may be operatively coupled to transmit data or both. However, the computer need not have these devices. Computers can also be used with other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning Systems (GPSs), to name just a few. ) receiver, or a portable storage device (eg, a universal serial bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media and memory devices including CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated with special purpose logic circuitry.

To provide interaction with a user, embodiments of the subject matter described herein include a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, and It may be implemented on a computer with a keyboard and pointing device, such as a mouse or trackball, through which a user can provide input to the computer. Other types of devices may also be used to provide interaction with a user, eg, the feedback provided to the user may be any form of sensory feedback, eg, visual feedback, auditory feedback or tactile feedback. may be, and the input from the user may be received in any form including acoustic, voice, or tactile input. In addition, the computer sends documents to and receives documents from the device used by the user, for example, by sending web pages to a web browser on the user's user device in response to requests received from the web browser. can interact with

Embodiments of the subject matter described herein include, for example, a back-end component as a data server, or a middleware component, such as an application server, or a front-end component, for example, including a user computer having a graphical user interface or web browser through which a user may interact with an implementation of the subject matter described herein, or including any combination of one or more such back-end, middleware, or front-end components It can be implemented as a computing system that The components of the system may be interconnected by digital data communication in any form or medium, for example, a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), inter-networks (eg, the Internet), and peer-to-peer networks ( for example, ad hoc peer-to-peer networks).

A computing system may include users and servers. A user and a server are generally remote from each other and typically interact through a communication network. The user-server relationship arises by virtue of computer programs running on individual computers and having a user-server relationship to each other. In some embodiments, the server sends data (eg, an HTML page) to the user device (eg, for the purpose of displaying data to, and receiving user input from, a user interacting with the user device) . Data generated at the user device (eg, a result of a user interaction) may be received from the user device at a server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed or any features, but rather as descriptions of features specific to particular embodiments. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Also, although features may be described above as acting in particular combinations and may even initially be claimed as such, one or more features from a claimed combination may in some cases be eliminated from the combination, and the claimed combination is a sub-combination. or to a variant of a sub-combination.

Similarly, while acts are shown in a particular order in the figures, it is understood that this requires that such acts be performed in the particular order shown or sequential order or that all illustrated acts be performed to achieve desirable results. shouldn't be In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, that the described program components and systems are generally integrated together into a single software product or It should be understood that it may be packaged into multiple software products.

Accordingly, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the appended drawings do not necessarily require the specific order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (40)

As an image sensor,
a first integrated circuit layer comprising pixel sensors grouped into pixel sensor groups based on position;
a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer comprising image processing circuitry groups each configured to receive pixel information from a corresponding pixel sensor group; groups (image processing circuitry groups) further configured to perform image processing operations on the pixel information to provide pixel information processed during operation of the image sensor;
a third integrated circuit layer in electrical communication with the second integrated circuit layer, the third integrated circuit layer comprising groups of neural networks each configured to receive the processed pixel information from a corresponding group of image processing networks; neural network groups further configured to perform analysis for object detection on the processed pixel information during operation of the image sensor; and
A network for outputting information representing results of analysis for object detection by the neural network groups
including,
wherein the first integrated circuit layer is laminated on the second integrated circuit layer, and the second integrated circuit layer is laminated on the third integrated circuit layer. According to claim 1, wherein the results of the analysis for the object detection,
selected regions of interest for pixels representing detections;
metadata including time and geometric location information;
intermediate computational results before object detection;
statistical information regarding network certainty level; and
Classifications of detected objects
At least one of the group consisting of, the image sensor. The image sensor of claim 1 , wherein the neural network network groups each include circuitry configured to implement a convolutional neural network. The image sensor of claim 1 , wherein the convolutional neural networks each detect objects sensed by a pixel sensor group corresponding to an image processing network group corresponding to the neural network group. The image sensor of claim 1 , wherein the third integrated circuit layer comprises circuitry configured to implement a recurrent neural network. The image sensor of claim 1 , wherein the recurrent neural network receives information about objects detected by all of the neural network groups and detects objects sensed across multiple of the pixel sensor groups. The image sensor of claim 1 , wherein each of the neural network groups is located directly below an image processing network group that provides the processed pixel information to the neural network group. The image sensor of claim 1 , wherein each of the image processing circuitry groups is located directly below or near a pixel sensor group that provides the pixel information to the image processing circuitry group. The image sensor of claim 1 , wherein the first integrated circuit layer, the second integrated circuit layer, and the third integrated circuit layer are integrated on a single integrated chip. The image sensor of claim 1 , wherein each of the pixel sensor groups includes the same number of pixel sensors. The image sensor of claim 1 , wherein image processing operations performed on the pixel information include high dynamic range fusion prior to processing. The image sensor of claim 1 , wherein each of the image processing circuitry groups comprises an analog-to-digital converter. As a method,
obtaining pixel information by pixel sensors in the first integrated circuit layer and grouped into pixel sensor groups based on position;
performing image processing operations on pixel information from a corresponding pixel sensor group to provide pixel information processed by image processing circuitry groups of a second integrated circuit layer in electrical communication with the first integrated circuit layer step;
performing, by neural network groups of a third integrated circuit layer in electrical communication with the second integrated circuit layer, analysis for object detection on processed pixel information from a corresponding image processing network group; and
outputting information representing results of analysis for object detection by the neural network groups
including,
wherein the first integrated circuit layer is laminated on the second integrated circuit layer, and the second integrated circuit layer is laminated on the third integrated circuit layer. The method of claim 13, wherein the results of the analysis for object detection are:
selected regions of interest for pixels representing detections;
metadata including time and geometric location information;
intermediate calculation results before object detection;
statistical information regarding network certainty level; and
Classifications of detected objects
A method comprising at least one of the group consisting of 13. The method of claim 12, wherein the neural network network groups each comprise circuitry configured to implement a convolutional neural network. The method of claim 12 , wherein the convolutional neural networks each detect objects sensed by a pixel sensor group corresponding to an image processing network group corresponding to the neural network network group. 13. The method of claim 12, wherein the third integrated circuit layer comprises circuitry configured to implement a recurrent neural network. 13. The method of claim 12, wherein the recurrent neural network receives information about objects detected by all of the neural network groups and detects objects sensed across multiple of the pixel sensor groups. 13. The method of claim 12, wherein each of the neural network groups is located directly below an image processing network group that provides the processed pixel information to the neural network group. A method of forming an image sensor, comprising:
forming a first integrated circuit layer comprising pixel sensors grouped into pixel sensor groups based on position;
electrically coupling the first integrated circuit layer and the second integrated circuit layer, the second integrated circuit layer comprising groups of image processing circuitry each configured to receive pixel information from a corresponding pixel sensor group; image processing circuitry groups further configured to perform image processing operations on the pixel information to provide pixel information processed during operation of the image sensor;
electrically coupling the second integrated circuit layer and a third integrated circuit layer, the third integrated circuit layer comprising neural network groups each configured to receive the processed pixel information from a corresponding image processing network group. and the neural network groups are further configured to perform analysis for object detection on detection objects from the processed pixel information during operation of the image sensor; and
electrically coupling a network for outputting information representing results of analysis for object detection by the neural network groups to the neural network groups;
including,
wherein the first integrated circuit layer is laminated on the second integrated circuit layer, and the second integrated circuit layer is laminated on the third integrated circuit layer. As a method,
acquiring a plurality of images captured by pixel sensors of the image sensor;
analyzing the plurality of images for object detection using a neural network integrated in the image sensor;
generating, for each of the plurality of images using a neural network integrated in the image sensor, neural network output data related to results of analysis of the plurality of images for object detection; and
transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images;
A method comprising 22. The method of claim 21, wherein transmitting neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images comprises:
before transmitting image data for a first image of the plurality of images and neural network output data for the first image, and transmitting neural network output data for a second image, before transmitting image data for the second image; Transmitting image data for a third image of the plurality of images and neural network output data for the third image without transmitting this
A method comprising 23. The method of claim 22,
analyzing a second image of the plurality of images for object detection while transmitting image data for the first image of the plurality of images and neural network output data for the first image;
A method comprising 23. The method of claim 22,
generating neural network output data related to results of analysis of the plurality of images for object detection while capturing the third image;
A method comprising 22. The method of claim 21, wherein transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images, comprises:
transmitting image data of the subset of the plurality of images at a specified number of frames per second, and transmitting neural network output data for the images while image data for the subset of the plurality of images is not being transmitted;
A method comprising 22. The method of claim 21, wherein the subset of the plurality of images includes all Nth images captured by the image sensor, where N is an integer greater than one. 22. The method of claim 21, wherein the image data for a particular image of the images comprises data representing values for each pixel in the image. The method of claim 21, wherein the neural network output data for a specific image of the images,
selected regions of interest for pixels representing detections;
metadata including time and geometric location information;
intermediate calculation results before object detection;
statistical information about the level of network certainty; and
Classifications of detected objects
A method comprising one or more of The method of claim 21 , wherein the neural network output data for a particular image of the images represents partially processed data provided to a processor for further processing. 22. The method of claim 21, wherein transmitting the neural network output data and the image data comprises sending the neural network output data and the image data to a central processing unit. An image sensor having circuitry configured to perform operations, the image sensor comprising:
actions are,
acquiring a plurality of images captured by pixel sensors of the image sensor;
analyzing a plurality of images for object detection using a neural network integrated in the image sensor;
generating, for each of the plurality of images, neural network output data related to results of analysis of the plurality of images for object detection using a neural network integrated in the image sensor; and
transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images
comprising, an image sensor, The method of claim 31 , wherein transmitting neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images comprises:
before transmitting image data for a first image of the plurality of images and neural network output data for the first image, and transmitting neural network output data for a second image, before transmitting image data for the second image; Transmitting image data for a third image of the plurality of images and neural network output data for the third image without transmitting this
Including, an image sensor. 33. The method of claim 32,
Analyze the second image of the plurality of images for object detection while transmitting image data for the first image of the plurality of images and neural network output data for the first image
Including, an image sensor. 34. The method of claim 33,
generating neural network output data related to results of analysis of the plurality of images for object detection while capturing the third image;
Including, an image sensor. The method of claim 31 , wherein transmitting, from the image sensor, neural network output data for each of the plurality of images and image data for a subset of the plurality of images instead of image data for each of the plurality of images, comprises:
transmitting image data of the subset of the plurality of images at a specified number of frames per second, and transmitting neural network output data for the images while the image data for the subset of the plurality of images is not being transmitted;
Including, an image sensor. The image sensor of claim 31 , wherein the subset of the plurality of images includes all Nth images captured by the image sensor, where N is an integer greater than one. 32. The image sensor of claim 31, wherein the image data for a particular image of the images comprises data representing values for each pixel in the image. The method of claim 31 , wherein the neural network output data for a specific image of the images comprises:
selected regions of interest for pixels representing detections;
metadata including time and geometric location information;
intermediate calculation results before object detection;
statistical information about the level of network certainty; and
Classifications of detected objects
An image sensor comprising one or more of: The image sensor of claim 31 , wherein the neural network output data for a particular image of the images represents partially processed data provided to a processor for further processing. The image sensor of claim 31 , wherein transmitting the neural network output data and the image data comprises sending the neural network output data and the image data to a central processing unit.

Images