TW202219890A - Sparse image sensing and processing - Google Patents

Abstract

In one example, an apparatus comprises: an image sensor comprising a plurality of pixel cells; a frame buffer; and a sensor compute circuit configured to: receive, from the frame buffer, a first image frame comprising first active pixels and first inactive pixels, the first active pixels being generated by a first subset of the pixel cells selected based on first programming data; perform an image-processing operation on a first subset of pixels of the first image frame, whereby a second subset of pixels of the first image frame are excluded from the image-processing operation, to generate a processing output; based on the processing output, generate second programming data; and transmit the second programming data to the image sensor to select a second subset of the pixel cells to generate second active pixels for a second image frame.

Description

Sparse Image Sensing and Processing

The present invention relates to an image sensor. More specifically, and not by way of limitation, this disclosure relates to techniques for performing sparse image sensing and processing operations.

A typical image sensor includes an array of pixel cells. Each pixel cell may include a photodiode to sense light by converting photons into electrical charges (eg, electrons or holes). The charge converted at each pixel cell can be quantized to become a digital pixel value, and an image can be generated from an array of digital pixel values.

The images generated by the image sensor can be processed to support different applications, such as virtual-reality (VR) applications, augmented-reality (AR), or mixed reality (mixed reality) ; MR) application. Image processing operations can then be performed on the image to detect an object of interest and its location in the image. Based on the detection of the object and its position in the image, the VR/AR/MR application can generate and update, for example, virtual image data for display to the user via a display, audio data for output to the user via a speaker, etc. , to provide users with an interactive experience.

To improve the spatial and temporal resolution of image manipulation, image sensors typically include a large number of pixel cells and generate images at high frame rates. Generating high-resolution image frames at high frame rates and transmitting and processing these high-resolution image frames can result in huge power consumption for image sensors and image processing operations. Furthermore, since typically only a small subset of pixel cells receive light from the object of interest, much power is wasted in generating, transmitting and processing pixel data that is not suitable for object detection/tracking operations, reducing image sensing and processing operations total efficiency.

The present invention relates to an image sensor. More specifically, and not by way of limitation, this disclosure relates to techniques for performing sparse image sensing and processing operations.

In one example, an apparatus is provided. The apparatus includes an image sensor, a frame buffer, and a sensor computing circuit. The image sensor includes a plurality of pixel cells, and the image sensor can be configured by programming data to select a subset of the pixel cells to generate active pixels. The sensor computing circuit is configured to: receive a first image frame including first active pixels and first inactive pixels from the frame buffer, the first active pixels being generated based on first programming data generating a first subset of the selected pixel units, the first inactive pixels corresponding to a second subset of the pixel units not selected for generating the first active pixels; a first subset of pixels of the first image frame performs an image processing operation to generate a processing output, wherein a second subset of pixels of the first image frame is excluded from the image processing operation; based on the processing output, generating second programming data; and transmitting the second programming data to the image sensor to select a second subset of the pixel cells to generate for a second image frame The second active pixel.

In some aspects, the image processing operation includes a processing operation of a neural network model to detect an object of interest in the first image frame. The first subset of pixels corresponds to the object of interest.

In some aspects, the sensor computing circuit is coupled to a host device for executing an application that uses the results of detection of the object of interest. The host device is configured to provide information about the object of interest to the sensor computing circuit.

In some aspects, the sensor computing circuit includes: a computing memory configured to store: input data to a neural network layer of the neural network, weight data for the neural network layer, and the neural network intermediate output data of the network layer; a data processing circuit configured to perform arithmetic operations of the neural network layer on the input data and the weight data to generate the intermediate output data; and a computation controller configured to: extract from the computing memory a first subset of the input data and a first subset of the weight data corresponding to the first subset of the input data, the first subset of the input data corresponding to on at least some of the first active pixels; controlling the data processing circuit to perform the arithmetic operations on the first subset of the input data and the first subset of the weight data to generate data for the first a first subset of the intermediate output data of the image frame, the first subset of the intermediate output data corresponding to the first subset of the input data; to be used for the intermediate output of the first image frame storing the first subset of data in the computational memory; and storing in the computational memory a predetermined value for a second subset of the intermediate output data of the first image frame, the intermediate The second subset of output data corresponds to the inactive pixels.

In some aspects, the predetermined value is stored based on resetting the computing memory prior to the image processing operation.

In some aspects, the computing controller is configured to: extract the input data from the computing memory; identify the first subset of the input data from the extracted input data; and identify the first subset of the input data A subset is provided to the compute controller.

In some aspects, the computing controller is configured to: determine an address region of the computing memory that stores the first subset of the input data; and retrieve the first region of the input data from the computing memory Subset.

In some aspects, the address region is determined based on at least one of: the first programmed data, or information about connectivity between neural network layers of the neural network model.

In some aspects, the first active pixels include static pixels and non-static pixels; the static pixels correspond to a first subset of the first active pixels, for the first one of the first active pixels a subset, the degree of change in pixel values between the first image frame and a previous image frame is higher than a change threshold; the non-static pixels correspond to a second subset of the first active pixels , for the second subset of the first active pixels, the degree of change of the pixel values between the first image frame and the previous image frame is lower than the change threshold; and the computing controller is configured to extract the first subset of the input data corresponding to the non-static pixels of the first active pixels.

In some aspects, the predetermined value is a first predetermined value. The frame buffer is configured to store a second predetermined value for each of the static pixels to identify the static pixels. The computing controller is configured to exclude the static pixels from the data processing circuit based on detecting that the static pixels have the second predetermined value.

In some aspects, the frame buffer is configured to store the second predetermined value for a pixel across a threshold number of frames based on a determination that the degree of change of the pixel is below the change threshold .

In some aspects, the frame buffer is configured to set the time to update a pixel based on a leaky integrator function having a time constant and based on the last time a pixel experienced a degree of change greater than one of the change thresholds A pixel value.

In some aspects, the computational controller is configured to: determine a data change propagation map based on a topology of the neural network model, the data change propagation map indicating how changes to the non-static pixels are routed through the neural network Different neural network layers of the network model are propagated; the propagation map is changed based on the data, a first address region of the computing memory is determined, the first subset of the input data is extracted, and the computing memory is determined a second address area for storing the first subset of the intermediate output data; extracting the first subset of the input data from the first address area; and storing the first subset of the input data in the second address area The first subset of intermediate output data.

In some aspects, the computational controller is configured to determine the change threshold based on a depth of the neural network model and a quantitative accuracy at each neural network layer of the neural network model.

In some aspects, the change threshold is a first change threshold. The computing controller is configured to: track the degree of change in the pixel values of the first active pixels between two non-consecutive frames; and based on the degree of change exceeding a second change threshold A third subset of the first active pixels is determined to be a non-static pixel.

In some aspects, the image sensor is implemented in a first semiconductor substrate. The frame buffer and the sensor computing circuit are implemented in one or more second semiconductor substrates. The first semiconductor substrate and the one or more second semiconductor substrates form a stack and are housed in a single semiconductor package.

In some instances, a method is provided. The method includes: transmitting first programming data to an image sensor including a plurality of pixel units to select a first subset of the pixel units to generate a first active pixel; receiving from a frame buffer including A first image frame of the first active pixels and first inactive pixels, the first inactive pixels corresponding to a second one of the pixel units not selected for generating the first active pixels subset; performing an image processing operation on a first subset of pixels of the first image frame to generate a processing output in which a second subset of pixels of the first image frame is excluded from the image In addition to processing operations; based on the processing output, generating second programming data; and transmitting the second programming data to the image sensor to select a second subset of the pixel cells to generate for a first The second active pixel of the two image frames.

In some aspects, the image processing operation includes a processing operation of a neural network to detect an object of interest in the first image frame. The first subset of pixels corresponds to the object of interest.

In some aspects, the method further includes: storing in a computational memory input data to a neural network layer of the neural network, weight data of the neural network layer; extracting the input data from the computational memory a first subset and a first subset of the weight data corresponding to the first subset of the input data corresponding to at least some of the first active pixels ; using a data processing circuit to perform arithmetic operations on the first subset of input data and the first subset of weight data to generate a first subset of intermediate output data for the first image frame, the first subset of the intermediate output data corresponds to the first subset of the input data; storing the first subset of the intermediate output data for the first image frame in the computing memory; and A predetermined value for a second subset of the intermediate output data of the first image frame is stored in the computing memory, the second subset of the intermediate output data corresponding to the inactive pixels.

In some aspects, the first active pixels include static pixels and non-static pixels. The static pixels correspond to a first subset of the first active pixels, and for the first subset of the first active pixels, pixel values between the first image frame and a previous image frame The degree of change is higher than a change threshold. The non-static pixels correspond to a second subset of the first active pixels, for the second subset of the first active pixels, the distance between the first image frame and the previous image frame The degree of change of some pixel values is lower than the change threshold value. The first subset of the input data corresponds to the non-static pixels of the first active pixels.

In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain inventive examples. It will be apparent, however, that various examples may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

As discussed above, image sensors typically include a large number of pixel cells and produce images at high frame rates to improve the spatial and temporal resolution of imaging operations. However, generating high-resolution image frames and transmitting and processing these high-resolution image frames at high frame rates results in huge power consumption for image sensors and image processing operations. Furthermore, since typically only a small subset of pixel cells receive light from the object of interest, much power is wasted in generating, transmitting and processing pixel data that is not suitable for object detection/tracking operations, reducing image sensing and processing operations total efficiency.

The present invention proposes image sensing and processing techniques that can solve at least some of the above problems. In some examples, the apparatus includes an image sensor, a frame buffer, and computing circuitry. An image sensor includes a plurality of pixel cells that can be configured by programming data to select a subset of pixel cells to generate active pixels. The frame buffer may store a first image frame including at least some of the active pixels generated by a first subset of pixel cells selected by the image sensor based on first programming data. The first image frame further includes inactive pixels corresponding to a second subset of pixel cells not selected for generating active pixels. The computing circuit may receive the first image frame from the frame buffer. The computing circuit may include an image processor to perform image processing operations on a first subset of pixels of the first image frame to generate a processing output, wherein a second subset of pixels of the first image frame are excluded from the image processing outside the operation. The computing circuit further includes a programmed map generator to generate second programmed data based on the processing output from the image processor, and to transmit the second programmed data to the image sensor to select a second subset of pixel cells to output pixel data for the second image frame. The first subset of pixels of the first image frame for which the image processing operation is performed may correspond, for example, to active pixels, which are non-static pixels that undergo some degree of change between frames, or the like.

In some examples, the apparatus may support object detection and tracking operations based on sparse image sensing operations. A first subset of pixel cells can be selectively enabled to capture only pixel data related to tracking and detecting objects as active pixels, or to transfer only active pixels to a frame buffer to support sparse image sensing operations . Because only a subset of pixel cells are enabled to generate and/or transmit active pixels, the amount of pixel data generated/transmitted for an image frame can be reduced, which can reduce power consumption at the image sensor. Sparse image sensing operations can be continuously adjusted based on the results of object detection and tracking operations to account for relative movement of objects relative to the image sensor, which improves the likelihood that active pixels include object image data and improves object-dependent The performance of applications that detect and track actions (eg, VR/AR/MR applications). In addition, computing circuitry performs image processing operations on only active pixels or a subset of active pixels that may include image data of an object, while non-active pixels are excluded from image processing operations, which further reduces power consumption of image processing operations. All of these operations can improve the overall power and computational efficiency and performance of image sensing and processing operations.

In some examples, image processing operations may include neural network operations. Specifically, the image processor may include data processing circuitry for providing hardware acceleration for neural network operations, such as a multi-layer convolutional neural network (CNN) including an input layer and an output layer. . The image processor may include computational memory to store the input image frame and a set of weights associated with each neural network layer. The set of weights may represent characteristics of the object to be detected. The image processor may include a controller that controls the data processing circuit to extract input image frame data and weights from the computing memory. The controller can control the data processing circuitry to perform arithmetic operations, such as multiply-and-accumulate (MAC) operations, between the input image frames and the weights to generate intermediate output data for the input layer. For example, intermediate output data is post-processed based on excitation functions, merge operations, etc., and then the post-processed intermediate output data may be stored in computational memory. Post-processed intermediate output data can be extracted from computational memory and provided to the next neural network layer as input. The arithmetic operations and the extraction and storage of intermediate output data are repeated for all layers up to the output layer to generate the neural network output. The neural network output may indicate, for example, the likelihood of the object being present in the input image frame and the pixel location of the object in the input image frame.

The controller can configure the data processing circuitry to process sparse image data in an efficient manner. For example, for the input layer, the controller may control the data processing circuit to extract only active pixels and corresponding weights from computational memory, and perform MAC operations on only the active pixels and corresponding weights to generate corresponding values for the input layer. A subset of the intermediate outputs of the active pixels. The controller can also determine a subset of intermediate output data at each subsequent neural network, which can be traced back to the active pixel, based on the topology of the neural network and connections among subsequent neural network layers. The controller can control the data processing circuitry to perform MAC operations to generate only a subset of the intermediate output data at each subsequent neural network layer. Additionally, to reduce access to computational memory, predetermined values (eg, zeros) for the intermediate output data of each layer may be stored in computational memory prior to neural network operations. Only the intermediate output data for active pixels is updated. All of these operations can reduce power consumption for neural network operations on sparse image data.

In some examples, to further reduce power consumption and improve power and computational efficiency, frame buffers and computational circuits may support temporal sparse operation. As part of the temporal thinning operation, static pixels and non-static pixels can be identified. Static pixels may correspond to the first portion of the scene captured by the image sensor that undergoes small changes (or no changes) between the first image frame and the previous image frame, while non-static pixels correspond to portions of the scene. The second part, which undergoes a large change between the first image frame and the previous image frame. If the degree of change of the pixel is below a threshold value, the pixel can be determined to be static. In some examples, non-static pixels can be identified from active pixels, while static pixels can be identified from both active and inactive pixels, which remain inactive (and unchanged) between frames.

To reduce power consumption, the data processing circuitry may perform image processing operations (eg, neural network operations) only on non-static pixels of the first image frame to generate updated outputs for the non-static pixels. For static pixels, the image processing operation can be skipped and the output from the image processing operation on the previous image frame can be retained. In the case where the image processing operations include neural network operations, the controller may control the data processing circuit to extract only non-static pixels and corresponding weight data from the computational memory to update the data corresponding to the non-static pixels used in the input layer. A subset of intermediate output data. The remaining intermediate output data in computational memory corresponding to static pixels (obtained from previous image frames) and corresponding to inactive pixels (eg, having predetermined values such as zero) may be retained for the input layer. The controller may also determine a subset of intermediate output data at each subsequent neural network that can be traced back to non-static pixels based on the topology of the neural network and connections among subsequent neural network layers, and Only a subset of the intermediate output data is updated to reduce access to computing memory and reduce power consumption.

In some examples, a frame buffer may detect static pixels from active pixels output by the image sensor, and store pixel values for those pixels to indicate to the image processor that the pixels are static pixels. For example, the frame buffer may store the latest pixel data (including active and inactive pixels) from each pixel unit of the image sensor as the first image frame. For each of the active pixels, the frame buffer may determine how much the pixel has changed relative to a previous frame, such as the image frame immediately preceding the first image frame. The frame buffer may set pixel values in various ways to indicate static pixels. For example, a frame buffer may be based on a leaky integrator function with a time constant and based on a number of consecutive image frames across which the pixels output by the image sensor remain static for use in the frame buffer pixel value in pixel. If the pixel remains static for a large number of consecutive image frames, the pixel value of the pixel may stabilize at a predetermined pixel value. As another example, if the pixel remains static for a threshold number of consecutive image frames (eg, 10), the frame buffer may set a predetermined pixel value for the pixel in the frame buffer. The predetermined pixel value may correspond to black (zero), white (255), gray (128), or any value indicative of a static pixel. In all such cases, the image processor may distinguish static pixels from non-static pixels based on identifying pixel values that identify static pixels, and perform image processing operations only on non-static pixels as described above.

In some examples, the image processor may also generate additional information to facilitate processing of non-static pixels. For example, the image processor may determine a data change propagation map based on the topology of the model that tracks the propagation of data changes from the input layer to the output layer of the neural network model. Based on the propagation map, and static pixels from the frame buffer, the image processor can identify non-static input data for each neural network and extract only those inputs for neural network operations at each layer material. In addition, the image processor can also determine the threshold change degree for static/non-static pixel determination based on the topology of the neural network model to ensure that pixels determined to be non-static can cause the necessary changes at the output layer degree. In addition, the image processor can also track pixel changes between consecutive frames and between non-consecutive frames. The image processor can identify pixels that exhibit small changes between consecutive frames and also identify large changes between non-consecutive frames as non-static pixels so that the image processor can perform image processing operations on those pixels.

With the disclosed techniques, an image sensor can be configured to perform sparse image sensing operations to generate sparse images, which can reduce power consumption at the image sensor. Furthermore, the image processor can be configured to perform image processing operations only on active and/or non-static pixels, while skipping image processing operations on inactive and/or static pixels, which can further reduce power consumption. Additionally, the selection of pixel cells to generate active pixels may be based on image processing results to ensure that active pixels contain relevant information (eg, an image of the object of interest). All of these operations can improve the power and computational efficiency of image sensors and image processors.

The disclosed techniques may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality or a combination and/or derivative thereof. Artificial reality content may include generated content entirely or in combination with captured (eg, real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels, such as stereo video that produces a three-dimensional effect on the viewer. Additionally, in some instances, an artificial environment may also be associated with applications, products, and/or applications that are used, for example, to create content in the artificial environment and/or otherwise be used in the artificial environment (eg, to perform activities in the artificial environment). , accessories, services, or some combination thereof. AR systems that provide AR content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to host computer systems, stand-alone HMDs, mobile devices or computing systems, or Any other hardware platform where one or more viewers provide artificial reality content.

FIG. 1A is a diagram of an example of a near-eye display 100 . The near-eye display 100 presents media to the user. Examples of media presented by near-eye display 100 include one or more images, video, and/or audio. In some examples, audio is presented via external devices (eg, speakers and/or headphones) that receive audio information from near-eye display 100, a console, or both, and present audio data based on the audio information. The near-eye display 100 is generally configured to function as a virtual reality (VR) display. In some examples, the near-eye display 100 is modified for use as an augmented reality (AR) display and/or a mixed reality (MR) display.

The near-eye display 100 includes a frame 105 and a display 110 . Frame 105 is coupled to one or more optical elements. Display 110 is configured to allow a user to see content presented by near-eye display 100 . In some examples, display 110 includes a waveguide display assembly for directing light from one or more images to the user's eye.

The near-eye display 100 further includes image sensors 120a, 120b, 120c, and 120d. Each of image sensors 120a, 120b, 120c, and 120d may include an array of pixels configured to generate image data representing different fields of view along different directions. For example, sensors 120a and 120b can be configured to provide image data representing two fields of view oriented in direction A along the Z-axis, while sensor 120c can be configured to provide image data representing two fields of view oriented along the X-axis image data for the field of view in direction B, and sensor 120d may be configured to provide image data representing the field of view toward direction C along the x-axis.

In some examples, the sensors 120a - 120d may be configured as input devices to control or affect the display content of the near-eye display 100 to provide an interactive VR/AR/MR experience to a user wearing the near-eye display 100 . For example, the sensors 120a to 120d may generate physical image data of the physical environment in which the user is located. The physical image data can be provided to a location tracking system to track the user's location and/or movement path in the physical environment. The system may then update the image data provided to display 110 based on, for example, the user's location and orientation to provide an interactive experience. In some examples, the location tracking system may operate a simultaneous localization and mapping (SLAM) algorithm to track a set of objects in the physical environment and within the user's field of view as the user moves within the physical environment. The location tracking system can construct and update a map of the physical environment based on the set of objects, and track the user's location within the map. By providing image data corresponding to multiple fields of view, sensors 120a-120d can provide the location tracking system with a more holistic view of the physical environment, which can result in more objects being included in the construction and updating of the map. With this configuration, the accuracy and stability of tracking the user's position within the physical environment can be improved.

In some examples, the near-eye display 100 may further include one or more active illuminators 130 to project light into the physical environment. The projected light can be associated with different spectrums (eg, visible light, infrared (IR) light, ultraviolet light) and can be used for various purposes. For example, the illuminator 130 may project light in a dark environment (or in an environment with low intensity (IR) light, ultraviolet light, etc.) to assist the sensors 120a - 120d in capturing the different objects within the dark environment images, to enable, for example, location tracking of users. Illuminators 130 may project certain markers onto objects within the environment to assist the location tracking system in identifying objects for map building/updating.

In some instances, the illuminator 130 may also enable stereoscopic imaging. For example, one or more of sensors 120a or 120b may include both a first pixel array for visible light sensing and a second pixel array for (IR) light sensing. The first pixel array may be overlaid with a color filter (eg, a Bayer filter), where each pixel in the first pixel array is configured to measure a specific color (eg, red, green, or blue). , green or blue; one of RGB)) the associated light intensity. The second pixel array (for IR light sensing) can also be covered with a filter that allows only IR light to pass through, wherein each pixel in the second pixel array is configured to measure the intensity of the IR light. The pixel array can generate an RGB image and an IR image of the object, where each pixel of the IR image is mapped to each pixel of the RGB image. The illuminator 130 can project a set of IR markers on the object, and the image of the object can be captured by the IR pixel array. Based on the distribution of the IR markers of the object as shown in the image, the system can estimate the distances of different parts of the object from the IR pixel array and generate a stereoscopic image of the object based on the distances. Based on the stereoscopic image of the object, the system can determine, for example, the relative position of the object relative to the user, and can update the image data provided to the display 100 based on the relative position information to provide an interactive experience.

As discussed above, the near-eye display 100 can operate in environments associated with a wide range of light intensities. For example, the near-eye display 100 may operate in an indoor environment or in an outdoor environment and/or at different times of the day. The near-eye display 100 may also operate with the active illuminator 130 on or off. Thus, image sensors 120a-120d would need to have a wide dynamic range to be able to operate properly across the very wide range of light intensities associated with the different operating environments for near-eye display 100 (eg, to generate a intensity-dependent output).

FIG. 1B is a diagram of another example of a near-eye display 100 . FIG. 1B illustrates the side of the near-eye display 100 that faces the eyeball 135 of the user wearing the near-eye display 100 . As shown in FIG. IB , the near-eye display 100 may further include a plurality of illuminators 140a, 140b, 140c, 140d, 140e, and 140f. The near-eye display 100 further includes a plurality of image sensors 150a and 150b. Illuminators 140a, 140b, and 140c may emit light in a certain frequency range (eg, near-infrared (NIR)) toward direction D, which is opposite direction A of FIG. 1A . The emitted light can be associated with a pattern and can be reflected by the user's left eyeball. Sensor 150a may include an array of pixels to receive reflected light and generate an image of the reflected pattern. Similarly, illuminators 140d, 140e, and 140f may emit pattern-bearing NIR light. The NIR light may be reflected by the user's right eyeball and may be received by the sensor 150b. The sensor 150b may also include an array of pixels to generate an image of the reflected pattern. Based on the images of the reflected patterns from sensors 150a and 150b, the system can determine the user's gaze point and update the image data provided to the display 100 based on the determined gaze point to provide an interactive experience to the user.

As discussed above, in order to avoid damage to the user's eye, the illuminators 140a, 140b, 140c, 140d, 140e, and 140f are typically configured to output low-intensity light. In the case where the image sensors 150a and 150b comprise the same sensor device as the image sensors 120a-120d of FIG. 1A , when the intensity of the incident light is low, the image sensors 120a-120d may need to be able to generate a The intensity-dependent output of the incident light can further increase the dynamic range requirements of the image sensor.

In addition, the image sensors 120a-120d would need to be able to generate output at high speed to track eye movement. For example, a user's eyeballs may perform extremely fast movements (eg, saccade movements), where there may be rapid jumps from one eyeball position to another. In order to track the fast movement of the user's eyeballs, the image sensors 120a to 120d need to generate images of the eyeballs at high speed. For example, the rate at which an image sensor generates an image frame (frame rate) needs to at least match the movement speed of the eyeball. A high frame rate requires a short total exposure time for all pixel units involved in generating an image frame, as well as a high speed for converting the sensor output into digital values for image generation. Furthermore, as discussed above, image sensors also need to be able to operate in environments with low light intensities.

FIG. 2 is an example of a cross-section 200 of the near-eye display 100 illustrated in FIG. 1 . Display 110 includes at least one waveguide display assembly 210 . The exit pupil 230 is where the user's single eyeball 220 is positioned in the orbital region when the user wears the near-eye display 100 . For purposes of illustration, Figure 2 shows a cross-section 200 associated with an eyeball 220 and a single waveguide display assembly 210, but with a second waveguide display for the user's second eye.

The waveguide display assembly 210 is configured to direct image light to the orbit at the exit pupil 230 and to the eyeball 220 . The waveguide display assembly 210 may be constructed of one or more materials (eg, plastic, glass) having one or more indices of refraction. In some examples, near-eye display 100 includes one or more optical elements between waveguide display assembly 210 and eyeball 220 .

In some examples, waveguide display assembly 210 includes a stack of one or more waveguide displays, including but not limited to stacked waveguide displays, zoom waveguide displays, and the like. Stacked waveguide displays are multicolor displays (eg, RGB displays) created by stacking waveguide displays with different colors for each monochromatic source. Stacked waveguide displays are also multi-color displays that can be projected on multiple planes (eg, multi-plane color displays). In some configurations, the stacked waveguide display is a monochrome display that can be projected on multiple planes (eg, a multi-plane monochrome display). A zoom waveguide display is a display that can adjust the focus position of the image light emitted from the waveguide display. In an alternate example, the waveguide display assembly 210 may include a stacked waveguide display and a zoom waveguide display.

FIG. 3 illustrates an isometric view of an example of a waveguide display 300 . In some examples, waveguide display 300 is a component of near-eye display 100 (eg, waveguide display assembly 210). In some examples, waveguide display 300 is part of some other near-eye display or other system that directs image light to a particular location.

The waveguide display 300 includes a source assembly 310 , an output waveguide 320 and a controller 330 . For purposes of illustration, FIG. 3 shows a waveguide display 300 associated with a single eyeball 220, but in some instances another waveguide display separate or partially separate from the waveguide display 300 provides image light to the other eye of the user .

Source assembly 310 produces image light 355 . Source assembly 310 generates image light 355 and outputs it to coupling element 350 on first side 370 - 1 of output waveguide 320 . The output waveguide 320 is an optical waveguide that outputs the expanded image light 340 to the eyeball 220 of the user. The output waveguide 320 receives the image light 355 at one or more coupling elements 350 located on the first side 370 - 1 and guides the received input image light 355 to the guide element 360 . In some examples, coupling element 350 couples image light 355 from source assembly 310 into output waveguide 320 . Coupling element 350 may be, for example, a diffraction grating, a holographic grating, one or more cascaded reflectors, one or more crystalline surface elements, and/or an array of holographic reflectors.

The directing element 360 redirects the received input image light 355 to the decoupling element 365 such that the received input image light 355 is decoupled from the output waveguide 320 via the decoupling element 365 . The guide element 360 is part of or attached to the first side 370 - 1 of the output waveguide 320 . The decoupling element 365 is part of or attached to the second side 370 - 2 of the output waveguide 320 such that the guiding element 360 is opposite the decoupling element 365 . The guiding element 360 and/or the decoupling element 365 may be, for example, a diffraction grating, a holographic grating, one or more cascaded reflectors, one or more homing surface elements, and/or an array of holographic reflectors.

The second side 370-2 represents a plane along the x- and y-dimensions. The output waveguide 320 may be constructed of one or more materials that promote total internal reflection of the image light 355 . The output waveguide 320 may be constructed of, for example, silicon, plastic, glass, and/or polymers. The output waveguide 320 has a relatively small external size. For example, the output waveguide 320 may be approximately 50 mm wide along the x dimension, approximately 30 mm long along the y dimension, and approximately 0.5 to 1 mm thick along the z dimension.

The controller 330 controls the scanning operation of the source assembly 310 . Controller 330 determines scan commands for source assembly 310 . In some examples, the output waveguide 320 outputs the expanded image light 340 to the eyeball 220 of the user with a large field of view (FOV). For example, the expanded image light 340 is provided to the user's eyeball 220 having a diagonal FOV (in x and y) of 60 degrees and/or more and/or 150 degrees and/or less. The output waveguide 320 is configured to provide an eye socket having a length of 20 mm or more and/or equal to or less than 50 mm; and/or a width of 10 mm or more and/or equal to or less than 50 mm.

In addition, controller 330 also controls image light 355 generated by source assembly 310 based on the image data provided by image sensor 370 . Image sensor 370 may be located on first side 370-1 and may include, for example, image sensors 120a-120d of FIG. 1A . Image sensors 120a-120d are operable to perform, for example, 2D sensing and 3D sensing of objects 372 in front of the user (eg, facing first side 370-1). For 2D sensing, each pixel cell of image sensors 120a-120d may be operated to generate pixel data representing the intensity of light 374 generated by light source 376 and reflected from object 372. For 3D sensing, each pixel cell of image sensors 120a-120d may be operated to generate pixel data representing a time-of-flight measurement of light 378 produced by illuminator 325. For example, each pixel unit of image sensors 120a-120d can determine a first time when illuminator 325 is enabled to project light 378 and a second time when the pixel unit detects light 378 reflected from object 372. The difference between the first time and the second time may indicate the time of flight of light 378 between image sensors 120a-120d and object 372, and the time-of-flight information may be used to determine the distance between image sensors 120a-120d and object 372. distance between. Image sensors 120a-120d are operable to perform 2D and 3D sensing at different times, and the image sensors provide 2D and 3D image data to a remote console 390, which may (or may not) ) within the waveguide display 300. The remote console can combine 2D and 3D images to, for example, generate a 3D model of the environment in which the user is located, to track the user's position and/or orientation, and the like. The remote console can determine the content of the image to be displayed to the user based on information derived from the 2D and 3D images. The remote console may transmit instructions related to the determined content to the controller 330 . Based on the instructions, the controller 330 can control the generation and output of the image light 355 by the source assembly 310 to provide an interactive experience to the user.

FIG. 4 illustrates an example of a cross-section 400 of waveguide display 300 . Cross section 400 includes source assembly 310 , output waveguide 320 and image sensor 370 . In the example of FIG. 4 , the image sensor 370 may include a set of pixel cells 402 on the first side 370-1 to generate an image of the physical environment in front of the user. In some examples, there may be a mechanical shutter 404 and an optical filter array 406 interposed between the set of pixel cells 402 and the physical environment. The mechanical shutter 404 can control the exposure of the group of pixel units 402 . In some examples, the mechanical shutter 404 may be replaced by an electronic shutter, as will be discussed below. Optical filter array 406 can control an optical wavelength range of light to which the set of pixel elements 402 are exposed, as will be discussed below. Each of pixel cells 402 may correspond to a pixel of the image. Although not shown in Figure 4 , it should be understood that each of the pixel cells 402 may also overlap a filter to control the optical wavelength range of light to be sensed by the pixel cells.

After receiving a command from the remote console, the mechanical shutter 404 can be opened and the set of pixel elements 402 will be exposed for an exposure cycle. During the exposure period, the image sensor 370 obtains samples of light incident on the set of pixel units 402 and generates image data based on the intensity distribution of the incident light samples detected by the set of pixel units 402 . Image sensor 370 may then provide image data to a remote console, which determines the display content and provides display content information to controller 330 . Controller 330 may then determine image light 355 based on the display content information.

Source assembly 310 generates image light 355 according to instructions from controller 330 . Source assembly 310 includes source 410 and optical system 415 . Source 410 is a light source that produces coherent or partially coherent light. The source 410 can be, for example, a laser diode, a vertical cavity surface emitting laser, and/or a light emitting diode.

Optical system 415 includes one or more optical components that condition light from source 410 . Adjusting the light from source 410 may include spreading, collimating, and/or adjusting orientation, eg, according to instructions from controller 330 . The one or more optical components may include one or more lenses, liquid lenses, mirrors, apertures and/or gratings. In some examples, the optical system 415 includes a liquid lens with a plurality of electrodes that allow scanning of the beam with a threshold value of scan angle to shift the beam to a region outside the liquid lens. Light emitted from optical system 415 (and source assembly 310 ) is referred to as image light 355 .

The output waveguide 320 receives the image light 355 . Coupling element 350 couples image light 355 from source assembly 310 into output waveguide 320 . In instances where the coupling element 350 is a diffraction grating, the diffraction grating spacing is selected such that total internal reflection occurs in the output waveguide 320 and the image light 355 is directed toward the solution inside the output waveguide 320 (eg, by total internal reflection) Coupling element 365 propagates.

Guiding element 360 redirects image light 355 towards decoupling element 365 to decouple from output waveguide 320 . In instances where the guiding elements 360 are diffraction gratings, the spacing between the diffraction gratings is selected so that the incident image light 355 exits the output waveguide 320 at an oblique angle relative to the surface of the decoupling element 365 .

In some examples, guide element 360 and/or decoupling element 365 are similar in structure. Expanded image light 340 exiting output waveguide 320 expands along one or more dimensions (eg, may be elongated along the x-dimension). In some examples, waveguide display 300 includes a plurality of source assemblies 310 and a plurality of output waveguides 320 . Each of the source assemblies 310 emits monochromatic image light having a particular wavelength band corresponding to a primary color (eg, red, green, or blue). Each of the output waveguides 320 can be stacked together by a separation distance to output multi-colored expanded image light 340 .

5 is a block diagram of an example of a system 500 including a near-eye display 100. System 500 includes near-eye display 100, imaging device 535, input/output interface 540, and image sensors 120a-120d and 150a-150b, each coupled to control circuit 510. System 500 may be configured as a head-mounted device, a mobile device, a wearable device, or the like.

The near-eye display 100 is a display that presents media to a user. Examples of media presented by near-eye display 100 include one or more images, video, and/or audio. In some examples, the audio is presented via an external device (eg, speakers and/or headphones) that receives the audio information from the near-eye display 100 and/or the control circuit 510 and presents the audio data based on the audio information to the user render. In some instances, the near-eye display 100 may also function as AR glasses. In some examples, the near-eye display 100 augments the view of the physical real-world environment with computer-generated elements (eg, images, video, sound).

The near-eye display 100 includes a waveguide display assembly 210 , one or more position sensors 525 and/or an inertial measurement unit (IMU) 530 . The waveguide display assembly 210 includes a source assembly 310 , an output waveguide 320 and a controller 330 .

The IMU 530 is an electronic device that generates fast calibration data based on measurement signals received from one or more of the position sensors 525 , the fast calibration data indicating the estimated position of the near-eye display 100 relative to the initial position of the near-eye display 100 .

Imaging device 535 may generate image data for various applications. For example, imaging device 535 may generate image data to provide slow calibration data based on calibration parameters received from control circuit 510 . Imaging device 535 may include, for example, image sensors 120a to 120d of FIG. 1A for generating image data of the physical environment in which the user is located for performing position tracking of the user. The imaging device 535 may further include, for example, the image sensors 150a to 150b of FIG. 1B for generating image data for determining the user's gaze point to identify the user's object of interest.

The input/output interface 540 is a device that allows the user to send action requests to the control circuit 510 . An action request is a request to perform a specific action. For example, an action request may be to start or end an application or to perform a specific action within the application.

Control circuitry 510 provides media to near-eye display 100 for presentation to the user based on information received from one or more of: imaging device 535 , near-eye display 100 , and input/output interface 540 . In some examples, control circuitry 510 may be housed within system 500 configured as a head-mounted device. In some examples, control circuit 510 may be a separate console device communicatively coupled with other components of system 500 . In the example shown in FIG. 5 , control circuit 510 includes application storage 545 , tracking module 550 , and engine 555 .

Application storage 545 stores one or more applications for execution by control circuit 510 . An application is a set of instructions that, when executed by a processor, produce content for presentation to a user. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

The tracking module 550 uses one or more calibration parameters to calibrate the system 500 and can adjust the one or more calibration parameters to reduce errors in the determination of the position of the near-eye display 100 .

The tracking module 550 uses the slow calibration information from the imaging device 535 to track the movement of the near-eye display 100 . The tracking module 550 also uses the position information from the quick calibration information to determine the position of the reference point of the near-eye display 100 .

Engine 555 executes applications within system 500 and receives position information, acceleration information, velocity information, and/or predicted future position of near-eye display 100 from tracking module 550 . In some examples, information received by engine 555 may be used to generate signals (eg, display commands) to waveguide display assembly 210 that determine the type of content presented to the user. For example, to provide an interactive experience, the engine 555 may be based on the user's location (eg, provided by the tracking module 550 ) or the user's gaze point (eg, based on image data provided by the imaging device 535 ) or object and The distance between users (eg, based on image data provided by imaging device 535 ) determines the content to be presented to the user.

6A , 6B , 6C , and 6D illustrate examples of image sensor 600 and its operation. As shown in FIG. 6A , image sensor 600 may include an array of pixel cells, including pixel cell 601, and may generate digital intensity data corresponding to pixels of an image. Pixel unit 601 may be part of pixel unit 402 of FIG. 4 . As shown in FIG. 6A , pixel cell 601 may include photodiode 602 , electronic shutter switch 603 , transfer switch 604 , charge storage device 605 , buffer 606 , and quantizer 607 . Photodiode 602 may include, for example, a PN diode, a PIN diode, a fixed diode, etc., while charge storage device 605 may be the floating drain node of transfer switch 604 . The photodiode 602 may generate and accumulate residual charge upon receiving light during an exposure period. After being saturated by the residual charge during the exposure period, the photodiode 602 can output the overflow charge to the charge storage device 605 via the switch 604 . Charge storage device 605 can convert the overflow charge to a voltage, which can be buffered by buffer 606 . The buffered voltage can be quantized by quantizer 607 to produce measurement data 608 representing, for example, the intensity of light received by photodiode 602 during an exposure period.

Quantizer 607 may include comparators to compare the buffered voltages to different thresholds for different quantization operations associated with different intensity ranges. For example, for high intensity ranges where the amount of overflow charge generated by photodiode 602 exceeds the saturation limit of charge storage device 605, quantizer 607 can detect whether the buffered voltage exceeds a static threshold representing the saturation limit by detecting whether the buffered voltage exceeds the saturation limit. value and if the buffered voltage exceeds the static threshold, the time-to-saturation (TTS) measurement operation is performed by measuring the time it takes for the buffered voltage to exceed the static threshold. The measured time may be inversely proportional to the light intensity. Also, quantizer 607 can perform a fully digital analog to digital converter for photodiodes that saturate with residual charge but the overflow charge remains below the saturation limit of charge storage device 605 in the mid-intensity range; FD ADC) operates to measure the amount of overflow charge stored in the charge storage device 605 . Additionally, for low intensity ranges where the photodiode is not saturated by residual charge and no overflow charge is accumulated in charge storage device 605, quantizer 607 may perform digital processing instrumentation (PD ADC) operation for analog sensors to The amount of residual charge accumulated in the photodiode 602 is measured. The output of one of the TTS, FD ADC, or PD ADC operations may be output as measurement data 608 to represent the intensity of the light.

6B illustrates an example sequence of operation of pixel cell 601. As shown in FIG. 6B , the exposure period may be defined based on the timing of the AB signal that controls the electronic shutter switch 603, which, when enabled, may be controlled by the photodiode 602, and based on the timing of the TG signal that controls the transfer switch 604. The generated charge is diverted away, and the switch can be controlled to transfer the overflow charge and then the residual charge to the charge storage device 605 for readout. For example, referring to Figure 6B , the AB signal may be deasserted at time TO to allow photodiode 602 to generate charge. T0 marks the start of an exposure period. During the exposure period, the TG signal may set the transfer switch 604 to a partially on state to allow the photodiode 602 to accumulate at least some of the charge as residual charge until the photodiode 602 is saturated, after which the overflow charge may is transferred to the charge storage device 605 . Between times T0 and T1, quantizer 607 may perform TTS operations to determine whether the overflow charge at charge storage device 605 exceeds the saturation limit, and then between times T1 and T2, quantizer 607 may perform FD ADC operations to The amount of overflow charge at the charge storage device 605 is measured. Between times T2 and T3, the TG signal may be asserted to bias transfer switch 604 in the fully on state to transfer residual charge to charge storage device 605. At time T3, the TG signal may be deasserted to isolate the charge storage device 605 from the photodiode 602, and the AB signal may be asserted to divert the charge generated by the photodiode 602 away. Time T3 may mark the end of the exposure period. Between times T3 and T4, the quantizer 607 may perform a PD operation to measure the amount of residual charge.

The AB and TG signals may be generated by a controller (not shown in FIG. 6A ), which may be part of pixel cell 601 to control the duration of the exposure period and the sequence of quantization operations. The controller can also detect whether the charge storage device 605 is saturated and whether the photodiode 602 is saturated to select the output as measurement data 608 from one of TTS, FD ADC or PD ADC operation. For example, if the charge storage device 605 is saturated, the controller may provide the TTS output as the measurement data 608 . If the charge storage device 605 is not saturated but the photodiode 602 is saturated, the controller may provide the FD ADC output as measurement data 608 . If the photodiode 602 is not saturated, the controller can provide the PD ADC output as measurement data 608 . The measurement data 608 from each pixel unit of the image sensor 600 generated during the exposure period can form an image frame. The controller may repeat the sequence of operations in Figure 6B in subsequent exposure cycles to generate subsequent image frames.

Image frame data from image sensor 600 may be transmitted to a host processor (not shown in FIGS. 6A and 6B ) to support various applications, such as tracking one or more objects, detecting motion (eg, as dynamic Visual sensing (dynamic vision sensing; DVS) operation part) and so on. 7A - 7D illustrate examples of applications that may be supported by image frame data from image sensor 600. FIG. 7A illustrates an example of an object tracking operation based on image frames from image sensor 600 . As shown in FIG. 7A , an application operating at the host processor can identify a group of pixels in a region of interest (ROI) 702 corresponding to object 704 from an image frame 700 captured at time T0 . The application can continue to track the position of object 704 in subsequent image frames, including image frame 710 captured at time T1, and identify groups of pixels in ROI 712 corresponding to object 704 . The image position of the tracking object 704 within the image frame can be performed to support the SLAM algorithm, which can construct/update the image sensor 600 based on the image position of the tracking object 704 in the scene captured by the image sensor 600 ( and a map of the environment in which the mobile device including the image sensor 600, such as the near-eye display 100, is located.

7B illustrates an example of an object detection operation on an image frame from image sensor 600. As shown on the left side of FIG. 7B , the host processor may identify one or more objects in the scene captured in image frame 720, such as vehicle 722 and person 724. As shown on the right side of FIG. 7B , based on the identification, the host processor may determine that the group of pixels 726 corresponds to a vehicle 722 and the group of pixels 728 corresponds to a person 724. Recognition of vehicle 722 and person 724 can be performed to support various applications such as surveillance applications where vehicle 722 and person 724 are targeted for surveillance, MR applications where vehicle 722 and person 724 are replaced with virtual objects, to reduce certain Gaze imaging operations at the resolution of some images (eg, license plate of vehicle 722, face of person 724), etc.

7C illustrates an example of an eye tracking operation on an image frame from image sensor 600. As shown in FIG. 7C , the host processor can identify the group of pixels 734 and 736 corresponding to pupil 738 and glint 739 from images 730 and 732 of the eyeball. Pupil 738 and blink 739 identification may be performed to support eye tracking operations. For example, based on the image positions of pupil 738 and blink 739, the application can determine the user's gaze direction at different times, which can be provided as input to the system to determine, for example, what to display to the user.

FIG. 7D illustrates an example of dynamic vision sensing (DVS) operations on image frames from image sensor 600 . In DVS operation, image sensor 600 may only output pixels that undergo a predetermined degree of change in luminance (reflected in pixel value), and pixels that have not undergone a degree of change will not be output by image sensor 600 . DVS operations may be performed to detect motion of objects and/or reduce the amount of pixel data output. For example, referring to FIG. 7D , an image 740 is captured at time T0 containing a group of pixels 742 for light sources and a group of pixels 744 for people. The group of both pixels 742 and 744 may be output as part of image 740 at time TO. Image 750 is captured at time T1. The pixel values of the group of pixels 742 corresponding to the light source remain the same between times T0 and T1 , and the group of pixels 742 is not output as part of the image 750 . On the other hand, a person changes from standing to walking between times T0 and T1 , causing the pixel values of the group of pixels 744 to change between times T0 and T1 . Thus, the group of people's pixels 744 is output as part of the image 750 .

In the operations of FIGS. 7A - 7D , image sensor 600 may be controlled to perform a sparse extraction operation in which only a subset of pixel cells are selected to output pixel data of interest to the host processor. The pixel data of interest may include pixel data required to support a particular operation at the host processor. For example, in the object tracking operation of Figure 7A , image sensor 600 may be controlled to transmit only the group of pixels in ROIs 702 and 712 of object 704 in image frames 700 and 710, respectively. In the object detection operation of FIG. 7B , image sensor 600 may be controlled to transmit only groups of pixels 726 and 728 of vehicle 722 and person 724, respectively. Additionally, in the eye tracking operation of FIG. 7C , image sensor 600 may be controlled to transmit only the group of pixels 734 and 736 containing pupil 738 and glint 739. Furthermore, in the DVS operation of Figure 7D , the image sensor 600 can be controlled to transmit only the group of pixels 744 of the moving person at time Tl rather than the group of pixels 742 of the static light source. All of these configurations may allow higher resolution images to be generated and transmitted without a corresponding increase in power and bandwidth. For example, larger pixel cell arrays including more pixel cells may be included in image sensor 600 to improve image resolution, while only a subset of pixel cells generate pixel data of interest at high resolution and will Transmission of high-resolution pixel data to the host processor while the remaining pixel units do not generate/transmit pixel data or generate/transmit pixel data at low resolution can reduce the bandwidth and power required to provide improved image resolution. Furthermore, although image sensor 600 can be operated to generate images at higher frame rates, when each image includes only a small set of pixel values at high resolution and represented by a large number of bits and the remaining pixel values are not When transmitted or represented by a smaller number of bits, the increase in bandwidth and power can be reduced.

In the case of 3D sensing, the amount of pixel data transmission can also be reduced. For example, referring to FIG. 6D , illuminator 640 may project a pattern 642 of structured light onto object 650. The structured light can be reflected on the surface of the object 650, and the pattern 652 of the reflected light can be captured by the image sensor 600 to generate an image. The host processor may match pattern 652 to pattern 642 and determine the depth of object 650 relative to image sensor 600 based on the image position of pattern 652 in the image. For 3D sensing, only the group of pixel cells 660, 662, 664, and 666 contains relevant information (eg, pixel data for pattern 652). To reduce the amount of pixel data transmitted, image sensor 600 may be configured to send pixel data only from groups of pixel cells 660, 662, 664, and 666, or to send high resolution from pixel cells 660, 662 , 664, and 666 groups of pixel data to the host processor, while the rest of the pixel data is low-resolution.

Figures 8A and 8B illustrate an example of an imaging system 800 that may perform sparse extraction operations to support the operations illustrated in Figures 7A - 7D . As shown in FIG. 8A , imaging system 800 includes image sensor 802 and host processor 804. The image sensor 802 includes a sensor computing circuit 806 and a pixel unit array 808 . The sensor computing circuit 806 includes an image processor 810 and a programmed map generator 812 . In some examples, the sensor computing circuit 806 may be implemented as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or execute instructions to implement an image processor 810 and a hardware processor that programs the functions of the map generator 812. Additionally, the host processor 804 includes a general-purpose central processing unit (CPU) that can execute the application program 814 .

Each pixel cell or block of pixel cells in pixel cell array 808 can be individually programmed to, for example, enable/disable the output of pixel values, set the resolution of pixel values output by the pixel cells, and the like. Pixel cell array 808 may receive first programmed signal 820 from programmed map generator 812 of sensor computing circuit 806, which may be in the form of a programmed map containing programming for each pixel cell material. The pixel cell array 808 can sense light from the scene based on the first programming signal 820 and generate a first image frame 822 of the scene. In particular, pixel cell array 808 may be controlled by first programming signal 820 to operate in different thinning modes, such as in full frame mode where first image frame 822 includes a complete image frame of pixels, and/or in The first image frame 822 operates in a sparse mode that includes only a subset of the pixels specified by the stylized map. The pixel cell array 808 can output the first image frame 822 to both the host processor 804 and the sensor computing circuit 806 . In some examples, pixel cell array 808 may also output first image frames 822 with different pixel sparsities to host processor 804 and sensor computing circuit 806 . For example, pixel cell array 808 may output a first image frame 822 with a full image frame of pixels back to sensor computing circuit 806 and a first image frame with sparse pixels defined by first programming signal 820 An image frame 822 is output to the host processor 804 .

The sensor computing circuit 806 and the host processor 804 together with the image sensor 802 can form a two-layer feedback system based on the first image frame 822 to control the image sensor to generate subsequent image frames 824 . In a two-layer feedback operation, the image processor 810 of the sensor computing circuit 806 may perform image processing operations on the first image frame 822 to obtain processing results, and then the programmed map generator 812 may update the first image frame 822 based on the processing results. Stylized signal 820. Image processing operations at the image processor 810 may be directed/configured based on the second stylized signal 832 received from the application 814 , which may generate the second stylized signal 832 based on the first image frame 822 . The pixel cell array 808 may then generate a subsequent image frame 824 based on the updated first programming signal 820 . The host processor 804 and the sensor computing circuit 806 can then update the first programming signal 820 and the second programming signal 832, respectively, based on the subsequent image frame 824.

In the aforementioned two-layer feedback system, the second programmed signal 832 from the host processor 804 may be in the form of a teach/pilot signal, the result of a neural network training operation (eg, backpropagation results), etc., to affect Image processing operations and/or stylized map generation at sensor computing circuit 806 . The host processor 804 may be based not only on the first image frame but also on other sensor data (eg, other image frames captured by other image sensors, audio information, motion sensor output, input) to generate a teach/guidance signal to determine the context of the light sensing operation of the image sensor 802, and then to determine the teach/guidance signal. The context may include, for example, the environmental conditions in which the image sensor 802 is operating, the location of the image sensor 802 , or any other requirements of the application 814 . Given that the context typically changes at a much slower rate than the frame rate, the teach/pilot signal may be updated at a relatively low rate (eg, lower than the frame rate) based on the context, while the sensor computing circuit 806 The image processing operations and updating of the stylized map can be performed at a relatively high rate (eg, at the frame rate) to accommodate the image captured by the pixel cell array 808 .

Although FIG. 8A illustrates pixel cell array 808 transmitting first image frame 822 and second image frame 824 to both host processor 804 and sensor computing circuit 806, in some cases pixel cell array 808 may Image frames with different sparsity are transmitted to host processor 804 and sensor computing circuit 806 . For example, pixel cell array 808 may transmit a first image frame 822 and a second image frame 824 with full pixels to image processor 810 , each including a subset of pixels selected based on first programming signal 820 . The sparse versions of the two image frames are sent to the host processor 804 .

8B illustrates an example of the operation of imaging system 800 to support the object tracking operation of FIG. 7A . Specifically, at time TO, pixel cell array 808 (not shown in FIG. 8B ) generates first image frame 822 based on first programmed signal 820 indicating a full frame of pixels to be generated, and the first image Frame 822 is transmitted to both host processor 804 and image processor 810, the first image frame including complete pixels of the scene including object 704. Based on executing application 814, host processor 804 may determine that object 704 is to be tracked. Such determinations may be based on, for example, user input, requirements of the application 814, and the like. Host processor 804 may also process first image frame 822 to extract spatial features of object 704, such as features 840 and 842. Based on the processing results, host processor 804 may determine a region of interest (ROI) 850 that includes pixels of object 704 (or other objects, such as pupil 738 and glint 739 of FIG. 7C ) in first image frame 822 approximate location, size and shape. Additionally, based on other outputs from other sensors (eg, IMUs), the host processor 804 also determines that the image sensor 802 is moving relative to the object 704 at a certain speed, and can estimate the ROI 852 in subsequent image frames new location in. Host processor 804 may then transmit target features of object 704 (eg, features 840 and 842), ROI information (eg, initial position, shape, size of ROI 850), velocity, etc. as part of second programming signal 832, To image the processor 810 and the stylized map generator 812.

Based on the second programmed signal 832, the image processor 810 may process the first image frame 822 to detect the target image feature of the object 704 and determine the precise location, size and shape of the ROI 852 based on the detection results. The image processor 810 may then transmit the ROI information 854 including the precise location, size and shape of the ROI 850 in the first image frame 822 to the stylized map generator 812. Based on the ROI information 854, and the second stylized signal 832, the stylized map generator 812 can estimate the expected location, size, and shape of the ROI 852 to be in the subsequent image frame to be captured at time T1. For example, based on the velocity information included in the second stylized signal 832, the stylized map generator 812 may determine that the ROI 850 will move a distance d between times T0 and T1 to become the ROI 852, and based on the distance d determine Location of ROI 852 at time T1. As another example, where pupil 738 and flicker 739 of FIG. 7C are tracked as part of an eye tracking operation, stylized map generator 812 may obtain information about the user's gaze change and determine to include pupil based on the gaze change 738 and the expected location of the ROI (eg, ROI 852) of blink 739 at time T1. Stylized map generator 812 may then update first stylized signal 820 at time T1 to select pixel cells within ROI 852 to output object 704 (or pupil 738 and blink 739, or other) for subsequent image frames object) pixel data.

9A , 9B , 9C , and 9D illustrate examples of internal components of the imaging system 800 of FIG. 8A . FIG. 9A illustrates an example of an array 808 of pixel cells. As shown in FIG. 9A , pixel cell array 808 may include row controller 904 , column controller 906 , and programming signal parser 920 . Row controllers 904 are connected to row buses 908 (eg, 908a, 908b, 908c... 908n), and column controllers 906 are connected to column buses 910 (eg, 910a, 910b... 908n). One of the row controller 904 or the column controller 906 is also connected to the programming bus 912 to transmit pixel-level programming signals 926 targeted to a particular pixel cell or group of pixel cells. Each box labeled P ₀₀ , P ₀₁ , P _0j , etc. may represent a pixel cell or a group of pixel cells (eg, a group of 299′ 1×2 pixel cells). Each pixel cell or group of pixel cells may be connected to one of row bus 908, one of column bus 910, programming bus 912, and output data bus to output pixel data (not shown in FIG. 9A ). exhibit). Each pixel cell (or each group of pixel cells) can be accessed by row address signals 930 on row bus 908 provided by row controller 904 and column addresses on column bus 910 provided by column controller 906 Signals 932 are individually addressed to receive pixel-level programming signals 926 once via pixel-level programming bus 912 . Row address signals 930 , column address signals 932 , and pixel-level programming signals 926 may be generated based on the first programming signal 820 from the programming map generator 812 .

Additionally, FIG. 9A includes a programming signal parser 920 that can extract pixel-level programming signals from the first programming signal 820 . In some examples, the first programming signal 820 may include a programming map, which may include programming data for each pixel cell or each group of pixel cells of the pixel cell array 808 . 9B illustrates an example of a pixel array stylized map 940. As shown in FIG. 9B , pixel array stylization map 940 may include a two-dimensional array of pixel-level stylization data, where each pixel-level stylization data of the two-dimensional array is in the form of a pixel cell or group of pixel cells of pixel cell array 808 group as the target. For example, where each pixel-level programming data targets a pixel cell, and assume that pixel cell array 808 has a width of M pixels (eg, M rows of pixels) and N pixels (eg, pixels of N columns), the pixel array stylized map 940 may also have a width of M entries (eg, M rows of entries) and a height of N entries (eg, N columns of entries), where each entry stores a value for Pixel-level programming data for the corresponding pixel unit. For example, pixel-level programming data A ₀₀ at entry (0, 0) of pixel array programming map 940 targets pixel cell P ₀₀ at pixel position (0, 0) of pixel cell array 808 , while pixel Pixel-level programming data A ₀₁ at entry (0, 1) of array programming map 940 targets pixel cell P ₀₁ at pixel position (0, 1) of pixel cell array 808 . Where pixel-level stylized data targets groups of pixel cells, the number of entries along height and width of pixel array stylized map 940 can be scaled based on the number of pixel cells in each group .

The pixel array stylized map 940 can be configured to support the feedback operation described in FIG. 9B . For example, the pixel-level programming data stored at each entry can process each pixel cell (or each group of pixel cells) individually to, for example, power up or down, to enable or disable the output of pixel data , to set the quantization resolution, set the accuracy of the output pixel data, select the quantization operation (eg, one of TTS, FD ADC, PD ADC), set the frame rate, etc. As described above, the stylized map generator 812 may generate a pixel-array stylized map 940 based on predictions of, for example, one or more ROIs, where the pixel-level stylized data for pixel cells within the ROI is different from that used outside the ROI Pixel-level stylized data for pixel cells. For example, the pixel array programming map 940 may enable a subset of pixel cells (or groups of pixel cells) to output pixel data, while the remaining pixel cells do not output pixel data. As another example, the pixel array programmed map 940 may control a subset of pixel cells to output pixel data at a higher resolution (eg, using a larger number of bits to represent each pixel), while the remaining pixel cells are at a lower resolution Output pixel data in degrees.

Referring back to FIG. 9A , stylized map parser 920 may parse pixel array stylized map 940 to identify pixel-level stylization data for each pixel cell (or each group of pixel cells) whose pixel array stylizes A map can be a serial data stream. The identification of pixel-level programming data can be based on, for example, a predetermined scan pattern by which a two-dimensional pixel array programmed map is converted into a serial format, and by the programming signal parser 920 receiving pixel-level programming data from a serial data stream. according to the order. For each entry of programming data, programming signal parser 920 may generate column address signal 930 and row address signal 832, and transmit column address signal 830 and row address signal 832, respectively, to column sensor computing Circuitry 806 and row controller 904 to select pixel cells and transmit pixel-level programming signals 826 to the selected pixel cell (or group of pixel cells).

Figure 9C illustrates example internal components of pixel cell 950 of pixel cell array 808, which may include at least some of the components of pixel cell 601 of Figure 6A . Pixel unit 950 may include one or more photodiodes, including photodiodes 952a, 952b, and the like. In some examples, one or more of the photodiodes of pixel unit 950 may be configured to detect light in different frequency ranges. For example, photodiode 952a can detect visible light (eg, monochromatic, or one of red, green, or blue), while photodiode 952b can detect infrared light. In some examples, some or all of the photodiodes of pixel unit 950 may detect light of the same wavelength. Pixel unit 950 further includes switches 954 (eg, transistors, controller barrier layers) to control which photodiode outputs charge for pixel data generation. In situations where the photodiodes detect light in different frequency ranges, the output from each photodiode can correspond to a pixel to support juxtaposed 2D/3D sensing. In situations where the photodiodes detect light of the same frequency range, the outputs from the photodiodes can be combined in analog grouping operation to, for example, increase the signal-to-noise ratio in situations where low intensity light is measured -to-noise ratio; SNR).

In addition, pixel unit 950 further includes electronic shutter switch 603, transfer switch 604, charge storage device 605, buffer 606, quantizer 607, and reset switch 951 and memory 955 as shown in FIG. 6A . The charge storage device 605 may have a configurable capacitance to set the charge-to-voltage conversion gain. In some examples, the capacitance of the charge storage device 605 can be increased to store overflow charge for medium light intensity FD ADC operation to reduce the likelihood of the charge storage device 605 being saturated by overflow charge. The capacitance of the charge storage device 605 can also be reduced to increase the charge-to-voltage conversion gain for low light intensity PD ADC operation. The increase in charge-to-voltage conversion gain reduces quantization error and increases quantization resolution. In some examples, the capacitance of charge storage device 605 may also be reduced during FD ADC operation to increase quantization resolution. The reset switch 951 can reset the charge storage device 605 before capturing an image frame and/or between FD ADC and PD ADC operations. The buffer 606 includes a current source 956 whose current can be set by the bias signal BIAS1, and a power gate 958 that can be controlled by the PWR_GATE signal to turn the buffer 606 on/off. As part of disabling pixel cell 950, buffer 606 may be turned off.

Additionally, quantizer 607 includes comparator 960 and output logic 962 . Comparator 960 may compare the output of the buffer to a reference voltage (VREF) to generate an output. Depending on the quantization operation (eg, TTS, FD ADC, and PD ADC operation), comparator 960 may compare the buffered voltage to different VREF voltages to generate an output that is further processed by output logic 962 to cause memory 955 to store the output from free The value of the running counter or digital ramp is output as a pixel. The bias current of comparator 960 can be controlled by bias signal BIAS2, which can set the bandwidth of comparator 960, which can be set based on the frame rate to be supported by pixel unit 950. In addition, the gain of the comparator 960 may be controlled by the gain control signal GAIN. The gain of comparator 960 may be set based on the quantization resolution to be supported by pixel unit 950 . The comparator 960 further includes a power switch 961a and a power switch 961b, which are also controlled by the PWR_GATE signal to turn on/off the comparator 960 and the memory 955, respectively. As part of disabling pixel cell 950, comparator 960 may be turned off.

Additionally, output logic 962 may select the output of one of TTS, FD ADC, or PD ADC operation, and based on the selection, determine whether to forward the output of comparator 960 to memory 955 to store the value from the counter/digital ramp. Output logic 962 may include internal memory to store indications based on the output of comparator 960 as to whether photodiode 952 (eg, photodiode 952a) is saturated with residual charge and whether charge storage device 605 is Saturation by overflow charge. If charge storage device 605 is saturated by overflow charge, output logic 962 selects the TTS output to be stored in memory 955 and prevents memory 955 from overwriting the TTS output with the FD ADC/PD ADC output. If charge storage device 605 is not saturated but photodiode 952 is saturated, output logic 962 may select the FD ADC output to be stored in memory 955; otherwise, output logic 962 may select the PD ADC to be stored in memory 955 output. In some examples, instead of a counter value, an indication of whether the photodiode 952 is saturated with residual charge and whether the charge storage device 605 is saturated with overflow charge can be stored in memory 955 to provide the lowest accuracy pixel data.

Additionally, pixel unit 950 may include a pixel unit controller 970, which may include logic circuits to generate control signals, such as AB, TG, BIAS1, BIAS2, GAIN, VREF, PWR_GATE, and the like. Pixel cell controller 970 may also be programmed by pixel-level programming signal 926 . For example, to disable pixel cell 950, pixel cell controller 970 may be programmed to deassert PWR_GATE via pixel-level programming signal 926 to turn off buffer 606 and comparator 960. Additionally, to increase the quantization resolution, the pixel cell controller 970 can be programmed by the pixel-level programming signal 926 to reduce the capacitance of the charge storage device 605, increase the gain of the comparator 960 by the GAIN signal, and so on. To increase the frame rate, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to increase the BIAS1 and BIAS2 signals to increase the bandwidth of buffer 606 and comparator 960, respectively. Furthermore, in order to control the accuracy of the pixel data output by pixel cell 950, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to, for example, only bits of a counter (eg, the most significant bit) A subset of are connected to memory 955 so that memory 955 stores only a subset of the bits, or the indication stored in output logic 962 is stored to memory 955 as pixel data. Additionally, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to control the sequence and timing of the AB and TG signals to, for example, adjust exposure periods and/or select specific quantization operations (eg, TTS, FD ADC, or PD) one of the ADCs) while skipping other quantization operations based on operating conditions, as described above.

9D illustrates an example of the internal components of image processor 810. As shown in FIG. 9D , image processor 810 may include feature extraction circuitry 972 and memory 976. Features to be extracted/detected by the image processor 810 may include, for example, spatial features and key points, temporal contrast, etc. of predetermined objects (eg, human faces, body parts, certain physical objects in a scene). In some examples, feature extraction circuitry 972 may implement a machine learning model 973, such as a convolutional neural network (CNN), recurrent neural network (RNN), etc., which may be trained to The input image frame (eg, first image frame 822 ) generated by pixel cell array 808 performs image feature operations. In some examples, feature extraction circuitry 972 may also include comparison circuitry 975 that compares pixel data to threshold values to identify pixels with predetermined temporal contrast. Feature extraction circuitry 972 may include other circuitry, such as a digital signal processor (DSP), linear solver unit, microcontroller, arithmetic circuitry, etc., to perform feature extraction operations. Image processor 810 may receive target features/thresholds, machine learning parameters (eg, weights, back-propagation gradients) or other configuration parameters as part of second programmed signal 832 from host processor 804 to support the machine Feature extraction operations and/or training operations of the learning model 973 . As a result of this feature extraction operation, feature extraction circuit 972 may output, for example, the pixel locations of the detected features in the input image frame, which may then be fed to stylized map generator 812 to generate pixel array stylized map 940.

Additionally, memory 976 may provide on-chip memory to store pixel data of the input image frame, various configuration data for feature extraction operations, and the output of feature extraction circuit 972 (eg, pixel locations). In some examples, the current input image frame provided to feature extraction circuit 972 may include only sparse pixel data, rather than a full frame of pixel data. In this case, memory 976 may also store pixel data of previously input image frames, which may be fed to feature extraction circuit 972 and combined with the current input image to generate a reconstructed full frame of pixel data. Feature extraction circuitry 972 may then perform feature extraction operations based on the reconstructed full frame of pixel data. The memory 976 may include, for example, spin tunneling random access memory (STRAM), non-volatile random access memory (NVRAM), and the like. In some examples, image processor 810 may also include an interface to off-chip memory (eg, dynamic random access memory) to support feature extraction operations at feature extraction circuit 880 .

Feature extraction circuitry 972 may employ various techniques to perform this feature extraction operation. In one example, feature extraction circuitry 972 may use a machine learning model 973, such as a CNN, to perform convolution operations between blocks of pixel data and filters. The filter may include a set of weights representing the target features to be extracted. As part of the convolution operation, the filter is superimposed with a portion of the block of pixel data at a particular stride position, and the sum of the products of each element of the filter and each pixel within the portion can be determined. When the filter is shifted around within a block of pixels, the distribution relative to the sum of the products of different stride positions can be determined as the convolution output. The convolution output may indicate, for example, the probability that a particular pixel captures the target feature, the probability that the pixel belongs to the target object, and the like. Based on these probabilities, feature extraction circuit 972 may output pixel locations of pixels determined to likely include the target feature or be part of the target object. The pixel locations may then be output as part of the ROI information 852 of Figure 8B to adjust the sparse extraction operation of the pixel cell array 808 as described above.

The filter weights for the convolution operation can be obtained from a training procedure, which can be performed offline, online, or a combination of the two. In an offline training procedure, the weights may be pre-stored in memory 976 prior to the feature extraction operation. The weights may be obtained from the training program based on a training data set covering a series of image data expected to be processed by the image processor 810 . The training data set can be stored in the cloud environment, and the training can also be performed in the cloud environment as an offline training program. The weights obtained from the offline training procedure may be common to all image processors 810 of different imaging systems 800 .

In the online training procedure, the weights used by the image processor 810 may be obtained when the image processor 810 receives image data of the actual object to be detected. An example application may be eye tracking (eg, based on images of eyes captured by an image sensor). As part of the online training program, the image processor 810 may operate in a training mode in which the image processor receives pixel data of the user's eyes when the user is asked to look at a particular object or location in space. Throughout the training procedure, the image processor 810 may adjust the weights to maximize the likelihood of correctly identifying the user's eyes. In this case, the weights used by the image processor 810 of a particular imaging system 800 may be different from the weights used by the image processor 810 of another imaging system 800 because of specific user and/or specific operating conditions Optimization weights. In some examples, the weights used by image processor 810 may be obtained through a combination of offline and online training procedures. For example, the weights used by the first neural network layer may be generic weights used to extract general characteristics of objects, while the weights of the upper neural network layers may be trained in an online training process to become user-specific and/or specific operating conditions.

In addition, in order to support dynamic vision sensing (DVS) operations, the feature extraction circuit 972 can use the comparison circuit 975 to compare the pixels in the input image frame with the correspondence of the previous image frame stored in the memory 976 pixels to obtain the temporal contrast for those pixels. The comparison circuit 975 may also compare the temporal contrast to a target threshold value (received as part of the second programming signal 832) to output pixel locations of pixels having (or exceeding) the predetermined threshold value of the temporal contrast.

Feature extraction operations at feature extraction circuit 972 may be configured based on second programmed signal 832 . For example, the host processor 804 may encode the target features to be extracted as filter weights, and supply the filter weights to a CNN model to perform a convolution operation. Additionally, the host processor 804 may set temporal contrast thresholds for DVS operation and send the temporal contrast thresholds as part of the second programming signal 832 . The pixel locations may then be output as part of the ROI information 852 of Figure 8B to adjust the sparse extraction operation of the pixel cell array 808 as described above.

In addition to target features and thresholds, the host processor 804 may affect feature extraction operations at the feature extraction circuit 972 based on other configuration parameters included in the second programming signal 832 . For example, the host processor 804 may be part of an online training operation and may determine back-propagated gradients based on training operations involving images received from a single imaging system 800 or multiple imaging systems 800. The host processor 804 may then provide the back-propagating gradients back to each imaging system 800 as part of the second programmed signal 832 to adjust the weights locally at each imaging system. As another example, host processor 804 may provide intermediate results of image processing operations (such as outputs of lower level neural network layers) as part of second programming signal 832 to feature extraction circuitry 972, which may then use These outputs are used to perform neural network computations at higher level neural network layers. As another example, host processor 804 may provide as feedback the predicted accuracy of image processing operations performed by the neural network, which allows the neural network of feature extraction circuit 972 to update weights to improve the predicted accuracy of image processing operations Spend.

As another example, host processor 804 may provide the location of an initial ROI (eg, ROI 850 of Figure 8B ). The image processor 810 may perform feature extraction operations (eg, convolution operations, motion sensing operations) in a two-step method. For example, the image processor 810 may first perform a feature extraction operation on the pixels identified by the initial ROI. If the extraction results indicate that the initial ROI is broken (eg, the identified pixels do not resemble the shape of the target object), the image processor 810 may use the initial ROI as a baseline to search for additional pixels that may include target features in a second step. At the end of the second step, the image processor 810 may determine improved pixel locations to provide a more improved ROI.

In addition, host processor 804 may also perform evaluation of feature extraction operations and provide evaluation results back to feature extraction circuit 972 . Host processor 804 may provide evaluation results as feedback to influence feature extraction operations at feature extraction circuit 972 . The evaluation results may include, for example, an indication (and/or percentage) as to whether the sparse pixels output by the pixel cell array 808 contain the data required by the application 814 . In the case where sparse pixels are output based on the ROI defined in the first stylized signal 820 resulting from the feature extraction operation, the feature extraction circuit 972 may adjust the ROI and/or the feature extraction operation based on the evaluation results. For example, in the case of object tracking/detection operations, the host processor 804 can evaluate whether the sparse pixels in the image frame output by the pixel cell array 808 contain all the pixels of the target object, and provide the evaluation results back to to feature extraction circuit 972. Feature extraction circuitry 972 may then adjust, for example, the selection of pixels to perform feature extraction operations based on the evaluation results. In the case where the evaluation result indicates that the sparse pixels do not contain all the pixels of the target object, the feature extraction circuit 972 can expand the ROI to process more pixels, or even discard the ROI and process all pixels of the input image frame to extract/detect the target feature .

Image frame data from image sensor 600 may be transmitted to a host processor (not shown in FIGS. 6A and 6B ) to support various applications, such as tracking one or more objects, detecting motion (eg, as dynamic Visual Sensing (DVS) operation part) etc. 7A - 7D illustrate examples of applications that may be supported by image frame data from image sensor 600. FIG. 7A illustrates an example of an object tracking operation based on image frames from image sensor 600 . As shown in FIG. 7A , an application operating at the host processor can identify a group of pixels in a region of interest (ROI) 702 corresponding to object 704 from an image frame 700 captured at time T0 . The application can continue to track the position of object 704 in subsequent image frames, including image frame 710 captured at time T1, and identify groups of pixels in ROI 712 corresponding to object 704 . The image position of the tracking object 704 within the image frame can be performed to support the SLAM algorithm, which can construct/update the image sensor 600 based on the image position of the tracking object 704 in the scene captured by the image sensor 600 ( and a map of the environment in which the mobile device including the image sensor 600, such as the near-eye display 100, is located.

7B illustrates an example of an object detection operation on an image frame from image sensor 600. As shown on the left side of FIG. 7B , the host processor may identify one or more objects in the scene captured in image frame 720, such as vehicle 722 and person 724. As shown on the right side of FIG. 7B , based on the identification, the host processor may determine that the group of pixels 726 corresponds to a vehicle 722 and the group of pixels 728 corresponds to a person 724. The identification of vehicles 722 and people 724 can be performed to support various applications, such as surveillance applications in which vehicles 722 and people 724 are targeted for surveillance, mixed reality (MR) applications in which vehicles 722 and people 724 are replaced with virtual objects, for Gaze imaging operations that reduce the resolution of certain images (eg, the license plate of a vehicle 722, the face of a person 724) for privacy, and the like.

7C illustrates an example of an eye tracking operation on an image frame from image sensor 600. As shown in FIG. 7C , the host processor can identify the group of pixels 734 and 736 corresponding to pupil 738 and glint 739 from images 730 and 732 of the eyeball. Pupil 738 and blink 739 identification may be performed to support eye tracking operations. For example, based on the image positions of pupil 738 and blink 739, the application can determine the user's gaze direction at different times, which can be provided as input to the system to determine, for example, what to display to the user.

FIG. 7D illustrates an example of dynamic vision sensing (DVS) operations on image frames from image sensor 600 . In DVS operation, image sensor 600 may only output pixels that undergo a predetermined degree of change in luminance (reflected in pixel value), and pixels that have not undergone a degree of change will not be output by image sensor 600 . DVS operations may be performed to detect motion of objects and/or reduce the amount of pixel data output. For example, referring to FIG. 7D , an image 740 is captured at time T0 containing a group of pixels 742 for light sources and a group of pixels 744 for people. The group of both pixels 742 and 744 may be output as part of image 740 at time TO. Image 750 is captured at time T1. The pixel values of the group of pixels 742 corresponding to the light source remain the same between times T0 and T1 , and the group of pixels 742 is not output as part of the image 750 . On the other hand, a person changes from standing to walking between times T0 and T1 , causing the pixel values of the group of pixels 744 to change between times T0 and T1 . Thus, the group of human pixels 744 is output as part of the image 750 .

In the operations of FIGS. 7A - 7D , image sensor 600 may be controlled to perform a sparse extraction operation in which only a subset of pixel cells are selected to output pixel data of interest to the host processor. The pixel data of interest may include pixel data required to support a particular operation at the host processor. For example, in the object tracking operation of Figure 7A , image sensor 600 may be controlled to transmit only the group of pixels in ROIs 702 and 712 of object 704 in image frames 700 and 710, respectively. In the object detection operation of FIG. 7B , image sensor 600 may be controlled to transmit only groups of pixels 726 and 728 of vehicle 722 and person 724, respectively. Additionally, in the eye tracking operation of FIG. 7C , image sensor 600 may be controlled to transmit only the group of pixels 734 and 736 containing pupil 738 and glint 739. Furthermore, in the DVS operation of Figure 7D , the image sensor 600 can be controlled to transmit only the group of pixels 744 of the moving person at time Tl rather than the group of pixels 742 of the static light source. All of these configurations may allow higher resolution images to be generated and transmitted without a corresponding increase in power and bandwidth. For example, larger pixel cell arrays including more pixel cells may be included in image sensor 600 to improve image resolution, while only a subset of pixel cells generate pixel data of interest at high resolution and will Transmission of high-resolution pixel data to the host processor while the remaining pixel units do not generate/transmit pixel data or generate/transmit pixel data at low resolution can reduce the bandwidth and power required to provide improved image resolution. Furthermore, although image sensor 600 can be operated to generate images at higher frame rates, when each image includes only a small set of pixel values at high resolution and represented by a large number of bits and the remaining pixel values are not When transmitted or represented by a smaller number of bits, the increase in bandwidth and power can be reduced.

In the case of 3D sensing, the amount of pixel data transmission can also be reduced. For example, referring to FIG. 6D , illuminator 640 may project a pattern 642 of structured light onto object 650. The structured light can be reflected on the surface of the object 650, and the pattern 652 of the reflected light can be captured by the image sensor 600 to generate an image. The host processor may match pattern 652 to pattern 642 and determine the depth of object 650 relative to image sensor 600 based on the image position of pattern 652 in the image. For 3D sensing, only the group of pixel cells 660, 662, 664, and 666 contains relevant information (eg, pixel data for pattern 652). To reduce the amount of pixel data transmitted, image sensor 600 may be configured to send pixel data only from groups of pixel cells 660, 662, 664, and 666, or to send high resolution from pixel cells 660, 662 , 664, and 666 groups of pixel data to the host processor, while the rest of the pixel data is low-resolution.

Figures 8A and 8B illustrate an example of an imaging system 800 that may perform sparse extraction operations to support the operations illustrated in Figures 7A - 7D . As shown in FIG. 8A , imaging system 800 includes image sensor 802 and host processor 804. The image sensor 802 includes a sensor computing circuit 806 , a pixel cell array 808 and a frame buffer 809 . The sensor computing circuit 806 includes an image processor 810 and a programmed map generator 812 . In some examples, the sensor computing circuit 806 may be implemented as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or execute instructions to implement an image processor 810 and a hardware processor that programs the functions of the map generator 812. Frame buffer 809 may include memory to store image frames output by pixel cell array 808 and provide the image frames to sensor computing circuit 806 for processing. Frame buffer 809 may include on-chip memory (eg, static random access memory (SRAM)) integrated on the same wafer as sensor computing circuit 806, or off-chip memory ( For example, resistive random access memory (ReRAM), dynamic random access memory (DRAM)). Additionally, the host processor 804 includes a general-purpose central processing unit (CPU) that can execute the application program 814 .

Each pixel cell or block of pixel cells in pixel cell array 808 can be individually programmed to, for example, enable/disable the output of pixel values, set the resolution of pixel values output by the pixel cells, and the like. Pixel cell array 808 may receive first programmed signal 820 from programmed map generator 812 of sensor computing circuit 806, which may be in the form of a programmed map containing programming for each pixel cell material. The pixel cell array 808 can sense light from the scene based on the first programming signal 820 and generate a first image frame 822 of the scene. In particular, pixel cell array 808 may be controlled by first programming signal 820 to operate in different thinning modes, such as in full frame mode where first image frame 822 includes a complete image frame of pixels, and/or in The first image frame 822 operates in a sparse mode that includes only a subset of the pixels specified by the stylized map. The pixel cell array 808 can output the first image frame 822 to both the host processor 804 and the sensor computing circuit 806 . In some examples, pixel cell array 808 may also output first image frames 822 with different pixel sparsities to host processor 804 and sensor computing circuit 806 . For example, pixel cell array 808 may output a first image frame 822 with a full image frame of pixels back to sensor computing circuit 806 and a first image frame with sparse pixels defined by first programming signal 820 An image frame 822 is output to the host processor 804 .

In addition to generating the first programming signal 820 , the sensor computing circuit 806 may also generate a global signal that is sent to each pixel cell of the pixel cell array 808 . The global signal may include, for example, threshold voltages for quantization operations in TTS, FD ADC, and PD ADC operations (eg, global voltage ramps for FD ADC and PD ADC operations, flat voltages for TTS operations, etc.), and Global control signals such as the AB and TG signals of Figure 6B .

The sensor computing circuit 806 and the host processor 804 together with the image sensor 802 can form a two-layer feedback system based on the first image frame 822 to control the image sensor to generate subsequent image frames 824 . In a two-layer feedback operation, the image processor 810 of the sensor computing circuit 806 may perform image processing operations on the first image frame 822 to obtain processing results, and then the programmed map generator 812 may update the first image frame 822 based on the processing results. Stylized signal 820. Image processing operations at the image processor 810 may be directed/configured based on the second stylized signal 832 received from the application 814 , which may generate the second stylized signal 832 based on the first image frame 822 . The pixel cell array 808 may then generate a subsequent image frame 824 based on the updated first programming signal 820 . The host processor 804 and the sensor computing circuit 806 can then update the first programming signal 820 and the second programming signal 832, respectively, based on the subsequent image frame 824.

In the aforementioned two-layer feedback system, the second programmed signal 832 from the host processor 804 may be in the form of a teach/pilot signal, the result of a neural network training operation (eg, backpropagation results), etc., to affect Image processing operations and/or stylized map generation at sensor computing circuit 806 . The host processor 804 may be based not only on the first image frame but also on other sensor data (eg, other image frames captured by other image sensors, audio information, motion sensor output, input) to generate a teach/guidance signal to determine the context of the light sensing operation of the image sensor 802, and then to determine the teach/guidance signal. The context may include, for example, the environmental conditions in which the image sensor 802 is operating, the location of the image sensor 802 , or any other requirements of the application 814 . Given that the context typically changes at a much slower rate than the frame rate, the teach/pilot signal may be updated at a relatively low rate (eg, lower than the frame rate) based on the context, while the sensor computing circuit 806 The image processing operations and updating of the stylized map can be performed at a relatively high rate (eg, at the frame rate) to accommodate the image captured by the pixel cell array 808 .

Although FIG. 8A illustrates pixel cell array 808 transmitting first image frame 822 and second image frame 824 to both host processor 804 and sensor computing circuit 806, in some cases pixel cell array 808 may Image frames with different sparsity are transmitted to host processor 804 and sensor computing circuit 806 . For example, pixel cell array 808 may transmit a first image frame 822 and a second image frame 824 with full pixels to image processor 810 , each including a subset of pixels selected based on first programming signal 820 . The sparse versions of the two image frames are sent to the host processor 804 .

8B illustrates an example of the operation of imaging system 800 to support the object tracking operation of FIG. 7A . Specifically, at time TO, pixel cell array 808 (not shown in FIG. 8B ) generates first image frame 822 based on first programmed signal 820 indicating a full frame of pixels to be generated, and the first image Frame 822 is transmitted to both host processor 804 and image processor 810, the first image frame including complete pixels of the scene including object 704. Based on executing application 814, host processor 804 may determine that object 704 is to be tracked. Such determinations may be based on, for example, user input, requirements of the application 814, and the like. Host processor 804 may also process first image frame 822 to extract spatial features of object 704, such as features 840 and 842. Based on the processing results, the host processor 804 can determine the approximate location, size and shape of the ROI 850 including the pixels of the object 704 in the first image frame 822 (or other objects, such as the pupil 738 and glint 739 of FIG. 7C ). Additionally, based on other outputs from other sensors (eg, IMUs), the host processor 804 also determines that the image sensor 802 is moving relative to the object 704 at a certain speed, and can estimate the ROI 852 in subsequent image frames new location in. Host processor 804 may then transmit target features of object 704 (eg, features 840 and 842), ROI information (eg, initial position, shape, size of ROI 850), velocity, etc. as part of second programming signal 832, To image the processor 810 and the stylized map generator 812.

Based on the second programmed signal 832, the image processor 810 may process the first image frame 822 to detect the target image feature of the object 704 and determine the precise location, size and shape of the ROI 852 based on the detection results. The image processor 810 may then transmit the ROI information 854 including the precise location, size and shape of the ROI 850 in the first image frame 822 to the stylized map generator 812. Based on the ROI information 854, and the second stylized signal 832, the stylized map generator 812 can estimate the expected location, size, and shape of the ROI 852 to be in the subsequent image frame to be captured at time T1. For example, based on the velocity information included in the second stylized signal 832, the stylized map generator 812 may determine that the ROI 850 will move a distance d between times T0 and T1 to become the ROI 852, and based on the distance d determine Location of ROI 852 at time T1. As another example, where pupil 738 and flicker 739 of FIG. 7C are tracked as part of an eye tracking operation, stylized map generator 812 may obtain information about the user's gaze change and determine to include pupil based on the gaze change 738 and the expected location of the ROI (eg, ROI 852) of blink 739 at time T1. Stylized map generator 812 may then update first stylized signal 820 at time T1 to select pixel cells within ROI 852 to output object 704 (or pupil 738 and blink 739, or other) for subsequent image frames object) pixel data.

Figures 9A , 9B , and 9C illustrate examples of internal components of the pixel cell array 808 of Figure 8A . As shown in FIG. 9A , pixel cell array 808 may include row controller 904 , column controller 906 , and programming signal parser 920 . Row controllers 904 are connected to row buses 908 (eg, 908a, 908b, 908c... 908n), and column controllers 906 are connected to column buses 910 (eg, 910a, 910b... 908n). One of the row controller 904 or the column controller 906 is also connected to the programming bus 912 to transmit pixel-level programming signals 926 targeted to a particular pixel cell or group of pixel cells. Each box labeled P ₀₀ , P ₀₁ , P _0j , etc. may represent a pixel cell or a group of pixel cells (eg, a group of 2×2 pixel cells). Each pixel cell or group of pixel cells may be connected to one of row bus 908, one of column bus 910, programming bus 912, and output data bus to output pixel data (not shown in FIG. 9A ). exhibit). Each pixel cell (or each group of pixel cells) can be accessed by row address signals 930 on row bus 908 provided by row controller 904 and column addresses on column bus 910 provided by column controller 906 Signals 932 are individually addressed to receive pixel-level programming signals 926 once via pixel-level programming bus 912 . Row address signals 930 , column address signals 932 , and pixel-level programming signals 926 may be generated based on the first programming signal 820 from the programming map generator 812 .

Additionally, FIG. 9A includes a programming signal parser 920 that can extract pixel-level programming signals from the first programming signal 820 . In some examples, the first programming signal 820 may include a programming map, which may include programming data for each pixel cell or each group of pixel cells of the pixel cell array 808 . 9B illustrates an example of a pixel array stylized map 940. As shown in FIG. 9B , pixel array stylization map 940 may include a two-dimensional array of pixel-level stylization data, where each pixel-level stylization data of the two-dimensional array is in the form of a pixel cell or group of pixel cells of pixel cell array 808 group as the target. For example, where each pixel-level programming data targets a pixel cell, and assume that pixel cell array 808 has a width of M pixels (eg, M rows of pixels) and N pixels (eg, pixels of N columns), the pixel array stylized map 940 may also have a width of M entries (eg, M rows of entries) and a height of N entries (eg, N columns of entries), where each entry stores a value for Pixel-level programming data for the corresponding pixel unit. For example, pixel-level programming data A ₀₀ at entry (0, 0) of pixel array programming map 940 targets pixel cell P ₀₀ at pixel position (0, 0) of pixel cell array 808 , while pixel Pixel-level programming data A ₀₁ at entry (0, 1) of array programming map 940 targets pixel cell P ₀₁ at pixel position (0, 1) of pixel cell array 808 . Where pixel-level stylized data targets groups of pixel cells, the number of entries along height and width of pixel array stylized map 940 can be scaled based on the number of pixel cells in each group .

The pixel array stylized map 940 can be configured to support the feedback operation described in FIG. 9B . For example, the pixel-level programming data stored at each entry can process each pixel cell (or each group of pixel cells) individually to, for example, power up or down, to enable or disable the output of pixel data , to set the quantization resolution, set the accuracy of the output pixel data, select the quantization operation (eg, one of TTS, FD ADC, PD ADC), set the frame rate, etc. As described above, the stylized map generator 812 may generate a pixel-array stylized map 940 based on predictions of, for example, one or more ROIs, where the pixel-level stylized data for pixel cells within the ROI is different from that used outside the ROI Pixel-level stylized data for pixel cells. For example, the pixel array programming map 940 may enable a subset of pixel cells (or groups of pixel cells) to output pixel data, while the remaining pixel cells do not output pixel data. As another example, the pixel array programmed map 940 may control a subset of pixel cells to output pixel data at a higher resolution (eg, using a larger number of bits to represent each pixel), while the remaining pixel cells are at a lower resolution Output pixel data in degrees.

Referring back to FIG. 9A , stylized map parser 920 may parse pixel array stylized map 940 to identify pixel-level stylization data for each pixel cell (or each group of pixel cells) whose pixel array stylizes A map can be a serial data stream. The identification of pixel-level programming data can be based on, for example, a predetermined scan pattern by which a two-dimensional pixel array programmed map is converted into a serial format, and by the programming signal parser 920 receiving pixel-level programming data from a serial data stream. according to the order. For each entry of programming data, programming signal parser 920 may generate column address signal 930 and row address signal 832, and transmit column address signal 830 and row address signal 832, respectively, to column sensor computing Circuitry 806 and row controller 904 to select pixel cells and transmit pixel-level programming signals 826 to the selected pixel cell (or group of pixel cells).

Figure 9C illustrates example internal components of pixel cell 950 of pixel cell array 808, which may include at least some of the components of pixel cell 601 of Figure 6A . Pixel unit 950 may include one or more photodiodes, including photodiodes 952a, 952b, etc., each of which may be configured to detect light in different frequency ranges. For example, photodiode 952a can detect visible light (eg, monochromatic, or one of red, green, or blue), while photodiode 952b can detect infrared light. Pixel unit 950 further includes switches 954 (eg, transistors, controller barrier layers) to control which photodiode outputs charge for pixel data generation.

In addition, pixel unit 950 further includes electronic shutter switch 603, transfer switch 604, charge storage device 605, buffer 606, quantizer 607, and memory 955 as shown in FIG. 6A . In some examples, pixel cell 950 may include a separate switch 604 and/or a separate charge storage device 605 for each photodiode. The charge storage device 605 may have a configurable capacitance to set the charge-to-voltage conversion gain. In some examples, the capacitance of the charge storage device 605 can be increased to store overflow charge for medium light intensity FD ADC operation to reduce the likelihood of the charge storage device 605 being saturated by overflow charge. The capacitance of the charge storage device 605 can also be reduced to increase the charge-to-voltage conversion gain for low light intensity PD ADC operation. The increase in charge-to-voltage conversion gain reduces quantization error and increases quantization resolution. In some examples, the capacitance of charge storage device 605 may also be reduced during FD ADC operation to increase quantization resolution. The buffer 606 includes a current source 956 whose current can be set by the bias signal BIAS1, and a power gate 958 that can be controlled by the PWR_GATE signal to turn the buffer 606 on/off. As part of disabling pixel cell 950, buffer 606 may be turned off.

Additionally, quantizer 607 includes comparator 960 and output logic 962 . Comparator 960 may compare the output of the buffer to a reference voltage (VREF) to generate an output. Depending on the quantization operation (eg, TTS, FD ADC, and PD ADC operation), comparator 960 may compare the buffered voltage to different VREF voltages to generate an output that is further processed by output logic 962 to cause memory 955 to store the output from free The value of the running counter is output as a pixel. The bias current of comparator 960 can be controlled by bias signal BIAS2, which can set the bandwidth of comparator 960, which can be set based on the frame rate to be supported by pixel unit 950. In addition, the gain of the comparator 960 may be controlled by the gain control signal GAIN. The gain of comparator 960 may be set based on the quantization resolution to be supported by pixel unit 950 . The comparator 960 further includes a power switch 961, which can also be controlled by the PWR_GATE signal to turn the comparator 960 on/off. As part of disabling pixel cell 950, comparator 960 may be turned off.

Additionally, output logic 962 may select the output of one of TTS, FD ADC, or PD ADC operation, and based on that selection, determine whether to forward the output of comparator 960 to memory 955 to store the value from the counter. Output logic 962 may include internal memory to store indications based on the output of comparator 960 as to whether photodiode 952 (eg, photodiode 952a) is saturated with residual charge and whether charge storage device 605 is Saturation by overflow charge. If charge storage device 605 is saturated by overflow charge, output logic 962 selects the TTS output to be stored in memory 955 and prevents memory 955 from overwriting the TTS output with the FD ADC/PD ADC output. If charge storage device 605 is not saturated but photodiode 952 is saturated, output logic 962 may select the FD ADC output to be stored in memory 955; otherwise, output logic 962 may select the PD ADC to be stored in memory 955 output. In some examples, instead of a counter value, an indication of whether the photodiode 952 is saturated with residual charge and whether the charge storage device 605 is saturated with overflow charge can be stored in memory 955 to provide the lowest accuracy pixel data.

Additionally, pixel unit 950 may include a pixel unit controller 970, which may include logic circuits to generate control signals, such as AB, TG, BIAS1, BIAS2, GAIN, VREF, PWR_GATE, and the like. Pixel cell controller 970 may also be programmed by pixel-level programming signal 926 . For example, to disable pixel cell 950, pixel cell controller 970 may be programmed to deassert PWR_GATE via pixel-level programming signal 926 to turn off buffer 606 and comparator 960. Additionally, to increase the quantization resolution, the pixel cell controller 970 can be programmed by the pixel-level programming signal 926 to reduce the capacitance of the charge storage device 605, increase the gain of the comparator 960 by the GAIN signal, and so on. To increase the frame rate, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to increase the BIAS1 and BIAS2 signals to increase the bandwidth of buffer 606 and comparator 960, respectively. Furthermore, in order to control the accuracy of the pixel data output by pixel cell 950, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to, for example, only bits of a counter (eg, the most significant bit) A subset of are connected to memory 955 so that memory 955 stores only a subset of the bits, or the indication stored in output logic 962 is stored to memory 955 as pixel data. Additionally, pixel cell controller 970 may be programmed by pixel-level programming signal 926 to control the sequence and timing of the AB and TG signals to, for example, adjust exposure periods and/or select specific quantization operations (eg, TTS, FD ADC, or PD) one of the ADCs) while skipping other quantization operations based on operating conditions, as described above.

10A , 10B , and 10C illustrate examples of the internal components of image processor 810. As shown in FIG. 10A , image processor 810 may include computing memory 1002 , controller 1004 , and data processing circuitry 1006 . Computing memory 1002 may store pixels of an image frame (such as image frames 822/824 of Figure 8A ) to be processed by image processor 810. The controller 1004 may receive the image processing configuration parameters as part of the second programming signal 832 from the host processor 804 . The controller 1004 can then control the data processing circuit 1006 to extract image frames from the computing memory 1002 to perform image processing operations based on the configuration parameters. For example, where the image processing operation is to detect the object of interest and track its position in the image frame, the second programmed signal 832 may include image characteristics of the object of interest. Data processing circuitry 1006 may generate image processing output 1008 that indicates, for example, the pixel location of an object within an image frame.

In some examples, data processing circuitry 1006 may implement a machine learning model, such as a convolutional neural network (CNN) model, to perform object detection and tracking operations. FIG. 10B illustrates an example architecture of CNN 1020 that may be implemented by data processing circuit 1006 . Referring to Figure 10B , CNN 1020 may include four main operations: (1) convolution; (2) processing by excitation functions (eg, ReLU); (3) binning or subsampling; and (4) classification (fully connected Floor). These operations can be the basic building blocks of every convolutional neural network. Different CNNs can have different combinations of these four main operations.

An image to be classified, such as input image 1022, may be represented by a matrix of pixel values. The input image 1022 may include multiple channels, each channel representing a certain component of the image. For example, an image from a digital camera may have a red channel, a green channel, and a blue channel. Each channel can be represented by a 2-D matrix of pixels having pixel values in the range 0 to 255 (ie, 8 bits). Grayscale images can have only one channel. In the following description, the processing of a single video channel using CNN 1020 is described. Other channels can be handled similarly.

As shown in FIG . 10B , the input image 1022 may be processed by a first convolutional layer (eg, input layer) 1024 using a first weight array (labeled [W ₀ ] in FIG. 10B ). The first convolutional layer 1024 may include a plurality of nodes, where each node is assigned to multiply a pixel of the input image 1022 with a corresponding weight in the first weight array. As part of the convolution operation, in a multiply-add (MAC) operation, a block of pixels of the input image 1022 may be multiplied by a first array of weights to generate products, and the products are then accumulated to generate a sum. Each summation can then be post-processed by an excitation function to produce an intermediate output. The excitation function simulates the behavior of a linear perceptron in a neural network. The excitation function may include a linear function or a nonlinear function (eg, ReLU, softmax). The intermediate outputs may form an intermediate output tensor 1026 . The first weight array can be used, for example, to extract certain basic features (eg, edges) from the input image 1022, and the intermediate output tensor 1026 can represent the distribution of basic features as a basic feature map. Intermediate output tensors 1026 may be passed to binning layer 1028 , where intermediate output tensors 1026 may be subsampled or downsampled by binning layer 1028 to produce intermediate output tensors 1030 .

The intermediate output tensor 1030 may be processed by the second convolutional layer 1032 using a second weight array (labeled [W ₁ ] in FIG. 10B ). The second array of weights can be used, for example, to identify patterns from the intermediate output tensors 1030 that are specific to features of objects such as hands. As part of the convolution operation, the pixel blocks of tensor 1030 can be multiplied by the second weight array to produce products, and the products can be accumulated to produce a sum. Each sum can then also be processed by an excitation function to produce an intermediate output, and the intermediate output can form an intermediate output tensor 1034 . Intermediate output tensors 1034 may represent a distribution of features that represent hands. Intermediate output tensors 1034 may be passed to merge layer 1036 , where intermediate output tensors 1034 may be subsampled or downsampled to produce intermediate output tensors 1038 .

The intermediate output tensors 1038 may then pass through a fully connected layer 1040, which may include a multi-layer perceptron (MLP). The fully connected layer 1040 may perform classification operations based on the intermediate output tensors 1038 to, for example, classify whether an object in the image 1022 represents a hand, possible pixel locations of the hand in the image 1022, and the like. The fully connected layer 1040 can multiply the intermediate output tensor 1038 by a third weight array (labeled [W ₂ ] in FIG. 10B ) to generate sums, and these sums can be processed by the excitation function to generate the neural network output 1042. The neural network output 1042 may indicate, for example, whether an object of interest (eg, a hand) is present in the image frame and its pixel location and size.

FIG. 10C illustrates the internal components of data processing circuit 1006 and an example of implementing the operation of CNN 1020. As shown in FIG. 10C , the data processing circuit 1006 may include an array of arithmetic circuits 1050 . Each arithmetic circuit, such as 1050a-1050f, may include a multiplier 1054 to multiply an input data element (represented by "i") and a weight data element (represented by "w") to generate a local partial sum. The input data elements may correspond, for example, to pixels in an image frame, and the weight data elements may be corresponding weights of a weight matrix (eg, [W ₀ ], [W ₁ ], [W ₂ ]) of the neural network layer. Each arithmetic circuit 1050 may also include a method to add the local partial sum to the input partial sum (labeled "p_in") received from an adjacent arithmetic circuit and generate an output partial sum (labeled "p_out") The adder 1052. This output partial sum is then input to another adjacent arithmetic circuit. For example, arithmetic circuit 1050a may receive the input partial sum from arithmetic circuit 1050b, add its local partial sum to the input partial sum to generate an output partial sum, and provide the output partial sum to arithmetic circuit 1050c. Thus, each arithmetic circuit produces a local partial sum, and the local partial sums are accumulated in an array of arithmetic circuits to form an intermediate output. Data processing circuitry 1006 further includes post-processing circuitry 1056 to perform post-processing (eg, excitation function processing, merging) on the intermediate outputs. In some examples, data processing circuitry 1006 may include other types of circuitry, such as look-up tables, to implement multiplier 1054 and post-processing circuitry 1056 .

To perform the convolution and post-processing operations of the first input convolution layer 1024 and the pooling layer 1028, the controller 1004 (not shown in FIG. 10C ) may control the data processing circuit 1006 according to the CNN 1020 based on a predetermined mapping between the input data and the arithmetic circuit To extract input data from computing memory 1002 . For example, a first group of arithmetic circuits including arithmetic circuits 1050a, 1050b, and 1050c may extract a group of input pixels 1064 from computing memory 1002, while a second group of arithmetic circuits including arithmetic circuits 1050d, 1050e, and 1050f Groups may fetch groups of input pixels 1066 from computing memory 1002 . Each group of arithmetic circuits may perform a convolution operation between the group of input pixels and the weight array to produce an intermediate output based on a multiply-add operation, as described above. The intermediate output may then be post-processed by post-processing circuitry 1056 . The post-processed output may be stored back to computing memory 1002 . For example, intermediate output 1068 may be generated from convolution and post-processing of groups of input pixels 1064, while intermediate output 1070 may be generated from convolution and post-processing of groups of input pixels 1066. After the operations of the first convolutional layer 1024 and the merging layer 1028 are completed and the intermediate outputs are stored in the computational memory 1002, the controller 1004 can control the array of arithmetic circuits 1050 to extract the intermediate outputs to perform the second convolutional layer 1032 and the merging layer 1036 convolution and post-processing operations to generate a second set of intermediate outputs and store them in computational memory 1002. The controller can then control the array of arithmetic circuits to extract a second set of intermediate outputs to generate the neural network output 1042 based on the topology of the fully connected layer 1040 .

As depicted in FIG. 8A , image processor 810 may receive from frame buffer 809 a sparse image including active pixels and inactive pixels. Active pixels may correspond to one or more objects of interest, while non-active pixels may contain no image information (eg, have an all-black color or other predetermined color). 11A illustrates an example of a sparse image 1100 stored in computing memory 1002 of image processor 810. As shown in FIG. 11A , to support applications that track the head and hands of an individual, the sparse image 1100 may include: a first group of active pixels 1102, which includes pixels of the individual's head; a second group of active pixels 1104 A group that includes pixels of the individual's left hand; and a third group of active pixels 1106 that includes pixels of the individual's right hand. Groups of active pixels can be generated and transmitted by the pixel cell array 808 based on the first programming signal 820 . The remaining pixels of the sparse image 1100 comprising the group of pixels 1108 are inactive and contain no image information. Each inactive pixel may have a pixel value of zero or another value to indicate that the pixel is inactive. In some examples, the memory device of frame buffer 809 may be reset prior to receiving active pixels of an image frame. The remaining memory devices that did not receive active pixels may retain their reset state (eg, logic zero) and become inactive pixels while the active pixels are being written to the corresponding memory devices of frame buffer 809 . The pixel values representing the active and inactive pixels of the image frame may then be stored in the computational memory 1002 of the image processor 810 .

Although the generation and transmission of the sparse image 1100 can reduce the power consumption of the pixel cell array 808, if the data processing circuit 1006 is to perform a processing operation (eg, a convolution operation) on each pixel of the sparse image 1100, the data processing circuit 1006 will still Can consume a lot of power. On the other hand, as shown in FIG. 11A , given that only a small subset of pixels are active pixels and contain image data and most of the pixels are inactive pixels, having data processing circuitry 1006 performing processing operations on inactive pixels would not be applicable Information on detecting and locating the object of interest. Thus, a huge amount of power is wasted in generating unsuitable information, which can reduce the overall power and computational efficiency of image processing operations.

Referring to FIG. 11B , to improve overall power and computational efficiency, the controller 1004 may include a sparse data handling circuit 1110. In some examples, sparse data handling circuitry 1110 may extract groups of input data (eg, pixels, intermediate outputs, etc.) from memory 1002 to a neural network layer and detect a subset of the groups of input data, where the input data The entire group has an inactive value (eg, zero) or otherwise contains no image information. The sparse data handling circuit 1110 may not include the groups of inactive input data from the data processing circuit 1006, and the data processing circuit 1006 does not generate intermediate outputs for the groups of inactive input data and does not write them back to computing memory 1002. On the other hand, groups of input data that include active values (eg, non-zero) representing image information may be forwarded by sparse data handling circuitry 1110 to data processing circuitry 1006, which may then process groups of active input data group to generate intermediate outputs and write them back to computing memory 1002 .

In some examples, sparse data handling circuit 1110 may also receive information about the sparsity of image frames stored in memory 1002. The sparsity information may be based on, for example, stylized map information from stylized map generator 812 or based on a neural network model topology as described below. The sparse data handling circuit 1110 can determine the memory addresses of the memory 1002 in which the active pixel data is stored, and extract the active pixel data from the memory addresses.

In the event that the computing memory 1002 is reset/re-initialized between different image frames (eg, reset/re-initialized to logic zero), the computing memory 1002 is assigned to store data for inactive input data. The memory device of the intermediate output of the group (based on the mapping assignment between the input and output of the neural network layer) may retain its initialized/reset state and not be accessed for the group of inactive input data. Meanwhile, the memory devices of computing memory 1002 that are assigned to store intermediate outputs for groups of actively input data may be updated by data processing circuitry 1006 . Such a configuration may reduce access to the computing memory 1002 for processing sparse image data, which may further reduce the power consumption of the entire computing memory 1002 and the image processor 810 .

For example, referring back to FIG. 10C , sparse data handling circuit 1110 may detect that groups of input pixels 1064 may be completely inactive and contain no image information, while groups of input pixels 1066 contain active pixels and image information. The sparse data handling circuit 1110 may exclude the group of input pixels 1064 and corresponding weights from the first group of arithmetic circuits (including arithmetic circuits 1050a-1050c), or otherwise disable the first group of arithmetic circuits so that there is no intermediate The output is written back to compute memory 1002 for the group of input pixels 1064 . The intermediate output 1068 for the group of input pixels 1064 may retain a reset value (eg, logic zero) at the end of processing for the first convolutional layer 1024 . On the other hand, sparse data handling circuit 1110 may provide the group of input pixels 1066 and the corresponding weights to a second group of arithmetic circuits (including arithmetic circuits 1050d-1050f) to generate an intermediate output 1070, which may then be processed by Write to computing memory 1002 . The sparse data handling circuit 1110 may also reuse the sparse data handling for other neural network layers based on detecting and excluding inactive groups of input data from the data processing circuit 1006 to prevent the data processing circuit 1006 from performing computations The intermediate outputs are written to computing memory 1002 for the inactive groups of input data.

In addition, the data processing circuit 1006 may also include a bypass mechanism to reduce power consumption associated with the processing of inactive/zero input data within the group of active input data forwarded by the sparse data handling circuit 1110. Specifically, referring to FIG. 11C , the arithmetic circuit 1050a may include a disable circuit 1120 and a multiplexer 1122. When one or more of the input data element (i) or the weight data element (w) are zero, the product will be zero. To avoid wasting power in arithmetic circuit 1050a computing zero, disabling circuit 1120 may disable adder 1052 and multiplier 1054 after detecting that one or more of the input data element (i) or weight data element (w) is zero (e.g. based on cutting off its power supply). Furthermore, when the product will be zero, the multiplexer 1122 may be controlled to pass the input partial sum (p_in) directly as the output partial sum (p_out).

In some examples, to further reduce power consumption and improve power and computational efficiency, image sensor 802 may support temporal sparsity operation. As part of the temporal sparsity operation among active pixels, static and non-static pixels can be identified. Image processor 810 can be configured to perform image processing operations only on non-static pixels, while sparse data handling circuit 1110 can exclude static and non-active pixels from data processing circuit 1006 to further reduce power consumption and improve power and Computational efficiency.

12A illustrates an example of a group of active pixels 1200 with static and non-static pixels. As shown in FIG. 12A , groups of active pixels 1200 are captured in two image frames at time T0 and at time T1. The group of active pixels 1200 may include an object of interest (eg, an individual's head) to be tracked. The group of active pixels 1200 may also include a subset of pixels 1202 and 1204 (which include the pixels for the eyes and mouth in FIG. 12A ) that undergo changes between times T0 and T1, while the remaining active pixels 1200 are between times T0 and T1. Static between T1. If the degree of change of the pixel is below a threshold value, the pixel can be determined to be static.

In some examples, frame buffer 809 may detect static pixels from active pixels output by the image sensor, and store pixel values for those pixels to indicate to image processor circuitry that the pixels are static pixels. In some examples, frame buffer 809 may also detect static pixels from pixels stored in the frame buffer, some of which may correspond to inactive pixels, which the image sensor does not provide. some inactive pixels, and thus remain static. 12B includes example internal components of frame buffer 809 to support signaling of static pixels . As shown in FIG. 12B , frame buffer 809 may include pixel update module 1212 , buffer 1204 , and pixel update tracking table 1216 . Specifically, pixel update module 1212 may receive the latest image frame received from pixel cell array 808, which includes active and inactive pixels, and overwrite the previous image frame in buffer memory 1214 with the latest image frame pixel. For each pixel in the buffer memory 1214, the pixel update module 1212 can determine the degree of change of the pixel relative to the previous image frame, and determine whether the pixel is static or non-static based on whether the degree of change exceeds a threshold value. The pixel update module 1212 may also update the last frame time at which a pixel was updated (and considered non-static) in the pixel update tracking table 1216 for each pixel. After pixel update module 1212 updates pixels using pixel values from previous frames generated by pixel cell array 808, pixel update module 1212 can track pixels based on information from pixel update tracking table 1216 to remain static (static frame time ), and the pixel value of the pixel is set based on the static frame time.

（等式1） 12C illustrates an example technique by which pixel update module 1212 may set pixel values for static pixels. As shown in table 1220 on the left side of Figure 12C , pixel update module 1212 can set pixel values for static pixels based on a leaky integrator function with a time constant C as follows:

(Equation 1)

In Equation 1, P represents the pixel value set by the pixel update module 1212, S0 represents the predetermined pixel value used to represent the static pixel, and S represents the difference between the original pixel value obtained from the previous frame and S0. The original pixel value is obtained from the previous frame when the pixel last experienced a degree of change greater than the change threshold and was therefore updated in the frame buffer. The pixel value of a static pixel is set to S0 + S and decreases as the pixel remains static. Finally, if the pixels remain static for the expanded number of frames, the pixel value of the static pixel stabilizes at S0.

As another example, as shown in table 1222 on the right side of Figure 12C , pixel update module 1212 may set the pixel values of static pixels based on a step function. Specifically, for a threshold number of frames denoted by _Tth , the pixel update module 1212 may cause the pixel value of the static pixel to be S0+S. After the threshold number of frames is exceeded, the pixel update module 1212 may set the pixel value to the pixel value S0.

The predetermined pixel value S0 may correspond to black (zero), white (255), gray (128), or any value indicative of a static pixel. In all such cases, the image processor may distinguish static pixels from non-static pixels based on identifying pixel values that identify static pixels, and perform image processing operations only on non-static pixels as described above.

In some examples, the controller 1004 of the image processor 810 may include additional components to further improve the handling of static pixels. As shown in FIG. 13A , the controller 1004 may include a neural network operation controller 1302 and a data spreading controller 1304, which may include the sparse data handling circuit 1110 of FIG. 11B . The neural network operation controller 1302 can determine operations for each neural network layer, including extracting input data, storing intermediate output data, arithmetic operations, and generating control signals 1306 reflecting the operations. Data dissemination controller 1304 may perform extraction of input data and storage of intermediate output data based on control signal 1306 .

Specifically, the neural network operation controller 1302 may have topology information for the neural network model implemented by the data processing circuit 1006, including, for example, for each neural network layer and between adjacent network layers The input/output connectivity of each neural network layer, the size of each neural network layer, the quantization operations at each neural network layer and other post-processing operations (eg, excitation function processing, merge operations), the receptive field of the neural network, etc. The neural network operation controller 1302 can generate control signals 1306 to control extraction of input data, storage of intermediate output data, and arithmetic operations based on topology information. For example, the neural network operation controller 1302 may include, for each neural network, a mapping between the addresses of the input data and the addresses of the intermediate outputs in the computing memory 1002 based on the connectivity information as part of the control signal 1306. part, allowing the data dissemination controller 1304 to extract the input data and store the intermediate output data at the correct memory locations within the computing memory 1002.

Additionally, the neural network operation controller 1302 may include additional information in the control signal 1306 to facilitate static pixel manipulation operations. For example, based on the topology information of the neural network model and the distribution of active and inactive pixels, the neural network operation controller 1302 can determine that the changes in the indicated pixels (between image frames) differ in the neural network The data of the propagation mode in the layer changes the propagation map 1310 and provides the map to the data propagation controller 1304. Based on the identification of static pixels and the data change propagation map, the data propagation controller 1304 (and the sparse data handling circuit 1110) may selectively extract input data that is predetermined to be non-static into the data processing circuit 1006 to generate a subset of the intermediate output data, And a subset of the intermediate output data is stored at computational memory 1002 . At the same time, intermediate output data corresponding to static pixels (generated from previous frames) are retained in computational memory 1002 and are not updated.

13B illustrates an example operation of identifying non-static inputs/outputs for different neural network layers based on a data change propagation map 1310. In Figure 13B , black areas 1314a, 1314b, 1314c, 1314d, and 1314n may correspond to active data addresses in compute memory 1002 that store non-static/active pixel data and non-static intermediate outputs at each neural network layer regions, while the white regions may correspond to address regions in computing memory 1002 that store static/inactive pixel data and static intermediate outputs. Based on the data change propagation map 1310, the data propagation controller 1304 may determine the data address regions in the computing memory 1002 that store or will store non-static/active pixel data, and will only come from those data regions in the computing memory 1002 The pixel data and/or the intermediate output data of the are extracted to the data processing circuit 1006 to perform neural network computations (eg, multiply and accumulate operations) for each neural network layer.

Referring back to Figure 13A , the neural network operation controller 302 may also determine to change the threshold value 1320 for use in determining whether a pixel is static. The change threshold 1320 may be determined based on the topology of the neural network model (such as the depth of the neural network model) and the quantization and merging operations at each layer. Specifically, although changes in pixel data may propagate through different layers of the neural network model, the degree of change in intermediate outputs and outputs of the neural network model generally decreases at higher neural network layers, especially to a large extent In the case of up-quantizing input and output data (eg, represented by a very large number of bits). Thus, for a given neural network model with a certain number of layers and a certain quantization scheme, the neural network operation controller 302 may determine a change threshold 1320 for pixel data such that pixels considered non-static At least some degree of change can be made at the output of the neural network model. In some instances, due to, for example, different merge operations performed at different neural network layers, different quantization accuracy at different neural network layers, different sparsity distributions of input data for different neural network layers, etc. The network operations controller 302 may also determine different change thresholds 1320 for different neural network layers to ensure that non-static input data selected based on change thresholds can be generated at the output data for the neural network layers meaningful change.

In some examples, data dissemination controller 1304 may include residual processing circuitry 1316 to track pixel changes between consecutive image frames, as well as non-consecutive image frames separated by sequences (eg, 10) of other image frames The pixels between the boxes change both. Residual handling circuitry 1316 can handle situations where a pixel is determined to be static due to having small changes between consecutive image frames, but the change in pixels between non-consecutive frames is large enough to require updating the intermediate output of the neural network. This situation is illustrated in Figure 13C . As shown in Figure 13C , between frames 1, 2, 3, and 4, only pixels 1330 and 1332 change between frames. Between successive frames, the change in pixel value (0.2 to 0.3) may be small, and the pixels may be determined to be static pixels. But between boxes 4 and 1, the changes are 0.5 to 0.7 and are significant, and these changes are reflected in the significant difference in convolution output between boxes 1 and 4 (0.119 vs. 0.646). To handle this situation, residual handling circuit 1316 may determine that a pixel is based not only on changes in pixels between two consecutive frames, but also on changes between two non-consecutive frames separated by a predetermined number of frames. still. If the pixel exhibits a small change between two consecutive frames but a large change between non-consecutive frames, the residual processing circuit 1316 may determine that the pixel is a non-static pixel and allow the data processing circuit 1006 to perform image processing on the pixel operate.

14A and 14B illustrate an example of a physical configuration of image sensor 802. As shown in Figure 14A , image sensor 802 may include: a semiconductor substrate 1400 that includes some of the components of pixel cell array 808, such as photodiodes of pixel cells; and one or more semiconductor substrates 1402, which Processing circuitry including pixel cell array 808 , such as buffer 606 , quantizer 607 and memory 955 , and sensor computation circuitry 806 . In some examples, the one or more semiconductor substrates 1402 include a semiconductor substrate 1402a and a semiconductor substrate 1402b. The semiconductor substrate 1402a may include the processing circuitry of the pixel cell array 808 , and the semiconductor substrate 1402b may include the sensor computing circuitry 806 . The semiconductor substrate 1400 and one or more semiconductor substrates 1402 can be housed within a semiconductor package to form a chip.

In some examples, semiconductor substrate 1400 and one or more semiconductor substrates 1402 may form a stack along a vertical direction (eg, represented by the z-axis), where vertical interconnects 1404 and 1406 provide electrical power in the substrates connect. Such a configuration can reduce the routing distance of the electrical connections between the pixel cell array 808 and the sensor computing circuit 806 , thereby increasing the speed of the transfer of data, particularly pixel data, from the pixel cell array 808 to the sensor computing circuit 806 And reduce the power required for transmission. In some examples, image sensor 802 may include an array of memory devices (eg, SRAM, RRAM, etc.) formed on or between a semiconductor substrate to provide frame buffer 809 and computational memory 1002 .

FIG. 14B illustrates an example of the details of the stack structure of image sensor 802 . As shown in FIG. 14B , the first semiconductor substrate 1000 may include: a backside surface 1408 configured as a light receiving surface and including a photodiode for each pixel cell; and a frontside surface 1410 on which the transfer electrons are implemented Crystal 604 and charge storage device 605 (eg, the floating drain of transfer transistor 604), while processing circuitry for pixel units including buffers 606, quantizers 607, memory 955, etc. is implemented under the front side surface 1412 of the semiconductor substrate 1402a . The front side surface 1410 of the semiconductor substrate 1400 may be electrically connected to the front side surface 1012 of the semiconductor substrate 1402a by vertical interconnects 1404 including die-to-die copper bonds. Wafer-to-wafer copper bonding can provide pixel interconnection, for example, between the transfer transistor 604 of each pixel cell and the buffer 606 of each pixel cell.

Additionally, imaging sensor 800 further includes through vertical interconnects between pixel cell array 808 and sensor computing circuitry 806, such as through silicon vias (TSVs), micro TSVs, copper-copper bumps, and the like . Vertical interconnects may be on the shoulder regions 1420 and 1422 of the stack and through the semiconductor substrates 1402a and 1402b. The vertical interconnect may be configured to transmit, for example, the first programming signal 820 and an image frame (eg, the first image frame 822). The vertical interconnect may support the transfer of a full frame of, eg, pixel data (eg, 1920 pixels by 1080 pixels) from the pixel cell array 808 to the image processor 810 for execution at normal frame rates (eg, 60 frames/second) Image feature extraction operations.

15 illustrates a method 1500 of operating an image sensor, such as image sensor 802 of FIG. 8A . Method 1500 may be performed by various components such as image sensor 802 including sensor computing circuit 806 , pixel cell array 808 , and frame buffer 809 . The sensor computing circuit 806 further includes an image processor 810 and a programmed map generator 812 . In some examples, the image sensor is implemented in a first semiconductor substrate, the frame buffer and sensor computing circuitry are implemented in one or more second semiconductor substrates, and the first semiconductor substrate and one or more second semiconductor substrates are Two semiconductor substrates form a stack and are housed in a single semiconductor package.

In step 1502, the programmed map generator 812 transmits first programmed data to an image sensor comprising a plurality of pixel cells to select a first subset of pixel cells to generate a first active pixel.

In some examples, first programming data may be generated based on sparse image sensing operations to support object detection and tracking operations at host processor 804 . A first subset of pixel cells can be selectively enabled to capture only pixel data related to tracking and detecting objects as active pixels, or to transfer only active pixels to a frame buffer to support sparse image sensing operations . The first programming data may be generated based on, for example, feedback data from the host processor 804 .

In step 1504, the sensor computing circuit 806 receives a first image frame from the frame buffer 809, the first image frame including at least some of the active pixels generated by the first subset of pixel cells, the The pixel cells are selected by the image sensor based on the first programming data. The first image frame further includes inactive pixels corresponding to a second subset of pixel cells not selected for generating active pixels. In some examples, as described in Figures 12A - 12C , frame buffer 809 may also overwrite some of the pixels in the first image frame with predetermined values to indicate that those pixels are static and have not yet experienced The threshold change level for multiple image frames.

Specifically, the frame buffer may detect static pixels from active pixels output by the image sensor, and store pixel values for the pixels to indicate to the image processor that the pixels are static pixels. For example, the frame buffer may store the latest pixel data (including active and inactive pixels) from each pixel unit of the image sensor as the first image frame. For each of the active pixels, the frame buffer may determine how much the pixel has changed relative to a previous frame, such as the image frame immediately preceding the first image frame. The frame buffer may set pixel values in various ways to indicate static pixels. For example, a frame buffer may be based on a leaky integrator function with a time constant and based on a number of consecutive image frames across which the pixels output by the image sensor remain static for use in the frame buffer pixel value in pixel. If the pixel remains static for a large number of consecutive image frames, the pixel value of the pixel may stabilize at a predetermined pixel value. As another example, if the pixel remains static for a threshold number of consecutive image frames (eg, 10), the frame buffer may set a predetermined pixel value for the pixel in the frame buffer. The predetermined pixel value may correspond to black (zero), white (255), gray (128), or any value indicative of a static pixel.

In step 1506, the image processor 810 performs an image processing operation on the first subset of pixels of the first image frame to generate a processing output, wherein the second subset of pixels of the first image frame is excluded from the image processing operation outside. The first subset of pixels of the first image frame for which the image processing operation is performed may correspond, for example, to active pixels, which are non-static pixels that undergo some degree of change between frames, or the like. For example, the first subset of pixels may correspond to the object of interest tracked/detected by object detection and tracking operations at the host processor 804 .

In some examples, image processing operations may include neural network operations. 10A , the image processor 810 may include data processing circuitry 1006, such as a multilayer convolutional neural network (CNN) including input and output layers, to provide hardware acceleration for neural network operations. The image processor may include computational memory 1002 to store input image frames and a set of weights associated with each neural network layer. The set of weights may represent characteristics of the object to be detected. The image processor may also include a controller 1004 to control the data processing circuit to extract input image frame data and weights from the computing memory. The controller can control the data processing circuitry to perform arithmetic operations, such as multiply-and-accumulate (MAC) operations, between the input image frames and the weights to generate intermediate output data for the input layer. For example, the intermediate output data is post-processed based on excitation functions, merge operations, etc., and then the post-processed intermediate output data may be stored in computational memory. Post-processed intermediate output data may be extracted from computational memory and provided to the next neural network layer as input. The arithmetic operations and the extraction and storage of intermediate output data are repeated for all layers up to the output layer to generate the neural network output. The neural network output may indicate, for example, the likelihood of the object being present in the input image frame and the pixel location of the object in the input image frame.

The controller can configure the data processing circuitry to process sparse image data in an efficient manner. For example, for the input layer, the controller may control the data processing circuitry to extract only a first subset of pixels and corresponding weights from computational memory, and perform MAC operations on only active pixels and corresponding weights to generate a corresponding Subset of intermediate outputs for active pixels of the input layer. The controller can also determine a subset of intermediate output data at each subsequent neural network, which can be traced back to the active pixel, based on the topology of the neural network and connections among subsequent neural network layers. The controller can control the data processing circuitry to perform MAC operations to generate only a subset of the intermediate output data at each subsequent neural network layer. Additionally, to reduce access to computational memory, predetermined values (eg, zeros) for the intermediate output data of each layer may be stored in computational memory prior to neural network operations. Only the intermediate output data for active pixels is updated. All of these operations can reduce power consumption for neural network operations on sparse image data.

In some examples, to further reduce power consumption and improve power and computational efficiency, the data processing circuitry may perform image processing operations (eg, neural network operations) only on non-static pixels of the first image frame to generate images for non-static pixels The updated output of the pixel. For static pixels (which may include inactive pixels), the image processing operation may be skipped, while the output from the image processing operation on the previous image frame may be retained. In the case where the image processing operations include neural network operations, the controller may control the data processing circuit to extract only non-static pixels and corresponding weight data from the computational memory to update the data corresponding to the non-static pixels used in the input layer. A subset of intermediate output data. The remaining intermediate output data in computational memory corresponding to static pixels (obtained from previous image frames) and corresponding to inactive pixels (eg, having predetermined values such as zero) may be retained for the input layer. The controller may also determine a subset of intermediate output data at each subsequent neural network that can be traced back to non-static pixels based on the topology of the neural network and connections among subsequent neural network layers, and Only a subset of the intermediate output data is updated to reduce access to computing memory and reduce power consumption.

In some examples, the image processor may also generate additional information to facilitate processing of non-static pixels. For example, the image processor may determine a data change propagation map based on the topology of the model that tracks the propagation of data changes from the input layer to the output layer of the neural network model. Based on the propagation map, and static pixels from the frame buffer, the image processor can identify non-static input data for each neural network and extract only those inputs for neural network operations at each layer material. In addition, the image processor can also determine the threshold change degree for static/non-static pixel determination based on the topology of the neural network model to ensure that pixels determined to be non-static can cause the necessary changes at the output layer degree. In addition, the image processor can also track pixel changes between consecutive frames and between non-consecutive frames. The image processor can identify pixels that exhibit small changes between consecutive frames and also identify large changes between non-consecutive frames as non-static pixels so that the image processor can perform image processing operations on those pixels.

In step 1508, the stylized map generator 812 generates second stylized data based on the processing output. The second programmed data may reflect, for example, movement of the object, changes in portions of the tracked object, etc. based on the processing output.

In step 1510, the programmed map generator 812 transmits the second programmed data to the image sensor to generate second active pixels for the second image frame.

Portions of this description describe examples of the invention in terms of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing often use these algorithmic descriptions and representations to effectively convey the substance of their work to others of ordinary skill in the art. These operations, although described functionally, computationally, or logically, should be understood to be implemented by computer programs or equivalent circuits, microcode, and the like. Furthermore, it has proven convenient at times, without loss of generality, to refer to these operational configurations as modules. The described operations and their associated modules may be embodied in software, firmware, and/or hardware.

The steps, operations or procedures described may be performed or implemented by one or more hardware or software modules, alone or in combination with other means. In some examples, a software module is implemented using a computer program product comprising a computer-readable medium containing computer code that can be executed by a computer processor for performing any or all of the steps, operations, or procedures described .

Examples of the invention may also relate to apparatus for performing the described operations. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. The computer program may be stored on a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to in this specification may include a single processor, or may be an architecture designed using multiple processors for increased computing power.

Examples of the present invention may also relate to products produced by the computational programs described herein. Such a product may include information generated by a computing program, and may include any instance of the computer program product or other combination of data described herein, where the information is stored on a non-transitory tangible computer-readable storage medium.

The language used in this specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or define the inventive subject matter. Therefore, it is intended that the scope of the present invention be limited not by this detailed description, but rather by the scope of any claims issued with respect to applications based herein. Accordingly, the disclosure of examples is intended to illustrate, but not to limit, the scope of the invention set forth in the following claims.

100: Near Eye Display 105: Frame 110: Display 120a: Image Sensor 120b: Image Sensor 120c: Image Sensor 120d: Image Sensor 130: Active Illuminator 135: Eyeball 140a: Illuminator 140b: Illuminator 140c: Illuminator 140d: Illuminator 140e: Illuminator 140f: Illuminator 150a: Image Sensor 150b: Image Sensor 200: Cross Section 210: Waveguide Display Assembly 220: Eyeball 230: Exit Pupil 300: Waveguide Display 310: source assembly 320: output waveguide 325: illuminator 330: controller 340: image light 350: coupling element 355: image light 360: guiding element 365: decoupling element 370: image sensor 370-1: th Side 370-2: Second Side 372: Object 374: Light 376: Light Source 378: Light 390: Remote Console 400: Cross Section 402: Pixel Unit 404: Mechanical Shutter 406: Optical Filter Array 410: Source 415 : Optical System 500: System 510: Control Circuit 525: Position Sensor 530: Inertial Measurement Unit (IMU) 535: Imaging Device 540: Input/Output Interface 545: Application Storage 550: Tracking Module 555: Engine 600 : image sensor 601 : pixel unit 602 : photodiode 603 : electronic shutter switch 604 : switch 605 : charge storage device 606 : buffer 607 : quantizer 608 : measurement data 640 : illuminator 642 : structured Pattern of light 650: Object 652: Pattern of reflected light 660: Pixel unit 662: Pixel unit 664: Pixel unit 666: Pixel unit 700: Image frame 702: Region of interest (ROI) 704: Object 710: Image frame 712 :ROI 720:ImageFrame722:Vehicle 724:People726:Pixel728:Pixel730:Image732:Image734:Pixel736:Pixel738:Pupil739:Flicker740:Image742:Pixel744:Pixel750:Image800 : Imaging System 802 : Image Sensor 804 : Host Processor 806 : Sensor Computing Circuit 808 : Pixel Cell Array 809 : Frame Buffer 810 : Image Processor 812 : Programmable Map Generator 814 : Application 820 : First Stylized Signal 822: First Image Frame 824: Image Frame 832: Second Stylized Signal 840: Feature 842: Feature 850: Region of Interest (ROI) 852: ROI Information 854: ROI Information 904: Line Control Controller 906:Column Controller 908a:Row Bus 908b:Row Bus 908c:Row Bus 908n:Row Bus 910a:Column Bus 910b:Column Bus 910n:Column Bus 912:Programmed Bus 920:Program Programmable Signal Parser 926: Pixel-Level Programmable Signals 930: Row Address Signals 932: Column Address Signals 940: Pixel Array Stylized Map 950: Pixel Cell 951: Reset Switch 952: Photodiode 952a: Photodiode 952b: Photodiode 954: Switch 955 : Memory 956: Current Source 958: Power Gate 960: Comparator 961: Power Switch 961a: Power Switch 961b: Power Switch 962: Output Logic 970: Pixel Cell Controller 972: Feature Extraction Circuit 973: Machine Learning Model 975: Comparison Circuit 976: Memory 1002: Computational Memory 1004: Controller 1006: Data Processing Circuit 1012: Front Surface 1020: CNN 1022: Input Image 1024: First Input Convolutional Layer 1026: Intermediate Output Tensor 1028: Merging Layer 1030: Intermediate output tensor 1032: Second convolutional layer 1034: Intermediate output tensor 1036: Merging layer 1038: Intermediate output tensor 1040: Fully connected layer 1042: Neural network output 1050: Arithmetic circuit 1050a: Arithmetic circuit 1050b: Arithmetic circuit 1050c : Arithmetic circuit 1050d: Arithmetic circuit 1050e: Arithmetic circuit 1050f: Arithmetic circuit 1052: Adder 1054: Multiplier 1056: Post-processing circuit 1064: Input pixel 1066: Input pixel 1068: Intermediate output 1070: Intermediate output 1100: Sparse image 1102: Active Pixel 1104: Active Pixel 1106: Active Pixel 1108: Pixel 1110: Sparse Data Handling Circuit 1120: Disable Circuit 1122: Multiplexer 1200: Active Pixel 1202: Pixel 1204: Buffer 1212: Pixel Update Module 1214: Buffer memory 1216: pixel update tracking table 1220: table 1222: table 1302: neural network operation controller 1304: data propagation controller 1306: control signal 1310: data change propagation map 1314a: black area 1314b: black area 1314c: black area 1314d: Black area 1314n: Black area 1316: Residual processing circuit 1320: Change threshold value 1330: Pixel 1332: Pixel 1400: Semiconductor substrate 1402a: Semiconductor substrate 1402b: Semiconductor substrate 1404: Vertical interconnect 1406: Vertical interconnect Connector 1408: Back Surface 1410: Front Surface 1412: Front Surface 1420: Shoulder Region 1422: Shoulder Region 1500: Method 1502: Step 1504: Step 1506: Step 1508: Step 1510: Step A: Direction A ₀₀ : Pixel Stage programming data AB: Control signal BIAS1: Bias signal BIAS2: Bias signal B: Direction C: Time constant D: Direction GAIN: Gain control signal i: Input data element P ₀₀ : Box/pixel unit P ₀₁ : Box/ Pixel unit P _0j : frame/pixel unit p_in: input partial sum p_out: output partial sum PWR_GATE: control signal S: predetermined pixel value S0: difference T0: time T1: time T2: time T3: time T4: time TG: control signal VREF: reference voltage/control signal [ W ₀ ]: weight matrix [W ₁ ]: second weight array [W ₂ ]: third weight array w: weight data element

Illustrative examples are described with reference to the following figures.

[ FIG. 1A] and [ FIG. 1B] are diagrams of examples of near-eye displays.

[ FIG. 2] is an example of a cross section of a near-eye display.

[ FIG. 3] An isometric view illustrating an example of a waveguide display with a single source assembly.

[ FIG. 4] A cross section illustrating an example of a waveguide display.

[ FIG. 5] is a block diagram of an example of a system including a near-eye display.

[ FIG. 6A] and [ FIG. 6B] illustrate an example of an image sensor and its operation.

[ FIG. 7A] , [ FIG. 7B] , [ FIG. 7C] , and [ FIG. 7D] illustrate examples of applications supported by the output of the image sensor of FIGS. 6A and 6B .

[ FIG. 8A ] and [ FIG. 8B ] illustrate an example of an imaging system to support the operations illustrated in FIGS. 7A - 7D .

[ FIG. 9A] , [ FIG. 9B] , and [ FIG. 9C] illustrate example internal components and operations of the imaging system of FIGS. 8A and 8B .

[ FIG. 10A] , [ FIG. 10B] , and [ FIG. 10C] illustrate example internal components and operations of the image processor of FIGS. 8A and 8B .

[ FIG. 11A] , [ FIG. 11B] , and [ FIG. 11C] illustrate example internal components and operations of the image processor of FIGS. 10A to 10C .

[ FIG. 12A] , [ FIG. 12B] and [ FIG. 12C] illustrate example internal components of the frame buffer of FIGS. 8A and 8B and their operation.

[ FIG. 13A] , [ FIG. 13B] , and [ FIG. 13C] illustrate example internal components and operations of the image processor of FIGS. 10A - 10C .

[ FIG. 14A] and [ FIG. 14B] illustrate an example of a physical configuration of the image sensor of FIGS. 8A to 13C .

[ FIG. 15] A flowchart illustrating an example procedure for operating an image sensor.

The figures depict examples of the invention for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative examples of the structures and methods illustrated may be employed without departing from the principles or claimed advantages of the invention.

In the drawings, similar components and/or features may have the same reference numerals. In addition, various components of the same type may be distinguished by adding a dash after the reference symbol and a second designation to distinguish between similar components. If only the first reference number is used in the specification, the description applies to any of the similar components having the same first reference number regardless of the second reference number.

800: Imaging System

802: Image Sensor

804: Host processor

806: Sensor calculation circuit

808: Pixel cell array

809: Frame buffer

810: Image Processor

812: Stylized Map Generator

814: Application

820: First programmed signal

822: First image frame

824: Image frame

832: Second programmed signal

Claims (20)

A device comprising: an image sensor including a plurality of pixel units, the image sensor configurable by programming data to select a subset of the pixel units to generate active pixels; a frame buffer; and A sensor computing circuit configured to: A first image frame is received from the frame buffer, including first active pixels and first inactive pixels, the first active pixels being one of the pixel units selected based on the first programming data generating a first subset, the first inactive pixels corresponding to a second subset of the pixel units not selected for generating the first active pixels; performing an image processing operation on a first subset of pixels of the first image frame to generate a processing output in which a second subset of pixels of the first image frame is excluded from the image processing operation outside; generating second programmed data based on the processing output; and The second programming data is transmitted to the image sensor to select a second subset of the pixel cells to generate second active pixels for a second image frame. The apparatus of claim 1, wherein the image processing operation includes a processing operation of a neural network model to detect an object of interest in the first image frame; and wherein the first subset of pixels corresponds to the object of interest. The apparatus of claim 2, wherein the sensor computing circuit is coupled to a host device configured to execute an application that uses the results of detection of the object of interest; and wherein the sensor computing circuit is configured to receive information about the object of interest from the host device. The apparatus of claim 2, wherein the sensor computing circuit comprises: a computational memory configured to store: input data to a neural network layer of the neural network, weight data for the neural network layer, and intermediate output data for the neural network layer; a data processing circuit configured to perform arithmetic operations of the neural network layer on the input data and the weight data to generate the intermediate output data; and A compute controller configured to: A first subset of the input data and a first subset of the weight data corresponding to the first subset of input data are extracted from the computing memory, the first subset of the input data corresponding to the at least some of the first active pixels; controlling the data processing circuit to perform the arithmetic operations on the first subset of input data and the first subset of weight data to generate a first one of the intermediate output data for the first image frame a subset, the first subset of the intermediate output data corresponds to the first subset of the input data; storing the first subset of the intermediate output data for the first image frame in the computing memory; and A predetermined value is stored in the computing memory for a second subset of the intermediate output data of the first image frame, the second subset of the intermediate output data corresponding to the inactive pixels. The apparatus of claim 4, wherein the x value is stored based on resetting the computational memory prior to the image processing operation. The apparatus of claim 4, wherein the computing controller is configured to: extracting the input data from the computing memory; identifying the first subset of the input data from the extracted input data; and The identified first subset of the input data is provided to the computing controller. The apparatus of claim 4, wherein the computing controller is configured to: determining an address region of the computing memory that stores the first subset of the input data; and The first subset of the input data is extracted from the computing memory. 7. The apparatus of claim 7, wherein the address zone is determined based on at least one of: the first programmed data, or about connectivity between neural network layers of the neural network model News. The apparatus of claim 4, wherein: the first active pixels include static pixels and non-static pixels; The static pixels correspond to a first subset of the first active pixels, and for the first subset of the first active pixels, pixel values between the first image frame and a previous image frame The degree of change is higher than a change threshold; The non-static pixels correspond to a second subset of the first active pixels, for the second subset of the first active pixels, the distance between the first image frame and the previous image frame some pixel values are changed less than the change threshold; and The computing controller is configured to extract the first subset of the input data corresponding to the non-static pixels of the first active pixels. The apparatus of claim 9, wherein the predetermined value is a first predetermined value; wherein the frame buffer is configured to store a second predetermined value for each of the static pixels to identify the static pixels; and wherein the computing controller is configured to exclude the static pixels from the data processing circuit based on detecting that the static pixels have the second predetermined value. The apparatus of claim 10, wherein the frame buffer is configured to store the second for a pixel across a threshold number of frames based on a determination that the degree of change of the pixel is below the change threshold predetermined value. The apparatus of claim 10, wherein the frame buffer is configured to update the setting based on a leaky integrator function having a time constant and based on the last time a pixel experienced a degree of change greater than one of the change thresholds A pixel value of a pixel. The apparatus of claim 9, wherein the computing controller is configured to: determining a data change propagation map based on a topology of the neural network model, the data change propagation map indicating how changes to the non-static pixels propagate through different neural network layers of the neural network model; Based on the data change propagation map, determine a first address region of the computing memory for extracting the first subset of the input data, and determine a second address region of the computing memory for storing the first subset of the intermediate output data; extract the first subset of the input data from the first address area; and The first subset of the intermediate output data is stored in the second address area. 9. The apparatus of claim 9, wherein the computing controller is configured to determine the change threshold based on a depth of the neural network model and a quantization accuracy at each neural network layer of the neural network model . The apparatus of claim 9, wherein the change threshold is a first change threshold; and where the compute controller is configured to: tracking the degree of change in the pixel values of the first active pixels between two non-consecutive frames; and A third subset of the first active pixels is determined to be non-static pixels based on the degree of change exceeding a second change threshold. The apparatus of claim 1, wherein the image sensor is implemented in a first semiconductor substrate; wherein the frame buffer and the sensor computing circuit are implemented in one or more second semiconductor substrates; and Wherein the first semiconductor substrate and the one or more second semiconductor substrates form a stack and are accommodated in a single semiconductor package. A method that includes: transmitting the first programming data to an image sensor including a plurality of pixel units to select a first subset of the pixel units to generate a first active pixel; A first image frame including the first active pixels and first inactive pixels corresponding to not selected for generating the first active pixels is received from a frame buffer a second subset of the pixel units; performing an image processing operation on a first subset of pixels of the first image frame to generate a processing output in which a second subset of pixels of the first image frame is excluded from the image processing operation outside; generating second programmed data based on the processing output; and The second programming data is transmitted to the image sensor to select a second subset of the pixel cells to generate second active pixels for a second image frame. The method of claim 17, wherein the image processing operation includes a processing operation of a neural network to detect an object of interest in the first image frame; and wherein the first subset of pixels corresponds to the object of interest. The method of claim 18, further comprising: Store input data to the neural network layer of the neural network, weight data of the neural network layer in a computing memory; A first subset of the input data and a first subset of the weight data corresponding to the first subset of input data are extracted from the computing memory, the first subset of the input data corresponding to the at least some of the first active pixels; using a data processing circuit to perform arithmetic operations on the first subset of input data and the first subset of weight data to generate a first subset of intermediate output data for the first image frame, the the first subset of intermediate output data corresponds to the first subset of the input data; storing the first subset of the intermediate output data for the first image frame in the computing memory; and A predetermined value for a second subset of the intermediate output data of the first image frame is stored in the computing memory, the second subset of the intermediate output data corresponding to the inactive pixels. The method of claim 19, wherein: the first active pixels include static pixels and non-static pixels; The static pixels correspond to a first subset of the first active pixels, and for the first subset of the first active pixels, pixel values between the first image frame and a previous image frame The degree of change is higher than a change threshold; The non-static pixels correspond to a second subset of the first active pixels, for the second subset of the first active pixels, the distance between the first image frame and the previous image frame some pixel values are changed less than the change threshold; and The first subset of the input data corresponds to the non-static pixels of the first active pixels.

Images