WO2022047651A1 - 视觉传感器芯片及其操作方法和电子设备 - Google Patents

视觉传感器芯片及其操作方法和电子设备 Download PDF

Info

Publication number
WO2022047651A1
WO2022047651A1 PCT/CN2020/112982 CN2020112982W WO2022047651A1 WO 2022047651 A1 WO2022047651 A1 WO 2022047651A1 CN 2020112982 W CN2020112982 W CN 2020112982W WO 2022047651 A1 WO2022047651 A1 WO 2022047651A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
light intensity
circuit
intensity change
read
Prior art date
Application number
PCT/CN2020/112982
Other languages
English (en)
French (fr)
Inventor
董思维
方运潭
方舒
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080093015.0A priority Critical patent/CN115004677B/zh
Priority to PCT/CN2020/112982 priority patent/WO2022047651A1/zh
Publication of WO2022047651A1 publication Critical patent/WO2022047651A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • Example embodiments of the present disclosure relate to the field of vision sensors, and more particularly, to a vision sensor chip, a method for operating a vision sensor chip, and an electronic device.
  • Visual sensing technology is widely used in video surveillance, digital cameras, robot navigation, automotive autonomous navigation, biomedical pixel analysis, human-machine interface, virtual reality, industrial control, wireless long-distance sensing, microscope technology, scientific instruments and many other fields. .
  • vision sensors can acquire image information from the external environment, and realize operations such as image processing, image storage, and image output.
  • the clarity and detail of an image depends on the resolution of the vision sensor, that is, the number of pixels. In general, an image can be represented by 256 ⁇ 256 pixels, 512 ⁇ 512 pixels, or 1024 ⁇ 1024 pixels, etc.
  • vision sensors For a bionic vision sensor, the principle is to use an integrated circuit to simulate the biological retina, and each pixel in the pixel array circuit simulates a biological neuron, and the changes in light intensity are expressed in the form of events.
  • vision sensors usually have two modes for reading event data, a frame scan-based synchronous read mode and an event stream-based asynchronous read mode.
  • the vision sensor scans all the pixels in the pixel array circuit in scan order and arranges the data signals generated by the pixels into a frame of data, and then reads each frame of data at a fixed rate.
  • the vision sensor determines the pixels where the light intensity change event occurs by scanning the pixel array circuit, and only reads the data signals output by this part of the pixels.
  • a single vision sensor can only use one data reading mode.
  • vision sensors using a single reading mode are difficult to adapt to various environmental changes and motion states, resulting in poor performance of vision sensors in some application scenarios.
  • example embodiments of the present disclosure propose solutions related to vision sensor chips.
  • a vision sensor chip includes: a pixel array circuit configured to generate a plurality of data signals corresponding to a plurality of pixels therein by measuring the amount of light intensity change, the plurality of data signals indicating at least one light intensity change event, at least one light intensity change a change event representing an amount of change in light intensity measured by a corresponding pixel in the pixel array circuit that exceeds a predetermined threshold; and a read circuit coupled to the pixel array circuit and configured to: read a plurality of from the pixel array circuit in a first read mode at least one of the data signals, providing the at least one data signal to the control circuit, and switching the first read mode to the second read mode if a mode switch signal generated based on the at least one data signal is received from the control circuit .
  • a method for operating a vision sensor chip includes: a pixel array circuit of a visual sensor chip generates a plurality of data signals corresponding to a plurality of pixels in the pixel array circuit by measuring the amount of light intensity change, the plurality of data signals indicating at least one light intensity change event, at least one light intensity change event, and at least one light intensity change event.
  • the strong change event indicates that the amount of light intensity change measured by the corresponding pixel in the pixel array circuit exceeds a predetermined threshold; the read circuit of the vision sensor chip reads at least one data of the plurality of data signals from the pixel array circuit in the first read mode signal; the read circuit provides at least one data signal to the control circuit; and if a mode switch signal generated based on the at least one data signal is received from the control circuit, the read circuit switches the first read mode to the second read mode.
  • an electronic device in a third aspect, includes the vision sensor chip according to the first aspect and a parsing circuit.
  • the parsing circuit is coupled to the vision sensor chip and is configured to parse the data signals received from the reading circuit in a parsing mode corresponding to a reading mode of the reading circuit of the vision sensor chip.
  • Figure 1A shows a schematic diagram of the amount of data read over time in an event stream-based asynchronous read mode
  • FIG. 1B shows a schematic diagram of the change of the amount of read data with time in the synchronous reading mode based on frame scanning
  • FIG. 2 illustrates a block diagram of an example vision sensor in which example embodiments of the present disclosure may be implemented
  • FIG. 3 illustrates a block diagram of another example vision sensor in which example embodiments of the present disclosure may be implemented
  • FIG. 4 shows a schematic diagram of the principles of a frame scan-based synchronous read mode and an event stream-based asynchronous read mode according to an example embodiment of the present disclosure
  • FIG. 5A shows a schematic diagram of a vision sensor operating in a frame scan based reading mode according to an example embodiment of the present disclosure.
  • 5B shows a schematic diagram of a vision sensor operating in an event stream based reading mode according to an example embodiment of the present disclosure.
  • FIG. 6A shows a schematic diagram of a vision sensor operating in an event stream based reading mode according to an example embodiment of the present disclosure.
  • 6B shows a schematic diagram of a vision sensor operating in a frame scan based reading mode according to an example embodiment of the present disclosure.
  • FIG. 7 shows a flowchart of a method for operating a vision sensor chip according to an example embodiment of the present disclosure
  • FIG. 8 shows a block diagram of a control circuit for a vision sensor according to an example embodiment of the present disclosure
  • FIG. 9 shows a block diagram of an electronic device according to an example embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram illustrating a change in the amount of data over time in an adaptive read-based mode according to an example embodiment of the present disclosure.
  • the term “including” and variations thereof mean open-ended inclusion, ie, “including but not limited to”.
  • the term “or” means “and/or” unless specifically stated otherwise.
  • the term “based on” means “based at least in part on”.
  • the terms “example embodiment” and “certain embodiments” mean “at least one example embodiment.” Other explicit and implicit definitions may also be included below.
  • FIG. 1A and FIG. 1B are schematic diagrams showing the relationship between the amount of read data and time in the asynchronous read mode based on event flow and the synchronous read mode based on frame scan, respectively.
  • the pixels in the vision sensor's pixel array circuit measure the amount of light intensity variation in the environment. If the measured amount of light intensity change exceeds a predetermined threshold, the pixel may output a data signal indicative of the light intensity change event. Therefore, in the event stream based asynchronous read mode, the pixels of the vision sensor are further differentiated into pixels that generate light intensity change events and pixels that do not.
  • the light intensity change event can be characterized by the coordinate information (x, y) of the pixel generating the event, the light intensity change information at the pixel, and the time t when the light intensity change information is read.
  • the coordinate information (x, y) can be used to uniquely identify a pixel in the pixel array circuit, eg, x represents the row index in the pixel array circuit where the pixel is located, and y represents the column index in the pixel array circuit where the pixel is located.
  • the event data to be read is sparse and asynchronous.
  • the vision sensor works in the asynchronous reading mode based on the event stream.
  • the rate of light intensity change events occurring in the pixel array circuit changes, the amount of data that the vision sensor needs to read also varies with the Change of time. In a static scene, fewer light-intensity-change events are generated, so the total amount of data that the vision sensor needs to read is lower. In dynamic scenes, such as vigorous exercise, a large number of light intensity change events are generated, and the total amount of data that the vision sensor needs to read also increases.
  • the data amount of one frame of data is 528 bits.
  • the frames are output at equal time intervals, for example, they can be output at a rate of 30 frames per second, 60 frames per second, and 120 frames per second.
  • the pixel values in a frame are arranged in order from top left to bottom right.
  • the same scan order ie, from top left to bottom right
  • the same scan order is used to obtain the light intensity value at each pixel position, where the pixel coordinates (x, y) are the same as the pixel in one frame The scan number in the one-to-one correspondence.
  • the output and storage costs of adopting the two modes may have significant differences. For example, when shooting a static scene, only a small number of pixels produce light intensity change events over a period of time. By way of example, for example, in one scan, only three pixels in the pixel array circuit generate light intensity change events. In the asynchronous reading mode, only the coordinate information (x, y), time information t and light intensity change of the three pixels are read, and three light intensity change events can be represented. Assuming that in the asynchronous reading mode, 4, 2, 2, and 2 bits are allocated for the coordinates, reading time stamp, and light intensity change of a pixel, respectively, the total amount of data to be read in this reading mode is 30 bits.
  • the data signals output by all pixels in the entire array still need to be read to form a complete frame of data.
  • the total number of pixels of the pixel array circuit is 66
  • the total amount of data to be read is 528 bits. It can be seen that even if there are a large number of unevented pixels in the pixel array circuit, so many bits still need to be allocated in the synchronous read mode. It is uneconomical from the standpoint of presentation cost and increases the pressure on data transmission and storage. Therefore, in this case, it is more economical to use the asynchronous read mode.
  • a large number of pixels in the vision sensor measure the light in a short period of time when there is a lot of motion within the scene or when the light intensity in the environment changes drastically, such as a large number of people walking around, or suddenly switching lights on and off, etc. strong changes and generate data signals indicative of light intensity change events. Since the amount of data that characterizes a single event in asynchronous read mode is greater than that of a single event in synchronous read mode, the use of asynchronous read mode in this case may require significant representation costs. Specifically, in each row of the pixel array circuit, there may be multiple consecutive pixels generating light intensity change events, and for each event, coordinate information (x, y), time information t, and light intensity change information a need to be transmitted.
  • Coordinate changes between these events tend to be off by only one unit, and read times are about the same.
  • the asynchronous read mode is expensive for the representation of coordinate and time information, which will cause a surge in data volume.
  • each pixel outputs a data signal that only indicates the light intensity change, without the need for coordinate information for each pixel. and time information allocation bits. Therefore, it is more economical to use the synchronous read mode for event-heavy situations.
  • the present disclosure proposes a scheme for adaptively switching reading modes.
  • the vision sensor compares the amount of data in the two reading modes based on the statistical results of light intensity change events generated by the pixel array circuit, so that it can switch to the reading mode suitable for the current application scene and motion state.
  • data can be read from the pixel array circuit in a lower cost read mode at all times, so that the needs of image transmission and storage can be better met.
  • FIG. 2 shows a block diagram of an example vision sensor 200 in which example embodiments of the present disclosure may be implemented.
  • the vision sensor 200 may be implemented as a vision sensor chip and capable of reading data signals indicative of light intensity change events in at least one of a frame scan based read mode and an event stream based read mode.
  • the visual sensor 200 includes a pixel array circuit 210 and a reading circuit 220 .
  • Vision sensor 200 is coupled to control circuit 230 . It should be understood that the vision sensor 200 shown in FIG. 2 is for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be embodied in different sensor architectures.
  • the visual sensor 200 may also include other elements or entities for realizing image acquisition, image processing, image transmission and other purposes, which are not shown for the convenience of description, but do not mean that the embodiments of the present disclosure do not have these elements or entities.
  • the pixel array circuit 210 may include one or more pixel arrays, and each pixel array includes a plurality of pixels, each pixel having position information for unique identification, such as coordinates (x, y).
  • the pixel array circuit 210 can be used to measure the variation of light intensity and generate a plurality of data signals corresponding to a plurality of pixels.
  • each pixel is configured to respond independently to changes in light intensity in the environment. The pixel compares the measured light intensity change to a predetermined threshold, and if the measured light intensity change exceeds the predetermined threshold, the pixel generates a first data signal, eg, +1 or -1, indicative of a light intensity change event.
  • the first data signal may indicate a light intensity variation trend or light intensity value at the corresponding pixel. If the measured light intensity change does not exceed a predetermined threshold, the pixel generates a second data signal, eg, 0, that is different from the first data signal.
  • the data signal may indicate, but not limited to, light intensity polarity, absolute light intensity value, change value of light intensity, and the like.
  • Light intensity polarity can indicate the tendency of light intensity to change, for example, increase or decrease, and is usually represented by +1 and -1.
  • the absolute light intensity value may represent the light intensity value measured at the current moment. Depending on the structure, use and kind of the sensor, there may be different physical meanings regarding the light intensity or the amount of light intensity change. The scope of the present disclosure is not limited in this regard.
  • Readout circuitry 220 is coupled to, and can communicate with, pixel array circuitry 210 and control circuitry 230 .
  • the reading circuit 220 may be configured to scan the pixels in the pixel array circuit 210 in a predetermined order to read data signals generated by the corresponding pixels.
  • the reading circuit 220 is configured to be able to read the data signals output by the pixel array circuit 210 in more than one signal reading mode. For example, the read circuit 220 may read in one of the first read mode and the second read mode.
  • the first read mode and the second read mode correspond to one of a frame scan-based read mode and an event stream-based read mode, respectively, and further, the first read mode may refer to Instead of the current read mode of the read circuit 220, the second read mode may refer to a switchable alternate read mode.
  • FIG. 4 there is shown a schematic diagram of the principles of the frame scan based synchronous read mode and the event stream based asynchronous read mode according to an example embodiment of the present disclosure.
  • the black dots represent pixels that generate light intensity change events
  • the white dots represent pixels that do not generate light intensity change events.
  • the dotted box on the left represents a frame scan-based synchronous reading mode, in which all pixels generate voltage signals based on received optical signals, and then output data signals after analog-to-digital conversion.
  • the reading circuit 220 constitutes one frame of data by reading data signals generated by all pixels.
  • the dotted box on the right represents the asynchronous reading mode based on the event stream.
  • the reading circuit 220 scans the pixel that generates the light intensity change event, the coordinate information (x, y) of the pixel can be obtained. Then, only the data signal (ie, the first data n) generated by the pixel generating the light intensity change event indicating the light intensity change amount a is read, and the read time t is recorded.
  • the reading circuit 220 sequentially reads the data signals generated by the multiple pixels according to the scanning sequence, and forms an event stream as an output.
  • the lower part of Figure 4 describes the two read modes in terms of representational cost (eg, the amount of data required to read).
  • representational cost eg, the amount of data required to read.
  • the amount of data read by the read circuit 220 each time is the same, for example, one frame of data.
  • the 1st frame data 401-1 and the 2nd frame data 401-2 are shown.
  • the amount of data representing a single pixel eg, the number of bits B p
  • the total number M of pixels in the pixel array circuit the amount of data to be read in one frame can be determined as M ⁇ B p .
  • the read circuit 220 reads the data signal indicating the light intensity change event, and then forms an event stream 402 for all events in the order of occurrence.
  • the amount of data read each time by the read circuit 220 is related to the amount of event data B ev for representing a single event (eg, coordinates (x, y) representing the pixel generating the event, read time stamp t and The sum of the number of bits of the light intensity change amount a) and the number of light intensity change events N ev are related.
  • the read circuit 220 may be configured to provide the read at least one data signal to the control circuit 230 .
  • the reading circuit 220 may provide the control circuit 230 with data signals read within a period of time for the control circuit 230 to perform historical data statistics and analysis.
  • the read circuit 220 scans the pixel array circuit 210 at a predetermined frame rate fHz in this mode to read all the pixels in the pixel array circuit 210 The data signal generated by the pixel.
  • These data signals may indicate the amount of change a of light intensity measured by each pixel in the pixel array circuit 210, and may include both the first data signal and the second data signal.
  • the amount of data per frame read by the reading circuit 220 can be expressed as M ⁇ B p bits, and the amount of data read per second is M ⁇ B p f bits/second (bits/s), i.e., the read data rate of the read circuit 220 is M ⁇ Bp ⁇ f bits/s, where the total number of pixels in a given vision sensor 200 is M, and Bp is the The amount of pixel data (eg, the number of bits) allocated for each pixel in the frame scan-based read mode.
  • the read circuit 220 when the currently adopted first read mode is an event stream-based read mode, the read circuit 220 reads data generated by pixels in the pixel array circuit 210 that generate light intensity change events For the convenience of description, these data signals are also referred to as first data signals hereinafter. Specifically, the reading circuit 220 scans the pixel array circuit 210 to determine the position information (x, y) of the pixels related to the light intensity change event. Based on the position information (x, y) of the pixel, the reading circuit 220 reads the first data signal generated by the pixel among the plurality of data signals to obtain the light intensity change amount a indicated by the first data signal and read time information t.
  • the control circuit 230 is coupled to the read circuit 220 and is configured to control the read circuit 220 to read the data signals generated by the pixel array circuit 210 in a particular read mode.
  • the control circuit 230 may obtain at least one data signal from the read circuit 220 and determine which one of the current read mode and the alternative read mode is more suitable based at least on the at least one data signal The current application scene and motion state. Further, in some embodiments, the control circuit 230 may instruct the read circuit 220 to switch from the current data read mode to another data read mode based on the determination.
  • control circuit 230 may send an indication to the reading circuit 220 to switch the reading mode based on historical statistics of light intensity change events. For example, control circuit 230 may determine statistical data related to at least one light intensity change event based on at least one data signal received from read circuit 220 . If the statistical data is determined to satisfy the predetermined switching condition, the control circuit 230 sends a mode switching signal to the reading circuit 220 to cause the reading circuit 220 to switch to the second reading mode. For ease of comparison, the statistics may be used to measure the read data rate of the first read mode and the second read mode, respectively.
  • the statistical data may include the total data volume of the number of events measured by the pixel array circuit 210 per unit time. If the total data volume of the light intensity change events read by the reading circuit 220 in the first reading mode has been greater than or equal to the total data volume of the light intensity change events in the second reading mode, it indicates that the reading circuit 220 should The first read mode is switched to the second read mode.
  • the given first read mode is a frame scan based read mode and the second read mode is an event stream based read mode.
  • the control circuit 230 may determine the total data amount M ⁇ Bp ⁇ f of light intensity change events read in the first read mode based on the pixel number M, the frame rate f, and the pixel data amount Bp of the pixel array circuit.
  • the control circuit 230 may determine the total data amount of light intensity change events B ev ⁇ N ev based on the number of light intensity change events N ev and the event data amount B ev associated with the event stream-based read mode, that is, The total data amount B ev ⁇ N ev of light intensity change events read in the second reading mode.
  • a switching parameter can be used to adjust the relationship between the total data amount in the two reading modes, as shown in the following formula (1), the light intensity change event read in the first reading mode If the total data volume M ⁇ B p ⁇ f is greater than or equal to the total data volume B ev ⁇ N ev of the light intensity change event of the second reading mode, the reading circuit 220 should switch to the second reading mode:
  • the first threshold data amount d 1 M ⁇ Bp ⁇ f ⁇ . That is, if the total data volume B ev ⁇ N ev of the light intensity change events is less than or equal to the threshold data volume d 1 , it indicates that the total data volume of the light intensity change events read in the first reading mode has been greater than or equal to the second data volume
  • the control circuit 230 may determine that the statistical data of the light intensity change events in the read mode satisfy a predetermined switching condition.
  • the threshold data amount d 1 may be determined based on at least the pixel number M of the pixel array circuit, the frame rate f associated with the frame scan based read mode, and the pixel data amount B p .
  • the total data amount M ⁇ B p ⁇ f of the light intensity change events read in the first reading mode is greater than or equal to the total data of the light intensity change events in the second reading mode
  • the quantity B ev ⁇ N ev can be expressed as the following formula (2):
  • the control circuit 230 may determine that the statistical data of the light intensity change event satisfies a predetermined switching condition.
  • the threshold data amount d 2 may be determined based on at least the pixel number M of the pixel array circuit, the frame rate f associated with the frame scan based read mode, and the pixel data amount B p .
  • the first read mode is an event stream based read mode and the second read mode is a frame scan based read mode. Since in the event stream-based read mode, the read circuit 220 only reads the data signal generated by the pixel generating the event, ie, the first data signal. Thus, the control circuit 230 can directly determine the number N ev of light intensity change events generated in the pixel array circuit 210 based on the number of the first data signals provided by the read circuit 220 . The control circuit 230 may determine the total data volume of light intensity change events, that is, the total number of events read in the first reading mode, based on the event number N ev and the event data volume B ev associated with the event stream-based reading mode.
  • the amount of data B ev ⁇ N ev can also determine the total data amount M ⁇ B p of the light intensity change event read in the second reading mode based on the pixel number M, the frame rate f and the pixel data amount B p of the pixel array circuit ⁇ f. As shown in the following formula (3), the total data volume B ev ⁇ N ev of the light intensity change events read in the first reading mode is greater than or equal to the total data volume M ⁇ of the light intensity change events in the second reading mode B p f, the read circuit 220 should switch to the second read mode:
  • the control circuit 230 determines that the statistical data of the light intensity change event satisfies a predetermined switching condition.
  • the threshold data amount d 1 may be determined based on at least the pixel number M of the pixel array circuit, the frame rate f, and the pixel data amount B p .
  • the total data volume B ev ⁇ N ev of the light intensity change events read in the first reading mode is greater than or equal to the total data volume of the light intensity change events in the second reading mode M ⁇ B p ⁇ f can be expressed as the following formula (4)
  • the control circuit 230 determines that the statistical data of the light intensity change event satisfies the predetermined switching condition.
  • the threshold data amount d 2 may be determined based on at least the pixel number M of the pixel array circuit, the frame rate f, and the pixel data amount B p .
  • the statistical data may include the number of events N ev measured by the pixel array circuit 210 per unit time. If the first read mode is the frame scan-based read mode and the second read mode is the event stream-based read mode, the control circuit 230 is based on the difference of the first data signal among the plurality of data signals provided by the read circuit 220 . number to determine the number of light intensity change events N ev .
  • the control circuit 230 determines that the statistics of light intensity change events satisfy the predetermined switching condition may be based at least on the number M of pixels of the pixel array circuit, and based on the frame The frame rate f and the amount of pixel data Bp associated with the scanned read mode, and the event data amount Bev associated with the event stream-based read mode determine the first threshold number n 1 .
  • the following formula (5) can be further obtained based on formula (1):
  • the first threshold number n 1 may be determined as
  • the second threshold number n 2 can be determined as
  • the control circuit 230 is based on at least one data signal provided by the read circuit 220
  • the number of light intensity change events, N ev can be directly determined. If the statistical data indicates that the number N ev of light intensity change events is greater than or equal to the first threshold number n 1 , the control circuit 230 determines that the statistical data of the light intensity change events satisfies the predetermined switching condition.
  • n 1 M ⁇ B p ⁇ f/( ⁇ B ev ).
  • formula (7) can be further obtained based on formula (3):
  • the first threshold number n 1 may be determined as
  • the second threshold number n 2 can be determined as
  • the control circuit 230 may be connected to the vision sensor 200 through an interface as a separate circuit or chip external to the vision sensor 200 as shown in FIG. 2 .
  • the control circuit 230 can also be used as a circuit or chip inside the vision sensor, integrated with the pixel array circuit and the reading circuit therein.
  • FIG. 3 shows a block diagram of another vision sensor 300 according to an example embodiment of the present disclosure. Vision sensor 300 may be implemented as an example of vision sensor 200 .
  • the visual sensor 300 includes a pixel array circuit 310 , a reading circuit 320 and a control circuit 330 .
  • the pixel array circuit 310 , the readout circuit 320 and the control circuit 330 are functionally the same as the pixel array circuit 210 , the readout circuit 220 and the control circuit 230 shown in FIG. 2 , and thus will not be repeated here.
  • the vision sensor 300 is for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be embodied in different vision sensors.
  • the vision sensor 300 may also include other elements, modules or entities, which are not shown for the sake of clarity, but do not mean that the embodiments of the present disclosure do not have these elements or entities.
  • 5A shows a schematic diagram of a vision sensor operating in a frame scan based reading mode according to an embodiment of the present disclosure.
  • 5B shows a schematic diagram of a vision sensor operating in an event stream based reading mode according to an embodiment of the present disclosure.
  • the readout circuit 220 or 320 is currently operating in a first readout mode, a frame scan based readout mode. Since the control circuit 230 or 330 determines that the number of events generated in the pixel array circuit 210 or 310 is currently relatively small, for example, there are only four valid data in one frame of data, based on historical statistics, and then predicts the possible event generation rate in the next time period lower.
  • the control circuit 230 or 330 sends a mode switching signal to the reading circuit 220 or 320 to switch the reading circuit 220 or 320 from the first reading mode to the second reading mode. After switching, as shown in FIG. 5B , the read circuit 220 or 320 operates in the second read mode and only reads valid data signals, thereby avoiding transmission bandwidth and storage resources occupied by a large number of invalid data signals.
  • 6A shows a schematic diagram of a vision sensor operating in an event stream based reading mode according to an embodiment of the present disclosure.
  • 6B shows a schematic diagram of a vision sensor operating in a frame scan based reading mode according to an embodiment of the present disclosure.
  • the read circuit 220 or 320 is currently operating in a first read mode, ie, an event stream-based read mode.
  • the control circuit 230 or 330 determines based on historical statistics, the current number of events generated in the pixel array circuit 210 or 310 is relatively large. For example, in a short period of time, almost all the pixels in the pixel array circuit 210 or 310 generate an indication that the amount of change in light intensity is high. data signal at the threshold.
  • the read circuit 220 or 320 can predict that the likely event generation rate is higher in the next time period. Since a large amount of redundant data exists in the read data signal, for example, nearly the same pixel position information, read time stamp, etc., if the read circuit 220 or 320 continues to use the event stream-based read mode to read, Will cause a surge in the amount of read data. Thus, in this case, the control circuit 230 or 330 sends a mode switching signal to the reading circuit 220 or 320 to switch the reading circuit 220 or 320 from the first reading mode to the second reading mode. After switching, as shown in FIG. 6B , the read circuit 220 or 320 operates in a frame scan-based mode to read data signals in a read mode with a less expensive representation of a single pixel, alleviating the need to store and transmit data signals. pressure.
  • the vision sensor 200 or 300 may further include a parsing circuit, which may be configured to parse the data signal output by the reading circuit 220 or 320 .
  • the parsing circuit may employ a parsing mode compatible with the current data reading mode of the read circuit 220 or 320 to parse the data signal. This will be described in detail below.
  • example embodiments of the present disclosure may switch between more than two data read modes.
  • a vision sensor chip capable of adaptively switching between multiple reading modes according to historical statistics of light intensity change events generated in a pixel array circuit. In this way, no matter in dynamic scenes or static scenes, the vision sensor chip can always achieve good reading and parsing performance, avoid the generation of redundant data, and relieve the pressure of image processing, transmission and storage.
  • FIG. 7 shows a flowchart of a method 700 for operating a vision sensor chip according to an example embodiment of the present disclosure.
  • method 700 may be implemented in vision sensor 200 shown in FIG. 2 or vision sensor 300 shown in FIG. 3 and electronic device 900 shown in FIG. 9 below, or any suitable device may be used. to implement, including various devices currently known or to be developed in the future.
  • the method 700 will be described below in conjunction with the vision sensor 200 shown in FIG. 2 .
  • the pixel array circuit 210 generates a plurality of data signals corresponding to the plurality of pixels in the pixel array circuit 210 by measuring the amount of change in light intensity.
  • the plurality of data signals may indicate whether a light intensity change event is generated at the corresponding pixel.
  • a light intensity change event means that the amount of light intensity change measured by a corresponding pixel in the pixel array circuit 210 exceeds a predetermined threshold.
  • a first data signal is generated indicating a light intensity change event, eg, +1 or -1. If the amount of change in light intensity measured by the pixels in the pixel array circuit 210 does not exceed a predetermined threshold, a second data signal that is different from the first data signal, eg, 0, is generated.
  • the read circuit 220 reads at least one data signal of the plurality of data signals from the pixel array circuit 210 in a first read mode.
  • the data signals may indicate the light intensity change amount a, and these data signals occupy certain storage and transmission resources in the vision sensor 200 after being read.
  • the way in which the vision sensor chip 200 reads the data signals may vary.
  • read circuit 220 scans pixel array circuit 210 to determine position information (x, y) of pixels associated with light intensity change events. Based on the position information, the readout circuit 220 may read out the first data signal among the plurality of data signals.
  • the reading circuit 220 obtains the light intensity change amount a, the position information (x, y) of the pixel generating the light intensity change event, and the time stamp t of the read data signal by reading the data signal.
  • the first read mode may be a frame scan based read mode.
  • vision sensor 200 scans pixel array circuit 210 at the frame frequency associated with the frame scan based read mode to read all data signals generated by pixel array circuit 210 .
  • the reading circuit 220 obtains the light intensity variation a by reading the data signal.
  • the read circuit 220 provides the read at least one data signal to the control circuit 230 for the control circuit 230 to perform data statistics and analysis.
  • the control circuit 230 may determine statistics related to at least one light intensity change event based on the at least one data signal.
  • the control circuit 230 may utilize the switching strategy module to analyze the statistical data. If it is determined that the statistical data satisfies the predetermined switching condition, the control circuit 230 sends a mode switching signal to the reading circuit 220 .
  • the control circuit 230 may be based on the first read mode of the plurality of data signals. A number of data signals to determine the number of at least one light intensity change event. In turn, the control circuit 230 compares the number of light intensity change events with a first threshold number. If the statistical data indicates that the number of light intensity change events is less than or equal to the first threshold number, the control circuit 230 determines that the statistical data of the light intensity change events satisfies a predetermined switching condition, and sends a mode switch signal.
  • control circuit 230 may be based on the number of pixels of the pixel array circuit, the frame rate and amount of pixel data associated with the frame scan based read mode, and the event data associated with the event stream based read mode amount to determine or adjust the first threshold number.
  • the control circuit 230 may further based on the light intensity change event The number and amount of event data associated with the event stream-based read mode determine the total amount of data for light intensity change events. Further, the control circuit 230 compares the total data amount of the light intensity change event with the first threshold data amount. If the total data amount of the light intensity change event does not exceed the first threshold data amount, the control circuit 230 determines that the statistical data of the light intensity change event satisfies a predetermined switching condition, and sends a mode switching signal. In this embodiment, the control circuit 230 may determine or adjust the first threshold data amount based on the pixel number, frame rate, and pixel data amount of the pixel array circuit.
  • the control circuit 230 may The first data signal determines statistical data related to the light intensity change event. In turn, the control circuit 230 compares the number of light intensity change events to a second threshold number. If the number of light intensity change events is greater than or equal to the second threshold number, the control circuit 230 determines that the statistical data of the light intensity change events satisfy a predetermined switching condition, and sends a mode switching signal.
  • control circuit 230 may be based on the number of pixels of the pixel array circuit, the frame rate and amount of pixel data associated with the frame scan based read mode, and the event data associated with the event stream based read mode amount to determine or adjust the second threshold number.
  • the control circuit 230 may further change the light intensity according to the number of light intensity change events and the Based on the event data volume associated with the read mode of the event stream, the total data volume of the light intensity change event is determined. Further, the control circuit 230 compares the total data amount of the light intensity change event with the second threshold data amount. If the total data volume of the light intensity change event does not exceed the second threshold data volume, the control circuit 230 determines that the statistical data of the light intensity change event satisfies a predetermined switching condition, and sends a mode switching signal.
  • control circuit 230 may determine or adjust the second threshold data amount based on the pixel number, frame rate, and pixel data amount of the pixel array circuit. It should be understood that when the switching strategy is configured with multiple switching conditions, the control circuit 230 can select from multiple switching conditions as required, for example, according to the type of the visual sensor 200, the characteristics of the light intensity change event, the attributes of the external environment, the motion state, and other factors. Choose one for analysis and decision-making. In the exemplary embodiments of the present disclosure, other suitable handover policy modules and handover conditions or policies may also be employed, and the scope of the present disclosure is not limited in this respect.
  • the read circuit 220 switches the first read mode to the second read mode based on the mode switch signal received from the control circuit 220 . Further, the read circuit 220 reads at least one data signal generated by the pixel array circuit 210 in the second read mode. The control circuit 230 can then continue to perform historical statistics on the light intensity change events generated by the pixel array circuit 210, and when the switching conditions are met, send a mode switching signal to make the reading circuit 220 switch from the second reading mode to the first reading mode. .
  • control circuit continuously performs historical statistics and real-time analysis on the light intensity change events generated in the pixel array circuit during the whole process of reading and parsing, and sends the mode switching once the switching condition is satisfied. signal to cause the read circuit to switch from the current read mode to a more appropriate alternate switch mode. This adaptive switching process is repeated until the reading of all data signals is completed.
  • FIG. 8 shows a block diagram of a control circuit 800 of an example embodiment of the fundamental disclosure.
  • the control circuit 800 may be used to implement the control circuit 230 in FIG. 2 , the control circuit 330 in FIG. 3 , etc., and may also be implemented by other suitable devices.
  • the control circuit 800 is for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be embodied in different control circuits.
  • the control circuit 800 may also include other elements, modules or entities, which are not shown for the purpose of clarity, but do not mean that the embodiments of the present disclosure do not have these elements or entities.
  • the control circuit 800 includes at least one processor 802 , at least one memory 804 coupled to the processor 802 , and a communication mechanism 812 coupled to the processor 802 .
  • the memory 804 is used to store at least the computer program and the data signals obtained from the reading circuit.
  • a statistical model 806 and a handover decision module 808 are preconfigured on the processor 802 .
  • the control circuit 830 may be communicatively coupled to the reading circuit 220 of the vision sensor 200 as shown in FIG. 2 or a reading circuit external to the vision sensor through the communication mechanism 812 to implement control functions therewith.
  • control circuit 800 may be configured to control read circuit 220 in a particular data read mode (eg, frame scan-based synchronous read mode, event-based read mode Asynchronous read mode of a stream, etc.) to read a plurality of data signals generated by the pixel array circuit 210 . Additionally, the control circuit 800 may be configured to obtain data signals from the read circuit 220 that may indicate, but are not limited to, light intensity polarity, absolute light intensity values, changes in light intensity, and the like. For example, light intensity polarity can indicate the trend of light intensity change, such as increase or decrease, usually expressed as +1/-1. The absolute light intensity value may represent the light intensity value measured at the current moment. Information about light intensity or changes in light intensity can have different physical meanings, depending on the structure, use, and kind of sensor.
  • the control circuit 800 determines statistical data related to at least one light intensity change event based on the data signal obtained from the read circuit 220 .
  • the control circuit 800 may acquire data signals generated by the pixel array circuit 210 over a period of time from the reading circuit 220 and store these data signals in the memory 804 for historical statistics and analysis.
  • the first read mode and the second read mode may be one of an event stream based asynchronous read mode and a frame scan based synchronous read mode, respectively. It should be noted, however, that all features described herein with regard to adaptively switching read modes are equally applicable to other types of sensors and data read modes now known or developed in the future, as well as any of more than two data read modes. switch between.
  • control circuit 800 may utilize one or more preconfigured statistical models 806 to perform historical statistics of light intensity change events generated by the pixel array circuit 210 provided by the readout circuit 220 over a period of time.
  • the statistical model 806 may then transmit the statistical data to the handover decision module 808 as an output.
  • the statistical data may indicate the number of light intensity change events, or may indicate the total data volume of light intensity change events. It should be understood that any suitable statistical model, statistical algorithm can be applied to the exemplary embodiments of the present disclosure, and the scope of the present disclosure is not limited in this respect.
  • the decision module 808 can be preconfigured with a rate switch decision module 808 for analyzing and predicting the occurrence of events in the next period or multiple handover decisions.
  • the control circuit 800 can select one of the multiple switching decisions for Analysis and decision making.
  • other suitable switching decision modules and mode switching conditions or strategies may also be employed, and the scope of the present disclosure is not limited in this respect.
  • the switching decision module 808 determines that the statistics satisfy the mode switching condition, an indication is output to the reading circuit 220 to switch the reading mode. In another embodiment, if the switching decision module 808 determines that the statistics do not meet the mode switching conditions, no indication is output to the reading circuit 220 to switch the reading mode. In some embodiments, the indication to switch the read mode may take an explicit form as described in the above embodiments, for example in the form of a switch signal or flag bit to notify the read circuit 220 to switch the read mode.
  • FIG. 9 shows a block diagram of an electronic device 900 according to an example embodiment of the present disclosure.
  • the electronic device 900 includes a visual sensor chip 901 , communication interfaces 902 and 903 , a control circuit 930 , and an analysis circuit 904 .
  • the electronic device 900 is for exemplary purposes and may be implemented using any suitable device, including various sensor devices currently known and developed in the future. Embodiments of the present disclosure may also be embodied in different sensor systems.
  • the electronic device 900 may also include other elements, modules or entities, which are not shown for the purpose of clarity, but do not mean that the embodiments of the present disclosure do not have these elements, modules or entities.
  • vision sensor 901 includes pixel array circuit 910 and readout circuit 920, wherein readout components 920-1 and 920-2 of readout circuit 920 are coupled to control circuit 930 via communication interfaces 902 and 903, respectively.
  • the reading components 920-1 and 920-2 may be implemented by separate devices respectively, or may be integrated in the same device.
  • the read circuit 220 shown in FIG. 2 is an example implementation of integration.
  • the reading components 920-1 and 920-2 may be configured to implement a data reading function in a frame scan-based reading mode and an event stream-based reading mode, respectively.
  • the pixel array circuit 910 may be implemented using the pixel array circuit 210 in FIG. 2 or the pixel array circuit 310 in FIG. 3 , or may be implemented using any suitable other device, and the present disclosure is not limited in this respect. The features of the pixel array circuit 910 are not repeated here.
  • the read circuit 920 may read the data signals generated by the pixel array circuit 910 in a specific read mode. For example, in the example where the read component 920-1 is turned on and the read component 920-2 is turned off, the read circuit 920 initially reads the data signal using a frame scan based read mode. In the example where the read component 920-2 is turned on and the read component 920-1 is turned off, the read circuit 920 initially reads the data signal using an event stream based read mode.
  • the reading circuit 920 can be implemented by using the reading circuit 220 in FIG. 2 or the reading circuit 320 in FIG. 3 , and can also be implemented by using any other suitable device. The characteristics of the reading circuit 920 are not repeated here. .
  • control circuit 930 may instruct the reading circuit 920 to switch from the first reading mode to the second reading mode by means of an indication signal or a flag bit.
  • the reading circuit 920 may receive an instruction from the control circuit 930 to switch the reading mode, eg, turn on the reading component 920-1 and turn off the reading component 920-2, or turn on the reading component 920-2 and disconnect the reading assembly 920-1.
  • the electronic device 900 may also include parsing circuitry 904, as previously described.
  • the parsing circuit 904 may be configured to parse the data signals read by the reading circuit 920 .
  • the parsing circuit may employ a parsing mode that is compatible with the current data reading mode of the reading circuit 920 .
  • the parsing circuit parses the data accordingly based on the first amount of data Bev ⁇ Nev associated with the read mode.
  • the parsing circuit starts parsing according to the second data amount, that is, the data size of one frame M ⁇ B p data signal and vice versa.
  • the parsing circuit 904 can implement switching of parsing modes of the parsing circuit without explicitly switching signals or flag bits.
  • parsing circuit 904 may employ the same or corresponding statistical model and switching strategy as control circuit 930 to perform the same statistical analysis of data signals provided by read circuit 920 as control circuit 930 and make consistent switching predictions.
  • the parsing circuit initially parses the data based on the first amount of data B ev ⁇ N ev associated with the read mode.
  • the first b x bits parsed by the parsing circuit indicate the coordinate x of the pixel
  • the next b y bits indicate the coordinate y of the pixel
  • the following b t bits indicate the reading time
  • b a bits indicate the light Strong variability.
  • the parsing circuit obtains the at least one data signal from the reading circuit 920 and determines statistical data related to the at least one light intensity change event. If the analysis circuit 904 determines that the statistical data satisfies the switching condition, it switches to the analysis mode corresponding to the frame scan-based read mode, and analyzes the data signal with the frame data size M ⁇ B p .
  • the parsing circuit 904 sequentially fetches the data in the frame according to each B p bit in the parsing mode corresponding to the read mode. The value for each pixel location, where pixel locations that did not generate light intensity change events have the value 0.
  • the parsing circuit 904 can count the number of non-zeros in a frame based on the data signal, that is, the number of light intensity change events in the frame.
  • the parsing circuit 904 obtains at least one data signal from the reading circuit 920 and determines, based on at least the at least one data signal, which of the current parsing mode and the alternative parsing mode is related to the reading circuit 920 corresponding to the read mode. Further, in some embodiments, the parsing circuit 904 may switch from the current parsing mode to another parsing mode based on the determination.
  • the parsing circuit 904 may determine whether to switch parsing modes based on historical statistics of light intensity change events. For example, parsing circuit 904 may determine statistical data related to at least one light intensity change event based on at least one data signal received from read circuit 920 . If the statistical data is determined to satisfy the switching condition, the parsing circuit 904 switches from the current parsing mode to an alternate parsing mode. For ease of comparison, the statistics may be used to measure the read data rate of the first read mode and the second read mode of the read circuit 920, respectively.
  • the statistical data may include the total data volume of the number of events measured by the pixel array circuit 910 per unit time. If the parsing circuit 904 determines based on at least one data signal that the total data volume of the light intensity change events read by the reading circuit 920 in the first reading mode has been greater than or equal to the total data of the light intensity change events in the second reading mode amount, it indicates that the reading circuit 220 has switched from the first reading mode to the second reading mode. In this case, the parsing circuit 904 should switch to the parsing mode corresponding to the current reading mode accordingly.
  • the given first read mode is a frame scan based read mode and the second read mode is an event stream based read mode.
  • parsing circuit 904 initially parses the data signals acquired from reading circuit 920 in a frame-based parsing mode corresponding to the first reading mode.
  • the parsing circuit 904 may determine the total data amount M ⁇ B of the light intensity change event read by the reading circuit 920 in the first reading mode based on the pixel number M, the frame rate f, and the pixel data amount B p of the pixel array circuit 910 p ⁇ f.
  • the parsing circuit 904 may determine the light intensity change read by the read circuit 920 in the second read mode based on the number N ev of light intensity change events and the amount of event data B ev associated with the event stream-based read mode The total data volume of the event B ev ⁇ N ev . In some embodiments, a switching parameter may be used to adjust the relationship between the total data volume in the two read modes. Further, the parsing circuit 904 can determine whether the total data amount M ⁇ B p ⁇ f of the light intensity change events read by the reading circuit 920 in the first reading mode is greater than or It is equal to the total data amount B ev ⁇ N ev of the light intensity change events of the second reading mode. If so, the parsing circuit 904 determines that the reading circuit 920 has switched to the event stream based read mode, and switches from the frame based parsing mode to the event stream based parsing mode accordingly.
  • the parsing circuit 904 can determine the total data amount M ⁇ B of the light intensity change events read by the reading circuit 920 in the first reading mode according to the above formula (2). Whether p ⁇ f is greater than or equal to the total data amount B ev ⁇ N ev of light intensity change events read in the second reading mode.
  • the parsing circuit 904 determines that the reading circuit 920 has switched to the event stream based read mode, and switches from the frame based parsing mode to the event stream based parsing mode accordingly.
  • the first read mode is an event stream based read mode and the second read mode is a frame scan based read mode.
  • parsing circuit 904 initially parses data signals acquired from reading circuit 920 in an event stream-based parsing mode corresponding to the first reading mode.
  • the parsing circuit 904 can directly determine the number N ev of light intensity change events generated in the pixel array circuit 910 based on the number of the first data signals provided by the reading circuit 220 .
  • the parsing circuit 904 may determine the total amount of data B ev of events read by the read circuit 920 in the first read mode based on the number of events N ev and the amount of event data B ev associated with the event stream-based read mode. Nev .
  • the parsing circuit 904 can also determine the total data amount of the light intensity change event read by the read circuit 920 in the second read mode based on the pixel number M, the frame rate f and the pixel data amount B p of the pixel array circuit M ⁇ B p ⁇ f. Then, the parsing circuit 904 may, for example, determine whether the total data amount B ev ⁇ N ev of the light intensity change events read in the first reading mode is greater than or equal to the second reading mode according to formula (3) above The total amount of data M ⁇ B p ⁇ f of light intensity change events.
  • the parsing circuit 904 determines that the reading circuit 920 has switched to the frame scan-based reading mode, and switches from the event stream-based parsing mode to the frame-based parsing mode accordingly.
  • the parsing circuit 904 can determine the total data amount B ev ⁇ Whether N ev is greater than or equal to the total data amount M ⁇ B p ⁇ f of light intensity change events read in the second reading mode. Similarly, after determining that the total data amount B ev ⁇ N ev of the light intensity change events read by the reading circuit 920 in the first reading mode is greater than or equal to the total data amount M of the light intensity change events in the second reading mode In the case of ⁇ Bp ⁇ f, the parsing circuit 904 determines that the reading circuit 920 has switched to the frame scan based read mode, and switches from the event stream based parsing mode to the frame scan based parsing mode accordingly.
  • T 0 is the start time of the first frame
  • k is the frame serial number
  • the time required for digital-to-analog conversion for one pixel in the M pixels can be determined by the following formula (10):
  • the time when the light intensity change event occurs at the i-th pixel in the k-th frame can be determined by the following formula (11):
  • the current read mode is the synchronous read mode, switch to the asynchronous read mode, and parse the data according to the B ev bits of each event.
  • the switching of parsing modes can be implemented without an explicit switching signal or flag bit.
  • the parsing circuit can also parse the data in a similar manner suitable for the data reading mode, which will not be repeated here.
  • a vision sensor with multiple data reading modes is proposed.
  • the vision sensor can adaptively switch between multiple data reading modes, so that the reading data rate always does not exceed the predetermined reading data rate threshold, thereby reducing the cost of data transmission, parsing and storage of the vision sensor, significantly Improved sensor performance.
  • a visual sensor can perform data statistics on events generated in a period of time to predict the possible rate of event generation in the next period of time, so it is possible to select a reading that is more suitable for the current external environment, application scenario and motion state. model.
  • FIG. 10 is a schematic diagram illustrating the variation of data amount with time in a single data read mode and an adaptively switched read mode according to an example embodiment of the present disclosure.
  • the left half of FIG. 10 depicts a schematic diagram of the amount of read data over time for a conventional vision sensor or sensor system simply using a synchronous read mode or an asynchronous read mode.
  • the synchronous read mode as shown in the curve 1001
  • the read data rate the read data per unit time
  • the right half of FIG. 10 depicts a schematic diagram of the amount of data changing over time in an adaptive data read mode according to an example embodiment of the present disclosure.
  • the adaptive data reading mode may be implemented using the vision sensor 200 shown in FIG. 2 , the vision sensor 300 shown in FIG. 3 , or the electronic device 900 shown in FIG. 9 , or a conventional vision sensor or sensor system may be implemented by using the Control circuit 800 is shown to implement an adaptive data read mode.
  • the vision sensor 200 selects eg an asynchronous read mode in the initialization state.
  • the vision sensor 200 can Calculates the read data rate in the current mode.
  • the number of bits B p used to represent each pixel of each frame in the synchronous read mode is also predetermined, so that the read data rate using the synchronous read mode during this period can be calculated.
  • Vision sensor 200 can then determine whether the relationship between the data rates in the two read modes satisfies the mode switching condition. For example, vision sensor 200 may compare which of the two read modes has a lower read data rate based on a predefined threshold.
  • the vision sensor 200 switches to another reading mode, eg, from an initial asynchronous reading mode to a synchronous reading mode.
  • the above steps are continued during the process of reading and parsing the data signal until the output of all data is completed.
  • the vision sensor 200 adaptively selects the optimal reading mode during the entire data reading process, and the two reading modes appear alternately, so that the reading data rate of the vision sensor 200 always keeps not exceeding the synchronous reading rate. This reduces the cost of data transmission, parsing, and storage for vision sensors.
  • the visual sensor 200 can perform historical data statistics on events to predict the possible event generation rate in the next time period, so it can select a more suitable application scenario and Read mode for motion status.
  • the various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a control circuit, microprocessor or other computing device. While aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it is to be understood that the blocks, apparatus, systems, techniques, or methods described herein may be taken as non-limiting Examples are implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or control circuits or other computing devices, or some combination thereof.
  • blocks in the flowcharts may be viewed as method steps, and/or operations generated by operation of computer program code, and/or as multiple coupled logic circuit elements that perform the associated functions.
  • embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code configured to implement the methods described above.
  • a machine-readable medium can be any tangible medium that contains or stores a program for or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination thereof. More detailed examples of machine-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only Memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.
  • Computer program code for implementing the methods of the present disclosure may be written in one or more programming languages. Such computer program code may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus such that the program code, when executed by the computer or other programmable data processing apparatus, causes the flowchart and/or block diagrams The functions/operations specified in are implemented.
  • the program code may execute entirely on the computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)

Abstract

本公开的实施例涉及视觉传感器芯片及其操作方法和电子设备。该视觉传感器芯片包括：像素阵列电路，被配置为通过测量光强变化量来生成与其中的多个像素对应的多个数据信号，多个数据信号指示至少一个光强变化事件，至少一个光强变化事件表示像素阵列电路中的对应像素测量到的光强变化量超过预定阈值；以及读取电路，耦合到像素阵列电路并且被配置为：以第一读取模式从像素阵列电路读取多个数据信号中的至少一个数据信号，向控制电路提供至少一个数据信号，如果从控制电路接收到基于至少一个数据信号而生成的模式切换信号，将第一读取模式切换为第二读取模式。通过在第一和第二读取模式之间自适应地切换，视觉传感器芯片的传输和存储成本能够得到有效降低。

Description

视觉传感器芯片及其操作方法和电子设备 技术领域
本公开的示例实施例涉及视觉传感器领域，并且更加具体地，涉及一种视觉传感器芯片、用于操作视觉传感器芯片的方法以及电子设备。
背景技术
视觉传感技术在视频监控、数码相机、机器人导航、汽车自主导航、生物医学像素分析、人机界面、虚拟现实、工业控制、无线远距离传感、显微镜技术、科学仪器等诸多领域具有广泛用途。通过利用光学元件和成像装置，视觉传感器可以从外部环境获取图像信息，并实现图像处理、图像存储和图像输出等操作。图像的清晰和细腻程度取决于视觉传感器的分辨率，即，像素的数量。一般而言，一幅图像可以由256×256像素、512×512像素或1024×1024像素等来表示。
经过数十年的发展，已经出现了许多不同类型的视觉传感器。例如，对于仿生视觉传感器而言，其原理是利用集成电路对生物视网膜进行仿真，像素阵列电路中的每个像素模拟生物神经元，对光强度的变化以事件的形式进行表达。在实践中，视觉传感器通常具有两种用于读取事件数据的模式，分别是基于帧扫描的同步读取模式和基于事件流的异步读取模式。在基于帧扫描的同步读取模式中，视觉传感器按照扫描顺序扫描像素阵列电路中的所有像素并将像素生成的数据信号排列成一帧数据，继而以固定的速率读取每一帧数据。在基于事件流的异步读取模式中，视觉传感器通过扫描像素阵列电路确定发生光强变化事件的像素，并且只读取这部分像素输出的数据信号。目前由于硬件设计、芯片结构等因素，单个视觉传感器仅能使用一种数据读取模式。然而，采用单一读取模式的视觉传感器很难适应各种环境变化及运动状态，导致在某些应用场景下视觉传感器的性能较差。
发明内容
总体上，本公开的示例实施例提出了与视觉传感器芯片有关的方案。
在第一方面，提出了一种视觉传感器芯片。该视觉传感器芯片包括：像素阵列电路，被配置为通过测量光强变化量来生成与其中的多个像素对应的多个数据信号，多个数据信号指示至少一个光强变化事件，至少一个光强变化事件表示像素阵列电路中的对应像素测量到的光强变化量超过预定阈值；以及读取电路，耦合到像素阵列电路并且被配置为：以第一读取模式从像素阵列电路读取多个数据信号中的至少一个数据信号，向控制电路提供至少一个数据信号，以及如果从控制电路接收到基于至少一个数据信号而生成的模式切换信号，将第一读取模式切换为第二读取模式。
在第二方面，提出了一种用于操作视觉传感器芯片的方法。该方法包括：视觉传感器芯片的像素阵列电路通过测量光强变化量来生成与像素阵列电路中的多个像素对应的多个数据信号，多个数据信号指示至少一个光强变化事件，至少一个光强变化事件表示像素阵列电路中的对应像素测量到的光强变化量超过预定阈值；视觉传感器芯片的读取电路以 第一读取模式从像素阵列电路读取多个数据信号中的至少一个数据信号；读取电路向控制电路提供至少一个数据信号；以及如果从控制电路接收到基于至少一个数据信号而生成的模式切换信号，读取电路将第一读取模式切换为第二读取模式。
在第三方面，提出了一种电子设备。该电子设备包括根据第一方面所述的视觉传感器芯片以及解析电路。该解析电路耦合到视觉传感器芯片，并且被配置为以与视觉传感器芯片的读取电路的读取模式相对应的解析模式，解析从读取电路接收的数据信号。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1A示出了在基于事件流的异步读取模式下读取的数据量随时间变化的示意图；
图1B示出了基于帧扫描的同步读取模式下读取数据量随时间变化的示意图；
图2示出了本公开的示例实施例可以在其中实施的示例视觉传感器的框图；
图3示出了本公开的示例实施例可以在其中实施的另一示例视觉传感器的框图；
图4示出了根据本公开的示例实施例的基于帧扫描的同步读取模式和基于事件流的异步读取模式的原理的示意图；
图5A示出了根据本公开的示例实施例的视觉传感器在基于帧扫描的读取模式下操作的示意图。
图5B示出了根据本公开的示例实施例的视觉传感器在基于事件流的读取模式下操作的示意图。
图6A示出了根据本公开的示例实施例的视觉传感器在基于事件流的读取模式下操作的示意图。
图6B示出了根据本公开的示例实施例的视觉传感器在基于帧扫描的读取模式下操作的示意图。
图7示出了根据本公开的示例实施例的用于操作视觉传感器芯片的方法的流程图；
图8示出了根据本公开的示例实施例的用于视觉传感器的控制电路的框图；
图9示出了根据本公开的示例实施例的电子设备的框图；以及
图10示出了根据本公开的示例实施例的在基于自适应读取模式下的数据量随时间变化的示意图。
具体实施方式
下面将参考附图描述一些示例实施例。虽然附图中显示了本公开的示例实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本文中使用的术语“包括”及其变形表示开放性包括，即“包括但不限于”。除非特别申明，术语“或”表示“和/或”。术语“基于”表示“至少部分地基于”。术语“示例实施例”和“某些实施例”表示“至少一个示例实施例”。下文还可能包括其他明确的和隐含的定义。
在传统技术中，视觉传感器普遍采用基于事件流的异步读取模式(下文也简称“基于事件流的读取模式”或“异步读取模式”)和基于帧扫描的同步读取模式(下文也简称“基 于帧扫描的读取模式”或“同步读取模式”)。对于已经制造完成的视觉传感器，仅能够采用两种模式中的一种。根据具体的应用场景及运动状态，上述两种读取模式在单位时间内所需读取的信号数据量可能具有显著差异，进而输出读取数据所需的代价也不尽相同。图1A和图1B分别示出了基于事件流的异步读取模式和基于帧扫描的同步读取模式下读取数据量与时间的关系的示意图。
以现有仿生视觉传感器为例，因其具有对运动敏感的特性，而环境中的静态区域通常不产生光强变化事件(在本文在也被称为“事件”)，这类传感器几乎全部采用基于事件流的异步读取模式。具体而言，视觉传感器的像素阵列电路中的像素测量环境中的光强变化量。如果测量到的光强变化量超过预定阈值，则像素可以输出指示光强变化事件的数据信号。因此，在基于事件流的异步读取模式下，视觉传感器的像素被进一步区分为产生光强变化事件的像素和未产生光强变化事件的像素。光强变化事件可以通过产生事件的像素的坐标信息(x，y)、该像素处的光强变化信息以及光强变化信息被读取的时间t等来表征。坐标信息(x，y)可用于在像素阵列电路中惟一地标识像素，例如，x表示像素在像素阵列电路中所位于的行索引，y表示像素在像素阵列电路中所位于的列索引。通过标识与像素相关联的坐标和时间戳，能够唯一地确定发生光强变化事件的时空位置，继而可以将所有的事件按发生的先后顺序构成一个事件流。在异步读取模式下，只有产生光强变化事件的像素处的数据信号才会被读取。因而，对于仿生视觉传感器而言，所需读取的事件数据具有稀疏异步的特性。如图1A的曲线101所示，视觉传感器工作在基于事件流的异步读取模式下，当像素阵列电路中发生的光强变化事件的速率变化时，视觉传感器所需读取的数据量也随时间变化。在静态场景中产生的光强变化事件较少，因而视觉传感器所需读取的总数据量也较低。而在动态场景中，例如剧烈运动时，大量产生光强变化事件，视觉传感器所需读取的总数据量也随之上升。
另一方面，传统的视觉传感器，诸如手机摄像头、数字摄像机等通常采用基于帧扫描的同步读取模式。该读取模式并未对视觉传感器的像素是否产生光强变化事件进行区分。无论在某一像素处是否产生光强变化事件，该像素所生成的数据信号均被读取。在读取数据信号时，视觉传感器按照预定顺序扫描像素阵列电路，同步读取各个像素处的指示光强变化量的数据信号，并按照顺序输出为第1帧数据、第2帧数据等等。因而，如图1B的曲线102所示，在同步读取模式下，视觉传感器读取的每一帧数据量具有相同的大小，数据量随时间保持不变。例如，假定采用8比特来表示一个像素的光强度值，而视频传感器中的像素总数为66个，则一帧数据的数据量为528比特。通常，帧与帧之间按照相等的时间间隔输出，例如可以按照每秒30帧、每秒60帧、每秒120帧等速率输出。在视频传感器进行采样时，一帧内的像素值按照从左上到右下的顺序排列。同样，在解析一帧数据时，也采用相同的扫描顺序(即，从左上到右下)来获取每个像素位置处的光强度值，其中像素坐标(x,y)与该像素在一帧内的扫描序号一一对应。由于读取前后帧的时间间隔固定，基于帧扫描的同步读取模式不容易产生时域抖动的问题，因而像素的一致性较好。近年来在仿生视觉传感器中也开始借鉴传统摄像头的同步读取模式来输出事件数据。
如前所述，在不同应用场景以及运动状态下，采用两种模式的输出和存储代价可能具有显著的差异。例如，拍摄静态场景时，在一段时间内仅有少量像素产生光强变化事件。通过示例的方式，例如，在一次扫描中，像素阵列电路中只有三个像素处产生光强变化事件。在异步读取模式下只需读取这三个像素的坐标信息(x，y)、时间信息t以及光强变 化量，就可以表征着三个光强变化事件。假定在异步读取模式中，针对一个像素的坐标、读取时间戳和光强变化量分别分配4、2、2、2个比特，则该读取模式下所需读取的总数据量为30比特。而在同步读取模式下，尽管仅有三个像素产生指示光强变化事件的有效数据信号，但是仍需读取整个阵列的所有像素输出的数据信号，以形成完整的一帧数据。假定在同步读取模式中，针对每个像素分配8比特，像素阵列电路的像素总数为66，则所需读取的总数据量为528比特。由此可见，即便像素阵列电路中存在大量未产生事件的像素，在同步读取模式中仍需分配如此之多的比特。从表示代价的角度来看是不经济的，并且增加了数据传输和存储方面的压力。因而，在这种情况下，采用异步读取模式更为经济。
在另一示例中，当场景内发生剧烈运动或者环境中的光强度发生剧烈变化时，诸如有大量人员走动、或突然开关灯等等，视觉传感器中有大量像素在短时间内测量到光强变化并产生指示光强变化事件的数据信号。由于在异步读取模式下表征单个事件的数据量多于在同步读取模式下表征单个事件的数据量，在这种情况下采用异步读取模式可能需要极大表示代价。具体而言，像素阵列电路中的每一行可能存在连续多个像素产生光强变化事件，对于每个事件均需传输坐标信息(x，y)、时间信息t以及光强变化信息a。这些事件之间的坐标变化往往只有一个单位的偏差，读取时间也基本相同。在这种情况下，异步读取模式对于坐标和时间信息的表示代价较大，会引起数据量激增。而在同步读取模式中，无论任一时刻在像素阵列电路中产生的光强变化事件的数目为多少，各个像素都输出仅指示光强变化量的数据信号，无需为每个像素的坐标信息和时间信息分配比特。因而，对于事件稠密的情形，采用同步读取模式更为经济。
鉴于此，本公开提出了一种用于自适应切换读取模式的方案。根据该方案，视觉传感器基于像素阵列电路产生的光强变化事件的统计结果，来比较两种读取模式下的数据量，从而可以切换至适合当前应用场景和运动状态的读取模式。以此方式，可以始终保持以具有较低成本的读取模式来从像素阵列电路读取数据，进而可以更好地满足图像传输和存储的需求。
以下结合图2至图10进一步描述本公开的示例实施例。图2示出了本公开的示例实施例可以在其中实施的示例视觉传感器200的框图。视觉传感器200可以被实现为视觉传感器芯片，并能够以至少基于帧扫描的读取模式和基于事件流的读取模式中的一种来读取指示光强变化事件的数据信号。如图2所示，视觉传感器200包括像素阵列电路210以及读取电路220。视觉传感器200与控制电路230相耦合。应当理解，图2所示的视觉传感器200仅用于示例性目的，而不暗示对于本公开的范围的任何限制。本公开的实施例还可以被体现在不同的传感器架构中。另外，还应当理解，视觉传感器200还可以包括用于实现图像采集、图像处理、图像传输等目的的其他元件或实体，为了便于描述未被示出，但不意味着本公开的实施例不具备这些元件或实体。
像素阵列电路210可以包括一个或多个像素阵列，并且每个像素阵列包括多个像素，每个像素具有用于唯一标识的位置信息，例如坐标(x,y)。像素阵列电路210可用于测量光强变化量，生成与多个像素对应的多个数据信号。在一些示例实施例中，每个像素被配置为独立地对环境中的光强变化做出响应。像素将所测量的光强变化量与预定阈值进行比较，如果测量到的光强变化量超过预定阈值，则该像素生成指示光强变化事件的第一数据信号，例如，+1或-1。在该示例中，第一数据信号可以指示对应像素处的光强变化趋势或 光强度值。如果测量到的光强变化量未超过预定阈值，则该像素生成与第一数据信号不同的第二数据信号，例如0。在本公开的实施例中，数据信号可以指示包括，但不限于光强极性、绝对光强度值、光强度的变化值等等。光强极性可以表示光强变化的趋势，例如，增强或减弱，通常用+1和-1表示。绝对光强度值可以表示在当前时刻测量的光强度值。取决于传感器的结构、用途和种类，关于光强度或光强变化量可以具有不同的物理意义。本公开的范围在此方面不受限制。
读取电路220被耦合至像素阵列电路210和控制电路230，并且可以与二者通信。读取电路220可以被配置为按照预定顺序扫描像素阵列电路210中的像素，以读取对应像素生成的数据信号。在本公开的实施例中，读取电路220被配置为能够以多于一种信号读取模式来读取像素阵列电路210输出的数据信号。例如，读取电路220可以以第一读取模式和第二读取模式中的一者进行读取。在本文的上下文中，第一读取模式和第二读取模式分别对应于基于帧扫描的读取模式和基于事件流的读取模式中的一者，进一步地，第一读取模式可以指代读取电路220的当前读取模式，第二读取模式可以指代可切换的备选读取模式。
现在参考图4，其示出了根据本公开的示例实施例的基于帧扫描的同步读取模式和基于事件流的异步读取模式的原理的示意图。如图4的上半部分所示，黑色圆点表示产生光强变化事件的像素，白色圆点表示未产生光强变化事件的像素。左侧虚线方框表示基于帧扫描的同步读取模式，其中所有像素基于所接收的光信号生成的电压信号，继而经过模数转换后输出数据信号。在该模式下，读取电路220通过读取所有像素生成的数据信号来构成一帧数据。右侧虚线方框表示基于事件流的异步读取模式，在该模式下，当读取电路220扫描到产生光强变化事件的像素时，可以获取该像素的坐标信息(x，y)。然后，仅读取产生光强变化事件的像素所生成的指示光强变化量a的数据信号(即，第一数据忆)，并记录读取时间t。在像素阵列电路中存在多个产生光强变化事件的像素的情况下，读取电路220按照扫描顺序，依次读取多个像素生成的数据信号，并构成事件流作为输出。
图4的下半部分从表示代价(例如，所需读取的数据量)的角度来描述两种读取模式。如图4所示，在同步读取模式下，读取电路220每次读取的数据量是相同的，例如1帧数据。在图4中示出为第1帧数据401-1和第2帧数据401-2。根据表示单个像素的数据量(例如，比特数目B p)和像素阵列电路中的像素总数M即可确定要读取的一帧数据量为M·B p。在异步读取模式下，读取电路220读取指示光强变化事件的数据信号，继而将所有的事件按发生的先后顺序构成一个事件流402。在此情况下，读取电路220每次读取的数据量与用于表示单个事件的事件数据量B ev(例如，表示产生事件的像素的坐标(x，y)、读取时间戳t以及光强变化量a的比特数目的总和)以及光强变化事件的数目N ev有关。
继续参考图2，在一些示例实施例中，读取电路220可以被配置向控制电路230提供所读取的至少一个数据信号。例如，读取电路220可以向控制电路230提供在一段时间内读取的数据信号，以供控制电路230进行历史数据统计和分析。在当前采用的第一读取模式为基于帧扫描的读取模式的情况下，读取电路220以该模式下的预定帧速率fHz扫描像素阵列电路210，以读取像素阵列电路210中的所有像素生成的数据信号。这些数据信号可以指示像素阵列电路210中的每个像素所测量的光强变化量a，并且可以包括第一数据信号和第二数据信号二者。通过示例的方式，在基于帧扫描的读取模式中，读取电路220所读取的每一帧数据量可以表示为M·B p个比特，每秒读取的数据量为M·B p·f比特/秒 (bits/s)，即，读取电路220的读取数据速率为M·B p·f bits/s，其中给定视觉传感器200中的像素总数为M，B p为与基于帧扫描的读取模式下针对每个像素分配的像素数据量(例如，比特数目)。
在另一些示例实施例中，在当前采用的第一读取模式为基于事件流的读取模式的情况下，读取电路220读取像素阵列电路210中产生光强变化事件的像素生成的数据信号，为了描述方便，以下将这些数据信号也称为第一数据信号。具体而言，读取电路220通过扫描像素阵列电路210，确定与光强变化事件有关的像素的位置信息(x,y)。基于像素的位置信息(x,y)，读取电路220读取多个数据信号中的、由像素生成的第一数据信号，以获取该第一数据信号指示的光强变化量a和读取时间信息t。通过示例的方式，在基于事件流的读取模式中，读取电路220每秒读取的事件数据量可以表示为B ev·N ev个比特，即读取电路220的读取数据速率为B ev·N ev bits/s，其中B ev为基于事件流的读取模式下针对每个光强变化事件所分配的事件数据量(例如，比特数目)，其中前b x和b y个比特用于表示像素坐标(x,y)，接下来的b t个比特用于表示数据信号被读取的时间戳t，最后b a个比特用于表示数据信号所指示的光强变化量a，即B ev＝b x+b y+b t+b a，N ev为读取电路220基于一段时间内对像素阵列电路210中产生的光强变化事件的数目历史统计而得出的每秒产生的平均事件数目。
控制电路230耦合至读取电路220，并且被配置为控制读取电路220以特定读取模式来读取像素阵列电路210生成的数据信号。在一些示例实施例中，控制电路230可以从读取电路220获取至少一个数据信号，并至少基于该至少一个数据信号来判断当前读取模式和备选的读取模式中的哪一种更适合当前的应用场景和运动状态。进而，在一些实施例中，控制电路230可以基于该判断来指示读取电路220从当前数据读取模式切换到另一数据读取模式。
在一些示例实施例中，控制电路230可以基于对光强变化事件的历史统计来向读取电路220发送关于切换读取模式的指示。例如，控制电路230可以基于从读取电路220接收的至少一个数据信号，确定与至少一个光强变化事件有关的统计数据。如果统计数据被确定为满足预定切换条件，则控制电路230向读取电路220发送模式切换信号，以使读取电路220切换到第二读取模式。为了便于比较，统计数据可以分别用于衡量第一读取模式和第二读取模式的读取数据速率。
在一些实施例中，统计数据可以包括单位时间内像素阵列电路210测量到的事件数目的总数据量。如果读取电路220在第一读取模式下读取的光强变化事件的总数据量已经大于或等于第二读取模式的光强变化事件的总数据量，则表明读取电路220应当从第一读取模式切换至第二读取模式。在一些实施例中，给定第一读取模式为基于帧扫描的读取模式并且第二读取模式为基于事件流的读取模式。控制电路230可以基于像素阵列电路的像素数目M、帧速率f和像素数据量B p来确定与第一读取模式下读取的光强变化事件的总数据量M·B p·f。控制电路230可以基于光强变化事件的数目N ev和与基于事件流的读取模式相关联的事件数据量B ev，来确定光强变化事件的总数据量B ev·N ev，也即，在第二读取模式下读取的光强变化事件的总数据量B ev·N ev。在一些实施例中，可以利用切换参数来调节两个读取模式下的总数据量之间的关系，如以下公式(1)所示，第一读取模式下读取的光强变化事件的总数据量M·B p·f大于或等于第二读取模式的光强变化事件的总数据量B ev·N ev，读取电路220应当切换到第二读取模式：
η·M·B P·f≥B ev·N ev  (1)
其中η为用于调节的切换参数。由以上公式(1)可以进一步得出，第一阈值数据量d 1＝M·B p·f·η。即，如果光强变化事件的总数据量B ev·N ev小于或等于阈值数据量d 1，则表明第一读取模式下读取的光强变化事件的总数据量已经大于或等于第二读取模式的光强变化事件的总数据量，控制电路230可以确定光强变化事件的统计数据满足预定切换条件。在该实施例中，可以至少基于像素阵列电路的像素数目M、与基于帧扫描的读取模式相关联的帧速率f和像素数据量B p来确定阈值数据量d 1。
作为上述实施例的备选实施方式中，第一读取模式下读取的光强变化事件的总数据量M·B p·f大于或等于第二读取模式的光强变化事件的总数据量B ev·N ev可以如以下公式(2)所示：
M·B P·f-B ev·N ev≥θ  (2)
其中θ为用于调节的切换参数。由以上公式(2)可以进一步得出，第二阈值数据量
d 2＝M·B p·f-θ
即，如果光强变化事件的总数据量B ev·N ev小于或等于第二阈值数据量d 2，则表明第一读取模式下读取的光强变化事件的总数据量已经大于或等于第二读取模式的光强变化事件的总数据量，控制电路230可以确定光强变化事件的统计数据满足预定切换条件。在该实施例中，可以至少基于像素阵列电路的像素数目M、与基于帧扫描的读取模式相关联的帧速率f和像素数据量B p来确定阈值数据量d 2。
在一些实施例中，第一读取模式为基于事件流的读取模式并且第二读取模式为基于帧扫描的读取模式。由于在基于事件流的读取模式，读取电路220仅读取产生事件的像素生成的数据信号，即，第一数据信号。因而，控制电路的230基于读取电路220所提供的第一数据信号的数目可以直接确定像素阵列电路210中产生的光强变化事件的数目N ev。控制电路230可以基于事件数目N ev和与基于事件流的读取模式相关联的事件数据量B ev，确定光强变化事件的总数据量，即第一读取模式下读取的事件的总数据量B ev·N ev。类似地，控制电路230还可以基于像素阵列电路的像素数目M、帧速率f和像素数据量B p来确定与第二读取模式下读取的光强变化事件的总数据量M·B p·f。如以下公式(3)所示，第一读取模式下读取的光强变化事件的总数据量B ev·N ev大于或等于第二读取模式的光强变化事件的总数据量M·B p·f，读取电路220应当切换到第二读取模式：
B ev·N ev≥η·M·B P·f  (3)
其中η为用于调节的切换参数。由以上公式(3)可以进一步得出，第一阈值数据量d 1＝η·M·B P·f。如果光强变化事件的总数据量B ev·N ev大于或等于阈值数据量d 1，则控制电路230确定光强变化事件的统计数据满足预定切换条件。在该实施例中，可以至少基于像素阵列电路的像素数目M、帧速率f和像素数据量B p来确定阈值数据量d 1。
作为上述实施例的备选实施方式中，第一读取模式下读取的光强变化事件的总数据量B ev·N ev大于或等于第二读取模式的光强变化事件的总数据量M·B p·f可以如以下公式(4)所示
M·B P·f-B ev·N ev≤θ  (4)
其中θ为用于调节的切换参数。由以上公式(4)可以进一步得出，第二阈值数据量d 2＝M·B P·f-θ，如果光强变化事件的总数据量B ev·N ev大于或等于阈值数据量d 2，则 控制电路230确定光强变化事件的统计数据满足预定切换条件。在该实施例中，可以至少基于像素阵列电路的像素数目M、帧速率f和像素数据量B p来确定阈值数据量d 2。
在另一些实施例中，统计数据可以包括单位时间内像素阵列电路210测量到的事件数目N ev。如果第一读取模式为基于帧扫描的读取模式并且第二读取模式为基于事件流的读取模式，控制电路230基于读取电路220提供的多个数据信号中的第一数据信号的数目，确定光强变化事件的数目N ev。如果统计数据指示光强变化事件的数目N ev小于第一阈值数目n 1，则控制电路230确定光强变化事件的统计数据满足预定切换条件可以至少基于像素阵列电路的像素数目M、与基于帧扫描的读取模式相关联的帧速率f和像素数据量B p、以及与基于事件流的读取模式相关联的事件数据量B ev来确定第一阈值数目n 1。例如，在前述实施例中，基于公式(1)可以进一步得到如下公式(5)：
Figure PCTCN2020112982-appb-000001
即，第一阈值数目n 1可以被确定为
Figure PCTCN2020112982-appb-000002
作为上述实施例的备选实施方式中，基于公式(2)可以进一步得到如下公式(6)：
Figure PCTCN2020112982-appb-000003
相应地，第二阈值数目n 2可以被确定为
Figure PCTCN2020112982-appb-000004
在又一些实施例中，如果第一读取模式为基于事件流的读取模式并且第二读取模式为基于帧扫描的读取模式，控制电路230基于读取电路220提供的至少一个数据信号的数目，可以直接确定光强变化事件的数目N ev。如果统计数据指示光强变化事件的数目N ev大于或等于第一阈值数目n 1，则控制电路230确定光强变化事件的统计数据满足预定切换条件。可以至少基于像素阵列电路210的像素数目M、与基于帧扫描的读取模式相关联的帧速率f和像素数据量B p、以及与基于事件流的读取模式相关联的事件数据量B ev来确定第一阈值数目n 1＝M·B p·f/(η·B ev)。例如，在前述实施例中，基于公式(3)可以进一步得到如下公式(7)：
Figure PCTCN2020112982-appb-000005
即，第一阈值数目n 1可以被确定为
Figure PCTCN2020112982-appb-000006
作为上述实施例的备选实施方式中，基于公式(4)可以进一步得到如下公式(8)：
Figure PCTCN2020112982-appb-000007
相应地，第二阈值数目n 2可以被确定为
Figure PCTCN2020112982-appb-000008
应当理解的是，以上给出的公式、切换条件及相关的计算方法仅仅是本公开的实施例的一种示例实现，其他合适的模式切换条件、切换策略及计算方法也可以被采用，本公开的范围在此方面不受限制。
根据本公开的示例实施例，控制电路230可以如图2所示的作为视觉传感器200外部 的独立电路或芯片，通过接口连接到视觉传感器200。在另一些示例实施例中，控制电路230也可以作为视觉传感器内部的电路或芯片，与其中的像素阵列电路和读取电路集成在一起。图3示出了根据本公开的示例实施例的另一视觉传感器300的框图。视觉传感器300可以作为视觉传感器200的一种示例实现。
如图3所示，视觉传感器300包括像素阵列电路310、读取电路320和控制电路330。像素阵列电路310、读取电路320和控制电路330在功能上与图2所示的像素阵列电路210、读取电路220以及控制电路230相同，因而在此不再进行赘述。应当理解的是，视觉传感器300仅用于示例性目的，而不暗示对于本公开的范围的任何限制。本公开的实施例还可以被体现在不同的视觉传感器中。另外，还应当理解，视觉传感器300还可以包括其他元件、模块或实体，出于清楚的目的未被示出，但不意味着本公开的实施例不具备这些元件或实体。
图5A示出了根据本公开的实施例的视觉传感器在基于帧扫描的读取模式下操作的示意图。图5B示出了根据本公开的实施例的视觉传感器在基于事件流的读取模式下操作的示意图。如图5A所示，读取电路220或320当前以第一读取模式，即基于帧扫描的读取模式操作。由于控制电路230或330基于历史统计，确定当前像素阵列电路210或310中产生的事件数目较少，例如一帧数据中仅有四个有效数据，继而预测下一时间段内可能的事件产生速率较低。如果读取电路220或320继续采用基于帧扫描的读取模式进行读取，将需要重复地为生成事件的像素分配比特，从而产生大量冗余数据。在这种情况下，控制电路230或330向读取电路220或320发送模式切换信号，以使读取电路220或320从第一读取模式切换为第二读取模式。在切换后，图5B所示，读取电路220或320在第二读取模式下操作，仅读取有效的数据信号，从而避免了大量无效数据信号所占用的传输带宽和存储资源。
图6A示出了根据本公开的实施例的视觉传感器在基于事件流的读取模式下操作的示意图。图6B示出了根据本公开的实施例的视觉传感器在基于帧扫描的读取模式下操作的示意图。如图6A所示，读取电路220或320当前以第一读取模式，即基于事件流的读取模式操作。由于控制电路230或330基于历史统计，确定当前像素阵列电路210或310中产生的事件数目较多，例如，短时间内像素阵列电路210或310中的近乎所有像素均生成指示光强变化量高于阈值的数据信号。继而，读取电路220或320可以预测下一时间段内可能的事件产生速率较高。由于所读取的数据信号中存在大量冗余数据，例如，近乎相同的像素位置信息、读取时间戳等等，如果读取电路220或320继续采用基于事件流的读取模式进行读取，将造成读取数据量激增。因而，在这种情况下，控制电路230或330向读取电路220或320发送模式切换信号，以使读取电路220或320从第一读取模式切换为第二读取模式。在切换后，图6B所示，读取电路220或320在基于帧扫描的模式下操作，以单个像素的表示代价更小的读取模式来读取数据信号，缓解了存储和传输数据信号的压力。
在一些示例实施例中，视觉传感器200或300还可以包括解析电路，其可以被配置为针对读取电路220或320输出的数据信号进行解析。在一些示例实施例中，解析电路可以采用与读取电路220或320的当前数据读取模式相适应的解析模式来解析数据信号。这将在下面进行详细描述。
应当理解的是，其他现有的或未来待开发的数据读取模式、数据读取模式、数据解析 模式等也适用于本公开的示例实施例，并且本公开的实施例中的所有数值均是说明性而非限制性的，例如，本公开的示例实施例可以在多于两种数据读取模式之间进行切换。
根据本公开的示例实施例，提供了一种视觉传感器芯片，能够根据对像素阵列电路中产生的光强变化事件的历史统计情况，自适应地在多种读取模式之间进行切换。这样，无论在动态场景还是静态场景中，视觉传感器芯片始终可以实现良好的读取和解析性能，避免冗余数据的产生，缓解了图像处理、传输和存储的压力。
图7示出了根据本公开的示例实施例的用于操作视觉传感器芯片的方法700的流程图。在一些示例实施例中，方法700可以在图2所示的视觉传感器200或者图3所示的视觉传感器300以及下文的图9所示的电子设备900中实现，或者也可以使用任意适当的设备来实现，包括目前已知的或将来待开发的各种设备。为了方便讨论，下面将结合图2所示的视觉传感器200来描述方法700。
如图7所示，在框701，像素阵列电路210通过测量光强变化量来生成与像素阵列电路210中的多个像素对应的多个数据信号。多个数据信号可以指示在对应像素处是否产生光强变化事件。在本文的上下文中，光强变化事件表示像素阵列电路210中的对应像素测量到的光强变化量超过预定阈值。在一些实施例中，如果像素阵列电路210中的像素测量到的光强变化量超过预定阈值，则生成指示光强变化事件的第一数据信号，例如+1或-1。如果像素阵列电路210中的像素测量到的光强变化量未超过预定阈值，则生成与第一数据信号不同的第二数据信号，例如，0。
在框702，读取电路220以第一读取模式从像素阵列电路210读取多个数据信号中的至少一个数据信号。数据信号可以指示光强变化量a，并且这些数据信号经读取之后在视觉传感器200内占用一定的存储和传输资源。取决于特定的读取模式，视觉传感器芯片200读取数据信号的方式可以不同。在一些示例实施例中，例如，在基于事件流的读取模式中，读取电路220通过扫描像素阵列电路210，确定与光强变化事件有关的像素的位置信息(x,y)。基于该位置信息，读取电路220可以读出多个数据信号中的第一数据信号。在该实施例中，读取电路220通过读取数据信号，获取光强变化量a、产生光强变化事件的像素的位置信息(x,y)、读取数据信号的时间戳t等。
在另一些示例实施例中，第一读取模式可以是基于帧扫描的读取模式。在该模式下，视觉传感器200以与基于帧扫描的读取模式相关联的帧频率扫描像素阵列电路210，来读取像素阵列电路210产生的所有数据信号。在该实施例中，读取电路220通过读取数据信号，获取光强变化量a。
在框703，读取电路220向控制电路230提供所读取的至少一个数据信号，以供控制电路230进行数据统计和分析。在一些实施例中，控制电路230可以基于该至少一个数据信号，确定与至少一个光强变化事件有关的统计数据。控制电路230可以利用切换策略模块来分析统计数据。如果确定统计数据满足预定切换条件，则控制电路230向读取电路220发送模式切换信号。
在第一读取模式为基于帧扫描的读取模式并且第二读取模式为基于事件流的读取模式的情况下，在一些实施例中，控制电路230可以基于多个数据信号中的第一数据信号的数目，确定至少一个光强变化事件的数目。进而，控制电路230将光强变化事件的数目与第一阈值数目比较。如果统计数据指示光强变化事件的数目小于或等于第一阈值数目，则控制电路230确定光强变化事件的统计数据满足预定切换条件，并发送模式切换信号。在 该实施例中，控制电路230可以基于像素阵列电路的像素数目、与基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与基于事件流的读取模式相关联的事件数据量来确定或调整第一阈值数目。
在第一读取模式为基于帧扫描的读取模式并且第二读取模式为基于事件流的读取模式的情况下，在另一些实施例中，控制电路230可以进一步基于光强变化事件的数目和与基于事件流的读取模式相关联的事件数据量，确定光强变化事件的总数据量。进而，控制电路230将光强变化事件的总数据量与第一阈值数据量进行比较。如果光强变化事件的总数据量未超过第一阈值数据量，则控制电路230确定光强变化事件的统计数据满足预定切换条件，并发送模式切换信号。在该实施例中，控制电路230可以基于像素阵列电路的像素数目、帧速率和像素数据量来确定或调整第一阈值数据量。
在第一读取模式为基于事件流的读取模式并且第二读取模式为基于帧扫描的读取模式的情况下，在一些实施例中，控制电路230可以基于从读取电路220接收的第一数据信号，确定与光强变化事件有关的统计数据。进而，控制电路230将光强变化事件的数目与第二阈值数目比较。如果光强变化事件的数目大于或等于第二阈值数目，则控制电路230确定光强变化事件的统计数据满足预定切换条件，并发送模式切换信号。在该实施例中，控制电路230可以基于像素阵列电路的像素数目、与基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与基于事件流的读取模式相关联的事件数据量来确定或调整第二阈值数目。
与此不同，在第一读取模式为基于事件流的读取模式并且第二读取模式为基于帧扫描的读取模式的情况下，控制电路230可以进一步根据光强变化事件的数目和与基于事件流的读取模式相关联的事件数据量，确定光强变化事件的总数据量。进而，控制电路230将光强变化事件的总数据量与第二阈值数据量进行比较。如果光强变化事件的总数据量未超过第二阈值数据量，则控制电路230确定光强变化事件的统计数据满足预定切换条件，并发送模式切换信号。在该实施例中，控制电路230可以基于像素阵列电路的像素数目、帧速率和像素数据量来确定或调整第二阈值数据量。应当理解当切换策略配置有多种切换条件时，控制电路230可以根据需要，例如根据视觉传感器200的类型、光强变化事件的特性、外部环境的属性、运动状态等因素，从多个切换条件中选择一个用于分析和决策。在本公开的示例实施例中，其他合适的切换策略模块及切换条件或策略也可以被采用，本公开的范围在此方面不受限制。
在框704，读取电路220基于从控制电路220接收的模式切换信号，将第一读取模式切换为第二读取模式。进而，读取电路220以第二读取模式来读取像素阵列电路210生成的至少一个数据信号。控制电路230随后可以继续对像素阵列电路210产生的光强变化事件进行历史统计，当满足切换条件时，发送模式切换信号以使读取电路220从第二读取模式切换为第一读取模式。
根据本公开的示例实施例提供的方法，控制电路在整个读取和解析的过程中持续地对像素阵列电路中产生的光强变化事件进行历史统计和实时分析，一旦满足切换条件就发送模式切换信号，以使读取电路从当前读取模式切换为更为合适的备选切换模式。该自适应地切换过程不断重复，直至对所有数据信号的读取完成。
图8示出了根本公开的示例实施例的控制电路800的框图。控制电路800可以用于实现图2中的控制电路230、图3中的控制电路330等，也可以利用其他合适的设备来实现。 应当理解的是，控制电路800仅用于示例性目的，而不暗示对于本公开的范围的任何限制。本公开的实施例还可以被体现在不同的控制电路中。另外，还应当理解，控制电路800还可以包括其他元件、模块或实体，出于清楚的目的未被示出，但不意味着本公开的实施例不具备这些元件或实体。
如图8所示，控制电路800包括至少一个处理器802、耦合到处理器802的至少一个存储器804以及耦合到处理器802的通信机制812。存储器804至少用于存储计算机程序以及从读取电路获取的数据信号。在处理器802上预配置有统计模型806和切换决策模块808。控制电路830可以通过通信机制812被通信耦合到如图2所示的视觉传感器200的读取电路220或者视觉传感器外部的读取电路，以对其实现控制功能。为了便于描述，下面参考图2中的读取电路220，但是本公开的实施例同样适用于外设读取电路的配置。
与图2所示的控制电路230类似，在一些示例实施例中，控制电路800可以被配置为控制读取电路220以特定数据读取模式(例如，基于帧扫描的同步读取模式、基于事件流的异步读取模式等)来读取像素阵列电路210生成的多个数据信号。另外，控制电路800可以被配置为从读取电路220获取数据信号，数据信号可以指示，但不限于光强极性、绝对光强度值、光强度的变化值等等。例如，光强极性可以表示光强变化的趋势，如增强或减弱，通常用+1/-1表示。绝对光强度值可以表示在当前时刻测量的光强度值。取决于传感器的结构、用途和种类，关于光强度或光强变化的信息可以具有不同的物理意义。
控制电路800基于从读取电路220获取数据信号，确定与至少一个光强变化事件有关的统计数据。在一些实施例中，控制电路800可以从读取电路220获取像素阵列电路210在一段时间内生成的数据信号，并将这些数据信号存储在存储器804中，以供历史统计和分析。在本公开的上下文中，第一读取模式和第二读取模式可以分别是基于事件流的异步读取模式和基于帧扫描的同步读取模式中的一者。但是应当注意，本文关于自适应地切换读取模式所描述的所有特征同样适用于其他目前已知的或将来开发的其他类型的传感器以及数据读取模式，以及多于两种数据读取模式之间的切换。
在一些示例实施例中，控制电路800可以利用预配置的一个或多个统计模型806，对由读取电路220提供的像素阵列电路210在一段时间内产生光强变化事件进行历史统计。统计模型806然后可以向切换决策模块808传输统计数据以作为输出。如前文所述，统计数据可以指示光强变化事件的数目，也可以指示光强变化事件的总数据量。应当理解，任何合适的统计模型、统计算法均可应用于本公开的示例实施例，本公开的范围在此方面不受限制。
由于统计数据是对视觉传感器在一段时间内产生光强变化事件的历史情况的统计结果，因而可供决策模块808分析和预测下一时间段内事件发生的速率切换决策模块808可以预配置有一个或多个切换决策。当存在多个切换决策时，控制电路800可以根据需要，例如根据视觉传感器200的类型、光强变化事件的特性、外部环境的属性、运动状态等因素，从多个切换决策中选择一个用于分析和决策。在本公开的示例实施例中，其他合适的切换决策模块及模式切换条件或策略也可以被采用，本公开的范围在此方面不受限制。
在一些实施例中，如果切换决策模块808确定统计数据满足模式切换条件，则向读取电路220输出关于切换读取模式的指示。在另一实施例中，如果切换决策模块808确定统计数据不满足模式切换条件，则不向读取电路220输出关于切换读取模式的指示。在一些实施例中，关于切换读取模式的指示可以如以上实施例所述的采用显式形式，例如以切换 信号或标记位的形式通知读取电路220切换读取模式。
图9示出了根据本公开的示例实施例的电子设备900的框图。如图9所示，电子设备900包括视觉传感器芯片901、通信接口902和903、控制电路930、以及解析电路904。应当理解的是，电子设备900用于示例性目的，其可以利用任何合适的设备来实现，包括当前已知的和未来开发的各种传感器设备。本公开的实施例还可以被体现在不同的传感器系统中。另外，还应当理解，电子设备900还可以包括其他元件、模块或实体，出于清楚的目的未被示出，但不意味着本公开的实施例不具备这些元件、模块或实体。
如图9所示，视觉传感器901包括像素阵列电路910以及读取电路920，其中读取电路920的读取组件920-1和920-2分别经由通信接口902和903耦合至控制电路930。在本公开的实施例中，读取组件920-1和920-2可以分别利用独立的设备实现，也可以被集成在同一设备中。例如，图2所示的读取电路220是一种集成的示例实现。为了便于描述，读取组件920-1和920-2可以分别被配置为以基于帧扫描的读取模式和基于事件流的读取模式实现数据读取功能。
像素阵列电路910可以利用图2中的像素阵列电路210或图3中的像素阵列电路310来实现，也可以利用任何合适的其他设备来实现，本公开在这方面不受限制。关于像素阵列电路910的特征在此不再进行赘述。
读取电路920可以以特定的读取模式读取像素阵列电路910生成的数据信号。例如，在接通读取组件920-1并断开读取组件920-2的示例中，读取电路920初始采用基于帧扫描的读取模式读取数据信号。在接通读取组件920-2并断开读取组件920-1的示例中，读取电路920初始采用基于事件流的读取模式读取数据信号。读取电路920以利用图2中的读取电路220或图3中的读取电路320来实现，也可以利用任何合适的其他设备来实现，关于读取电路920的特征在此不再进行赘述。
在本公开的实施例中，控制电路930可以通过指示信号或标记位的方式指示读取电路920从第一读取模式切换到第二读取模式。在这种情况下，读取电路920可以从控制电路930接收关于切换读取模式的指示，例如，接通读取组件920-1并断开读取组件920-2，或接通读取组件920-2并断开读取组件920-1。
如前文所述，电子设备900还可以包括解析电路904。解析电路904可以被配置为解析由读取电路920读取的数据信号。在本公开的示例实施例中，解析电路可以采用与读取电路920的当前数据读取模式相适应的解析模式。作为示例，如果读取电路920初始以基于事件流的读取模式读取数据信号，则解析电路相应地基于与该读取模式相关联的第一数据量B ev·N ev来解析数据。当读取电路920基于控制电路930的指示从基于事件流的读取模式切换为基于帧扫描的读取模式时，解析电路开始按照第二数据量，即一帧数据大小M·B p来解析数据信号，反之亦然。
在一些实施例中，解析电路904可以在无需显式切换信号或标记位的情况下实现解析电路的解析模式的切换。例如，解析电路904可以采用与控制电路930相同或相应的统计模型和切换策略，以对读取电路920提供的数据信号做出与控制电路930相同的统计分析并做出一致的切换预测。作为示例，如果读取电路920初始以基于事件流的读取模式读取数据信号，相应地，解析电路初始基于与该读取模式相关联的第一数据量B ev·N ev来解析数据。例如，解析电路解析出的前b x个比特指示像素的坐标x，接下来b y个比特指示像素的坐标y，其后的b t个比特指示读取时间，最后取b a个比特指示光强变化量。解析电路从读 取电路920获取至少一个数据信号，并确定与至少一个光强变化事件有关的统计数据。如果解析电路904确定该统计数据满足切换条件，则切换至与基于帧扫描的读取模式相对应的解析模式，以帧数据大小M·B p来解析数据信号。
作为另一示例，如果读取电路920初始以基于帧扫描的读取模式读取数据信号，则解析电路904以与该读取模式相对应的解析模式，按照每B p比特依次取出帧内的每个像素位置的值，其中未产生光强变化事件的像素位置的值为0。解析电路904基于数据信号可以统计出一帧内的非0个数，即该帧内的光强变化事件的数目。
在一些示例实施例中，解析电路904从读取电路920获取至少一个数据信号，并至少基于该至少一个数据信号来判断当前解析模式和备选的解析模式中的哪一种与读取电路920的读取模式相对应。进而，在一些实施例中，解析电路904可以基于该判断来从当前解析模式切换到另一解析模式。
在一些示例实施例中，解析电路904可以基于对光强变化事件的历史统计来确定是否切换解析模式。例如，解析电路904可以基于从读取电路920接收的至少一个数据信号，确定与至少一个光强变化事件有关的统计数据。如果统计数据被确定为满足切换条件，则解析电路904从当前解析模式切换到备选的解析模式。为了便于比较，统计数据可以分别用于衡量读取电路920的第一读取模式和第二读取模式的读取数据速率。
在一些实施例中，统计数据可以包括单位时间内像素阵列电路910测量到的事件数目的总数据量。如果解析电路904基于至少一个数据信号确定读取电路920在第一读取模式下读取的光强变化事件的总数据量已经大于或等于其第二读取模式的光强变化事件的总数据量，则表明读取电路220已从第一读取模式切换至第二读取模式。这种情况下，解析电路904应当相应地切换至与当前读取模式相对应的解析模式。
在一些实施例中，给定第一读取模式为基于帧扫描的读取模式并且第二读取模式为基于事件流的读取模式。在该实施例中，解析电路904初始以与第一读取模式相对应的、基于帧的解析模式来解析从读取电路920获取的数据信号。解析电路904可以基于像素阵列电路910的像素数目M、帧速率f和像素数据量B p来确定读取电路920在第一读取模式下读取的光强变化事件的总数据量M·B p·f。解析电路904可以基于光强变化事件的数目N ev和与基于事件流的读取模式相关联的事件数据量B ev，来确定读取电路920在第二读取模式下读取的光强变化事件的总数据量B ev·N ev。在一些实施例中，可以利用切换参数来调节两个读取模式下的总数据量之间的关系。进而，解析电路904可以根据，例如，上文中的公式(1)来确定读取电路920在第一读取模式下读取的光强变化事件的总数据量M·B p·f是否大于或等于第二读取模式的光强变化事件的总数据量B ev·N ev。如果是，则解析电路904确定读取电路920已切换至基于事件流的读取模式，并相应地从基于帧的解析模式切换到基于事件流的解析模式。
作为上述实施例的备选实施方式中，解析电路904可以根据上文中的公式(2)来确定读取电路920在第一读取模式下读取的光强变化事件的总数据量M·B p·f是否大于或等于其在第二读取模式下读取的光强变化事件的总数据量B ev·N ev。类似地，在确定读取电路920在第一读取模式下读取的光强变化事件的总数据量M·B p·f大于或等于第二读取模式的光强变化事件的总数据量B ev·N ev的情况下，解析电路904确定读取电路920已切换至基于事 件流的读取模式，并相应地从基于帧的解析模式切换到基于事件流的解析模式。
在一些实施例中，第一读取模式为基于事件流的读取模式并且第二读取模式为基于帧扫描的读取模式。在该实施例中，解析电路904初始以与第一读取模式相对应的、基于事件流的解析模式来解析从读取电路920获取的数据信号。如前所述，解析电路904基于读取电路220所提供的第一数据信号的数目可以直接确定像素阵列电路910中产生的光强变化事件的数目N ev。解析电路904可以基于事件数目N ev和与基于事件流的读取模式相关联的事件数据量B ev，确定读取电路920在第一读取模式下读取的事件的总数据量B ev·N ev。类似地，解析电路904还可以基于像素阵列电路的像素数目M、帧速率f和像素数据量B p来确定读取电路920在第二读取模式下读取的光强变化事件的总数据量M·B p·f。然后，解析电路904可以，例如，根据上文中的公式(3)来确定第一读取模式下读取的光强变化事件的总数据量B ev·N ev是否大于或等于第二读取模式的光强变化事件的总数据量M·B p·f。类似地，当解析电路904确定读取电路920在第一读取模式下读取的光强变化事件的总数据量B ev·N ev大于或等于第二读取模式的光强变化事件的总数据量M·B p·f时，解析电路904确定读取电路920已切换至基于帧扫描的读取模式，并相应地从基于事件流的解析模式切换到基于帧的解析模式。
作为上述实施例的备选实施方式中，解析电路904可以根据上文中的公式(4)来确定读取电路920在第一读取模式下读取的光强变化事件的总数据量B ev·N ev是否大于或等于其在第二读取模式下读取的光强变化事件的总数据量M·B p·f。类似地，在确定读取电路920在第一读取模式下读取的光强变化事件的总数据量B ev·N ev大于或等于第二读取模式的光强变化事件的总数据量M·B p·f的情况下，解析电路904确定读取电路920已切换至基于帧扫描的读取模式，并相应地从基于事件流的解析模式切换到基于帧扫描的解析模式。
对于基于帧扫描的读取模式下事件的读取时间t，默认同一帧内的所有事件均具有相同的读取时间t。在对事件读取时间的精度要求较高的情况下，可以进一步通过如下方式来确定每个事件的读取时间。以上述实施例为例，在基于帧扫描的读取模式下，读取电路920扫描像素阵列电路的频率为f Hz，则读取相邻两帧数据的时间间隔为S＝1/f，并且给定每一帧的起始时间为：
T k＝T 0+kS  (9)
其中T 0为第一帧的起始时间，k为帧序号，则针对M个像素中的一个像素进行数模转换所需的时间可由如下公式(10)来确定：
Figure PCTCN2020112982-appb-000009
第k帧中的第i个像素处产生光强变化事件的时间可由如下公式(11)来确定：
Figure PCTCN2020112982-appb-000010
其中i为正整数。如当前读取模式为同步读取模式，则切换为异步读取模式，按照每个事件B ev比特解析数据。在上述实施例中，解析模式的切换可以在没有显式切换信号 或标记位的情况下实现。对于其他目前已知或未来开发的数据读取模式，解析电路也可采用类似的与数据读取模式相适应的方式解析数据，在此不再赘述。
根据本公开的示例实施例，提出了一种具有多种数据读取模式的视觉传感器。视觉传感器能够自适应地在多种数据读取模式之间切换，使得读取数据速率始终保持未超过预定的读取数据速率阈值，从而降低了视觉传感器的数据传输、解析和存储的代价，显著提高了传感器的性能。另外，这样的视觉传感器可以对一段时间内产生的事件进行数据统计以用于预测下一时间段内可能的事件产生速率，因而能够选出更加适合当前外部环境、应用场景和运动状态的读取模式。
图10示出了单一数据读取模式与根据本公开的示例实施例的自适应切换读取模式的数据量随时间变化的示意图。图10的左半部分描绘了单纯采用同步读取模式或异步读模式的传统的视觉传感器或传感器系统的读取数据量随时间变化的示意图。在单纯采用同步读取模式的情形中，如曲线1001所示，由于每帧具有固定的数据量，读取的数据量随时间保持不变，即，读取数据速率(单位时间内读取的数据量)稳定。如前所述，当像素阵列电路中产生大量事件时，采用基于帧扫描的读取模式来读取数据信号较为合理，帧数据中绝大部分为表示产生事件的有效数据，存在的冗余较少。而当像素阵列电路中产生的事件较少时，一帧中存在大量表示并产生事件的无效数据，此时仍以帧数据结构表示和读取像素处的光强度信息将产生冗余，浪费传输带宽和存储资源。
在单纯采用异步读取模式的情形中，如曲线1002所示，读取的数据量随事件产生的速率而发生变化，因而读取数据速率不是固定的。当像素阵列电路中产生的事件较少时，仅需要为少量事件分配用于表示像素的坐标信息(x,y)、数据信号被读取的时间戳t和光强变化量a的比特，所需读取的总数据量较小，在这种情况下采用异步读取模式较为合理。当短时间内在像素阵列电路中产生大量事件时，需要分配大量用于表示这些事件的比特。然而，这些像素坐标几乎相邻，数据信号的读取时间也几乎相同。换言之，读取的事件数据中存在大量重复的数据，因而在异步读取模式中同样存在冗余的问题，在这种情况下的读取数据速率甚至超过同步读取模式下的读取数据速率，如果仍旧采用异步读取模式是不合理的。
图10的右半部分描绘了根据本公开的示例实施例的自适应数据读取模式下的数据量随时间变化的示意图。自适应数据读取模式可以利用图2所示的视觉传感器200、图3所示的视觉传感器300或者图9所示的电子设备900来实现，或者传统的视觉传感器或传感器系统可以通过利用图8所示的控制电路800来实现自适应的数据读取模式。为了便于描述，下面参照图2所示的视觉传感器200来描述关于自适应数据读取模式的特征。如曲线1003所示，视觉传感器200在初始化状态下选择例如异步读取模式。由于该模式下用于表示每个事件的比特数目B ev是预定的(例如，B ev＝b x+b y+b t+b a)，随着事件的产生和读取，视觉传感器200可以统计出当前模式下的读取数据速率。另一方面，同步读取模式中用于表示每一帧的每个像素的比特数目B p也是预定的，因此可以计算出在该段时间内使用同步读取模式的读取数据速率。视觉传感器200然后可以确定两种读取模式下的数据速率之间的关系是否满足模式切换条件。例如，视觉传感器200可以基于预定义的阈值比较两种读取模式中哪种模式的读取数据速率更小。一旦确定满足模式切换条件，则视觉传感器200切换为另一种读取模式，例如从初始的异步读取模式切换为同步读取模式。上 述步骤在数据信号的读取和解析的过程中持续进行，直至完成所有数据的输出。如曲线1003所示，视觉传感器200在整个数据读取过程中自适应地选择最优的读取模式，两种读取模式交替出现，使得视觉传感器200的读取数据速率始终保持未超过同步读取模式的读取数据速率，从而降低了视觉传感器的数据传输、解析和存储的代价。
另外，根据本公开的实施例所提出的自适应数据读取方式，视觉传感器200可以对事件进行历史数据统计以预测下一时间段内可能的事件产生速率，因此能够选出更加适合应用场景和运动状态的读取模式。
一般而言，本公开的各种示例实施例可以在硬件或专用电路、软件、逻辑，或其任何组合中实施。某些方面可以在硬件中实施，而其他方面可以在可以由控制电路、微处理器或其他计算设备执行的固件或软件中实施。当本公开的实施例的各方面被图示或描述为框图、流程图或使用某些其他图形表示时，将理解此处描述的方框、装置、系统、技术或方法可以作为非限制性的示例在硬件、软件、固件、专用电路或逻辑、通用硬件或控制电路或其他计算设备，或其某些组合中实施。
而且，流程图中的各框可以被看作是方法步骤，和/或计算机程序代码的操作生成的操作，和/或理解为执行相关功能的多个耦合的逻辑电路元件。例如，本公开的实施例包括计算机程序产品，该计算机程序产品包括有形地实现在机器可读介质上的计算机程序，该计算机程序包含被配置为实现上文描述方法的程序代码。
在公开的上下文内，机器可读介质可以是包含或存储用于或有关于指令执行系统、装置或设备的程序的任何有形介质。机器可读介质可以是机器可读信号介质或机器可读存储介质。机器可读介质可以包括但不限于电子的、磁的、光学的、电磁的、红外的或半导体系统、装置或设备，或其任意合适的组合。机器可读存储介质的更详细示例包括带有一根或多根导线的电气连接、便携式计算机磁盘、硬盘、随机存储存取器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或闪存)、光存储设备、磁存储设备，或其任意合适的组合。
用于实现本公开的方法的计算机程序代码可以用一种或多种编程语言编写。这些计算机程序代码可以提供给通用计算机、专用计算机或其他可编程的数据处理装置的处理器，使得程序代码在被计算机或其他可编程的数据处理装置执行的时候，引起在流程图和/或框图中规定的功能/操作被实施。程序代码可以完全在计算机上、部分在计算机上、作为独立的软件包、部分在计算机上且部分在远程计算机上或完全在远程计算机或服务器上执行。
另外，尽管操作以特定顺序被描绘，但这并不应当理解为要求此类操作以示出的特定顺序或以相继顺序完成，或者执行所有图示的操作以获取期望结果。在某些情况下，多任务或并行处理会是有益的。同样地，尽管上述讨论包含了某些特定的实施细节，但这并不应解释为限制任何发明或权利要求的范围，而应解释为对可以针对特定发明的特定实施例的描述。本说明书中在分开的实施例的上下文中描述的某些特征也可以整合实施在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以分离地在多个实施例或在任意合适的子组合中实施。
针对前述本公开的示例实施例的各种修改、改变将在连同附图查看前述描述时对相关技术领域的技术人员变得明显。任何及所有修改将仍落入非限制的和本公开的示例实施例范围。此外，前述说明书和附图存在启发的益处，涉及本公开的这些实施例的技术领域的 技术人员将会想到此处阐明的本公开的其他实施例。
将会理解，本公开的实施例不限于所公开的特定实施例，并且修改和其他实施例都应当包含于所附的权利要求范围内。尽管此处使用了特定的术语，但是它们仅在通用和描述的意义上使用，而并不用于限制目的。

Claims (21)

  1. 一种视觉传感器芯片，包括：
    像素阵列电路，被配置为通过测量光强变化量来生成与所述像素阵列电路中的多个像素对应的多个数据信号，所述多个数据信号指示至少一个光强变化事件，所述至少一个光强变化事件表示所述像素阵列电路中的对应像素测量到的光强变化量超过预定阈值；以及
    读取电路，耦合到所述像素阵列电路并且被配置为：
    以第一读取模式从所述像素阵列电路读取所述多个数据信号中的至少一个数据信号；
    向控制电路提供所述至少一个数据信号；以及
    如果从所述控制电路接收到基于所述至少一个数据信号而生成的模式切换信号，将所述第一读取模式切换为第二读取模式。
  2. 根据权利要求1所述的视觉传感器芯片，其中所述像素阵列电路还被配置为：
    如果所述像素阵列电路中的像素测量到的光强变化量超过所述预定阈值，生成指示所述光强变化事件的第一数据信号；以及
    如果所述像素阵列电路中的像素测量到的光强变化量未超过所述预定阈值，生成与所述第一数据信号不同的第二数据信号。
  3. 根据权利要求1所述的视觉传感器芯片，其中所述第一读取模式为基于帧扫描的读取模式并且所述第二读取模式为基于事件流的读取模式，并且其中所述读取电路被配置为通过以下方式来读取所述多个数据信号中的所述至少一个数据信号：
    通过以与所述基于帧扫描的读取模式相关联的帧频率扫描所述像素阵列电路，来读取所述多个数据信号中的所有数据信号；并且
    其中所述读取电路被配置为通过以下方式来提供所述至少一个数据信号：
    向所述控制电路提供所述所有数据信号。
  4. 根据权利要求1所述的视觉传感器芯片，其中所述第一读取模式为基于事件流的读取模式并且所述第二读取模式为基于帧扫描的读取模式，并且所述读取电路被配置为通过以下方式来读取所述多个数据信号中的所述至少一个数据信号：
    通过扫描所述像素阵列电路，确定与所述光强变化事件有关的像素的位置信息；以及
    基于所述位置信息，读取所述多个数据信号中的、由所述像素生成的所述至少一个数据信号。
  5. 根据权利要求1所述的视觉传感器芯片，其中所述视觉传感器芯片还包括所述控制电路，所述控制电路被配置为：
    基于从所述读取电路接收的所述至少一个数据信号，确定与所述至少一个光强变化事件有关的统计数据；以及
    如果确定所述统计数据满足预定切换条件，向所述读取电路发送模式切换信号。
  6. 根据权利要求5所述的视觉传感器芯片，其中所述第一读取模式为基于帧扫描的读取模式并且所述第二读取模式为基于事件流的读取模式，并且其中所述控制电路被配置为通过以下方式来确定所述统计数据：
    基于所述多个数据信号中的第一数据信号的数目，确定所述至少一个光强变化事件的 数目，每个第一数据信号指示所述像素阵列电路中的相应像素测量到的光强变化量超过预定阈值。
  7. 根据权利要求5所述的视觉传感器芯片，其中所述第一读取模式为基于事件流的读取模式并且所述第二读取模式为基于帧扫描的读取模式，并且其中所述控制电路被配置为通过以下方式来确定所述统计数据：
    基于所述至少一个数据信号的数目，确定所述至少一个光强变化事件的数目，所述至少一个数据信号中的每个数据信号指示所述像素阵列电路中的相应像素测量到的光强变化量超过预定阈值。
  8. 根据权利要求6或7所述的视觉传感器芯片，其中所述控制电路还被配置为通过以下方式来确定所述统计数据：
    基于所述光强变化事件的数目和与所述基于事件流的读取模式相关联的事件数据量，确定所述光强变化事件的总数据量。
  9. 根据权利要求6所述的视觉传感器芯片，其中所述控制电路还被配置为：
    如果所述统计数据指示所述光强变化事件的数目未超过第一阈值数目，确定所述光强变化事件的统计数据满足所述预定切换条件，所述第一阈值数目至少基于所述像素阵列电路的像素数目、与所述基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与所述基于事件流的读取模式相关联的事件数据量被确定；以及
    如果所述统计数据指示所述光强变化事件的总数据量未超过阈值数据量，确定所述光强变化事件的统计数据满足所述预定切换条件，所述阈值数据量至少基于所述像素阵列电路的像素数目、所述帧速率和所述像素数据量被确定。
  10. 根据权利要求7所述的视觉传感器芯片，其中所述控制电路还被配置为：
    如果所述统计数据指示所述光强变化事件的数目不低于第二阈值数目，确定所述光强变化事件的统计数据满足所述预定切换条件，所述第二阈值数目至少基于所述像素阵列电路的像素数目、与所述基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与所述基于事件流的读取模式相关联的事件数据量被确定；以及
    如果所述统计数据指示所述光强变化事件的总数据量不低于阈值数据量，确定所述光强变化事件的统计数据满足所述预定切换条件，所述阈值数据量至少基于所述像素阵列电路的像素数目、所述帧速率和所述像素数据量被确定。
  11. 一种用于操作视觉传感器芯片的方法，包括：
    所述视觉传感器芯片的像素阵列电路通过测量光强变化量来生成与所述像素阵列电路中的多个像素对应的多个数据信号，所述多个数据信号指示至少一个光强变化事件，所述至少一个光强变化事件表示所述像素阵列电路中的对应像素测量到的光强变化量超过预定阈值；
    所述视觉传感器芯片的读取电路以第一读取模式从所述像素阵列电路读取所述多个数据信号中的至少一个数据信号；
    所述读取电路向控制电路提供所述至少一个数据信号；以及
    如果从所述控制电路接收到基于所述至少一个数据信号而生成的模式切换信号，所述读取电路将所述第一读取模式切换为第二读取模式。
  12. 根据权利要求11所述的方法，其中生成所述多个数据信号包括：
    如果所述像素阵列电路中的像素测量到的光强变化量超过所述预定阈值，生成指示所 述光强变化事件的第一数据信号；以及
    如果所述像素阵列电路中的像素测量到的光强变化量未超过所述预定阈值，生成与所述第一数据信号不同的第二数据信号。
  13. 根据权利要求1所述的方法，其中所述第一读取模式为基于帧扫描的读取模式并且所述第二读取模式为基于事件流的读取模式，并且其中读取所述至少一个数据信号包括：
    所述读取电路通过以与所述基于帧扫描的读取模式相关联的帧频率扫描所述像素阵列电路，来读取所述多个数据信号，以及
    其中提供所述至少一个数据信号包括：所述读取电路向所述控制电路提供所述多个数据信号。
  14. 根据权利要求11所述的方法，其中所述第一读取模式为基于事件流的读取模式并且所述第二读取模式为基于帧扫描的读取模式，并且以所述第一读取模式来读取所述至少一个数据信号包括：
    所述读取电路通过扫描所述像素阵列电路，确定与所述光强变化事件有关的像素的位置信息；以及
    所述读取电路基于所述位置信息，读取所述多个数据信号中的、由所述像素生成的所述至少一个数据信号。
  15. 根据权利要求11所述的方法，还包括：
    所述控制电路基于从所述读取电路接收的所述至少一个数据信号，确定与所述至少一个光强变化事件有关的统计数据；以及
    如果确定所述统计数据满足预定切换条件，所述控制电路向所述读取电路发送模式切换信号。
  16. 根据权利要求15所述的方法，其中所述第一读取模式为基于帧扫描的读取模式并且所述第二读取模式为基于事件流的读取模式，并且确定所述光强变化事件的统计数据包括：
    所述控制电路基于所述多个数据信号中的第一数据信号的数目，确定所述至少一个光强变化事件的数目，每个第一数据信号指示所述像素阵列电路中的相应像素测量到的光强变化量超过预定阈值。
  17. 根据权利要求15所述的方法，其中所述第一读取模式为基于事件流的读取模式并且所述第二读取模式为基于帧扫描的读取模式，并且其中确定所述光强变化事件的统计数据包括：
    所述控制电路基于所述至少一个数据信号的数目，确定所述至少一个光强变化事件的数目，所述至少一个数据信号中的每个数据信号指示所述像素阵列电路中的相应像素测量到的光强变化量超过预定阈值。
  18. 根据权利要求16或17所述的方法，其中确定所述光强变化事件的统计数据还包括：
    所述控制电路基于所述光强变化事件的数目和与所述基于事件流的读取模式相关联的事件数据量，确定所述光强变化事件的总数据量。
  19. 根据权利要求16所述的方法，还包括：
    如果所述统计数据指示所述光强变化事件的数目未超过第一阈值数目，所述控制电路 确定所述光强变化事件的统计数据满足所述预定切换条件，所述第一阈值数目至少基于所述像素阵列电路的像素数目、与所述基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与所述基于事件流的读取模式相关联的事件数据量被确定；以及
    如果所述统计数据指示所述光强变化事件的总数据量未超过阈值数据量，所述控制电路确定所述光强变化事件的统计数据满足所述预定切换条件，所述阈值数据量至少基于所述像素阵列电路的像素数目、所述帧速率和所述像素数据量被确定。
  20. 根据权利要求17所述的方法，还包括：
    如果所述统计数据指示所述光强变化事件的数目不低于第二阈值数目，所述控制电路确定所述光强变化事件的统计数据满足所述预定切换条件，所述第二阈值数目至少基于所述像素阵列电路的像素数目、与所述基于帧扫描的读取模式相关联的帧速率和像素数据量、以及与所述基于事件流的读取模式相关联的事件数据量被确定；以及
    如果所述统计数据指示所述光强变化事件的总数据量不低于阈值数据量，所述控制电路确定所述光强变化事件的统计数据满足所述预定切换条件，所述阈值数据量至少基于所述像素阵列电路的像素数目、所述帧速率和所述像素数据量被确定。
  21. 一种电子设备，包括：
    根据权利要求1至10中任一项所述的视觉传感器芯片；以及
    解析器，耦合到所述视觉传感器芯片，并且被配置为：以与所述视觉传感器芯片的读取电路的读取模式相对应的解析模式，解析从所述读取电路接收的数据信号。
PCT/CN2020/112982 2020-09-02 2020-09-02 视觉传感器芯片及其操作方法和电子设备 WO2022047651A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080093015.0A CN115004677B (zh) 2020-09-02 2020-09-02 视觉传感器芯片及其操作方法和电子设备
PCT/CN2020/112982 WO2022047651A1 (zh) 2020-09-02 2020-09-02 视觉传感器芯片及其操作方法和电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/112982 WO2022047651A1 (zh) 2020-09-02 2020-09-02 视觉传感器芯片及其操作方法和电子设备

Publications (1)

Publication Number Publication Date
WO2022047651A1 true WO2022047651A1 (zh) 2022-03-10

Family

ID=80492097

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/112982 WO2022047651A1 (zh) 2020-09-02 2020-09-02 视觉传感器芯片及其操作方法和电子设备

Country Status (2)

Country Link
CN (1) CN115004677B (zh)
WO (1) WO2022047651A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115022621A (zh) * 2022-06-27 2022-09-06 深圳锐视智芯科技有限公司 一种事件相机测试方法、装置、设备及可读存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110169050A (zh) * 2016-12-30 2019-08-23 因赛特内斯股份公司 动态视觉传感器结构
US20190364237A1 (en) * 2018-05-24 2019-11-28 Samsung Electronics Co., Ltd. Dynamic vision sensor, electronic device and data transfer method thereof
CN110891152A (zh) * 2018-09-07 2020-03-17 三星电子株式会社 含cmos图像传感器像素和动态视觉传感器像素的图像传感器

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986179B2 (en) * 2014-09-30 2018-05-29 Qualcomm Incorporated Sensor architecture using frame-based and event-based hybrid scheme
US10698068B2 (en) * 2017-03-24 2020-06-30 Samsung Electronics Co., Ltd. System and method for synchronizing tracking points
CN111107287B (zh) * 2019-11-29 2021-06-04 豪威芯仑传感器(上海)有限公司 一种像素采集电路及图像传感器

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110169050A (zh) * 2016-12-30 2019-08-23 因赛特内斯股份公司 动态视觉传感器结构
US20190364237A1 (en) * 2018-05-24 2019-11-28 Samsung Electronics Co., Ltd. Dynamic vision sensor, electronic device and data transfer method thereof
CN110536050A (zh) * 2018-05-24 2019-12-03 三星电子株式会社 动态视觉传感器、电子设备及其数据传输方法
CN110891152A (zh) * 2018-09-07 2020-03-17 三星电子株式会社 含cmos图像传感器像素和动态视觉传感器像素的图像传感器

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115022621A (zh) * 2022-06-27 2022-09-06 深圳锐视智芯科技有限公司 一种事件相机测试方法、装置、设备及可读存储介质

Also Published As

Publication number Publication date
CN115004677B (zh) 2024-03-29
CN115004677A (zh) 2022-09-02

Similar Documents

Publication Publication Date Title
TWI415472B (zh) 編碼資料流之最佳電源使用
US20190007678A1 (en) Generating heat maps using dynamic vision sensor events
CN100515022C (zh) 图像信号处理设备、照相机系统及图像信号处理方法
KR102108321B1 (ko) 영상 구동 장치, 영상 구동 장치를 포함하는 전자 장치 및 영상 구동 방법
CN106254868B (zh) 视频编码码率控制方法、装置及系统
JP5988577B2 (ja) 画像符号化装置、画像符号化方法及びプログラム
US10051276B2 (en) Image encoding apparatus, method and imaging apparatus
US20200267396A1 (en) Human visual system adaptive video coding
WO2021077878A1 (zh) 图像处理方法、装置及电子设备
US20110007186A1 (en) Image capturing system, image capturing method, and computer readable medium storing therein program
US10366301B1 (en) Block based non-maximum suppression
JPH07203451A (ja) テレビジョン信号における動きの階層的予測方法
CN110087071B (zh) 基于感知的图像处理装置及相关方法
WO2022047651A1 (zh) 视觉传感器芯片及其操作方法和电子设备
CN105578079A (zh) 图像传感器及其成像画质的调节方法和成像装置
US20120033854A1 (en) Image processing apparatus
JP2007199815A (ja) メモリ制御装置およびメモリ制御方法
US8054345B2 (en) Electronic zooming apparatus and digital camera
CN105100595B (zh) 摄像设备及其控制方法
KR102494123B1 (ko) 이미지 센서, 카메라 모듈 및 카메라 모듈을 포함하는 광학 기기
WO2022188120A1 (en) Event-based vision sensor and method of event filtering
KR102213765B1 (ko) 이미지 센서, 카메라 모듈 및 카메라 모듈을 포함하는 광학 기기
KR102499286B1 (ko) 이미지가 분할된 복수의 블록들의 압축과 관련된 압축 손실 데이터에 기반하여 이미지를 압축하는 전자 장치 및 그 동작 방법
JPH1153529A (ja) 記憶装置および記憶方法
CN109561262B (zh) 一种脉冲序列格式转换方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20951899

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20951899

Country of ref document: EP

Kind code of ref document: A1