CN113727079A

CN113727079A - Image signal processing method and device and electronic equipment

Info

Publication number: CN113727079A
Application number: CN202010451017.4A
Authority: CN
Inventors: 李强; 张阳; 史斌
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-05-25
Filing date: 2020-05-25
Publication date: 2021-11-30
Anticipated expiration: 2040-05-25
Also published as: CN113727079B

Abstract

The embodiment of the application provides an image signal processing method and device and electronic equipment. The method and the device can ensure the synchronization of dynamic visual information and image gray signals. The method is applied to an image sensor which comprises a plurality of pixel groups, each pixel group comprises at least two sub-pixels, and each sub-pixel comprises a photosensitive circuit, and the method comprises the following steps: acquiring an electric signal emitted by a sub-pixel in a pixel group of an image sensor at any time period in the exposure duration of a frame of image; sending an electric signal sent by any sub-pixel in the pixel group to a DVS circuit, wherein the DVS circuit determines to generate an event signal according to the electric signal sent by any sub-pixel; and sending the electric signals sent by the sub-pixels except any sub-pixel in the pixel group to an APS circuit, wherein the APS circuit generates image gray scale signals according to the electric signals sent by the sub-pixels except any sub-pixel in the pixel group.

Description

Image signal processing method and device and electronic equipment

Technical Field

The present disclosure relates to the field of image processing, and in particular, to an image signal processing method and apparatus, and an electronic device.

Background

Machine vision has increasingly replaced human beings for the acquisition of visual information to help address complex real-life environments. The advent of Dynamic Vision Sensors (DVS) provides a machine vision viable solution from a new direction. The light sensing circuitry (typically a photodiode) of the DVS sends a light intensity signal to the DVS circuitry, which responds by generating dynamic visual information (e.g., an "event" signal) upon determining from the received light intensity signal that the light intensity change exceeds a certain threshold. The event information only contains the polarity of the change event, does not contain information of absolute light intensity, and cannot generate an image gray signal for generating a complete image. In practical applications, the information of absolute light intensity is also one of the important information to be acquired for the convenience of system identification and judgment. An asynchronous time based image sensor (ATIS) combines a DVS with an Active Pixel Sensor (APS) to acquire an event signal and an image gray signal, respectively.

In the ATIS scheme, a complete ATIS picture element is composed of a change detector and a photo-measurement unit. The change detector is equivalent to a DVS circuit of a DVS and used for generating an event signal according to the variation of the light intensity induced by the light intensity signal of the photosensitive circuit. The photo-measurement unit is equivalent to an APS circuit of APS, and can detect light intensity information received by the photosensitive circuit and generate an image gray signal. In the working process of ATIS, when the change of input light intensity exceeds a trigger threshold value, a change detector is caused to generate an event signal, and a photo-measurement unit is sent out a trigger signal; and after the photo-measurement unit receives the trigger signal, starting to acquire an image gray signal.

The photo-metric unit is triggered by the change detector, and the image gray scale signal is acquired after the photo-metric unit is triggered, so that the time stamp of the event signal is not completely consistent with the time stamp of the image gray scale signal, and the event signal and the image gray scale signal are asynchronous. In addition, the event signal and the image gray scale signal are obtained through two photosensitive circuits respectively, namely, one ATIS pixel needs to comprise two photodiodes and a matched operating and reading circuit thereof, so that the occupied area is at least doubled compared with that of a single DVS or APS under the same resolution, the use efficiency of the photosensitive area of a chip is low, and a part of spatial information of an image is lost.

Disclosure of Invention

The embodiment of the application provides an image signal processing method and device and electronic equipment, which can ensure the synchronization of dynamic visual information and an image gray signal.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, an image signal processing method is provided. The method is applied to an image sensor which comprises a plurality of pixel groups, wherein each pixel group comprises at least two sub-pixels, and the method comprises the following steps: acquiring an electric signal emitted by a sub-pixel in a pixel group of an image sensor at any time period in the exposure duration of a frame of image; sending an electric signal sent by any sub-pixel in the pixel group to a dynamic vision sensor DVS circuit, wherein the DVS circuit determines to generate dynamic vision information according to the electric signal sent by any sub-pixel; and sending the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group to an Active Pixel Sensor (APS) circuit, wherein the APS circuit generates image gray signals according to the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group.

In this way, in the exposure duration of one frame of image, no matter the exposure duration of the one frame of image includes one or more time periods, in any time period, the image signal processing apparatus can send the electrical signal emitted by one sub-pixel in the pixel group to the DVS circuit (i.e. the single sub-pixel operates in the DVS mode), and the DVS circuit determines to generate dynamic visual information according to the electrical signal emitted by the sub-pixel; and sending the electrical signals emitted by the other sub-pixels to an APS circuit (i.e., a single sub-pixel operates in APS mode), the APS circuit generating image gray scale signals from the electrical signals emitted by the other sub-pixels; the problem of synchronous acquisition of dynamic visual information and image gray scale signals in the same image sensor is solved.

In one possible implementation, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image; sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, including: sending an electrical signal emitted by a first sub-pixel in the pixel group to the DVS circuit, wherein the DVS circuit determines to generate dynamic visual information according to the electrical signal emitted by the first sub-pixel; sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: and sending the electric signal sent by the second sub-pixel in the pixel group to the APS circuit, wherein the APS circuit generates an image gray scale signal according to the electric signal sent by the second sub-pixel. In this way, in the above process, the image sensor sends the electric signals of n-1 sub-pixels to the APS circuit to generate the image gray scale signal in the exposure time of one frame of image, and sends the electric signals of one sub-pixel to the DVS circuit to generate the dynamic visual information in the exposure time of one frame of image. In the exposure time of one frame of image, the dynamic visual information and the image gray signal are synchronously acquired.

In one possible implementation, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration; sending the electric signal sent by any sub-pixel in the pixel group to the DVS circuit; the method comprises the following steps: sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: and the APS generates image gray signals corresponding to the xth sub-pixel according to signals in other sub-frame time lengths except the xth sub-frame time length sent by the xth sub-pixel. In this way, within the exposure time of each frame of image, any sub-pixel in any pixel group sends an electric signal to the DVS circuit within at least one sub-frame time, namely, the DVS circuit works in a DVS mode; and transmitting the electric signal to the APS circuit for the rest of the sub-frame time, namely, operating in an APS mode. Thus, during any sub-frame duration, at least one sub-pixel always operates in the DVS mode and the remaining sub-pixels operate in the APS mode. The dynamic visual information and the image gray signal are ensured to be synchronously acquired in the exposure time of one frame of image.

In one possible implementation, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image; sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, including: sending an electrical signal emitted by a first sub-pixel in the pixel group to the DVS circuit, wherein the DVS circuit determines to generate dynamic visual information according to the electrical signal emitted by the first sub-pixel; sending the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group to the APS circuit; the method comprises the following steps: and sending the electric signals sent by the other sub-pixels except the first sub-pixel in the pixel group to the APS circuit, wherein the APS circuit generates an image gray signal according to the electric signals sent by the other sub-pixels except the first sub-pixel in the exposure time of the one-frame image. In this way, in the above process, the image sensor is equivalent to an n-1-binning APS sub-pixel, that is, the exposure time of one frame of image is sent to the APS circuit for combining (summing or averaging) to generate an image gray signal, and the exposure time of one frame of image is sent to the DVS circuit for generating dynamic visual information, that is, the exposure time of one frame of image is 1 sub-pixel. Therefore, in the exposure time period of one frame image, the dynamic visual information and the image gradation signal are acquired synchronously.

In one possible implementation, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration; sending the electric signal sent by any sub-pixel in the pixel group to the DVS circuit; sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS circuit determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: and sending the electric signals sent by other sub-pixels except the xth sub-pixel to the APS circuit within the xth sub-frame time, wherein the APS circuit generates an image gray signal according to the electric signals of all the sub-pixels received within the exposure time of the frame image. In the process, the image sensor is equivalent to APS sub-pixels with the exposure duration of n-1 one-frame images, namely, each sub-pixel has the time of n-1/n and sends the electric signals to an APS circuit to be combined (summed or averaged) to generate image gray signals, and a DVS sub-pixel with the exposure duration of one-frame images, namely, each sub-pixel has the time of 1/n and sends the electric signals to the DVS circuit to generate dynamic visual information. In the exposure time of one frame of image, dynamic visual information and image gray signals are acquired synchronously in time.

In a second aspect, an image signal processing apparatus is provided for implementing the various methods described above. The image signal processing apparatus includes modules, units, or means (means) corresponding to the above methods, and the modules, units, or means may be implemented by hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions.

In a third aspect, there is provided an image signal processing apparatus comprising: a processor and a memory; the memory is used for storing computer instructions which, when executed by the processor, cause the image signal processing apparatus to perform the method of any of the above aspects.

In a fourth aspect, there is provided an image signal processing apparatus comprising: a processor and a transmission interface; the processor is configured to invoke program instructions stored in the memory to perform a method as in any of the above aspects.

In one possible design, the image signal processing apparatus further includes a memory for storing necessary program instructions and data. When the image signal processing apparatus is a chip system, the image signal processing apparatus may be constituted by a chip, or may include a chip and other discrete devices.

In a fifth aspect, there is provided a computer readable storage medium having stored therein program instructions which, when run on a computer or processor, cause the computer or processor to perform the method of any of the above aspects.

In a sixth aspect, there is provided a computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to perform the method of any of the above aspects.

A seventh aspect provides an electronic device, comprising an image sensor, the image signal processing apparatus as described above connected to the image sensor; the pixel sensor comprises a pixel group arranged in an array, wherein the pixel group comprises at least two sub-pixels, and each sub-pixel comprises a photosensitive circuit.

In one possible design, the image sensor includes a photosensitive layer and a sensor circuit layer located on a side of the photosensitive layer opposite to light, where the photosensitive layer includes the pixel groups arranged in the array; the sensor circuit layer includes circuit regions in one-to-one correspondence with the pixel groups, the circuit regions including a DVS circuit. Optionally, the circuit area further includes an APS circuit.

For technical effects brought by any one of the design manners in the second aspect to the seventh aspect, reference may be made to the technical effects brought by different design manners in the first aspect, and details are not described here.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 2a is a schematic structural diagram of an image sensor according to an embodiment of the present application;

fig. 2b is a schematic structural diagram of an image sensor according to another embodiment of the present application;

fig. 3 is a schematic flowchart of an image signal processing method according to an embodiment of the present application;

fig. 4 is a schematic circuit structure diagram of an electronic device according to an embodiment of the present application;

fig. 5 is a schematic circuit structure diagram of an electronic device according to an embodiment of the present application;

fig. 6 is a schematic diagram illustrating a first output manner of a sub-pixel in an image signal processing method according to an embodiment of the present application;

fig. 7 is a schematic diagram illustrating an output manner of a sub-pixel in an image signal processing method according to an embodiment of the present application;

fig. 8 is a schematic diagram illustrating a third output manner of a sub-pixel in an image signal processing method according to an embodiment of the present application;

fig. 9 is a schematic diagram illustrating a fourth output manner of a sub-pixel in an image signal processing method according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of an image signal processing apparatus according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an image signal processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments.

In the following, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature.

Further, in the present application, directional terms such as "upper", "lower", "left", "right", and the like are defined with respect to a schematically placed orientation of a component in the drawings, and it is to be understood that these directional terms are relative concepts, which are used for descriptive and clarifying purposes, and may vary accordingly depending on the orientation in which the component is placed in the drawings.

At present, machine vision has increasingly replaced human beings to acquire visual information, and helps to process complex real-life environments. With the improvement of the algorithm, the processing speed of machine vision and the accuracy of tracking objects are greatly improved, but the problems of large error, large calculation amount, large power consumption and the like still exist when the traditional camera processes a complex real environment.

In a machine vision scheme, each pixel of the DVS contains a photosensitive circuit as well as a DVS circuit. The light sensing circuit (typically a photodiode) sends a light intensity signal to the DVS circuit, which generates dynamic visual information (e.g., an "event" signal) in response to determining from the received light intensity signal that the light intensity change exceeds a certain threshold. The pixel perception of DVS is the light intensity change, can effectively filter static redundant background information, has reduced the low efficiency data bulk, has reduced the pressure of data processing. In addition, each pixel is not subjected to synchronous time sequence control, but is triggered asynchronously, so that the response time is not limited by a frame rate, the response speed is high, the time precision is high, optical flow information can be directly output at a sensor end, and the method has great advantages for shooting, identifying and tracking moving objects. In a typical DVS application, a real-time intensity signal is continuously monitored, and when the magnitude of the change in intensity relative to a reference exceeds a threshold, a polarity signal corresponding to ON or OFF is generated according to the positive or negative direction of the change, and the new intensity signal is used as the reference. It can be seen that the DVS itself reacts only to received light intensity changes and is not responsive to static information and small changes. Moreover, the event signal generated by the method only contains the polarity of the change event, does not contain information of absolute light intensity, and even cannot generate an image gray scale signal for acquiring a complete image.

In the ATIS scheme, the event signal and the image gradation signal may be acquired separately. In a typical arrangement of ATIS, a complete ATIS picture element is formed by a change detector and a photo-measurement unit. The change detector is equivalent to a DVS circuit of a DVS pixel and used for sensing the variation of light intensity according to a light intensity signal of the photosensitive circuit, and when the variation of the light intensity exceeds a certain threshold value, an event signal can be generated. The photo-measurement unit is equivalent to an APS circuit of one APS pixel, and can detect light intensity information received by the photosensitive circuit through resetting and integrating operations to generate an image gray signal. In the working process of ATIS, the change of input light intensity exceeds a trigger threshold value, a change detector is caused to generate an event signal, and a photo-measure unit is sent out a trigger signal; after the photo-measurement unit receives the trigger signal, resetting and integrating the capacitor, outputting a low level by the comparator, and when the input light intensity is stronger, the capacitor discharges more quickly; when the voltage of the integrating capacitor is reduced to a certain threshold value, the output of the comparator restores to a high level, and the integration is finished; the integral duration represents the absolute light intensity of the input light and can be converted into an image gray signal. Therefore, the ATIS can obtain the event signal and the image gradation signal of its corresponding pixel.

Because the photo-metric unit is triggered by the change detector, and the image gray scale signal is acquired after the trigger, strictly speaking, the time stamp of the event signal and the time stamp of the image gray scale signal are not completely consistent, and the two signals are asynchronous. In addition, the event signal and the image gray scale signal are obtained by two photosensitive circuits (e.g. photodiodes) respectively, that is, one ATIS pixel needs to include two photodiodes and their associated operating and reading circuits, so that the occupied area is at least doubled compared with the single DVS or APS at the same resolution, the utilization efficiency of the photosensitive area of the chip is low, and a part of the spatial information of the image is lost.

To solve the above problem, embodiments of the present application provide an electronic device having a function of image acquisition. The method is suitable for scenes needing to acquire traditional images and dynamic visual information at the same time, such as application scenes needing to perform target recognition, tracking, optical flow calculation, simultaneous localization and mapping (SLAM) and the like. The electronic device includes, for example: mobile phones, cameras, video cameras, smart terminals, tablet computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, Personal Digital Assistants (PDA), in-vehicle computers, and the like. The embodiment of the present application does not specifically limit the specific form of the electronic device.

Fig. 1 is a schematic structural diagram of an exemplary electronic device according to an embodiment of the present disclosure. As shown in fig. 1, the electronic device 01 includes: a processor 11, a Radio Frequency (RF) circuit 12, a power supply 13, a memory 14, an input unit 15, a display unit 16, an audio circuit 17, and the like. Those skilled in the art will appreciate that the configuration of the electronic device shown in fig. 1 does not constitute a limitation of the apparatus, and that the electronic device may include more or less components than those shown in fig. 1, or may combine some of the components shown in fig. 1, or may be arranged differently than those shown in fig. 1.

The processor 11 is a control center of the electronic device, connects various parts of the whole electronic device by various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 14 and calling data stored in the memory 14, thereby performing overall monitoring of the electronic device. Alternatively, processor 11 may include one or more processing units; preferably, the processor 11 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 11.

The RF circuit 12 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receives downlink information of a base station and then processes the received downlink information to the processor 11; in addition, the uplink data is transmitted to the base station. Typically, the RF circuitry includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 12 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to global system for mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

The electronic device includes a power supply 13 (e.g., a battery) for supplying power to various components, and optionally, the power supply may be logically connected to the processor 11 through a power management system, so that functions of managing charging, discharging, and power consumption are implemented through the power management system.

The memory 14 may be used to store software programs and modules, and the processor 11 executes various functional applications and data processing of the apparatus by operating the software programs and modules stored in the memory 14. The memory 14 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, image data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 14 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit 15 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device. Specifically, the input unit 15 may include a touch screen 151 and other input devices 152. The touch screen 151, also referred to as a touch panel, may collect a touch operation of a user on or near the touch screen (for example, an operation of the user on or near the touch screen 151 by using any suitable object or accessory such as a finger or a stylus pen), and drive a corresponding connection device according to a preset program. Alternatively, the touch screen 151 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 11, and can receive and execute commands sent by the processor 11. In addition, the touch screen 151 may be implemented in various types, such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 152 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 16 may be used to display information input by or provided to a user and various menus of the electronic device. The display unit 16 may include a display panel 161, and in the present application, the display panel 161 may be configured using an AMOLED display screen. Further, the touch screen 151 may cover the display panel 161, and when the touch screen 151 detects a touch operation on or near the touch screen 151, the touch operation is transmitted to the processor 11 to determine the type of the touch event, and then the processor 11 provides a corresponding visual output on the display panel 161 according to the type of the touch event. Although in fig. 1 the touch screen 151 and the display panel 161 are shown as two separate components to implement the input and output functions of the device, in some embodiments the touch screen 151 and the display panel 161 may be integrated to implement the input and output functions of the device.

Audio circuitry 17, a speaker 171 and a microphone 172 for providing an audio interface between the user and the device. The audio circuit 17 may transmit the electrical signal converted from the received audio data to the speaker 171, and convert the electrical signal into a sound signal by the speaker 171 for output; on the other hand, the microphone 172 converts the collected sound signals into electrical signals, converts the electrical signals into audio data after being received by the audio circuit 17, and then outputs the audio data to the RF circuit 12 to be transmitted to, for example, another device, or outputs the audio data to the memory 14 for further processing.

Optionally, the device shown in FIG. 1 may also include various sensors. Such as a gyroscope sensor, a hygrometer sensor, an infrared sensor, a magnetometer sensor, an image sensor, etc., and will not be described in detail herein. Optionally, the apparatus shown in fig. 1 may further include a wireless fidelity (WiFi) module, a bluetooth module, and the like, which are not described herein again.

It is understood that, in the embodiments of the present application, an electronic device may perform some or all of the steps in the embodiments of the present application, and these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of various operations. Further, the various steps may be performed in a different order presented in the embodiments of the application, and not all operations in the embodiments of the application may be performed. The embodiments of the present application may be implemented individually or in any combination, and the present application is not limited to these.

In order to implement the image signal processing method provided by the embodiment of the application, the electronic device provided by the embodiment of the application comprises an image sensor and an image signal processing device connected with the image sensor; the pixel sensor includes a pixel group (pixel group) arranged in an array, where the pixel group includes at least two sub-pixels, and each sub-pixel includes a photosensitive circuit, which may be a photodiode. As shown in fig. 2a, the image sensor includes a photosensitive layer 21 and a sensor circuit layer 22 located on a side of the photosensitive layer 21 opposite to the light, wherein the photosensitive layer 21 includes a pixel group 211 arranged in an array; the sensor circuit layer 22 includes circuit regions 221 corresponding to the pixel groups one by one, and the circuit regions 221 include at least a DVS circuit. In one example, as shown in fig. 2b, the circuit area 221 may also include an APS circuit. Wherein the APS circuit may also be disposed in a peripheral circuit of the image sensor. As an example, each pixel group is further shown in fig. 2a and 2b to include four sub-pixels. The pixel group referred to in the present invention is also referred to as a light sensing unit, and refers to a pixel structure in which a plurality of same-color sub-pixels are arranged adjacently, including but not limited to a four-in-one pixel group (also referred to as a Bayer array, a four-grid (four cell) or a four-pixel synthesis (tetracell), etc.), a nine-in-one pixel group (also referred to as a nonacell, etc.), a sixteen-in-one pixel group, and the like.

The realization principle of the invention is as follows: based on a conventional complementary metal oxide semiconductor image sensor (CIS), the DVS function is integrated in the CIS by using the principle of time-space multiplexing. In each pixel group of the CIS, at any time period within the exposure duration of each frame image, an electric signal sent by one sub-pixel in the pixel group is sent to a DVS circuit (namely, a single sub-pixel works in a DVS mode), and the DVS determines to generate dynamic visual information according to the electric signal sent by the sub-pixel; and sending the electrical signals emitted by the other sub-pixels to an APS circuit (i.e., a single sub-pixel operates in APS mode), the APS circuit generating image gray scale signals from the electrical signals emitted by the other sub-pixels; the problem of synchronous acquisition of dynamic visual information and image gray signals in the same image sensor is solved; in addition, each sub-pixel can work in a DVS mode or an APS mode, a photosensitive circuit (a photodiode) is not needed to be used separately, and the use efficiency of a photosensitive area is guaranteed; meanwhile, the sensor circuit layer is placed below the photosensitive layer through a chip stacking (stack) process, so that the additionally increased circuit area is reduced to the maximum extent.

Referring to fig. 3, an embodiment of the present application provides an image signal processing method including the steps of:

101. and acquiring the electric signals emitted by the sub-pixels in the pixel group of the image sensor in any time period in the exposure time of one frame of image.

102. And sending the electric signal emitted by any sub-pixel in the pixel group to a DVS circuit, wherein the DVS determines and generates dynamic visual information according to the electric signal emitted by any sub-pixel.

Wherein the dynamic visual information in the following schemes is illustrated by taking an event signal as an example.

103. And sending the electric signals sent by the sub-pixels except any sub-pixel in the pixel group to an APS circuit, wherein the APS circuit generates image gray scale signals according to the electric signals sent by the sub-pixels except any sub-pixel in the pixel group.

Illustratively, referring to fig. 4, a schematic circuit structure diagram of an electronic device is provided, which includes: the image sensor may include a photosensitive circuit 41, an image signal processing device 42, an APS circuit 43, and a DVS circuit 44, wherein the photosensitive circuit 41, the APS circuit 43, and the DVS circuit 44 may form a stacked CIS structure as shown in fig. 2. In this embodiment, a photosensitive circuit in any pixel group is taken as an example for description. In conjunction with the schematic circuit structure of the electronic device, the image signal processing apparatus 42 can implement the above steps 101-103 in conjunction with the photosensitive circuit 41, the APS circuit 43, and the DVS circuit 44.

The light sensing circuit 41 mainly includes a photodiode, and is responsible for a photoelectric conversion function, and converts a sensed light intensity into an electrical signal.

The image signal processing device 42 is responsible for controlling the switching of the sub-pixels between the APS mode and the DVS mode, and specifically, as shown in step 102, i.e., step 103, the image signal processing device 42 is specifically configured to send the electrical signal sent from the photosensitive circuit 41 to the APS circuit or the DVS circuit.

APS circuit 43 includes an integrating circuit 431 and a sample-and-hold circuit 432. The integrating circuit 431 is responsible for controlling the functions of photodiode reset, integration, charge transfer and the like in the APS mode. And the sample-hold circuit 432 is responsible for sampling and holding the integrated signal, and simultaneously can have a correlated double sampling function, and finally outputs an analog image gray scale signal.

The DVS circuit 44 includes a differential amplification circuit 441, an event detection circuit 442, and a handshake protocol circuit 443; the differential amplifier circuit 441 mainly includes an operational amplifier circuit and related components, and is responsible for differentially amplifying the electrical signal output by the photodiode in the DVS mode and the reference signal. The event detection circuit 442 is mainly composed of a comparator and peripheral elements, and is responsible for determining whether or not the differential signal reaches a threshold value, and if so, generates an event signal. Handshake protocol circuit 443, responsible for asynchronous communication control directly with the platform, asynchronously transfers event signals to the platform.

Further, a pixel peripheral circuit 45 is included, wherein the pixel peripheral circuit 45 mainly includes a timing control circuit 451 and an analog-to-digital conversion circuit 452. The timing control circuit 451 is generally referred to as a timing generation and control functional module in the whole operation process of the electronic device. The analog-to-digital conversion circuit 452 is responsible for converting the analog image gray scale signal into a digital signal. The timing signal generated by the timing control circuit 451 is specifically used in the following processes: the mode switching of the image signal processing device 42, the resetting and integration of the photodiode by the APS circuit 43, and the timing control of the analog-to-digital conversion by the analog-to-digital conversion circuit 452. For example: in step 101, the device needs to acquire the electric signals sent by the sub-pixels in the pixel group of the image sensor in any time period of the exposure time of one frame of image; therefore, the device needs to determine the exposure duration of one frame of image at a certain timing signal. Another example is: the APS circuit 43 needs to convert the image gradation signal by the integration duration time, and therefore needs to refer to a constant timing signal.

As shown in fig. 5, in an example where the image signal processing apparatus 42 may be composed of a logic driving circuit 421 and switches TG1 and TG2, in order to implement the functions described in step 102 and step 103, the mode switching according to the present invention includes, but is not limited to, an exemplary implementation manner, in which the photodiode 41 and the APS circuit 43 or the DVS circuit 44 are connected in a time-sharing manner through two MOS switches, as shown in fig. 5. When the logic drive circuit 421 controls the TG1 to be turned on and the TG2 to be turned off, the photodiode 41 is connected to the APS circuit 43; when the logic drive circuit 421 controls the TG1 to be turned off and the TG2 to be turned on, the photodiode 41 is connected to the DVS circuit 44.

In this way, in the exposure duration of one frame of image, no matter the exposure duration of the one frame of image includes one or more time periods, in any time period, the image signal processing apparatus can send the electrical signal emitted by one sub-pixel in the pixel group to the DVS circuit (i.e. the single sub-pixel operates in the DVS mode), and the DVS determines to generate dynamic visual information according to the electrical signal emitted by the sub-pixel; and sending the electrical signals emitted by the other sub-pixels to an APS circuit (i.e., a single sub-pixel operates in APS mode), the APS circuit generating image gray scale signals from the electrical signals emitted by the other sub-pixels; the problem of synchronous acquisition of dynamic visual information and image gray scale signals in the same image sensor is solved.

Each sub-pixel in each pixel group in the pixel group of the APS can independently sense light and respectively output an electric signal in a full size mode to generate an image gray signal; it is also possible to add or average the electric signals of the respective sub-pixels in the pixel group and output an image gradation signal in a binning (which is an image readout mode in which charges induced by adjacent pixels are added together or averaged and read out in a one-pixel mode).

The following description will be made by taking a full size mode as an example of the sub-pixels operating in APS in a pixel group:

example one: the pixel group comprises n sub-pixels, wherein n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of one frame of image; sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, comprising: sending an electric signal sent by a first sub-pixel in the pixel group to a DVS, wherein the DVS determines to generate dynamic visual information according to the electric signal sent by the first sub-pixel; sending the electric signals sent by the sub-pixels except any one sub-pixel in the pixel group to an APS circuit, comprising: sending the electric signals sent by the second sub-pixels in the pixel group to an APS circuit; wherein the APS circuit generates an image gray signal based on the electrical signal from the second subpixel. In this way, in the above process, the image sensor sends the electric signals of n-1 sub-pixels to the APS circuit to generate the image gray scale signal in the exposure time of one frame of image, and sends the electric signals of one sub-pixel to the DVS circuit to generate the dynamic visual information in the exposure time of one frame of image. In the exposure time of one frame of image, the dynamic visual information and the image gray signal are synchronously acquired.

Referring to fig. 6, a four-in-one pixel group structure is taken as an example, and practical applications of the present invention include, but are not limited to, a four-in-one pixel group arrangement mode, such that one pixel group includes four sub-pixels as shown in fig. 6. In a four-by-one pixel group, when the sub-pixels are output in full size mode, the four sub-pixels may be operated in different modes, for example, three sub-pixels are operated in APS mode, and one sub-pixel is operated in DVS mode. Specifically, in the exposure time period of one frame image (e.g., T1, T2, T3, T4 in fig. 6), the image signal processing apparatus outputs the electric signal output by the first subpixel to the DVS circuit; in the exposure time period of one frame image (e.g., T1, T2, T3, T4 in fig. 6), the image signal processing device outputs the electric signal output by the second subpixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output by the fourth sub-pixel to the APS circuit; thus, in the above process, the image sensor corresponds to an APS subpixel having an exposure duration of 3 one-frame images and a DVS subpixel having an exposure duration of one-frame images. In the exposure time of one frame of image, dynamic visual information and image gray signals are acquired synchronously in time. Since the first sub-pixel is fixedly adopted to always work in the DVS mode within the exposure duration of one frame of image, the loss of one fourth of image space information of the complete image is equivalent.

Example two: the pixel group comprises n sub-pixels, wherein n is a positive integer greater than or equal to 2; the exposure time of one frame of image comprises n subframe times; the duration of any time interval is equal to the duration of a subframe; sending an electric signal sent by any sub-pixel in the pixel group to a DVS circuit; the method comprises the following steps: sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; sending the electric signals sent by the sub-pixels except any one sub-pixel in the pixel group to an APS circuit, comprising: sending the electric signals sent by other sub-pixels except the x sub-pixel to an APS circuit within the x sub-frame time; and the APS circuit generates an image gray signal corresponding to the xth sub-pixel according to the electric signals in other sub-frame time lengths except the xth sub-frame time length sent by the xth sub-pixel. In this way, within the exposure time of each frame of image, any sub-pixel in any pixel group sends an electric signal to the DVS circuit within at least one sub-frame time, namely, the DVS circuit works in a DVS mode; and transmitting the electric signal to the APS circuit for the rest of the sub-frame time, namely, operating in an APS mode. Thus, during any sub-frame duration, at least one sub-pixel always operates in the DVS mode and the remaining sub-pixels operate in the APS mode. The dynamic visual information and the image gray signal are ensured to be synchronously acquired in the exposure time of one frame of image.

In the second example, a round-robin switching manner of time-space multiplexing is adopted, as shown in fig. 7. The exposure duration of one frame of image can be divided into n equal parts, where n is the number of sub-pixels in a pixel group. For example, in the four-in-one pixel group, the exposure time period of one frame image is equally divided into four sub-frame periods T1, T2, T3, and T4, and within each sub-frame period, three sub-pixels operate in the APS mode and one sub-pixel operates in the DVS mode. Meanwhile, the APS mode or DVS mode of the sub-pixels is switched in a continuous rotation, for example, in the sub-frame duration time T1, the first sub-pixel operates in the DVS mode, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel operate in the APS mode, and the image signal processing apparatus outputs the electric signal output by the first sub-pixel to the DVS circuit; the image signal processing device outputs the electric signal output by the second sub-pixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output from the fourth subpixel to the APS circuit. During the next sub-frame duration, i.e. T2, the second sub-pixel operates in DVS mode, the first sub-pixel returns to APS mode, and the third sub-pixel and the fourth sub-pixel operate in APS mode; the image signal processing device outputs the electric signal output by the second sub-pixel to the DVS circuit; the image signal processing device outputs the electric signal output by the first sub-pixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output from the fourth subpixel to the APS circuit. And so on. Thus, in the above process, the image sensor corresponds to the APS subpixel realizing the exposure time of 4 three-quarters frame images and the DVS subpixel corresponding to the exposure time of one frame image.

Therefore, within the exposure time of each frame image, any sub-pixel in any pixel group works in the DVS mode in at least one sub-frame time length, and works in the APS mode in the rest sub-frame time lengths. In any sub-frame duration, at least one sub-pixel always works in DVS mode, and the rest sub-pixels work in APS mode.

When the sub-pixels are in the DVS mode, the electrical signals output by the sub-pixels in each sub-frame can be directly output to the DVS circuit to generate the dynamic visual information. When the sub-pixel is in the APS mode, the problem of how to read or superimpose the image gray scale signals output from the APS circuit for each sub-frame period after the electrical signals of the sub-pixel are output. A typical implementation is to read out the image gray signals in each sub-frame for the same sub-pixel, and then superimpose them, so as to obtain the complete image gray signal of one frame of image at the end of the exposure time of one frame of image. Another typical implementation is that the image gray scale signals output by the APS circuit of each sub-pixel in each sub-frame are temporarily not read out, buffered in the charge storage area, and continuously overlapped until the exposure time of one frame of image is finished to uniformly read out the image gray scale signals of the complete one frame of image.

In the second example described above, when output in full size mode, the actual exposure time of the image gradation signal corresponding to each sub-pixel is three-quarters of the exposure time length of one frame image, while at the same time an event signal matching the time stamp of the exposure time length of one frame image can be acquired, and image space information of one quarter of the full size is not lost.

In the following, a description will be given by taking a binning mode as an example of a sub-pixel operating in an APS in a pixel group, where when the sub-pixel of the APS is output in the binning mode, image gray signals corresponding to the sub-pixels in the same pixel group are output in the form of one equivalent pixel by summing or averaging, so that the resolution is reduced, the sensitivity and the signal-to-noise ratio can be improved, and the frame rate can be improved. Each pixel group outputs only a single image gray signal for the exposure time of each frame image, and in this following example, each pixel group will also maintain the ability to acquire an event signal at all times.

Example three: the pixel group comprises n sub-pixels, wherein n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame image; sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, comprising: sending an electric signal sent by a first sub-pixel in a pixel group to the DVS circuit, wherein the DVS circuit determines to generate dynamic visual information according to the electric signal sent by the first sub-pixel; sending the electric signals sent by other sub-pixels except any sub-pixel in the pixel group to an APS circuit; the method comprises the following steps: sending the electric signals sent by other sub-pixels except the first sub-pixel in the pixel group to an APS circuit; wherein the APS circuit generates an image gray signal based on the electric signals from the other sub-pixels except the first sub-pixel during the exposure period of one frame of image. In this way, in the above process, the image sensor is equivalent to an n-1-binning APS sub-pixel, that is, the exposure time of one frame of image is sent to the APS circuit for combining (summing or averaging) to generate an image gray signal, and the exposure time of one frame of image is sent to the DVS circuit for generating dynamic visual information, that is, the exposure time of one frame of image is 1 sub-pixel. Therefore, in the exposure time period of one frame image, the dynamic visual information and the image gradation signal are acquired synchronously.

Taking the four-in-one pixel group structure as an example, practical applications of the present invention include, but are not limited to, four-in-one pixel group arrangement, such that one pixel group includes four sub-pixels as shown in fig. 8. In a four-by-one pixel group, when the sub-pixels are output in a binning mode, the four sub-pixels can be operated in different modes, for example, three sub-pixels are operated in an APS mode, and one sub-pixel is operated in a DVS mode. Specifically, in the exposure time period of one frame image (e.g., T1, T2, T3, T4 in fig. 8), the image signal processing apparatus outputs the electric signal output by the first subpixel to the DVS circuit; in the exposure time period of one frame image (e.g., T1, T2, T3, T4 in fig. 8), the image signal processing device outputs the electric signal output by the second subpixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output by the fourth sub-pixel to the APS circuit; the individual sub-pixels of each pixel group at the binning size will combine signals to generate image gray scale signals as electrical signals emitted by the sub-pixels, where the manner of combining signals includes, but is not limited to, addition or averaging of charges (e.g., directly adding or averaging electrical signals output by the sub-pixels), addition or averaging of voltages (e.g., adding or averaging analog image gray scale signals output by an APS circuit), addition or averaging of digital signals (e.g., adding or averaging after analog image gray scale signals are converted to digital signals by pixel peripheral circuit 45), and the like. In the exposure time of one frame of image, because three sub-pixels work in APS mode, the image sensor realizes the equivalent effect of one 3-binning APS sub-pixel in the process; meanwhile, one sub-pixel operates in the DVS mode, so that the image sensor in the above process is equivalent to implementing one DVS sub-pixel operating in a full period. In this embodiment, the image grayscale signal of the 3-binning APS subpixel is a perfect match in time and space to the event signal of the equivalent DVS subpixel.

Example four: the pixel group comprises n sub-pixels, wherein n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration; sending an electric signal sent by any sub-pixel in the pixel group to the DVS circuit; sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: and sending the electric signals sent by other sub-pixels except the xth sub-pixel to the APS circuit within the xth sub-frame time, wherein the APS circuit generates an image gray signal according to the electric signals of all the sub-pixels received within the exposure time of the frame image. In the process, the image sensor is equivalent to APS sub-pixels with the exposure duration of n-1 one-frame images, namely, each sub-pixel has the time of n-1/n and sends the electric signals to an APS circuit to be combined (summed or averaged) to generate image gray signals, and a DVS sub-pixel with the exposure duration of one-frame images, namely, each sub-pixel has the time of 1/n and sends the electric signals to the DVS circuit to generate dynamic visual information. In the exposure time of one frame of image, dynamic visual information and image gray signals are acquired synchronously in time.

Taking the four-in-one pixel group structure as an example, practical applications of the present invention include, but are not limited to, four-in-one pixel group arrangement, such that one pixel group includes four sub-pixels as shown in fig. 9. When the sub-pixels are output in the binning mode, the image gray signals corresponding to the four sub-pixels are combined into one signal, and each pixel group is output as an equivalent pixel. In the fourth example, a round-robin switching scheme of time-space multiplexing is adopted, as shown in fig. 9. Dividing the exposure time of each frame of image into four equal sub-frame time lengths T1, T2, T3 and T4, wherein in each sub-frame time length, at least one of four sub-pixels works in a DVS mode, and the rest of sub-pixels work in an APS mode; for example, during the sub-frame duration time T1, the first sub-pixel operates in the DVS mode, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel operate in the APS mode, and the image signal processing apparatus outputs the electric signal output by the first sub-pixel to the DVS circuit; the image signal processing device outputs the electric signal output by the second sub-pixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output from the fourth subpixel to the APS circuit. When the next sub-frame is long, namely T2, the second sub-pixel works in DVS mode, and the first sub-pixel is restored to APS mode; the image signal processing device outputs the electric signal output by the second sub-pixel to the DVS circuit; the image signal processing device outputs the electric signal output by the first sub-pixel to the APS circuit; similarly, the image signal processing device outputs the electric signal output by the third sub-pixel to the APS circuit; the image signal processing device outputs the electric signal output from the fourth subpixel to the APS circuit. And so on. At the output, the sub-pixels of each pixel group in the binning mode will combine the signals. The manner of combining the signals here includes, but is not limited to, addition or averaging of charges (for example, directly adding or averaging the electric signals output from the sub-pixels), addition or averaging of voltages (for example, adding or averaging the analog image gradation signals output from the APS circuit), addition or averaging of digital signals (for example, adding or averaging after the analog image gradation signals are converted into digital signals by the pixel peripheral circuit 45), and the like. Because each sub-pixel has three quarters of time to work in an APS mode, and an equivalent 3-binning effect can be obtained during output, the image sensor in the process is equivalent to the effect of realizing an equivalent 3-binning APS sub-pixel; meanwhile, each sub-pixel has a quarter of the time to work in the DVS mode, so the image sensor in the process is equivalent to a DVS sub-pixel which realizes one full-time work. In this embodiment, the image grayscale signal of the 3-binning APS subpixel is a time-space perfect match to the event signal of the equivalent DVS subpixel.

In the second embodiment of the present invention, taking fig. 6 and 7 as an example, when outputting in a binning size, each pixel group can output an image gray signal of 3-binning effect and an event signal that exactly matches it in time and space.

It is to be understood that, in the above embodiments, the method and/or the steps implemented by the image signal processing apparatus may also be implemented by a component (e.g., a chip or a circuit) that can be used in the image signal processing apparatus.

It is to be understood that the image signal processing apparatus includes hardware structures and/or software modules corresponding to the respective functions in order to implement the above-described functions. Those of skill in the art would readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the image signal processing apparatus may be divided into the functional modules according to the method embodiments, for example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.

Fig. 10 shows a schematic configuration diagram of an image signal processing apparatus. The image signal processing device is applied to an image sensor which comprises a plurality of pixel groups, wherein each pixel group comprises at least two sub-pixels, and each sub-pixel comprises a photosensitive circuit, and the device comprises: an acquisition module 1001 and a sending module 1002.

An obtaining module 1001, configured to obtain, at any time period in an exposure duration of a frame of image, an electrical signal emitted by a sub-pixel in a pixel group of an image sensor;

a sending module 1002, configured to send an electrical signal sent by any sub-pixel in the pixel group to the DVS circuit, where the DVS determines to generate dynamic visual information according to the electrical signal sent by the any sub-pixel;

the sending module 1002 is configured to send the electrical signals sent by the sub-pixels except for any sub-pixel in the pixel group to the APS circuit, where the APS circuit generates an image gray scale signal according to the electrical signals sent by the sub-pixels except for any sub-pixel in the pixel group.

Optionally, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image; the sending module 1002 is specifically configured to send an electrical signal sent by a first subpixel in the pixel group to the DVS circuit, where the DVS circuit determines to generate dynamic visual information according to the electrical signal sent by the first subpixel; the sending module 1002 is specifically configured to send an electrical signal sent by a second sub-pixel in the pixel group to the APS circuit, where the APS generates an image gray scale signal according to the electrical signal sent by the second sub-pixel.

Optionally, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration; the sending module 1002 is specifically configured to send the electrical signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame duration; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; the sending module 1002 is specifically configured to send, within an xth sub-frame duration, electrical signals sent by other sub-pixels except the xth sub-pixel to the APS circuit, where the APS generates an image gray signal corresponding to the xth sub-pixel according to signals sent by the xth sub-pixel within other sub-frame durations except the xth sub-frame duration.

Optionally, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image; the sending module 1002 is specifically configured to send an electrical signal sent by a first subpixel in the pixel group to the DVS, where the DVS determines to generate dynamic visual information according to the electrical signal sent by the first subpixel; the sending module 1002 is specifically configured to send the electrical signals sent by the other sub-pixels except the first sub-pixel in the pixel group to the APS circuit, where the APS generates an image gray signal according to the electrical signals sent by the other sub-pixels except the first sub-pixel in the exposure duration of the frame of image.

Optionally, the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration; the sending module 1002 is specifically configured to send the electrical signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame duration; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel; the sending module 1002 is specifically configured to send, within the xth sub-frame duration, the electrical signals sent by other sub-pixels except the xth sub-pixel to the APS circuit, where the APS circuit generates an image gray signal according to the electrical signals of all the sub-pixels received within the exposure duration of the frame image.

All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

As shown in fig. 11, an embodiment of the present application provides a hardware structure diagram of an image signal processing apparatus.

The image signal processing apparatus includes at least one processor (exemplarily illustrated in fig. 11 by including one processor 1101) and at least one transmission interface 1103 (for example, may be an interface circuit, exemplarily illustrated in fig. 11 by including one transmission interface 1103). Optionally, the image signal processing apparatus may further include at least one memory (fig. 11 exemplarily illustrates that one memory 1102 is included).

The processor 1101, the memory 1102 and the transmission interface 1103 are connected by a communication line. The communication link may include a path for transmitting information between the aforementioned components.

The processor 1101 may be a general processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs in accordance with the present disclosure. In a particular implementation, processor 1101 may also include multiple CPUs, and processor 1101 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor, as one embodiment. A processor herein may refer to one or more devices, circuits, or processing cores that process data (e.g., computer program instructions).

The memory 1102 may be a device having a storage function. Such as, but not limited to, read-only memory (ROM) or other types of static memory devices that may store static information and instructions, Random Access Memory (RAM) or other types of dynamic memory devices that may store information and instructions, electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 1102 may be separate and coupled to the processor 1101 via a communication link. The memory 1102 may also be integrated with the processor 1101.

The memory 1102 is used for storing computer-executable instructions for executing the present invention, and is controlled by the processor 1101. Specifically, the processor 1101 is configured to execute computer-executable instructions stored in the memory 1102, so as to implement the image signal processing method described in the embodiment of the present application.

Alternatively, in this embodiment of the application, the processor 1101 may perform a function related to processing in an image signal processing method provided in an embodiment of the application described below, and the transmission interface 1103 is responsible for being connected to other components to implement signal transmission, for example, transmitting an electrical signal of a sub-pixel to an image signal processing apparatus, which is not specifically limited in this embodiment of the application.

Optionally, the computer execution instruction in the embodiment of the present application may also be referred to as an application program code or a computer program code, which is not specifically limited in the embodiment of the present application.

In particular implementations, processor 1101 may include one or more CPUs such as CPU0 and CPU1 in fig. 11 for one embodiment.

In a specific implementation, the image signal processing apparatus may include a plurality of processors, such as the processor 1101 and the processor 1104 in fig. 11, as an example. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

The processor 1101 in the image signal processing apparatus may cause the image signal processing apparatus to execute the method in the above-described method embodiment by calling a computer stored in the memory 1102 to execute the instructions. Specifically, the functions/implementation processes of the acquiring module 1001 and the sending module 1002 in fig. 10 can be implemented by the processor 1101 in the image signal processing apparatus shown in fig. 11 calling the computer execution instructions stored in the memory 1102. Since the image signal processing apparatus provided in this embodiment can execute the method described above, the technical effects obtained by the image signal processing apparatus can refer to the method described above, and are not described herein again.

Optionally, an embodiment of the present application further provides an image signal processing apparatus (for example, the image signal processing apparatus may be a chip or a system-on-chip), where the image signal processing apparatus includes a processor, and is configured to implement the method in any of the method embodiments described above. In one possible design, the image signal processing apparatus further includes a memory. The memory is used for storing necessary program instructions and data, and the processor can call the program instructions stored in the memory to instruct the image signal processing device to execute the method in any method embodiment. Of course, the memory may not be in the image signal processing apparatus. When the image signal processing apparatus is a chip system, the image signal processing apparatus may be composed of a chip, or may include a chip and other discrete devices, which is not specifically limited in this embodiment of the present application.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application are all or partially generated upon loading and execution of computer program instructions on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or can comprise one or more data storage devices, such as a server, a data center, etc., that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others. In the embodiment of the present application, the computer may include the aforementioned apparatus.

While the present application has been described in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a review of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Although the present application has been described in conjunction with specific features and embodiments thereof, it will be evident that various modifications and combinations can be made thereto without departing from the spirit and scope of the application. Accordingly, the specification and figures are merely exemplary of the present application as defined in the appended claims and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. An image signal processing method applied to an image sensor, wherein the image sensor comprises a plurality of pixel groups, each of the pixel groups comprises at least two sub-pixels, and each of the sub-pixels comprises a photosensitive circuit, the method comprising:

acquiring an electric signal emitted by a sub-pixel in a pixel group of an image sensor at any time period in the exposure duration of a frame of image;

sending an electric signal sent by any sub-pixel in the pixel group to a dynamic vision sensor DVS circuit, wherein the DVS determines to generate dynamic vision information according to the electric signal sent by any sub-pixel;

and sending the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group to an Active Pixel Sensor (APS) circuit, wherein the APS circuit generates image gray signals according to the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group.

2. The image signal processing method according to claim 1, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image;

sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, including:

sending an electrical signal emitted by a first sub-pixel in the pixel group to the DVS circuit, wherein the DVS determines to generate dynamic visual information according to the electrical signal emitted by the first sub-pixel;

sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising:

sending the electrical signal sent by the second sub-pixel in the pixel group to the APS circuit;

wherein the APS generates an image gray signal according to the electrical signal emitted from the second sub-pixel.

3. The image signal processing method according to claim 1, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration;

sending the electric signal sent by any sub-pixel in the pixel group to the DVS circuit; the method comprises the following steps: sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS circuit determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel;

sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: sending the electric signals sent by other sub-pixels except the xth sub-pixel to the APS circuit within the xth sub-frame time length;

and the APS generates an image gray signal corresponding to the xth sub-pixel according to signals in other sub-frame time lengths except the xth sub-frame time length sent by the xth sub-pixel.

4. The image signal processing method according to claim 1, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image;

sending the electrical signal emitted by any sub-pixel in the pixel group to the DVS circuit, including: sending an electrical signal emitted by a first sub-pixel in the pixel group to the DVS circuit, wherein the DVS circuit determines to generate dynamic visual information according to the electrical signal emitted by the first sub-pixel;

sending the electric signals sent by other sub-pixels except any one sub-pixel in the pixel group to the APS circuit; the method comprises the following steps: sending the electric signals sent by other sub-pixels except the first sub-pixel in the pixel group to the APS circuit; and the APS circuit generates an image gray signal according to the electric signals sent by other sub-pixels except the first sub-pixel in the exposure time of the frame image.

5. The image signal processing method according to claim 1, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration;

sending the electric signal sent by any sub-pixel in the pixel group to the DVS circuit; sending the electric signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame time length; the DVS determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel;

sending the electrical signals sent by the sub-pixels except any one sub-pixel in the pixel group to the APS circuit, comprising: and sending the electric signals sent by other sub-pixels except the xth sub-pixel to the APS circuit within the xth sub-frame time, wherein the APS circuit generates an image gray signal according to the electric signals of all the sub-pixels received within the exposure time of the frame image.

6. An image signal processing device applied to an image sensor, wherein the image sensor comprises a plurality of pixel groups, wherein each pixel group comprises at least two sub-pixels, and each sub-pixel comprises a photosensitive circuit, the device comprises:

the acquisition module is used for acquiring electric signals sent by sub-pixels in a pixel group of the image sensor in any time period of the exposure duration of one frame of image;

a sending module, configured to send an electrical signal sent by any sub-pixel in the pixel group to a dynamic visual sensor DVS circuit, where the DVS determines to generate dynamic visual information according to the electrical signal sent by any sub-pixel;

the sending module is used for sending the electric signals sent by the sub-pixels except any sub-pixel in the pixel group to an active pixel sensor APS circuit, and the APS circuit generates image gray signals according to the electric signals sent by the sub-pixels except any sub-pixel in the pixel group.

7. The image signal processing apparatus according to claim 6, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image;

the sending module is specifically configured to send an electrical signal sent by a first subpixel in the pixel group to the DVS circuit, where the DVS circuit determines to generate dynamic visual information according to the electrical signal sent by the first subpixel;

the sending module is specifically configured to send an electrical signal sent by a second subpixel in the pixel group to the APS circuit, where the APS circuit generates an image grayscale signal according to the electrical signal sent by the second subpixel.

8. The image signal processing apparatus according to claim 6, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration;

the sending module is specifically configured to send the electrical signal sent by the xth sub-pixel to the DVS circuit within the xth sub-frame duration; the DVS circuit determines and generates dynamic visual information in the x subframe duration according to the electric signal sent by the x sub-pixel;

the sending module is specifically configured to send, within an xth sub-frame duration, electrical signals sent by other sub-pixels except the xth sub-pixel to the APS circuit, and the APS circuit generates an image gray signal corresponding to the xth sub-pixel according to signals sent by the xth sub-pixel within other sub-frame durations except the xth sub-frame duration.

9. The image signal processing apparatus according to claim 6, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the duration of any time interval is equal to the exposure duration of the frame of image;

the sending module is specifically configured to send the electrical signals sent by the other sub-pixels in the pixel group except the first sub-pixel to the APS circuit, where the APS circuit generates an image gray signal according to the electrical signals sent by the other sub-pixels except the first sub-pixel within the exposure duration of the frame of image.

10. The image signal processing apparatus according to claim 6, wherein the pixel group includes n sub-pixels, where n is a positive integer greater than or equal to 2; the exposure time of the frame of image comprises n subframe times; the duration of any time interval is equal to the subframe duration;

the sending module is specifically configured to send, within an xth sub-frame duration, electrical signals sent by other sub-pixels except the xth sub-pixel to the APS circuit, and the APS circuit generates an image gray signal according to signals of all sub-pixels received within an exposure duration of the frame of image.

11. An image signal processing apparatus, comprising: a processor and a transmission interface;

the processor is configured to invoke program instructions stored in the memory to perform the method of any of claims 1-5.

12. A computer-readable storage medium comprising instructions, in which program instructions are stored, which, when run on a computer or processor, cause the computer or processor to perform the method of any of claims 1-5.

13. An electronic device comprising an image sensor, the image signal processing apparatus according to any one of claims 6 to 11 connected to the image sensor; the pixel sensor comprises a pixel group arranged in an array, wherein the pixel group comprises at least two sub-pixels, and each sub-pixel comprises a photosensitive circuit.

14. The electronic device of claim 13, wherein the image sensor comprises a photosensitive layer and a sensor circuit layer on a side of the photosensitive layer opposite to light, wherein the photosensitive layer comprises the pixel groups arranged in the array; the sensor circuit layer includes circuit regions in one-to-one correspondence with the pixel groups, the circuit regions including a DVS circuit.

15. The electronic device of claim 14, wherein the circuit area further comprises an APS circuit.