WO2022135359A1

WO2022135359A1 - Dual-mode image signal processor and dual-mode image signal processing system

Info

Publication number: WO2022135359A1
Application number: PCT/CN2021/139857
Authority: WO
Inventors: 施路平; 杨哲宇; 赵蓉; 何伟; 王韬毅; 祝夭龙
Original assignee: 北京灵汐科技有限公司
Priority date: 2020-12-21
Filing date: 2021-12-21
Publication date: 2022-06-30

Abstract

Disclosed in the present application are a dual-mode image signal processor and an image sensor. The dual-mode image signal processor comprises: a synchronous image signal processor used for processing a synchronous signal in a dual-mode image signal; and an asynchronous image signal processor used for processing an asynchronous signal in the dual-mode image signal. The dual-mode image signal processor in the solution can simultaneously perform processing on the synchronous signal and the asynchronous signal in the dual-mode image signal, such that the efficiency of processing the dual-mode image signal by the dual-mode image signal processor can be improved, and the efficiency and real-time performance of dual-mode image signal processing are further improved. The dual-mode image sensor processed by the dual-mode image signal processor can not only achieve high signal fidelity when capturing an image at a high speed, but also achieve a high dynamic range and a high temporal resolution.

Description

Dual-modality image signal processor and dual-modality image signal processing system

technical field

Embodiments of the present invention relate to the technical field of image sensing, and in particular, to a dual-modality image signal processor and a dual-modality image signal processing system.

Background technique

Vision sensor refers to an instrument that uses optical elements and imaging devices to obtain image signals of the external environment. Referring to FIG. 1 , visual sensors in some related technologies generally include: Active Pixel Sensor (APS) and Dynamic Vision Sensor (DVS). Among them, the active pixel sensor is usually an image sensor based on a voltage signal or a current signal, and is widely used in the camera unit of a mobile phone or camera. This type of image sensor has the advantages of high color reproduction and high image quality. The dynamic range is smaller and the shooting speed is slower. Dynamic vision sensors are often used in the field of industrial control. They are characterized by the ability to perceive dynamic scenes. Due to the fast shooting speed and the large dynamic range of the acquired image signals, the image quality collected by such sensors is poor.

In some related technologies, the image signal processor used to process the image signal provided by the active pixel sensor is based on the "von Neumann" architecture, the calculation and storage are separated, the structure is simple, and it is easy to realize high-speed numerical calculation. However, when processing the image signal provided by the dynamic vision sensor, because the image signal of the dynamic vision sensor includes unstructured, space-time related information, the processor based on the "von Neumann" architecture exhibits low efficiency and high energy consumption. , poor real-time performance, etc. For example, when the image signal that the image signal processor needs to process includes high-speed, high-dynamic, low-resolution optical flow visual information (ie, the event-based image signal output by the dynamic vision sensor), the image signal processor in the related art cannot fully Due to the sparsity in the event-based image signal, the event-based image signal output by the dynamic vision sensor cannot be processed efficiently in real time, thus affecting the timeliness and time resolution of the image signal processing.

SUMMARY OF THE INVENTION

The invention provides a dual-mode image signal processor and a dual-mode image signal processing system, so as to improve the timeliness and time resolution of the dual-mode image signal processor.

In a first aspect, an embodiment of the present invention provides a dual-mode image signal processor, which includes: a synchronous image signal processor for processing a synchronization signal in the dual-mode image signal; an asynchronous image signal processor for Asynchronous signals in the bimodal image signal are processed.

Further, the synchronization signal in the dual-mode image signal is a color image signal, and the asynchronous signal in the dual-mode image signal is a grayscale gradient image signal.

Further, the dual-mode image signal processor in the embodiment of the present invention further includes: an adjustment unit configured to adjust the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor, The adjusted bimodal image signal is output.

Further, the adjustment unit is configured to fuse the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor to form a fusion signal; the adjusted dual-modal image output by the adjustment unit The signal is a fusion signal.

Further, the adjustment unit includes: a white balance subunit, configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor to form a fusion signal.

Further, the adjustment unit further includes at least one of the following: an exposure sub-unit for adjusting the exposure of the white-balanced fusion signal; and a focusing sub-unit for performing focus adjustment on the white-balanced fusion signal.

Further, the adjusted dual-mode image signal output by the adjustment unit includes a synchronous signal and an asynchronous signal.

Further, the adjustment unit includes: a white balance subunit, configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor.

Further, the adjustment unit further includes at least one of the following: an exposure sub-unit for performing exposure adjustment on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor; The unit is configured to perform focus adjustment on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor.

Further, the dual-modality image signal processor according to the embodiment of the present invention further includes: a wide dynamic range and spatiotemporal interpolation unit, configured to perform dynamic range adjustment and spatiotemporal interpolation on the adjusted dual-modality image signal output by the adjustment unit process, and output the processed dual-mode image signal; wherein, the processed dual-mode image signal is a fusion signal, or includes a synchronous signal and an asynchronous signal.

Further, the dual-modal image signal processor according to the embodiment of the present invention further includes: an encoding and compressing unit, configured to encode and compress the processed bi-modal image signal output by the wide dynamic range and spatiotemporal interpolation unit, and output to the external interface.

Further, the external interface includes at least one of the following: a universal serial bus interface, an Ethernet interface, and a high-definition multimedia interface.

Further, the dual-mode image signal processor according to the embodiment of the present invention further includes: a control unit configured to adjust the image sensor according to the adjusted dual-mode image signal output by the adjustment unit; A sensor for dual-modal image signals.

Further, the dual-modal image signal processor of the embodiment of the present invention further includes: a neural network unit, configured to process the synchronous signal output by the synchronous image signal processor and the output of the asynchronous image signal processor according to a neural network algorithm. Asynchronous signal, and output the dual-modal image signal processed by neural network.

Further, the neural network unit includes: an artificial neural network subunit for processing the synchronization signal output by the synchronous image signal processor according to an artificial neural network algorithm; an impulse neural network subunit for processing according to the impulse neural network algorithm The asynchronous signal output by the asynchronous image signal processor.

Further, the neural network unit further includes: a fusion subunit, configured to fuse the synchronous signal output by the artificial neural network subunit and the asynchronous signal output by the pulse neural network subunit to form a fusion signal, and according to the pulse A neural network algorithm processes the fused signal.

Further, the dual-mode image signal processor of the embodiment of the present invention further includes a control unit and an input and output bus; the control unit communicates with the synchronous image signal processor and the asynchronous image signal processor through the input and output bus. connected with the neural network unit; the control unit is used to control the synchronous image signal processor, the asynchronous image signal processor and the neural network unit according to the instruction information, and set the neural network according to the neural network parameters unit; the neural network unit outputs the dual-modal image signal processed by the neural network through the input and output bus.

Further, the dual-modal image signal processor of the embodiment of the present invention further includes a storage unit; the storage unit is connected to the control unit, and is used for caching the instruction information and the neural network parameter; the storage unit also It is connected to the input and output bus, and is used for buffering the dual-modal image signal processed by the neural network output by the neural network unit.

Further, the dual-mode image signal processor of the embodiment of the present invention further includes a mobile industry processor interface; the mobile industry processor interface is connected to the input and output bus; the dual-mode image signal passes through the mobile industry The processor interface and the I/O bus are transmitted to the synchronous image signal processor and the asynchronous image signal processor.

Further, the dual-mode image signal processor according to the embodiment of the present invention further includes an adjustment unit; the adjustment unit is configured to adjust the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor performing adjustment, and outputting the adjusted dual-mode image signal; the control unit is further configured to output the dual-mode image signal processed by the neural network according to the neural network unit and the adjusted dual-mode image signal output by the adjustment unit The image sensor is adjusted by the two-mode image signal; the image sensor is a sensor that collects the dual-mode image signal.

Further, the asynchronous image signal processor includes: an asynchronous signal encoding unit, configured to encode the asynchronous signal in the dual-modal image signal.

Further, the asynchronous image signal processor further includes: a pixel consistency correction unit, configured to perform pixel consistency correction on the asynchronous signal in the dual-modal image signal and output it to the asynchronous signal encoding unit.

Further, the asynchronous image signal processor further includes: a first black level correction unit, configured to perform black level correction on the asynchronous signal in the dual-mode image signal and output it to the pixel consistency correction unit .

Further, the synchronous image signal processor includes: a second black level correction unit, configured to perform black level correction on the synchronization signal in the dual-modal image signal and output it to the color interpolation unit; the color interpolation unit, It is used to perform color interpolation on the synchronization signal after black level correction and output to the color correction unit; the color correction unit is used to perform color correction on the synchronization signal after color interpolation.

In a second aspect, an embodiment of the present invention provides a dual-modality image signal processing system, which includes: any dual-modality image signal processor of the embodiments of the present invention; and an image sensor for acquiring the dual-modality image signal processor. image signal.

In the technical solution of the embodiment of the present invention, the dual-mode image signal processor includes two processors (a synchronous image signal processor and an asynchronous image signal processor), and the two processors are respectively used to process different signals (synchronous image signal processor and asynchronous image signal processor). signal and asynchronous signal), so each processor may have a special structure suitable for processing its corresponding signal; thus, the embodiment of the present invention can simultaneously process the synchronous signal and the asynchronous signal in the dual-mode image signal, The efficiency of the dual-mode image signal processor for processing the dual-mode image signal can be improved, thereby improving the efficiency and real-time performance of the dual-mode image signal processing. The dual-modality image signal processed by the dual-modality image signal processor can not only achieve high signal fidelity when capturing images at high speed, but also achieve high dynamic range and high temporal resolution at the same time.

Description of drawings

1 is a schematic structural diagram of an image signal processing system in some related technologies;

FIG. 2 is a schematic structural diagram of a dual-mode image signal processor according to an embodiment of the present invention;

3 is a schematic structural diagram of another dual-mode image signal processor provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a dual-modal image signal processor according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a dual-modal image signal processor according to an embodiment of the present invention;

6 is a schematic structural diagram of a dual-modal image signal processor provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a dual-modal image signal processor according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a dual-mode image signal processor according to an embodiment of the present invention;

9 is a schematic structural diagram of a dual-mode image signal processor provided by an embodiment of the present invention;

10 is a schematic structural diagram of a dual-mode image signal processor provided by an embodiment of the present invention;

11 is a schematic structural diagram of a dual-mode image signal processor provided by an embodiment of the present invention;

12 is a schematic structural diagram of a dual-mode image signal processor provided by an embodiment of the present invention;

13 is a schematic structural diagram of a dual-mode image signal processor provided by an embodiment of the present invention;

FIG. 14 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

FIG. 1 is a schematic structural diagram of an image signal processing system in the related art. As shown in FIG. 1 , the image signal processing system includes an active pixel sensor and an image signal processor.

Among them, the active pixel sensor generates a synchronization signal, and the synchronization signal contains all the color information of all the pixels of the image. The image signal processor can be a "von Neumann" architecture processor, which is used to perform a series of image processing on the synchronization signal generated by the active pixel sensor (for example: black level correction, color interpolation, color correction, white Balance, exposure, focus, etc.) and output, the color of the image can be restored with high precision, and a color image with higher quality can be obtained. However, when the image signal processor is used to process the asynchronous signal with high speed, high dynamic and low resolution, the image signal processor cannot make full use of the sparseness of the asynchronous signal, and cannot process the asynchronous signal efficiently in real time, thus affecting the image. Timeliness and time resolution of signal processing.

The present invention improves the above-mentioned defects existing in the existing image signal processor, and designs a dual-mode image signal processor, which can process the synchronous signal and the asynchronous signal in the dual-mode image signal at the same time, and can realize high-speed image capture. Very high signal fidelity, and at the same time high dynamic range and high temporal resolution can be achieved.

In a first aspect, an embodiment of the present invention provides a dual-modal image signal processor.

FIG. 2 is a schematic structural diagram of a dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 2 , the dual-mode image signal processor includes: a synchronous image signal processor 110 for processing a synchronization signal in a dual-mode image signal; an asynchronous image signal processor 120 for processing the dual-mode image signal Asynchronous signal in the state image signal.

Wherein, the dual-mode image signal includes a synchronous signal and an asynchronous signal. The dual-mode image signal processor includes a synchronous image signal processor 110 and an asynchronous image signal processor 120, and these two processors are used to process the synchronous signal and the asynchronous signal respectively, so each processor may have a The specialized structure of the corresponding signal.

Specifically, the synchronous image signal processor 110 processes the synchronous signal in the dual-modal image signal, and can make full use of the low-speed and high-resolution intensity information in the synchronous signal, that is, the image frame based on the pixel matrix, so as to improve the synchronous image The image quality of the processed synchronization signal output by the signal processor 110 . At the same time, the asynchronous image signal processor 120 can process the asynchronous signal in the dual-modal image signal, and can make full use of the high-speed, high-dynamic and low-resolution optical flow visual information in the asynchronous signal, that is, the event-based image signal, so as to improve the Dynamic range and temporal resolution of the processed asynchronous signal output by the asynchronous image signal processor 120 .

It can be seen from this that, by setting the synchronous image signal processor 110 and the asynchronous image signal processor 120 in the dual-mode image signal processor to process the synchronous signal and the asynchronous signal in the dual-mode image signal at the same time, not only can the dual-mode image signal processor be improved In addition, the asynchronous image signal processor 120 can make full use of the efficiency of the asynchronous signal when processing the asynchronous signal. sparseness, so that the dynamic range and time resolution of the processed asynchronous signal output by the asynchronous image signal processor 120 can be improved, so that the dual-modal image signal processed by the dual-modal image signal processor can not only capture images at high speed. Achieve high signal fidelity while simultaneously achieving high dynamic range and high temporal resolution.

Optionally, the synchronous signal in the dual-modal image signal is a color image signal, and the asynchronous signal in the dual-modal image signal is a grayscale gradient image signal.

Among them, the color image signal contains all the color information of all the pixels of the image, which can restore the color of the image with high precision and obtain an image with higher quality. The grayscale gradient image signal contains all the light intensity variation information of the image, which can reflect the grayscale variation of the image at high speed. By processing the gray gradient image signal by the asynchronous image signal processor 120, the sparsity of the gray gradient image signal can be fully utilized, and the dynamic range and time resolution of the processed asynchronous signal can be improved, so that the dual-modal image signal processor can Dual-modality image signals can be processed in real time with high-speed image capture, while achieving high fidelity, high dynamic range, and high temporal resolution of image signals.

Optionally, the asynchronous image signal processor 120 includes: an asynchronous signal encoding unit 124, configured to encode an asynchronous signal in the dual-modal image signal.

FIG. 3 is a schematic structural diagram of another dual-mode image signal processor provided in an embodiment of the present invention. As shown in FIG. 3 , the asynchronous image signal processor 120 includes an asynchronous signal encoding unit 124, and the asynchronous signal encoding unit 124 is configured to encode the asynchronous signal in the dual-modal image signal.

The asynchronous signal encoding unit 124 is used for encoding the asynchronous signal in the dual-mode image signal, that is, compressing the grayscale gradient image signal to a large extent, and then encoding it into an asynchronous mode to form an event signal in the form of an address.

The event signal in the form of an address can specifically be (X, Y, P, T), where "X, Y" is the event address, for example, "X, Y" can reflect the pixel position in the grayscale gradient image signal, and "P" For 4-value event output, for example, "P" can reflect the grayscale change of the pixel position, and "T" is the time when the event is generated, such as the shooting time. It can be seen from this that the information in the event signal in the form of an address includes characteristic information such as the event address, the event output and the time when the event is generated. The feature information of the event signal in the form of address can make full use of the discreteness and sparseness in the asynchronous signal, and improve the dynamic range and time resolution of the processed asynchronous signal, so that the dual-modal image signal processor can make the dual-mode image signal processor when shooting images at high speed. Achieve high signal fidelity while simultaneously achieving high dynamic range and high temporal resolution.

It should be noted that the address-form event signal in the above process is only an example, and in other embodiments, the address-form event signal may also be (X, Y, P) form, or (X, Y, P, ΔT) ) form, or (X, Y, ΔP, T) form, or (X, Y, ΔP, ΔT) form, etc. Among them, ΔP is the change of the output of the two 4-value events, and ΔT is the difference between the two events.

Optionally, the asynchronous image signal processor 120 further includes: a pixel consistency correction unit 123, configured to perform pixel consistency correction on the asynchronous signal in the dual-modal image signal and output it to the asynchronous signal encoding unit .

Optionally, continuing to refer to FIG. 3 , the asynchronous image signal processor 120 further includes a pixel consistency correction unit 123 for performing pixel consistency correction on the asynchronous signal in the dual-modal image signal and outputting it to the asynchronous signal encoding unit 124 .

The pixel consistency correction unit 123 can be connected to the asynchronous signal encoding unit 124. Before the asynchronous signal encoding unit 124 encodes the asynchronous signal in the dual-modal image signal, the pixel consistency correction unit 123 first encodes the dual-modal image signal. Pixel consistency correction is performed on the asynchronous signal in the asynchronous signal, which reduces the difference of the asynchronous signal caused by the pixel in the asynchronous signal, and reduces the influence of the pixel itself on the asynchronous signal. Then, the asynchronous signal after pixel consistency correction is encoded by the asynchronous signal encoding unit 124, which can improve the encoding accuracy of the asynchronous signal encoding unit 124.

Optionally, the asynchronous image signal processor 120 further includes: a first black level correction unit 122, configured to perform black level correction on the asynchronous signal in the dual-modal image signal and output it to the pixel consistent Sex correction unit.

Optionally, continuing to refer to FIG. 3 , the asynchronous image signal processor 120 further includes a first black level correction unit 122; the first black level correction unit 122 is connected to the pixel consistency correction unit 123, and is used for the dual mode image The asynchronous signal in the signal is subjected to black level correction and output to the pixel consistency correction unit 123 .

Among them, "black level" refers to the video signal level with no bright output on the display panel that has been calibrated to a certain extent, that is, the video signal level when the image data is 0. The main principle of black level correction is to first detect the level of the "light black" part of the luminance signal, and compare this level with the blanking level. To expand; if the blanking level has been reached, the expansion is stopped, that is, the blanking level is not exceeded. In this way, the original "light black" becomes "deep black", and the black level extension only changes the "light black" level in the luminance signal, while the white level and the luminance/chrominance signal ratio remain unchanged. Eliminates the blurry feeling of the image and improves the contrast of the image. It can be seen from this that the first black level correction unit 122 is used to adjust the image brightness in the asynchronous signal in the dual-modal image signal, so that the blurred shadow of the image is removed, the image becomes clearer, and the contrast of the image is improved. Then, the asynchronous signal of the first black level correction unit 122 to eliminate the blurring feature is output to the pixel consistency correction unit 123 for pixel consistency correction, which further improves the accuracy of the asynchronous signal.

Optionally, the synchronous image signal processor includes: a second black level correction unit 112, configured to perform black level correction on the synchronous signal in the dual-mode image signal and output it to the color interpolation unit 113; color The interpolation unit 113 is used to perform color interpolation on the synchronization signal after black level correction and output to the color correction unit 112; the color correction unit 114 is used to perform color correction on the synchronization signal after color interpolation.

Optionally, as shown in FIG. 3 , the synchronous image signal processor 110 includes a second black level correction unit 112 , a color interpolation unit 113 and a color correction unit 114 .

The second black level correction unit 112 is connected to the color interpolation unit 113 , and is used for performing black level correction on the synchronization signal in the dual-mode image signal and then outputting it to the color interpolation unit 113 ; the color interpolation unit 113 is connected to the color correction unit 114 , which is used to perform color interpolation on the synchronization signal after black level correction and output to the color correction unit 114 ; the color correction unit 114 is used to perform color correction on the synchronization signal after color interpolation.

Wherein, the second black level correction unit 112 is used to adjust the image brightness of the synchronization signal in the dual-modal image signal, so that the blurred shadow is removed from the image, the image becomes clearer, and the contrast of the image is improved.

Since each pixel can only sense one color after the action of the color filter, it is necessary to restore the information of the other two channels of the pixel and find the values of the other two channels of the point. Since the image changes continuously, the values of R, G, and B of a pixel should be related to the surrounding pixels, so the values of the surrounding pixels can be used to obtain the values of the other two channels at this point, In this way, the color is complemented and a more comprehensive image color information is obtained. Therefore, the color interpolation unit 113 can calculate the other two color components missing from each pixel according to the surrounding sampling points, so that a full-color synchronization signal can be obtained, the accuracy of the image color restored by the synchronization signal can be improved, and a more accurate image color can be obtained. High quality images.

The color correction unit 114 is to correct the color cast of the image, also known as color correction, which is an optical concept of the complementary color correction process of the three primary colors RGB and the three complementary colors CMY, which can ensure that the color of the synchronization signal can be accurately reproduced. Shooting what the human eye sees on site allows us to get a better and more accurate view.

In short, the synchronization signal in the dual-mode image signal first passes through the second black level correction unit 112 to compare the level of the "light black" part of the luminance signal with the blanking level to perform black level correction, Remove the blurred shadow of the image; then, the synchronizing signal processed by the second black level correction unit 112 is transmitted to the color interpolation unit 113 to restore the color information missing from the pixel, and obtain the color information of the complemented image; finally through the color interpolation unit The synchronizing signal processed by 113 is transmitted to the color correction unit 114 to correct the color cast of the image, so as to ensure that the color of the image can be more accurately reproduced as seen by the human eye at the shooting site.

It can be seen that after the synchronization signal in the dual-modal image signal is processed by the second black level correction unit 112, the color interpolation unit 113 and the color correction unit 114, a synchronization signal with better effect and more accurate color can be obtained.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes: an adjustment unit 130, configured to perform an adjustment on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor. Adjust, and output the adjusted bimodal image signal.

FIG. 4 is a schematic structural diagram of another dual-mode image signal processor provided in an embodiment of the present invention. As shown in FIG. 4 , the dual-mode image signal processor further includes an adjustment unit 130, and the adjustment unit 130 is configured to adjust the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 to adjust the Obtain the dual-modal image signal and output it.

Optionally, the adjustment unit 130 is configured to fuse the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 to form a fusion signal; The dual-modal image signal is a fusion signal.

As a way of the embodiment of the present invention, the adjustment unit 130 may fuse the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 to form a fusion signal. The bimodal image signal is the fusion signal.

The adjustment unit 130 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and can obtain the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120; thus, the adjustment unit 130 The feature information of the synchronous signal and the feature information of the asynchronous signal can be obtained separately, and the feature information of the asynchronous signal can be added to the feature information of the synchronous signal in a combined way, so as to complete the fusion of the synchronous signal and the asynchronous signal, and form a fusion signal. .

Since the fusion signal includes both the characteristic information of the synchronous signal and the characteristic information of the asynchronous signal, when the image obtained by the fusion signal (for example, the image is displayed according to the fusion signal), the image can use the pixel matrix-based pixel matrix in the synchronization signal. The image frame can improve the quality of the image, and the event-based image signal in the asynchronous signal can be used to improve the dynamic range and time resolution of the image, so that the image has a high signal fidelity.

Optionally, the adjustment unit 130 includes: a white balance sub-unit 131, configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, form a fusion signal.

FIG. 5 is a schematic structural diagram of another dual-mode image signal processor provided in an embodiment of the present invention. As shown in FIG. 5 , the adjustment unit 130 includes a white balance sub-unit 131 ; the white balance sub-unit 131 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120 , and is used for synchronizing signals output by the synchronous image signal processor 110 Perform automatic white balance processing with the asynchronous signal output by the asynchronous image signal processor 120, and fuse in the process of white balance processing to form a fusion signal.

The white balance sub-unit 131 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and can acquire the synchronous signal and the asynchronous signal at the same time, so as to obtain the feature information in the synchronous signal and the feature information in the asynchronous signal respectively, and The feature information of the asynchronous signal is added to the feature information of the synchronous signal in a combined manner, so as to complete the fusion of the synchronous signal and the asynchronous signal to form a fusion signal. Then the white balance sub-unit 131 performs automatic white balance adjustment on the fusion signal. In the process of automatic white balance adjustment, the white balance sub-unit 131 automatically detects the color temperature value of the subject according to the light conditions of its lens and white balance sensor, thereby judging the shooting conditions, and selects the closest color tone setting, Corrected by the color temperature correction circuit to adjust the white balance to an appropriate position.

Among them, white balance refers to "regardless of any light source, white objects can be restored to white", and the color cast phenomenon that occurs when shooting under a specific light source is compensated by strengthening the corresponding complementary color. For example, the white balance setting of the camera can calibrate the deviation of color temperature, so we can adjust the white balance to achieve the desired picture effect when shooting.

The white balance sub-unit 131 can correct the color temperature according to the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, restore the color of the subject to be photographed, and enable shooting under different light source conditions or different dynamics The color of the picture is similar to that of the picture viewed by human eyes; or, the white balance sub-unit 131 can also obtain images with different color effects by controlling the color temperature.

Optionally, the adjustment unit 130 further includes at least one of the following: an exposure subunit 132 for performing exposure adjustment on the white-balanced fusion signal; and a focusing subunit 133 for performing focus adjustment on the white-balanced fusion signal.

Optionally, continuing to refer to FIG. 5 , the adjustment unit 130 further includes an exposure sub-unit 132; the exposure sub-unit 132 is connected to the white balance sub-unit 131 for performing automatic exposure adjustment on the fusion signal after the automatic white balance.

Among them, the physical meaning of exposure means that light causes a photosensitive structure (such as a photosensitive layer coated with a photosensitive chemical, or a photosensitive element) to generate a latent image. Therefore, the quality of the image is related to the exposure, that is, how much light should be passed through so that the photosensitive element can get a clear image. Automatic exposure is based on the exposure value of the image measured by the metering system, and automatically sets the shutter speed and aperture value according to the combination of shutter and aperture exposure set during production. The exposure sub-unit 132 is connected with the white balance sub-unit 131, and the automatic exposure adjustment is performed on the fusion signal after the automatic white balance, which can make the color of the picture appear more vivid, and the light with obvious directionality can improve the picture quality well. texture.

Optionally, continuing to refer to FIG. 5 , the adjustment unit 130 further includes a focusing subunit 133; the focusing subunit 133 is connected to the white balance subunit 131, and is used to perform automatic focusing adjustment on the fusion signal after the automatic white balance.

Among them, auto focus uses the principle of light reflection from objects, determines the distance of the subject according to the reflection of the subject, and then adjusts the lens combination according to the measured results to achieve focusing. This AF method has the characteristics of high speed, easy implementation and low cost. The focusing sub-unit 133 is connected to the white balance sub-unit 131, and performs automatic focusing adjustment on the fusion signal after the automatic white balance, which can make the image clearer.

Optionally, the adjusted dual-mode image signal output by the adjustment unit 130 includes a synchronous signal and an asynchronous signal.

As another way of the embodiment of the present invention, the adjustment unit 130 may also adjust the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, but ultimately does not adjust the two The device is fused, so that the adjusted dual-mode image signal it outputs still includes a synchronous signal and an asynchronous signal.

The adjustment unit 130 can acquire the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, and process the synchronous image signal according to the acquired feature information in the synchronous signal and the feature information in the asynchronous signal. Adjust the synchronous signal output by the processor 110 and output the adjusted synchronous signal; and adjust the asynchronous signal output by the asynchronous image signal processor 120 and output the adjusted asynchronous signal; thus the adjustment unit 130 can output all color information at the same time The characteristic synchronous signal and the asynchronous signal with all gray gradient information characteristics realize the automatic adjustment of the synchronous signal and the asynchronous signal.

Optionally, the adjusting unit 130 includes: a white balance subunit 131 , configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 .

FIG. 6 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 6 , when fusion is not performed, the adjustment unit 130 may also include a white balance sub-unit 131; the white balance sub-unit 131 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and is used for The synchronous signal output by the image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 perform automatic white balance processing.

Optionally, the adjustment unit 130 further includes at least one of the following: an exposure sub-unit 130 configured to perform exposure adjustment on the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 ; The focusing sub-unit 132 is used to adjust the focus of the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 .

Optionally, continuing to refer to FIG. 6 , when the fusion is not performed, the adjustment unit 130 may further include an exposure sub-unit 132, and the exposure sub-unit 132 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120 for respectively Automatic exposure adjustment is performed on the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 .

Optionally, continuing to refer to FIG. 6 , when fusion is not performed, the adjustment unit 130 further includes a focusing sub-unit 133; the focusing sub-unit 133 can be connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120 for respectively The synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 are used for auto focus adjustment.

The meanings and effects of various adjustments (white balance, exposure, focus) have been described before, and will not be repeated here.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes: a wide dynamic range and spatiotemporal interpolation unit 150, configured to perform dynamic range adjustment on the adjusted dual-mode image signal output by the adjustment unit 130 and spatiotemporal interpolation processing, and output the processed dual-mode image signal; wherein, the processed dual-mode image signal is a fusion signal, or includes a synchronous signal and an asynchronous signal.

FIG. 7 is a schematic structural diagram of another dual-mode image signal processor provided in an embodiment of the present invention. As shown in FIG. 7 , the dual-modal image signal processor further includes a wide dynamic range and spatio-temporal interpolation unit 150; The signal is processed by dynamic range adjustment and spatiotemporal interpolation, and the processed bimodal image signal is output.

The processed dual-modal image signal output by the wide dynamic range and spatiotemporal interpolation unit 150 may include a synchronous signal and an asynchronous signal. For example, the wide dynamic range and spatiotemporal interpolation unit 150 may output a synchronous signal and an asynchronous signal to the adjustment unit 130 The signals are processed and output separately, or the wide dynamic range and spatiotemporal interpolation unit 150 may also process and decompose the fused signal output by the adjustment unit 130 to obtain a synchronous signal and an asynchronous signal.

Alternatively, the processed bimodal image signal output by the wide dynamic range and spatiotemporal interpolation unit 150 may also be a fusion signal. For example, the wide dynamic range and spatiotemporal interpolation unit 150 may directly process the fusion signal output by the adjustment unit 130 .

Among them, high dynamic range (High-Dynamic Range, HDR for short), also known as wide dynamic range technology, is a technology used to allow the camera to "see" the characteristics of the image under very strong contrast. "Dynamic range" refers to the adaptability of the camera to the light reflection of the scene in the shooting scene, and specifically refers to the variation range of brightness (contrast) and color temperature (contrast). Since missing data will reduce the integrity of spatiotemporal data, spatiotemporal interpolation is widely used in interpolation and estimation of missing spatiotemporal datasets. The wide dynamic range and spatiotemporal interpolation unit 150 is connected to the adjustment unit 130, and is used for performing dynamic range adjustment and spatiotemporal interpolation processing on the dual-modal image signal (such as a fusion signal, or a synchronous signal and an asynchronous signal) output by the adjustment unit 130 to obtain More dynamic range and image detail, higher temporal resolution, reflecting more realistic visuals.

In addition, the wide dynamic range and spatio-temporal interpolation unit 150 can output the processed fused signal, and can also output the processed synchronous signal and asynchronous signal respectively at the same time. When the wide dynamic range and spatiotemporal interpolation unit 150 outputs the fusion signal, the image signal with high fidelity, high dynamic range and high temporal resolution can be directly output; and when the wide dynamic range and spatiotemporal interpolation unit 150 simultaneously outputs the synchronization signal and When the asynchronous signal is used, the processed synchronous signal can be an image signal with high-precision restored image color, and the processed asynchronous signal can be an image signal with high dynamic range and high time resolution.

Optionally, the dual-modal image signal processor in the embodiment of the present invention further includes: an encoding and compressing unit, configured to encode and compress the processed dual-modal image signal output by the wide dynamic range and spatiotemporal interpolation unit, and output to the external interface.

FIG. 8 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 8 , the dual-modal image signal processor further includes a coding and compression unit 160 and an external interface 170; The output dual-mode image signal is encoded and compressed, and output to the external interface 170 .

The encoding and compression unit 160 is connected to the wide dynamic range and spatiotemporal interpolation unit 150, and can encode and compress the bimodal image signal output by the wide dynamic range and spatiotemporal interpolation unit 150 to eliminate a large amount of redundant information in the bimodal image signal. , the original data is represented by as few bytes as possible, so that the transmission efficiency of the image finally output from the external interface 170 can be improved.

The external interface 170 is a bridge for signal transmission between the dual-mode image signal processor and external devices, and is an input and output port for information interaction with external devices. An external device (such as a display, etc.) can acquire the dual-mode image signal processed by the dual-mode image signal processor by connecting with the external interface 170 .

In addition, corresponding to the form of the processed dual-mode image signal output by the wide dynamic range and spatiotemporal interpolation unit 150, the dual-mode image signal finally output from the external interface 170 may also be a fusion signal, or a synchronous signal and an asynchronous signal.

Optionally, the external interface includes at least one of the following: a universal serial bus interface, an Ethernet interface, and a high-definition multimedia interface.

Among them, the Universal Serial Bus (USB) interface is a serial bus standard for connecting computer systems and external devices, and is also a technical specification for input and output interfaces, which is widely used for information transmission between communication devices. The Ethernet interface is the port for network data connection. The Ethernet protocol defines a series of software and hardware standards, so that different communication devices are connected together through the Ethernet interface for information transmission. High Definition Multimedia Interface (HDMI) is a fully digital video and sound transmission interface, which can send uncompressed audio and video signals, and can send audio and video signals at the same time. Since the audio and video signals use the same wire, It greatly simplifies the installation difficulty of the system circuit.

It should be noted that the above-mentioned external interface types are only examples of the types of external interfaces. In specific implementation, the external interfaces need to be selected and set according to information transmission requirements.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes: a control unit 140, configured to adjust the image sensor according to the adjusted dual-mode image signal output by the adjustment unit 130; the image sensor is A sensor that acquires the dual-modality image signal.

FIG. 9 is a schematic structural diagram of another dual-mode image signal processor provided in an embodiment of the present invention. As shown in FIG. 9 , when there is an adjustment unit 130, the dual-mode image signal processor further includes a control unit 140, the control unit 140 is connected to the adjustment unit 130, and is used for adjusting the dual-mode image signal (for example, the fusion signal according to the adjustment unit 140). , or synchronous and asynchronous signals) to adjust the image sensor.

The image sensor is a sensor that acquires the dual-modal image processed by the dual-modal image signal processor of the embodiment of the present invention, that is, the dual-modal image signal processor can process the acquired dual-modal image, and according to the processing The resulting "feedback" adjusts the image sensor that acquired the dual-modality image signal.

The image sensor may be a dual-mode image sensor, that is, the image sensor includes a cone cell circuit and a rod cell circuit; the cone cell circuit is used to collect image color signals to form a synchronization signal in the dual-mode image signal, and the rod cells The cellular circuit is used to acquire grayscale gradient image signals to form asynchronous signals in the dual-modality image signal. The image sensor can also be composed of two independent sub-sensors. One of the sub-sensors includes a cone cell circuit, which is used to collect image color signals and form a synchronization signal in the dual-modal image signal; the other sub-sensor includes rod cells. The circuit is used to collect the grayscale gradient image signal to form an asynchronous signal in the dual-modal image signal.

The control unit 140 may be a control interface, and may acquire the dual-mode image signal output by the adjustment unit 130 . The dual-mode image signal may include the dual-mode image signal output by the exposure sub-unit 132 after automatic exposure adjustment, and may also include the dual-mode image signal output by the focusing sub-unit 133 after auto-focus adjustment. In this way, the control unit 140 can form a control signal for controlling the exposure parameters and focus parameters of the image sensor according to the dual-modality image signal after automatic exposure adjustment and the dual-modality image signal after automatic focus adjustment, and feed it back to the image sensor, thereby The exposure time and focal length of the image sensor are controlled according to the set exposure parameters and focus parameters.

Exemplarily, if the image sensor is a dual-mode image sensor, the dual-mode image sensor includes a cone cell circuit and a rod cell circuit, and the control unit 140 adjusts the dual-mode image signal according to the The dual-mode image signal forms a control signal for controlling the exposure parameters and focus parameters of the dual-mode image sensor, and is fed back to the dual-mode image sensor to control the dual-mode image sensor according to the set exposure parameters and focus parameters of the dual-mode image signal. Exposure time and focal length of the modal image sensor. If the image sensor is composed of two independent sub-sensors, one of which includes a cone cell circuit, and the other sub-sensor includes a rod cell circuit, the control unit 140 can adjust the automatic exposure according to the dual-mode image signal. The sync signal feature information, and the sync signal feature information in the auto-focus adjusted dual-modality image signal form a control signal that controls the exposure parameters and focus parameters of the image sensor including the cone circuit, and is adjusted according to the auto-exposure-adjusted dual-mode image signal. The asynchronous signal feature information in the modal image signal and the asynchronous signal feature information in the auto-focus adjusted dual-modal image signal form a control signal for controlling the exposure parameters and focus parameters of the image sensor including the rod cell circuit, respectively controlling Exposure time and focal length of the two sub-sensors in the image sensor.

It should be understood that the dual-modal image signal processor in the embodiment of the present invention may include the control unit 140 and the wide dynamic range and spatiotemporal interpolation unit 150 (as well as the encoding and compression unit 160 and the external interface 170) at the same time. In this case, referring to FIG. 8 , The control unit 140 and the wide dynamic range and spatiotemporal interpolation unit 150 should both be connected to the output of the adjustment unit 130 .

Optionally, referring to FIG. 10 , the dual-modal image signal processor in this embodiment of the present invention further includes: a neural network unit 180 for processing the synchronization signal output by the synchronized image signal processor 110 and the synchronization signal according to a neural network algorithm. The asynchronous image signal processor 120 outputs the asynchronous signal, and outputs the dual-modal image signal processed by the neural network.

In some related technologies, the intelligent image signal processor can only support the artificial neural network (ANN) represented by the convolutional neural network (CNN) to realize the machine learning task of traditional color images. However, the color image signal Due to the large amount of data, the intelligent image signal processor with artificial neural network is difficult to improve the frame rate of color image signals while ensuring image processing effect and low power consumption, and thus cannot process image signals at high speed and low power consumption.

Referring to FIG. 10 , the dual-modal image signal processor of the embodiment of the present invention further includes a neural network unit 180, which is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, so that the neural network unit 180 can pass through different neural networks. The network algorithm processes the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 simultaneously.

In the embodiment of the present invention, by setting the neural network unit 180 and using different neural network algorithms to process the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 respectively, the asynchronous signal can be extracted to the maximum extent. The sparseness, high speed and high dynamic range characteristics of the synchronous signal, as well as the high spatial resolution of the synchronization signal, can realize real-time marking of regions of interest (ROI), target recognition and other artificial intelligence-related tasks on the basis of ensuring low power consumption. , effectively improving the processing efficiency of dual-modal image signals.

It should be understood that, when the neural network unit 180 is provided, the dual-modal image signal processor may further include other units in the embodiments of the present invention. For example, referring to FIGS. 12 and 13 , the dual-modal image signal processor may also include one or more of an adjustment unit 130 , a control unit 140 , a wide dynamic range and spatiotemporal interpolation unit 150 , a coding and compression unit 160 , and an external interface 170 kind.

It should be understood that when the neural network unit 180 is provided, other units in the dual-modal image signal processor may also be specific forms in the embodiments of the present invention. For example, referring to FIG. 11 , the synchronous image signal processor 110 may include the second black level correction unit 112 , the color interpolation unit 113 , the color correction unit 114 , and the asynchronous image signal processor 120 may include the first black level correction unit 122 , a pixel consistency correction unit 113 , and an asynchronous signal encoding unit 114 . For another example, when there is an adjustment unit 130, the adjustment unit 130 may also include (not shown in the figure) a white balance sub-unit 131, an exposure sub-unit 132, a focusing sub-unit 133, etc., and the exposure sub-unit 132 and the focusing sub-unit 133 It can be connected to the output of the white balance sub-unit 131, or it can be connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120 respectively.

Optionally, the neural network unit 180 includes: an artificial neural network subunit 181 for processing the synchronization signal output by the synchronous image signal processor according to an artificial neural network algorithm; The neural network algorithm processes the asynchronous signal output by the asynchronous image signal processor.

FIG. 11 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 11 , the neural network unit 180 includes an artificial neural network subunit 181 and a spiking neural network subunit 182; the artificial neural network subunit 181 is used to process a synchronous image signal processor according to an artificial neural network (ANN, such as CNN) algorithm The synchronous signal output by 110, the spiking neural network sub-unit 182 is used for processing the asynchronous signal output by the asynchronous image signal processor 182 according to the spiking neural network (SNN) algorithm.

The artificial neural network subunit 181 can process the synchronization signal output by the synchronous image signal processor 110 according to the artificial neural network algorithm, so as to extract the high spatial resolution feature of the synchronization signal and output the processed synchronization signal. The processed synchronization signal is a region of interest signal, which can realize regions of interest (ROI) marking, target recognition, image classification and other tasks.

The spiking neural network sub-unit 182 can process the asynchronous signal output by the asynchronous image signal processor 120 according to the spiking neural network algorithm, so as to extract the sparsity and discreteness of the asynchronous signal and output the processed asynchronous signal. The signal can improve the processing efficiency and energy efficiency of the image signal, so that the dual-mode image signal processor can complete the task of low power consumption and high-speed computing.

Optionally, the neural network unit 180 further includes: a fusion subunit 183, configured to fuse the synchronous signal output by the artificial neural network subunit 181 and the asynchronous signal output by the spiking neural network subunit 182 to form a fusion signal, and process the fusion signal according to the spiking neural network algorithm.

As shown in FIG. 11 , the neural network unit 180 further includes a fusion subunit 183, which is used to fuse the synchronous signal output by the artificial neural network subunit 181 and the asynchronous signal output by the spiking neural network subunit 182 to form a fusion signal, and The fusion signal is processed according to the spiking neural network algorithm.

The fusion subunit 183 can acquire the synchronous signal output by the artificial neural network subunit 181 and the asynchronous signal output by the spiking neural network subunit 182, and the fusion subunit 183 respectively acquires the characteristic information of the synchronization signal extracted by the artificial neural network subunit 181 and the feature information of the asynchronous signal extracted by the spiking neural network subunit 182, and the feature information of the synchronous signal extracted by the artificial neural network subunit 181 is added to the asynchronous signal extracted by the spiking neural network subunit 182 in a combined manner. In the feature information, the fusion of the synchronous signal processed by the artificial neural network subunit 181 and the asynchronous signal processed by the spiking neural network subunit 182 is completed to form a fusion signal; then, the characteristics of the synchronous signal are processed according to the spiking neural network algorithm. The fusion signal is formed by the information and the characteristic information of the asynchronous signal, and the combined information of the fusion signal is used to obtain the optical flow result and output the processed fusion signal.

Therefore, the processed fusion signal output by the fusion sub-unit 183 can enable the dual-modal image signal processor to complete the real-time marking of regions of interest (Region Of Interest, ROI), target recognition and other manual tasks on the basis of ensuring low power consumption Intelligent related tasks to effectively improve the quality and processing efficiency of pictures.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes a control unit 140 and an input and output bus 190; the control unit 140 communicates with the synchronous image signal processor 110, all the The asynchronous image signal processor 120 is connected to the neural network unit 180; the control unit 140 is configured to control the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit according to the instruction information 180 , and set the neural network unit 180 according to the neural network parameters; the neural network unit 180 outputs the dual-modal image signal processed by the neural network through the input and output bus 190 .

FIG. 12 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 12 , when the neural network unit 180 is provided, the dual-modal image signal processor may also include a control unit 140, and also include an input and output bus 190; the control unit 140, the synchronous image signal processor 110, the asynchronous image The signal processor 120 and the neural network unit 180 are all connected to the input and output bus 190, so that the control unit 140 can control the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 180 according to the instruction information; the control unit 140 is also used for The neural network unit 180 is set according to the neural network parameters, and the neural network unit 180 outputs the processed bimodal image signal through the input and output bus 190 .

Among them, the control unit 140 is mainly responsible for the process management of the program, is the command and control center of the entire dual-mode image signal processor, and is extremely important for coordinating the orderly work of the entire equipment. In addition, the control unit 140 is also used to process the feedback information of the dual-mode image signal processor, and can form a control signal according to the dual-mode image signal processed by the adjustment unit 130 and the neural network unit 180, and feed it back to the synchronous image signal processor 110 and an asynchronous image signal processor 120.

The I/O bus 190 is a common communication trunk line for transmitting information between various functional components of the entire dual-mode image signal processor. For example, the control unit 140 is connected to the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 180 through the input and output bus 190, so that the control unit 140, the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 180 are connected. Information transmission and interaction can be implemented between the neural network units 180 through the input and output bus 190 . The instruction information sent by the control unit 140 is respectively transmitted to the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 180 through the input and output bus 190, thereby controlling the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 180. The neural network unit 180 responds to and executes operations corresponding to the instructions. In addition, the control unit 140 can also set the neural network unit 180 according to the neural network parameters, so that the neural network unit 180 can process the dual-modal image signal according to the preset parameters, so as to realize the high-speed completion of real-time marking of interest on the basis of ensuring low power consumption AI-related tasks such as Region Of Interest (ROI), target recognition, etc., and then output the processed dual-modal image signal through the input and output bus 190 .

Therefore, the control unit 140 is equivalent to the brain of the entire system in the entire dual-mode image signal processor, and can issue instructions to each functional module to coordinate and control the fast and stable operation of the entire system. At the same time, the input and output bus 190 is equivalent to the aorta of the whole system, which can quickly transmit information to each functional module, realize the timely transmission of information, and ensure that the system processes information in an orderly manner.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes an adjustment unit 130; the adjustment unit 130 is configured to perform the adjustment of the synchronous signal output by the synchronous image signal processor 110 and the asynchronous image signal processor The asynchronous signal output by 120 is adjusted, and the adjusted dual-mode image signal is output; the control unit 140 is also used for the dual-mode image signal processed by the neural network output by the neural network unit 180 and the adjustment unit. The adjusted dual-mode image signal output in 130 adjusts an image sensor; the image sensor is a sensor that collects the dual-mode image signal.

FIG. 13 is a schematic structural diagram of another dual-mode image signal processor according to an embodiment of the present invention. As shown in FIG. 13 , with the neural network unit 180 and the control unit 140 , the dual-modal image signal processor further includes an adjustment unit 130 , and the adjustment unit 130 is used to adjust the synchronous signal and the asynchronous image output by the synchronous image signal processor 110 The asynchronous signal output by the signal processor 120 ; the control unit 140 is further configured to adjust the image sensor according to the dual-mode image signal output by the neural network unit 180 and the dual-mode image signal output by the adjusting unit 130 .

It should be understood that, referring to FIG. 13 , when the neural network unit 180 and the adjusting unit 130 are provided at the same time, the neural network unit 180 can receive the dual-modal image signal output by the adjusting unit 130 ; further, referring to FIG. 13 , at this time, it can also have a wide The dynamic range and spatio-temporal interpolation unit 150, and may also have further corresponding coding and compression unit 160, external interface 170, etc., and when having the wide dynamic range and spatio-temporal interpolation unit 150, the wide dynamic range and spatio-temporal interpolation unit 150 can receive the adjustment unit 130 outputs the bimodal image signal, and the neural network unit 180 can receive the bimodal image signal output by the wide dynamic range and spatiotemporal interpolation unit 150 .

The adjustment unit 130 , the WDR and spatiotemporal interpolation unit 150 , the encoding and compression unit 160 , and the external interface 170 may be directly connected to other corresponding units, or may be connected to the input and output bus 190 respectively.

When the neural network unit 180 is provided, the control unit 140 can also adjust and control the image sensor. At this time, the control unit 140 can not only control the image sensor according to the processed dual-modal image signal output by the adjustment unit 130, but also simultaneously The image sensor is controlled according to the dual-modal image signal processed by the neural network output by the neural network unit 180, so as to realize more precise control and adjustment of the image sensor.

Optionally, the dual-modal image signal processor in this embodiment of the present invention further includes a storage unit 200; the storage unit 200 is connected to the control unit 140, and is configured to cache the instruction information and the neural network parameters; The storage unit 200 is also connected to the input and output bus 190 for buffering the dual-modal image signal processed by the neural network output by the neural network unit 180 .

Optionally, continue to refer to FIG. 12 , the dual-modal image signal processor further includes a storage unit 200, the storage unit 200 is used to cache instruction information and neural network parameters; the storage unit 200 is connected to the control unit 140 and is used for the control unit 140. Provide instruction information and neural network parameters; the storage unit 200 is also connected to the input and output bus 190 for buffering the processed dual-modal image signal output by the neural network unit 180 .

The storage unit 200 is used for buffering data information in the dual-modal image signal processor. For example, data information such as instruction information of the control unit 140, neural network parameters, and dual-modal image signals are cached. The storage unit 200 is connected to the control unit 140, and can transmit the instruction information and neural network parameters buffered in the storage unit 200 to the control unit 140, and the control unit 140 assigns the functional modules to which each instruction information and neural network parameters need to be transmitted. In addition, the storage unit 200 can also receive data information. For example, the storage unit 200 is also connected to the input and output bus 190, and can receive the dual-modal image signal processed by the neural network unit 180 transmitted through the input and output bus 190, and according to a certain The dual-modal image signal processed by the neural network unit 180 is cached in the storage unit 200 in a storage manner.

Optionally, the dual-mode image signal processor in this embodiment of the present invention further includes a mobile industry processor interface 210; the mobile industry processor interface 210 is connected to the input and output bus 190; the dual-mode image signal passes through The mobile industry processor interface 210 and the I/O bus 190 are transmitted to the synchronous image signal processor 110 and the asynchronous image signal processor 120 .

Optionally, continuing to refer to FIG. 12 , the dual-mode image signal processor further includes a mobile industry processor interface 210; the mobile industry processor interface 210 is connected to the input and output bus 190, and the dual-mode image signal passes through the mobile industry processor interface 210. The sum input-output bus 190 is transmitted to the synchronous image signal processor 110 and the asynchronous image signal processor 120 .

Among them, the Mobile Industry Processor Interface 210 (MIPI) is an open standard developed for mobile application processors initiated by the MIPI Alliance, which specially adopts low-amplitude signal swing in high-speed (data transmission) mode. Standardizing the interfaces inside mobile devices, such as camera, display, baseband, and RF interfaces, can increase design flexibility while reducing cost, design complexity, and power consumption. Since the mobile industry processor interface 210 , the synchronous image signal processor 110 and the asynchronous image signal processor 120 are all connected to the input and output bus 190 , the mobile industry processor interface 210 can quickly transfer the dual-mode image signal through the input and output bus 190 . Transmission to the synchronous image signal processor 110 and the asynchronous image signal processor 120 can effectively improve the information transmission rate.

In the second aspect, referring to FIG. 14 , an embodiment of the present invention provides a dual-modality image signal processing system, which includes: any dual-modality image signal processor according to the embodiment of the present invention; an image sensor 300 for acquiring the bimodal image signal.

FIG. 14 is a schematic structural diagram of a dual-modal image signal processing system according to an embodiment of the present invention. As shown in FIG. 14 , the dual-mode image signal processing system includes any dual-mode image signal processor according to the embodiment of the present invention, and an image sensor 300; the image sensor 300 is connected to the dual-mode image signal processor, and the image sensor 300 is connected to the dual-mode image signal processor. 300 is used to obtain the dual-mode image signal and transmit it to the dual-mode image signal processor for processing (specifically, the synchronous image signal processor 110 is used to process the synchronous signal in the dual-mode image signal, and the asynchronous image signal processor 120 for processing asynchronous signals in dual-modal image signals).

Wherein, the image sensor 300 may be a dual-mode image sensor, and the dual-mode image sensor includes both a cone cell circuit and a rod cell circuit. The cone cell circuit is used to collect image color signals, and the rod cell circuit is used to collect grayscale gradient image signals, so that the dual-modal image sensor can simultaneously output a synchronous signal with all color information and an asynchronous signal with all grayscale gradient information. .

Alternatively, the image sensor 300 may be a sensor formed by a combination of an Active Pixel Sensor (APS) and a Dynamic Vision Sensor (DVS), and the APS includes a cone cell circuit that can output a synchronization signal with all color information , DVS includes a rod cell circuit that can output asynchronous signals with full gray gradient information. APS and DVS combine to output a synchronous signal with all color information and an asynchronous signal with all gray gradient information.

It should be understood that the dual-modality image signal processor included in the dual-modality image signal processing system may be in any form of the embodiments of the present invention. For example, when the dual-mode image signal processor includes the above control unit, the control unit can also control the image sensor 300, which will not be repeated here.

It should be understood that the above "dual-modality image signal processor" may be integrated into the "image sensor 300", that is, the dual-modality image signal processing system of the embodiment of the present disclosure may appear to be a "device" of an image sensor; or, The "dual-modal image signal processor" and the "image sensor 300" may also be two relatively independent "devices" (of course, information transmission should be possible between the two devices).

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made to those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope is determined by the scope of the appended claims.

Claims

A dual-mode image signal processor, comprising:

A synchronous image signal processor for processing the synchronous signal in the dual-modal image signal;

An asynchronous image signal processor for processing asynchronous signals in the dual-mode image signal.
The dual-modal image signal processor according to claim 1, wherein,

The synchronous signal in the dual-mode image signal is a color image signal, and the asynchronous signal in the dual-mode image signal is a grayscale gradient image signal.
The dual-mode image signal processor according to claim 1 or 2, further comprising:

The adjustment unit is configured to adjust the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor, and output the adjusted dual-mode image signal.
The dual-modal image signal processor according to claim 3, wherein,

The adjustment unit is configured to fuse the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor to form a fusion signal;

The adjusted dual-mode image signal output by the adjustment unit is a fusion signal.
The dual-modal image signal processor according to claim 4, wherein the adjustment unit comprises:

The white balance subunit is configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor to form a fusion signal.
The dual-modal image signal processor according to claim 5, wherein the adjustment unit further comprises at least one of the following:

The exposure sub-unit is used to adjust the exposure of the fusion signal after white balance;

The focusing sub-unit is used to adjust the focus of the fused signal after white balance.
The dual-modal image signal processor according to claim 3, wherein,

The adjusted dual-mode image signal output by the adjustment unit includes a synchronous signal and an asynchronous signal.
The dual-mode image signal processor according to claim 7, wherein the adjustment unit comprises:

The white balance subunit is configured to perform white balance processing on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor.
The dual-modal image signal processor according to claim 8, wherein the adjustment unit further comprises at least one of the following:

an exposure subunit, configured to perform exposure adjustment on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor;

The focusing subunit is configured to perform focus adjustment on the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor.
The dual-mode image signal processor according to claim 4, further comprising:

a wide dynamic range and spatiotemporal interpolation unit, configured to perform dynamic range adjustment and spatiotemporal interpolation processing on the adjusted bimodal image signal output by the adjusting unit, and output the processed bimodal image signal;

Wherein, the processed dual-modal image signal is a fusion signal, or includes a synchronous signal and an asynchronous signal.
The dual-modal image signal processor according to claim 10, further comprising:

The coding and compression unit is used for coding and compressing the processed bimodal image signal output by the wide dynamic range and spatiotemporal interpolation unit, and outputting it to an external interface.
The dual-modality image signal processor according to claim 11, wherein the external interface comprises at least one of the following:

Universal serial bus interface, Ethernet interface, high-definition multimedia interface.
The dual-mode image signal processor according to claim 3, further comprising:

The control unit is configured to adjust the image sensor according to the adjusted dual-mode image signal output by the adjustment unit; the image sensor is a sensor that collects the dual-mode image signal.
The dual-mode image signal processor according to claim 1 or 2, further comprising:

The neural network unit is configured to process the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor according to the neural network algorithm, and output the dual-modal image signal processed by the neural network.
The dual-modal image signal processor according to claim 14, wherein the neural network unit comprises:

An artificial neural network subunit for processing the synchronous signal output by the synchronous image signal processor according to an artificial neural network algorithm;

The spiking neural network subunit is used for processing the asynchronous signal output by the asynchronous image signal processor according to the spiking neural network algorithm.
The dual-modal image signal processor according to claim 15, wherein the neural network unit further comprises:

A fusion subunit, configured to fuse the synchronous signal output by the artificial neural network subunit and the asynchronous signal output by the spiking neural network subunit to form a fusion signal, and process the fusion signal according to the spiking neural network algorithm.
The dual-mode image signal processor according to claim 14, further comprising a control unit and an input and output bus;

The control unit is connected with the synchronous image signal processor, the asynchronous image signal processor and the neural network unit through the input and output bus;

The control unit is configured to control the synchronous image signal processor, the asynchronous image signal processor and the neural network unit according to the instruction information, and set the neural network unit according to the neural network parameters;

The neural network unit outputs the dual-modal image signal processed by the neural network through the input and output bus.
The dual-mode image signal processor of claim 17, further comprising a storage unit;

The storage unit is connected to the control unit, and is used to cache the instruction information and the neural network parameter;

The storage unit is also connected to the input and output bus, and is used for buffering the dual-modal image signal processed by the neural network output by the neural network unit.
The dual-modal image signal processor of claim 18, further comprising a mobile industry processor interface;

the mobile industry processor interface is connected to the input and output bus;

The dual-modal image signal is transmitted to the synchronous image signal processor and the asynchronous image signal processor through the mobile industry processor interface and the input and output bus.
The dual-mode image signal processor according to claim 17, further comprising an adjustment unit;

The adjustment unit is configured to adjust the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor, and output the adjusted dual-mode image signal;

The control unit is further configured to adjust the image sensor according to the dual-mode image signal processed by the neural network output by the neural network unit and the adjusted dual-mode image signal output by the adjustment unit; the dual-modal image signal sensor.
The dual-mode image signal processor according to claim 1 or 2, wherein the asynchronous image signal processor comprises:

An asynchronous signal encoding unit, configured to encode an asynchronous signal in the dual-mode image signal.
The dual-modal image signal processor according to claim 21, wherein the asynchronous image signal processor further comprises:

A pixel consistency correction unit, configured to perform pixel consistency correction on the asynchronous signal in the dual-modal image signal and output it to the asynchronous signal encoding unit.
The dual-modal image signal processor according to claim 22, wherein the asynchronous image signal processor further comprises:

The first black level correction unit is configured to perform black level correction on the asynchronous signal in the dual-mode image signal and output it to the pixel consistency correction unit.
The dual-mode image signal processor according to claim 1 or 2, wherein the synchronous image signal processor comprises:

a second black level correction unit, configured to perform black level correction on the synchronization signal in the dual-mode image signal and output it to the color interpolation unit;

The color interpolation unit is used to perform color interpolation on the synchronization signal after black level correction and output it to the color correction unit;

The color correction unit is used to perform color correction on the synchronization signal after color interpolation.
A dual-modal image signal processing system, comprising:

The dual-modal image signal processor of any one of claims 1 to 24;

an image sensor for acquiring the dual-modality image signal.