CN112702588B - Dual-mode image signal processor and dual-mode image signal processing system - Google Patents

Dual-mode image signal processor and dual-mode image signal processing system Download PDF

Info

Publication number
CN112702588B
CN112702588B CN202011520252.9A CN202011520252A CN112702588B CN 112702588 B CN112702588 B CN 112702588B CN 202011520252 A CN202011520252 A CN 202011520252A CN 112702588 B CN112702588 B CN 112702588B
Authority
CN
China
Prior art keywords
image signal
signal processor
asynchronous
synchronous
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011520252.9A
Other languages
Chinese (zh)
Other versions
CN112702588A (en
Inventor
施路平
杨哲宇
赵蓉
何伟
王韬毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Lynxi Technology Co Ltd
Original Assignee
Beijing Lynxi Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Lynxi Technology Co Ltd filed Critical Beijing Lynxi Technology Co Ltd
Priority to CN202011520252.9A priority Critical patent/CN112702588B/en
Publication of CN112702588A publication Critical patent/CN112702588A/en
Priority to PCT/CN2021/139857 priority patent/WO2022135359A1/en
Application granted granted Critical
Publication of CN112702588B publication Critical patent/CN112702588B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3182—Colour adjustment, e.g. white balance, shading or gamut
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/73—Colour balance circuits, e.g. white balance circuits or colour temperature control
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the invention discloses a bimodal image signal processor and a bimodal image signal processing system, wherein the bimodal image signal processor comprises: a synchronous image signal processor for processing a synchronous signal in the bimodal image signal; an asynchronous image signal processor for processing an asynchronous signal in the bimodal image signal; and the neural network unit is used for processing and outputting the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor according to a neural network algorithm. The dual-mode image signal processor in the scheme can simultaneously process the synchronous signal and the asynchronous signal in the dual-mode image signal, can realize high signal fidelity when shooting an image at high speed, and can realize high dynamic range and high time resolution. Moreover, the neural network unit can be used for marking interested areas and other artificial intelligence related tasks in real time on the basis of ensuring low power consumption.

Description

Dual-mode image signal processor and dual-mode image signal processing system
Technical Field
The embodiment of the invention relates to the technical field of image sensing, in particular to a bimodal image signal processor and a bimodal image signal processing system.
Background
A conventional Image Signal Processor (ISP) generally includes an Active Pixel Sensor (APS) and a Dynamic Vision Sensor (DVS). Among them, the active pixel sensor can only process color images arranged in a pixel matrix image frame mode, and has the advantages of high color reproduction degree and high image quality, but the dynamic range of the acquired image signals is small, and the shooting speed is slow. The dynamic vision sensor can sense a dynamic scene, the shooting speed is high, the dynamic range of the obtained image signal is large, however, the quality of the image acquired by the sensor is poor, and the resolution ratio is low. The traditional image signal processor cannot process active pixel sensor signals and dynamic visual receptor signals at the same time, and the traditional image signal processor does not integrate an on-chip neural network and cannot realize image processing tasks related to artificial intelligence.
An intelligent image signal processor in the prior art can only support processing of color images adopted by a traditional active pixel sensor, cannot support simultaneous processing of image information with sparse events adopted by a similar dynamic vision sensor, and further cannot fully utilize the characteristics of sparsity, high speed and high dynamic range in an event-based image signal, so that the processing effect of images and the quality of images cannot be guaranteed. In addition, the current intelligent image signal processor can only support an Artificial Neural Network (ANN) represented by a convolutional Neural Network to implement a machine learning task of a conventional color image, but the data size of color image information is large, and it is difficult for the intelligent image signal processor having the Artificial Neural Network to improve the frame rate of the color image information while ensuring an image processing effect and low power consumption, and thus it is impossible to process image information at high speed and low power consumption.
Disclosure of Invention
The invention provides a dual-mode image signal processor and a dual-mode image signal processing system, which are used for improving timeliness and time resolution of the dual-mode image signal processor, reducing power consumption, marking interested areas, identifying targets and other artificial intelligent related tasks in real time.
In a first aspect, an embodiment of the present invention provides a dual-modality image signal processor, including:
a synchronous image signal processor for processing a synchronous signal in the bimodal image signal;
an asynchronous image signal processor for processing an asynchronous signal in the bimodal image signal;
and the neural network unit is used for processing and outputting the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor according to a neural network algorithm.
Further, the neural network unit comprises an artificial neural network subunit and a pulse neural network subunit;
the artificial neural network subunit is used for processing the synchronous signals output by the synchronous image signal processor according to an artificial neural network algorithm, and the impulse neural network subunit is used for processing the asynchronous signals output by the asynchronous image signal processor according to an impulse neural network algorithm.
Furthermore, the neural network unit also comprises a fusion subunit, and the fusion subunit is used for fusing the synchronous signal output by the artificial neural network subunit and the asynchronous signal output by the pulse neural network subunit to form a fusion signal and processing the fusion signal according to a pulse neural network algorithm.
Further, the asynchronous image signal processor includes an asynchronous signal encoding unit for encoding an asynchronous signal in the bimodal image signal.
Furthermore, the asynchronous image signal processor also comprises a pixel consistency correction unit which is used for carrying out pixel consistency correction on the asynchronous signals in the bimodal image signals and then outputting the asynchronous signals to the asynchronous signal coding unit.
Further, the asynchronous image signal processor further includes a first black level correction unit;
the first black level correction unit is connected with the pixel consistency correction unit and is used for performing black level correction on an asynchronous signal in the bimodal image signal and outputting the asynchronous signal to the pixel consistency correction unit.
Further, the synchronized image signal processor includes a second black level correction unit, a color interpolation unit, and a color correction unit;
the second black level correction unit is connected with the color interpolation unit and is used for performing black level correction on the synchronous signals in the bimodal image signals and outputting the signals to the color interpolation unit;
the color interpolation unit is connected with the color correction unit and used for performing color interpolation on the synchronous signal after the black level correction and outputting the synchronous signal to the color correction unit;
and a color correction unit for performing color correction on the color-interpolated synchronization signal.
Further, the dual-modality image signal processor further includes an automatic adjustment unit for adjusting the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor.
Further, the automatic adjustment unit includes an automatic white balance subunit;
the automatic white balance subunit is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic white balance processing according to the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
Furthermore, the automatic adjusting unit also comprises an automatic exposure photon unit;
the automatic exposure subunit is connected with the automatic white balance subunit and is used for carrying out automatic exposure adjustment on the bimodal image signal subjected to automatic white balance; or the image exposure control device is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic exposure adjustment on the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
Furthermore, the automatic adjustment unit also comprises an automatic focusing subunit;
the automatic focusing subunit is connected with the automatic white balance subunit and is used for carrying out automatic focusing adjustment on the bimodal image signal subjected to automatic white balance; or the automatic focusing device is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic focusing adjustment on the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
Further, the bimodal image signal processor also comprises a wide dynamic range and spatio-temporal interpolation unit; the wide dynamic range and space-time interpolation unit is connected with the automatic adjustment unit and is used for carrying out dynamic range adjustment and space-time interpolation processing on the bimodal image signals output by the automatic adjustment unit and outputting the processed bimodal image signals; the processed bimodal image signal comprises a synchronous signal and an asynchronous signal or is a processed fusion signal.
Further, the bimodal image signal processor also comprises an encoding compression unit and an external interface; the coding compression unit is connected with the wide dynamic range and space-time interpolation unit and is used for coding and compressing the bimodal image signals output by the wide dynamic range and space-time interpolation unit and outputting the bimodal image signals to an external interface.
Further, the external interface includes at least one of a universal serial bus interface, an ethernet interface, and a high definition multimedia interface.
Further, the bimodal image signal processor also comprises a control unit and an input/output bus;
the control unit is connected with the synchronous image signal processor, the asynchronous image signal processor and the neural network unit through an input/output bus, and is used for controlling the synchronous image signal processor, the asynchronous image signal processor and the neural network unit according to the instruction information; the control unit is also used for setting a neural network unit according to the neural network parameters, and the neural network unit outputs the processed bimodal image signals through the input and output bus.
Further, the bimodal image signal processor also comprises a storage unit, wherein the storage unit is used for caching the instruction information and the neural network parameters; the storage unit is connected with the control unit and used for providing instruction information and neural network parameters for the control unit; the storage unit is also connected with the input/output bus and is used for caching the processed bimodal image signals output by the neural network unit.
Further, the dual-mode image signal processor also comprises a mobile industry processor interface; the interface of the mobile industry processor is connected with the input/output bus, and the bimodal image signal is transmitted to the synchronous image signal processor and the asynchronous image signal processor through the interface of the mobile industry processor and the input/output bus.
Furthermore, the dual-mode image signal processor also comprises an automatic adjusting unit, wherein the automatic adjusting unit is used for adjusting a synchronous signal output by the synchronous image signal processor and an asynchronous signal output by the asynchronous image signal processor;
the control unit is also used for adjusting the image sensor according to the bimodal image signal output by the neural network unit and the bimodal image signal output by the automatic adjusting unit.
In a second aspect, embodiments of the present invention also provide a dual-modality image signal processing system, including an image sensor and a dual-modality image signal processor implementing any one of the first aspects; the image sensor is connected with the bimodal image signal processor, the image sensor is used for acquiring bimodal image signals, the synchronous image signal processor is used for processing synchronous signals in the bimodal image signals, and the asynchronous image signal processor is used for processing asynchronous signals in the bimodal image signals; and the neural network unit is used for processing and outputting the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor according to a neural network algorithm.
The technical scheme of the embodiment of the invention can improve the efficiency of processing the bimodal image signal by the bimodal image signal processor and can also improve the efficiency and the real-time property of processing the bimodal image signal by setting the synchronous image signal processor and the asynchronous image signal processor in the bimodal image signal processor to simultaneously process the synchronous signal and the asynchronous signal in the bimodal image signal. Moreover, when the asynchronous image signal processor processes the asynchronous signals, the sparsity of the asynchronous signals can be fully utilized, so that the dynamic range and the time resolution of the processed asynchronous signals output by the asynchronous image signal processor can be improved, and the dual-mode image signals processed by the dual-mode image signal processor not only can realize high signal fidelity when images are shot at high speed, but also can realize high dynamic range and high time resolution. In addition, by setting a neural network unit in the dual-mode image signal processor and respectively processing the synchronous signal output by the synchronous image signal processor and the asynchronous signal output by the asynchronous image signal processor by using different neural network algorithms, the sparsity, high-speed and high-dynamic range characteristics Of the asynchronous signal and the high spatial resolution Of the synchronous signal can be extracted to the maximum extent, artificial intelligence related tasks such as Region Of Interest (ROI) marking and target identification can be realized in real time on the basis Of ensuring low power consumption, and the processing efficiency Of dual-mode image information is effectively improved.
Drawings
FIG. 1 is a schematic structural diagram of a bimodal image signal processor according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention;
FIG. 7 is a block diagram of an alternative bimodal image signal processor according to an embodiment of the present invention;
FIG. 8 is a block diagram of an alternative bimodal image signal processor according to an embodiment of the present invention;
FIG. 9 is a block diagram of an alternative dual-mode image signal processor according to an embodiment of the present invention;
FIG. 10 is a block diagram of an alternative bimodal image signal processor according to an embodiment of the present invention;
FIG. 11 is a block diagram of an alternative bimodal image signal processor according to an embodiment of the present invention;
FIG. 12 is a block diagram of an alternative bimodal image signal processor according to an embodiment of the present invention;
fig. 13 is a schematic structural diagram of a bimodal image signal processing system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a schematic structural diagram of a dual-modality image signal processor according to an embodiment of the present invention. As shown in fig. 1, the bimodal image signal processing includes a synchronous image signal processor 110 for processing a synchronous signal in the bimodal image signal; an asynchronous image signal processor 120 for processing an asynchronous signal in the bimodal image signal; and a neural network unit 130 for processing and outputting the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 according to a neural network algorithm.
Wherein the bimodal image signal includes a synchronous signal and an asynchronous signal. The bimodal image signal processor includes a synchronous image signal processor 110, an asynchronous image signal processor 120, and a neural network unit 130. The synchronous image signal processor 110 processes the synchronous signal in the bimodal image signal, and can make full use of the low-speed and high-resolution intensity information in the synchronous signal, i.e. the image frame based on the pixel matrix, to improve the image quality of the processed synchronous signal output by the synchronous image signal processor 110. Meanwhile, the asynchronous image signal processor 120 may process the asynchronous signal in the bimodal image signal, and may fully utilize the optical flow visual information with high speed, high dynamic state and low resolution in the asynchronous signal, i.e., the image signal based on the event, so as to improve the dynamic range and the time resolution of the processed asynchronous signal output by the asynchronous image signal processor 120. The neural network unit 130 respectively and simultaneously processes the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 through different neural network algorithms, and can respectively extract the characteristic information of the synchronous signal and the characteristic information of the asynchronous signal by fully utilizing the characteristics of high spatial resolution of the synchronous signal output by the synchronous image signal processor 110 and the sparseness, high speed and high dynamic range of the asynchronous signal output by the asynchronous image signal processor 120, thereby improving the processing efficiency of the bimodal image signal and realizing the real-time output of the information of the region of interest. Therefore, by arranging the synchronous image signal processor 110 and the asynchronous image signal processor 120 in the dual-modality image signal processor to simultaneously process the synchronous signal and the asynchronous signal in the dual-modality image signal, the efficiency of the dual-modality image signal processor for processing the dual-modality image signal can be improved, and the efficiency and the real-time performance of the dual-modality image signal processing can be further improved. Moreover, when the asynchronous image signal processor 120 processes the asynchronous signal, the sparsity of the asynchronous signal can be fully utilized, so that the dynamic range and the time resolution of the processed asynchronous signal output by the asynchronous image signal processor 120 can be improved, and the bimodal image signal processed by the bimodal image signal processor not only can realize high signal fidelity when an image is shot at a high speed, but also can realize a high dynamic range and a high time resolution. In addition, by setting the neural network unit 130 in the bimodal image signal processor, the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 are respectively and simultaneously processed by different neural network algorithms, so that the sparsity, high-speed and high-dynamic range characteristics Of the asynchronous signal and the high spatial resolution Of the synchronous signal can be maximally extracted, artificial intelligence related tasks such as Region Of Interest (ROI) marking and target identification can be realized in real time on the basis Of ensuring low power consumption, and the processing efficiency Of the bimodal image signal is effectively improved.
Fig. 2 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 2, the neural network unit 130 includes an artificial neural network subunit 131 and a spiking neural network subunit 132; the artificial neural network subunit 131 is configured to process the synchronous signal output by the synchronous image signal processor 110 according to an artificial neural network algorithm, and the impulse neural network subunit 132 is configured to process the asynchronous signal output by the asynchronous image signal processor 120 according to an impulse neural network algorithm.
The artificial neural network subunit 131 may process the synchronization signal output by the synchronization image signal processor 110 according to an artificial neural network algorithm. The artificial neural network sub-unit 131 may extract the high spatial resolution feature of the synchronization signal and output the processed synchronization signal. The processed synchronous signal is a region-of-interest signal, and can realize the tasks of region-of-interest (ROI) marking, target identification, image classification and the like. The impulse neural network sub-unit 132 may process the asynchronous signal output from the asynchronous image signal processor 120 according to an impulse neural network algorithm. The spiking neural network subunit 132 may extract sparsity and discreteness of the asynchronous signal and output a processed asynchronous signal, which may improve processing efficiency and energy efficiency of the image signal, so that the dual-mode image signal processor may complete tasks of low power consumption and high-speed computation.
Fig. 3 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 3, the neural network unit 130 further includes a fusion subunit 133, and the fusion subunit 133 is configured to fuse the synchronous signal output by the artificial neural network subunit 131 and the asynchronous signal output by the impulse neural network subunit 132 to form a fusion signal, and process the fusion signal according to an impulse neural network algorithm.
The fusion subunit 133 may obtain a synchronous signal output by the artificial neural network subunit 131 and an asynchronous signal output by the impulse neural network subunit 132, where the fusion subunit 133 obtains feature information of the synchronous signal extracted by the artificial neural network subunit 131 and feature information of the asynchronous signal extracted by the impulse neural network subunit 132, respectively, and adds the feature information of the synchronous signal extracted by the artificial neural network subunit 131 to the feature information of the asynchronous signal extracted by the impulse neural network subunit 132 in a combined manner, so as to complete the fusion of the synchronous signal processed by the artificial neural network subunit 131 and the asynchronous signal processed by the impulse neural network subunit 132, and form a fusion signal. And then processing a fusion signal formed by the characteristic information of the synchronous signal and the characteristic information of the asynchronous signal according to a pulse neural network algorithm, acquiring an optical flow result by using the combined information of the fusion signal and outputting the processed fusion signal. The processed fusion signal output by the fusion subunit 133 can enable the dual-mode image signal processor to complete artificial intelligence related tasks such as real-time Region Of Interest (ROI) marking and target identification on the basis Of ensuring low power consumption, thereby effectively improving the quality and processing efficiency Of the picture.
Fig. 4 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 4, the asynchronous image signal processor 120 includes an asynchronous signal encoding unit 124, and the asynchronous signal encoding unit 124 is used to encode an asynchronous signal in the bimodal image signal.
The asynchronous signal encoding unit 124 is configured to encode an asynchronous signal in the bimodal image signal, that is, to compress the gray scale gradient image signal by a large margin, and then encode the gray scale gradient image signal into an asynchronous mode, so as to form an address form event signal. The address form event signal may be (X, Y, P, T), where "X, Y" is an event address, for example, "X, Y" may represent a pixel point position in the gray-scale gradient image signal, "P" is a 4-value event output, for example, "P" may represent a gray-scale variation of the pixel point position, and "T" is an event generation time, for example, a shooting time. Therefore, the information in the address form event signal includes characteristic information such as an event address, an event output, and a time when the event is generated. The characteristic information of the address form event signal can fully utilize the discreteness and sparsity in the asynchronous signal, and the dynamic range and the time resolution of the processed asynchronous signal are improved, so that the dual-mode image signal processor can realize high signal fidelity when shooting images at high speed, and can realize high dynamic range and high time resolution at the same time.
It should be noted that the address form event signal in the above process is only an example, and in other embodiments, the address form event signal may also be in the form of (X, Y, P), or in the form of (X, Y, P, Δ T), or in the form of (X, Y, Δ P, Δ T). Where Δ P is the variation of the two 4-value event outputs and Δ T is the variation of the time over which the two events occur.
Optionally, with continuing reference to fig. 4, the asynchronous image signal processor 120 further includes a pixel consistency correction unit 123, configured to perform pixel consistency correction on the asynchronous signal in the bimodal image signal and output the result to the asynchronous signal encoding unit 124.
The pixel consistency correcting unit 123 may be connected to the asynchronous signal encoding unit 124, and before the asynchronous signal encoding unit 124 encodes the asynchronous signal in the bimodal image signal, the pixel consistency correcting unit 123 performs pixel consistency correction on the asynchronous signal in the bimodal image signal, so as to reduce a difference value of the asynchronous signal caused by a pixel in the asynchronous signal, and reduce an influence of a pixel factor on the asynchronous signal. Then, the asynchronous signal after the pixel consistency correction is encoded by the asynchronous signal encoding unit 124, so that the encoding accuracy of the asynchronous signal encoding unit 124 can be improved.
Alternatively, with continued reference to fig. 4, the asynchronous image signal processor 120 further includes a first black level correction unit 122; the first black level correction unit 122 is connected to the pixel consistency correction unit 123, and is configured to perform black level correction on the asynchronous signal in the bimodal image signal and output the result to the pixel consistency correction unit 123.
The black level is a video signal level at which no light is output for one line on the display panel subjected to a certain calibration. The main principle of black level correction is to detect the level of the 'light black' part in the brightness signal, compare the level with the blanking level, if the blanking level is not reached, then expand towards the black level; if the blanking level has been reached, the extension is stopped, i.e. the blanking level is not exceeded. Thus, the original 'light black' is changed into 'dark black', the black level expansion only changes the 'light black' level in the luminance signal, and the white level and the luminance/chrominance signal ratio are not changed, thereby eliminating the fuzzy feeling of the image and improving the contrast of the image. Therefore, the first black level correction unit 122 is configured to adjust the image brightness in the asynchronous signal in the bimodal image signal, so that the blurred shadow of the image is removed, the image becomes clearer, and the contrast of the image is improved. Then, the asynchronous signal whose blur characteristics are removed by the first black level correction unit 122 is output to the pixel consistency correction unit 123 for pixel consistency correction, thereby further improving the accuracy of the asynchronous signal.
Fig. 5 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 5, the synchronized image signal processor 110 includes a second black level correction unit 112, a color interpolation unit 113, and a color correction unit 114; the second black level correction unit 112 is connected to the color interpolation unit 113, and configured to perform black level correction on the synchronization signal in the bimodal image signal and output the result to the color interpolation unit 113; the color interpolation unit 113 is connected to the color correction unit 114, and is configured to perform color interpolation on the synchronization signal after black level correction and output the result to the color correction unit 114; and a color correction unit 114 for performing color correction on the color-interpolated synchronization signal.
The second black level correction unit 112 is configured to adjust the image brightness in the synchronization signal in the bimodal image signal, so that the blurred shadow of the image is removed, the image becomes clearer, and the contrast of the image is improved. In addition, after the color filter is used, each pixel can only sense one color, so that the information of other two channels of the pixel needs to be restored to find the values of the other two channels of the pixel. Because the image is continuously changed, the values of R, G and B of one pixel point are related to the surrounding pixel points, so that the values of the surrounding pixel points can be used for obtaining the values of other two channels of the point, thereby completing the color image and obtaining more comprehensive color image information. Therefore, the color interpolation unit 113 can calculate the other two color components missing from each pixel according to the surrounding sampling points, so as to obtain a full-color synchronization signal, improve the accuracy of the synchronization signal in restoring the colors of the image, and further obtain an image with higher quality. The color correction unit 114 is a color deviation correction process for correcting color deviation of an image, which is also called color correction, and is a complementary color correction process of three primary colors RGB and three complementary colors CMY in an optical concept, so that it can be ensured that the color of a synchronization signal can be relatively accurately reproduced to be seen by human eyes in a shooting scene, and we can obtain a better and more accurate effect map.
The sync signal in the bimodal image signal passes through the second black level correction unit 112, and then the level of the "light black" part in the luminance signal in the sync signal is compared with the blanking level, so that the black level correction is performed to remove the image blur shadow. The synchronization signal processed by the second black level correction unit 112 is then transmitted to the color interpolation unit 113 to restore the missing color information of the pixel points, and obtain the color information of the complete image. Finally, the synchronization signal processed by the color interpolation unit 113 is transmitted to the color correction unit 114 to correct the color cast of the image, so as to ensure that the colors of the image can be reproduced more accurately to be seen by human eyes in the scene of shooting. Therefore, after the synchronization signal in the bimodal image signal is processed by the second black level correction unit 112, the color interpolation unit 113 and the color correction unit 114, a synchronization signal with better effect and more accurate color can be obtained.
Fig. 6 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 6, the dual-modality image signal processor further includes an auto adjustment unit 140, and the auto adjustment unit 140 is configured to adjust a synchronous signal output from the synchronous image signal processor 110 and an asynchronous signal output from the asynchronous image signal processor 120.
Before the neural network unit 130 processes the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, the automatic adjustment unit 140 is required to automatically adjust the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120. The automatic adjustment unit 140 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and the automatic adjustment unit 140 may obtain a synchronous signal output by the synchronous image signal processor 110 and an asynchronous signal output by the asynchronous image signal processor 120, adjust the synchronous signal output by the synchronous image signal processor 110 according to the feature information in the obtained synchronous signal and the feature information in the asynchronous signal, and output the adjusted synchronous signal; and adjusts the asynchronous signal output from the asynchronous image signal processor 120 and outputs the adjusted asynchronous signal; therefore, the automatic adjustment unit 140 can simultaneously output the synchronous signal with all color information characteristics and the asynchronous signal with all gray scale gradient information characteristics, thereby realizing automatic adjustment of the synchronous signal and the asynchronous signal.
In addition, the automatic adjusting unit 140 may add the obtained feature information of the asynchronous signal into the feature information of the synchronous signal in a merging manner, so as to realize the fusion of the synchronous signal and the asynchronous signal, and form a fused signal. The automatic adjusting unit 140 adjusts the fusion signal of the synchronous signal and the asynchronous signal according to the combination characteristic of the fusion signal and outputs the fusion signal. Because the fusion signal comprises the characteristic information of the synchronous signal and the characteristic information of the asynchronous signal, when the image is displayed through the fusion signal, the image can utilize the image frame based on the pixel matrix in the synchronous signal to improve the quality of the image, and simultaneously, the image signal based on the event in the asynchronous signal can be utilized to improve the dynamic range and the time resolution of the image, so that the image has high signal fidelity.
Fig. 7 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 7, the automatic adjustment unit 140 includes an automatic white balance subunit 141; the automatic white balance subunit 141 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and is configured to perform automatic white balance processing according to the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120.
The auto white balance subunit 141 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and can simultaneously acquire the synchronous signal and the asynchronous signal. The automatic white balance subunit 141 may perform automatic white balance adjustment on the acquired synchronous signal and asynchronous signal and output an adjusted synchronous signal and an adjusted asynchronous signal. In the automatic white balance adjustment process, the automatic white balance subunit 141 automatically detects the color temperature value of the object according to the light conditions of the lens and the white balance sensor, so as to determine the image capturing condition, and selects the closest hue setting, and the color temperature setting is corrected by the color temperature correction circuit, and the white balance automatic control circuit automatically adjusts the white balance to a proper position. The white balance means that "a white object can be reduced to white regardless of any light source", and a color cast phenomenon occurring when a picture is taken under a specific light source is compensated by enhancing a corresponding complementary color. For example: the white balance setting of the camera can calibrate the deviation of the color temperature, and the white balance can be adjusted greatly to achieve the desired picture effect during shooting. The automatic white balance subunit 141 can correct the color temperature according to the synchronous signal and the asynchronous image signal, and restore the color of the photographed main body, so that the color of the picture photographed under different light source conditions or in different dynamic states is similar to the color of the picture watched by human eyes. And images with different color effects can be obtained by controlling the color temperature.
Optionally, fig. 8 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 8, the automatic white balance subunit 141 may also fuse the synchronous signal and the asynchronous signal acquired simultaneously. The automatic white balance subunit 141 may obtain the feature information in the synchronous signal and the feature information in the asynchronous signal, respectively, and add the feature information of the asynchronous signal to the feature information of the synchronous signal in a combined manner, thereby completing the fusion of the synchronous signal and the asynchronous signal to form a fused signal. The automatic white balance subunit 141 then performs automatic white balance adjustment on the fusion signal. The automatic white balance subunit 141 can correct the color temperature according to the fusion signal, and restore the color of the photographed main body, so that the color of the picture photographed under different light source conditions or in different dynamic states is similar to the color of the picture viewed by human eyes. And images with different color effects can be obtained by controlling the color temperature.
Optionally, with continued reference to fig. 8, the automatic adjustment unit 140 further comprises an automatic exposure photon unit 142; the automatic exposure subunit 142 is connected to the automatic white balance subunit 141, and is configured to perform automatic exposure adjustment on the bimodal image signal after automatic white balance;
the physical meaning of exposure is that light causes a latent image to be formed on the photosensitive layer (coated with the photosensitive chemical). The post-exposure step is chemical development, and the development is performed to develop and cure the latent image. The quality of the image is related to the exposure, i.e. how much light should be passed through to enable the photosensitive element to obtain a sharp image. The automatic exposure is to automatically set a shutter speed and an aperture value according to a shutter and aperture exposure combination set at the time of production based on an exposure value of an image measured by a photometric system. The automatic exposure subunit 142 is connected with the automatic white balance subunit 140, and can perform automatic exposure adjustment on the fusion signal output after automatic white balance, so that the color of the picture can be more gorgeous, and the light with obvious directivity can well improve the texture of the picture.
In addition, in other embodiments, when the automatic white balancing sub-unit 141 performs white balancing on the synchronous signal and the asynchronous signal, respectively, the automatic exposure sub-unit 142 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and performs automatic exposure adjustment on the synchronous signal output from the synchronous image signal processor 110 and the asynchronous signal output from the asynchronous image signal processor 120. Illustratively, as shown in fig. 7, the automatic exposure subunit 142 is directly connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and can synchronously acquire the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, perform automatic exposure adjustment on the synchronous signal and the asynchronous signal, and output the synchronous signal and the asynchronous signal after the automatic exposure adjustment by the automatic exposure subunit 142, so that the color of the image frame appears more colorful, and the light with obvious directivity can well improve the texture of the frame.
Optionally, with continued reference to fig. 8, the auto-adjustment unit 140 further comprises an auto-focus subunit 143; the auto-focusing subunit 143 is connected to the auto-white-balancing subunit 141, and is configured to perform auto-focusing adjustment on the auto-white-balanced bimodal image signal;
the automatic focusing is realized by determining the distance of a shot object according to the reflection of the shot object by utilizing the principle of object light reflection and then adjusting a lens combination according to the measured result. The automatic focusing mode has the characteristics of high speed, easy realization and low cost. The auto-focus subunit 143 is connected to the auto-white balance subunit 141, and performs auto-focus adjustment on the fusion signal after auto-white balance, so as to make the image clearer.
Similarly, in other embodiments, when the auto white balance subunit 141 performs white balance on the synchronous signal and the asynchronous signal, respectively, the auto focus subunit 143 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and is configured to perform auto focus adjustment on the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120. Illustratively, as shown in fig. 7, the auto-focusing subunit 143 is connected to the synchronous image signal processor 110 and the asynchronous image signal processor 120, and can directly acquire the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120, and perform auto-focusing adjustment on the synchronous signal and the asynchronous signal and output the automatically focused synchronous signal and asynchronous signal, so as to make the image clearer.
Fig. 9 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 9, the bimodal image signal processor further comprises a wide dynamic range and spatio-temporal interpolation unit 150; the wide dynamic range and space-time interpolation unit 150 is connected to the automatic adjustment unit 140, and is configured to perform dynamic range adjustment and space-time interpolation processing on the bimodal image signal output by the automatic adjustment unit 140, and output the processed bimodal image signal; the processed bimodal image signal comprises a synchronous signal and an asynchronous signal or is a processed fusion signal.
Wherein the wide dynamic range and spatio-temporal interpolation unit 150 performs dynamic range adjustment and spatio-temporal interpolation processing on the processed bimodal image signals output from the automatic adjustment unit 140 before the neural network unit 130 performs processing on the synchronous signals output from the synchronous image signal processor 110 and the asynchronous signals output from the asynchronous image signal processor 120. High-Dynamic Range (HDR), also called a wide Dynamic Range technology, is a technology applied to make a camera see the image under very strong contrast. The "dynamic range" refers to the adaptability of the camera to the reflection of the scene illumination in the scene, specifically to the variation range of brightness (contrast) and color temperature (contrast). Spatiotemporal interpolation is widely used for interpolation and estimation of null data sets in the absence, since the absence of data degrades the integrity of spatiotemporal data. The wide dynamic range and spatio-temporal interpolation unit 150 is connected to the automatic adjustment unit 140, and is configured to perform dynamic range adjustment and spatio-temporal interpolation processing on the adjusted bimodal image signal output by the automatic adjustment unit 140 to obtain more dynamic ranges and image details, higher temporal resolution, and reflect a more realistic visual effect.
In addition, the wide dynamic range and spatio-temporal interpolation unit 150 may output a fusion signal, or may output a synchronous signal and an asynchronous signal, respectively, at the same time. When the wide dynamic range and spatio-temporal interpolation unit 150 outputs the fusion signal, it is possible to directly output an image signal having high fidelity, high dynamic range, and high temporal resolution. When the wide dynamic range and spatio-temporal interpolation unit 150 simultaneously outputs the synchronous signal and the asynchronous signal, respectively, the fused signal output by the automatic adjustment unit 140 may be decomposed to form the processed synchronous signal and asynchronous signal, such that the synchronous signal is an image signal with a high-precision restored image color, and the asynchronous signal is an image signal with a high dynamic range and a high time resolution.
Fig. 10 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 10, the bimodal image signal processor further includes an encoding compression unit 160 and an external interface 170; the encoding and compressing unit 160 is connected to the wide dynamic range and spatio-temporal interpolation unit 150, and is configured to encode and compress the bimodal image signal output by the wide dynamic range and spatio-temporal interpolation unit 150, and output the bimodal image signal to the external interface 170.
The encoding and compressing unit 160 is connected to the wide dynamic range and spatio-temporal interpolation unit 150, and can encode and compress the bimodal image signal output by the wide dynamic range and spatio-temporal interpolation unit 150, so as to eliminate a large amount of redundant information in the bimodal image signal, represent the original data by using as few bytes as possible, and output the original data to the external interface 170 to improve the image transmission efficiency. The external interface 170 is a bridge for signal transmission between the dual-mode image signal processor and the external device, and is an input/output port for information interaction with the external device. The external device may acquire the bimodal image signal processed by the bimodal image signal processor by connecting with an external interface 170. In addition, the bimodal image signal may be a fusion signal or a synchronous signal and an asynchronous signal.
Optionally, the external interface comprises at least one of a universal serial bus interface, an ethernet interface and a high definition multimedia interface.
Among them, a Universal Serial Bus (USB) interface is a Serial Bus standard for connecting a computer system and external devices, and is also a technical specification of an input/output interface, and is widely used for information transmission between communication devices. The ethernet interface is a port for network data connection, and the ethernet protocol defines a series of software and hardware standards, so that different communication devices are connected together through the ethernet interface for information transmission. High Definition Multimedia Interface (HDMI) is a full digital video and audio transmission Interface, can transmit uncompressed audio and video signals, can transmit audio and video signals simultaneously, and because audio and video signals adopt the same wire rod, the installation difficulty of system lines is greatly simplified.
It should be noted that the external interface type is only an example of the type of the external interface, and in a specific implementation, the external interface needs to be selectively set according to an information transmission requirement.
Fig. 11 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 11, the dual-modality image signal processor further includes a control unit 180 and an input-output bus 190; the control unit 180 is connected to the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 130 through an input/output bus 190, and the control unit 180 is configured to control the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 130 according to the instruction information; the control unit 180 is further configured to set the neural network unit 130 according to the neural network parameters, and the neural network unit 130 outputs the processed bimodal image signal through the input/output bus 190.
The control unit 180 is mainly responsible for program flow management, is a command control center of the whole bimodal image signal processor, and is very important for coordinating the ordered work of the whole equipment. In addition, the control unit 180 is further configured to process feedback control information of the bimodal image signal processor, and may feed back the feedback control information to the synchronous image signal processor and the asynchronous image signal processor according to a control signal formed by the bimodal image signals processed by the automatic adjusting unit 140 and the neural network unit 130. The input-output bus 190 is a common communications backbone that carries information between the various functional components of the overall dual-modality image signal processor. For example: the control unit 180 is connected to the synchronous image sensor signal processor 110, the asynchronous image signal processor sensor 120 and the neural network unit 130 through the input/output bus 190, and it can be known that information transmission interaction among the control unit 180, the synchronous image sensor signal processor 110, the asynchronous image signal processor sensor 120 and the neural network unit 130 needs to be realized through the input/output bus 190. The control unit 180 sends instruction information to be transmitted to the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 130 through the input-output bus 190, respectively, so as to control the synchronous image signal processor 110, the asynchronous image signal processor 120 and the neural network unit 130 to respond and execute the instruction content. In addition, the control unit 180 may further set the neural network unit 130 according to the neural network parameters, so that the neural network unit 130 processes the bimodal image signals according to the preset parameters, and on the basis Of ensuring low power consumption, the artificial intelligence related tasks such as real-time Region Of Interest (ROI) marking and target identification can be completed at a high speed, and then the processed bimodal image signals are output through the input/output bus 190. Therefore, the control unit 180 can issue instructions to each functional module when the whole dual-mode image signal processor is equivalent to the brain of the whole system, and coordinate and control the rapid and stable operation of the whole system. However, the input/output bus 190 is equivalent to the aorta of the whole system, and can rapidly transmit information to each functional module, so as to realize timely information transmission and ensure orderly information processing of the system.
Optionally, with continuing reference to fig. 11, the bimodal image signal processor further comprises a storage unit 200, the storage unit 200 being configured to cache the instruction information and the neural network parameters; the storage unit 200 is connected to the control unit 180, and is configured to provide the control unit 180 with instruction information and neural network parameters; the storage unit 200 is further connected to the input/output bus 190 for buffering the processed bimodal image signals output by the neural network unit 130.
The storage unit 200 is used for buffering data information in the bimodal image signal processor. For example: the instruction information, the neural network parameters, the bimodal image signals and other data information of the cache control unit 180. The storage unit 200 is connected to the control unit 180, and can transmit the instruction information and the neural network parameters cached in the storage unit 200 to the control unit 180, and the control unit 180 allocates the functional modules to which the instruction information and the neural network parameters need to be transmitted. In addition, the storage unit 200 may also receive data information, such as: the storage unit 200 is further connected to the input/output bus 190, and may receive the bimodal image signal processed by the neural network unit 130 and transmitted through the input/output bus 190, and buffer the bimodal image signal processed by the neural network unit 130 in the storage unit 200 according to a certain storage manner.
Optionally, with continued reference to fig. 11, the dual-modality image signal processor further includes a mobile industry processor interface 210; the MOL processor interface 210 is connected to the I/O bus 190, and the bimodal image signal is transmitted to the synchronous image signal processor 110 and the asynchronous image signal processor 120 through the MOL processor interface 210 and the I/O bus 190.
A Mobile Industry Processor Interface 210 (MIPI) is an open standard initiated by the MIPI alliance and established for a Mobile application Processor, and is dedicated to adopt a low amplitude signal swing in a high-speed (data transmission) mode. Standardizing interfaces such as a camera, a display screen, a baseband, a radio frequency interface and the like in the mobile equipment can increase the design flexibility and reduce the cost, the design complexity and the power consumption. Since the MOL interface 210, the CIS 110 and the ASP 120 are all connected to the I/O bus 190, the MOL interface 210 can rapidly transmit the bimodal image signals to the CIS 110 and the ASP 120 through the I/O bus 190, which can effectively improve the information transmission rate.
Fig. 12 is a schematic structural diagram of another bimodal image signal processor according to an embodiment of the present invention. As shown in fig. 12, the dual-modality image signal processor further includes an auto adjustment unit 140, the auto adjustment unit 140 being configured to adjust a synchronous signal output from the synchronous image signal processor 110 and an asynchronous signal output from the asynchronous image signal processor 120; the control unit 180 is also used to adjust the image sensor according to the bimodal image signal output by the neural network unit 130 and the bimodal image signal output by the automatic adjustment unit 140.
The image sensor is a sensor for acquiring an image signal. The image sensor may be a dual-modality image sensor, i.e., the image sensor includes a cone cell circuit and a rod cell circuit; the cone cell circuit is used for collecting image color signals to form synchronous signals in the bimodal image signals, and the rod cell circuit is used for collecting gray gradient image signals to form asynchronous signals in the bimodal image signals. The image sensor may also be comprised of two separate sub-sensors, one of which includes a cone cell circuit for acquiring image color signals, forming a synchronization signal in the bimodal image signal. The other sub-sensor includes a rod cell circuit for acquiring gray scale gradient image signals to form asynchronous signals in the bimodal image signals. The control unit 180 may adjust the image sensor according to the bimodal image signal output from the auto adjustment unit 140. The bimodal image signal output by the automatic adjustment unit 140 may be a fused signal of a synchronous signal and an asynchronous signal, and the fused signal may include an automatic exposure adjusted fused signal output by the automatic exposure subunit 142, and may further include an automatic focus adjusted fused signal output by the automatic focus subunit 143. The bimodal image signal output by the automatic adjustment unit 140 may be a fusion signal of a synchronous signal and an asynchronous signal, may include an automatic exposure adjusted fusion signal output by the automatic exposure subunit 142, and may further include an automatic focus adjusted fusion signal output by the automatic focus subunit 143. The control unit 180 controls the fusion signal after the automatic exposure adjustment and the fusion signal after the automatic focusing adjustment to form a control signal for controlling an exposure parameter and a focusing parameter of the image sensor, and feeds back the control signal to the image sensor for controlling the exposure time and the focal length of the image sensor according to the set exposure parameter and focusing parameter. Or, the control unit 180 controls the synchronous signal and the asynchronous signal after the automatic exposure adjustment and the synchronous signal and the asynchronous signal after the automatic focusing adjustment to form a control signal for controlling the exposure parameter and the focusing parameter of the image sensor, and feeds back the control signal to the image sensor for controlling the exposure time and the focal length of the image sensor according to the set exposure parameter and the set focusing parameter. The control unit 180 may adjust the image sensor according to the bimodal image signal output by the neural network unit 130. The bimodal image signal output by the neural network unit 130 may be a fused signal of a synchronous signal and an asynchronous signal, and the fused signal may include a fused signal after extracting features of high spatial resolution, sparsity, and high dynamic range, which are output by the artificial neural network subunit 131. The bimodal image signal output by the neural network unit 130 may also be a synchronous signal for extracting high spatial resolution features and an asynchronous signal for extracting sparsity, high dynamic range characteristics. The control unit 180 controls the fusion signal/synchronous signal and asynchronous signal after feature extraction to form a control signal for controlling the artificial neural network parameter and the impulse neural network parameter of the image sensor, and feeds back the control signal to the image sensor for controlling the feature extraction of the artificial neural network of the image sensor on the synchronous signal and the feature extraction of the impulse neural network on the asynchronous signal according to the set artificial neural network parameter and impulse neural network parameter.
Fig. 13 is a schematic structural diagram of another bimodal image signal processing system according to an embodiment of the present invention. As shown in fig. 13, an embodiment of the present invention further provides a dual-modality image signal processing system including an image sensor 300 and a dual-modality image signal processor implementing any one of the above embodiments; the image sensor 300 is connected to the bimodal image signal processor, the image sensor is used for acquiring bimodal image signals, the synchronous image signal processor 110 is used for processing synchronous signals in the bimodal image signals, and the asynchronous image signal processor 120 is used for processing asynchronous signals in the bimodal image signals; the neural network unit 130 is configured to process and output the synchronous signal output by the synchronous image signal processor 110 and the asynchronous signal output by the asynchronous image signal processor 120 according to a neural network algorithm.
The image sensor 300 may be a dual-modality image sensor that includes both a cone circuit and a rod circuit, among others. The cone cell circuit is used for collecting image color signals, and the rod cell circuit is used for collecting gray gradient image signals, so that the dual-mode image sensor can simultaneously output synchronous signals with all color information and asynchronous signals with all gray gradient information. Alternatively, the image Sensor 300 may be a Sensor formed by combining an Active Pixel Sensor (APS) including a cone cell circuit and outputting a synchronous signal having all color information and a Dynamic Vision Sensor (DVS) including a rod cell circuit and outputting an asynchronous signal having all gray gradient information. The APS and DVS combine to output a synchronous signal having all color information and an asynchronous signal having all gray gradient information.
The dual-mode image signal processing system comprises the image sensor and the dual-mode image signal processor provided by any embodiment of the invention, so that the dual-mode image signal processing system has the beneficial effects of the image sensor and the dual-mode image signal processor provided by the embodiment of the invention, and the details are not repeated here.
It is to be noted that the foregoing description is only exemplary of the invention and that the principles of the technology may be employed. Those skilled in the art will appreciate that the present invention is not limited to the particular embodiments described herein, and that various obvious changes, rearrangements and substitutions will now be apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in some detail by the above embodiments, the invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the invention, and the scope of the invention is determined by the scope of the appended claims.

Claims (19)

1. A bimodal image signal processor, comprising:
a synchronous image signal processor for processing a synchronous signal having a high spatial resolution in the bimodal image signal;
the asynchronous image signal processor is used for processing asynchronous signals with sparse, high-speed and high-dynamic-range characteristics in the bimodal image signals;
and the neural network unit is used for respectively and simultaneously processing and outputting the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor according to different neural network algorithms.
2. The dual-modality image signal processor of claim 1, wherein the neural network unit includes an artificial neural network sub-unit and a spiking neural network sub-unit;
the artificial neural network subunit is used for processing the synchronous signals output by the synchronous image signal processor according to an artificial neural network algorithm, and the impulse neural network subunit is used for processing the asynchronous signals output by the asynchronous image signal processor according to an impulse neural network algorithm.
3. The dual-modality image signal processor of claim 2, wherein the neural network unit further comprises a fusion subunit configured to fuse the synchronous signal output by the artificial neural network subunit and the asynchronous signal output by the impulse neural network subunit to form a fused signal, and process the fused signal according to the impulse neural network algorithm.
4. The bimodal image signal processor as claimed in claim 1, wherein the asynchronous image signal processor comprises an asynchronous signal encoding unit for encoding an asynchronous signal in the bimodal image signal.
5. The bimodal image signal processor as claimed in claim 4, wherein the asynchronous image signal processor further comprises a pixel consistency correction unit, for performing pixel consistency correction on the asynchronous signal in the bimodal image signal and outputting the result to the asynchronous signal encoding unit.
6. The bimodal image signal processor as claimed in claim 5, wherein the asynchronous image signal processor further comprises a first black level correction unit;
the first black level correction unit is connected with the pixel consistency correction unit and is used for performing black level correction on the asynchronous signals in the bimodal image signals and outputting the signals to the pixel consistency correction unit.
7. The dual-modality image signal processor according to any one of claims 1 to 6, characterized in that the synchronous image signal processor includes a second black level correction unit, a color interpolation unit, and a color correction unit;
the second black level correction unit is connected with the color interpolation unit and is used for performing black level correction on the synchronous signals in the bimodal image signals and outputting the signals to the color interpolation unit;
the color interpolation unit is connected with the color correction unit and used for performing color interpolation on the synchronous signals after the black level correction and outputting the synchronous signals to the color correction unit;
the color correction unit is used for performing color correction on the synchronous signal after color interpolation.
8. The dual-modality image signal processor of claim 7, further comprising an auto-adjustment unit for adjusting a synchronous signal output by the synchronous image signal processor and an asynchronous signal output by the asynchronous image signal processor.
9. The bimodal image signal processor as claimed in claim 8, wherein the automatic adjustment unit comprises an automatic white balance subunit;
the automatic white balance subunit is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic white balance processing according to the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
10. The dual-modality image signal processor of claim 9, wherein the auto-adjustment unit further comprises an auto-exposure photon unit;
the automatic exposure subunit is connected with the automatic white balance subunit and is used for carrying out automatic exposure adjustment on the bimodal image signal subjected to automatic white balance; or the image exposure control device is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic exposure adjustment on the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
11. The dual-modality image signal processor of claim 9, wherein the auto-adjustment unit further comprises an auto-focus subunit;
the automatic focusing subunit is connected with the automatic white balance subunit and is used for carrying out automatic focusing adjustment on the bimodal image signal subjected to automatic white balance; or the image signal processor is connected with the synchronous image signal processor and the asynchronous image signal processor and is used for carrying out automatic focusing adjustment on the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor.
12. The bimodal image signal processor of claim 8, further comprising a wide dynamic range and spatiotemporal interpolation unit; the wide dynamic range and space-time interpolation unit is connected with the automatic adjusting unit and is used for carrying out dynamic range adjustment and space-time interpolation processing on the bimodal image signals output by the automatic adjusting unit and outputting the processed bimodal image signals; the processed bimodal image signal comprises a synchronous signal and an asynchronous signal or is a processed fusion signal.
13. The bimodal image signal processor of claim 12, further comprising an encoding compression unit and an external interface; the coding compression unit is connected with the wide dynamic range and space-time interpolation unit and is used for coding and compressing the bimodal image signals output by the wide dynamic range and space-time interpolation unit and outputting the bimodal image signals to the external interface.
14. The dual-modality image signal processor of claim 13, wherein the external interface includes at least one of a universal serial bus interface, an ethernet interface, and a high definition multimedia interface.
15. The bimodal image signal processor as claimed in claim 1, further comprising a control unit and an input-output bus;
the control unit is connected with the synchronous image signal processor, the asynchronous image signal processor and the neural network unit through the input/output bus, and is used for controlling the synchronous image signal processor, the asynchronous image signal processor and the neural network unit according to instruction information; the control unit is also used for setting the neural network unit according to neural network parameters, and the neural network unit outputs the processed bimodal image signals through the input/output bus.
16. The bimodal image signal processor as claimed in claim 15, further comprising a storage unit for caching instruction information and the neural network parameters; the storage unit is connected with the control unit and used for providing the instruction information and the neural network parameters for the control unit; the storage unit is also connected with the input/output bus and is used for caching the processed bimodal image signals output by the neural network unit.
17. The dual-modality image signal processor of claim 15, further comprising a mobile industry processor interface; the mobile industry processor interface is connected with the input and output bus, and the bimodal image signals are transmitted to the synchronous image signal processor and the asynchronous image signal processor through the mobile industry processor interface and the input and output bus.
18. The dual-modality image signal processor of claim 15, further comprising an auto-adjustment unit for adjusting a synchronous signal output by the synchronous image signal processor and an asynchronous signal output by the asynchronous image signal processor;
the control unit is also used for adjusting the image sensor according to the bimodal image signal output by the neural network unit and the bimodal image signal output by the automatic adjusting unit.
19. A dual-modality image signal processing system, characterized by comprising an image sensor and the dual-modality image signal processor of any one of claims 1 to 18; the image sensor is connected with the bimodal image signal processor, the image sensor is used for acquiring the bimodal image signals, the synchronous image signal processor is used for processing synchronous signals with high spatial resolution in the bimodal image signals, and the asynchronous image signal processor is used for processing asynchronous signals with sparse, high-speed and high-dynamic-range characteristics in the bimodal image signals; and the neural network unit is used for simultaneously processing and outputting the synchronous signals output by the synchronous image signal processor and the asynchronous signals output by the asynchronous image signal processor according to different neural network algorithms.
CN202011520252.9A 2020-12-21 2020-12-21 Dual-mode image signal processor and dual-mode image signal processing system Active CN112702588B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202011520252.9A CN112702588B (en) 2020-12-21 2020-12-21 Dual-mode image signal processor and dual-mode image signal processing system
PCT/CN2021/139857 WO2022135359A1 (en) 2020-12-21 2021-12-21 Dual-mode image signal processor and dual-mode image signal processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011520252.9A CN112702588B (en) 2020-12-21 2020-12-21 Dual-mode image signal processor and dual-mode image signal processing system

Publications (2)

Publication Number Publication Date
CN112702588A CN112702588A (en) 2021-04-23
CN112702588B true CN112702588B (en) 2023-04-07

Family

ID=75509430

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011520252.9A Active CN112702588B (en) 2020-12-21 2020-12-21 Dual-mode image signal processor and dual-mode image signal processing system

Country Status (1)

Country Link
CN (1) CN112702588B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022135359A1 (en) * 2020-12-21 2022-06-30 北京灵汐科技有限公司 Dual-mode image signal processor and dual-mode image signal processing system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107302695A (en) * 2017-05-31 2017-10-27 天津大学 A kind of electronics compound eye system based on bionic visual mechanism

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11228758B2 (en) * 2016-01-22 2022-01-18 Peking University Imaging method and device
CN109190683A (en) * 2018-08-14 2019-01-11 电子科技大学 A kind of classification method based on attention mechanism and bimodal image
CN211296854U (en) * 2019-12-24 2020-08-18 清华大学 Pixel array and bionic vision sensor
CN111106807B (en) * 2019-12-24 2021-11-30 清华大学 Viewing cone and rod multiplexing bionic vision sensor
CN111083402B (en) * 2019-12-24 2020-12-01 清华大学 Bimodal bionic vision sensor
CN112085768B (en) * 2020-09-02 2023-12-26 北京灵汐科技有限公司 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107302695A (en) * 2017-05-31 2017-10-27 天津大学 A kind of electronics compound eye system based on bionic visual mechanism

Also Published As

Publication number Publication date
CN112702588A (en) 2021-04-23

Similar Documents

Publication Publication Date Title
EP3402179B1 (en) Image-capturing system, image-capturing method, and program
CN106713755A (en) Method and apparatus for processing panoramic image
CN108848354B (en) VR content camera system and working method thereof
CN112188093B (en) Bimodal signal fusion system and method
US11184553B1 (en) Image signal processing in multi-camera system
CN105141841A (en) Camera equipment and method therefor
US10600170B2 (en) Method and device for producing a digital image
US20100007766A1 (en) Camera device and image processing method
CN107968902A (en) Panoramic camera and its implementation based on single image sensor
CN112714301B (en) Bimodal image signal processor and image sensor
CN112702588B (en) Dual-mode image signal processor and dual-mode image signal processing system
KR101753261B1 (en) Photographing system and method for synchronizing image quality thereof
CN112188092B (en) Bimodal signal processing system and method
WO2022135359A1 (en) Dual-mode image signal processor and dual-mode image signal processing system
CN115802183A (en) Image processing method and related device
JP6750594B2 (en) Imaging system and imaging method
US20190052803A1 (en) Image processing system, imaging apparatus, image processing apparatus, control method, and storage medium
CN112738425A (en) Real-time video splicing system with multiple cameras for acquisition
CN112738357B (en) Video rod image signal processor and image sensor
TWI802015B (en) Dual-mode signal processing system and method
JP7295329B2 (en) Imaging system, image processing program, and image processing method
JP6083959B2 (en) Transmission apparatus, control method thereof, and control program
CN104660895A (en) Novel SDI wide dynamic range (WDR) human face capturing camera
CN113923341A (en) Electronic device
CN114531533A (en) Sliding window type high-definition color camera

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant