WO2021073140A1

WO2021073140A1 - Monocular camera, and image processing system and image processing method

Info

Publication number: WO2021073140A1
Application number: PCT/CN2020/097704
Authority: WO
Inventors: 陈开�; 黄普发
Original assignee: 华为技术有限公司
Priority date: 2019-10-15
Filing date: 2020-06-23
Publication date: 2021-04-22
Also published as: CN110798623A

Abstract

The embodiments of the present application relate to the field of image processing. Particularly provided are a monocular camera, and an image processing system and an image processing method. The monocular camera comprises a lens, a dual-pass filter, an image sensor and an image fusion processing unit, wherein the dual-pass filter filters light received by the lens, and retains an optical signal in a visible light waveband and an optical signal in an infrared waveband after filtering; the image sensor performs photoelectric conversion on the optical signal in the visible light waveband and then generates a color image frame, and performs photoelectric conversion on the optical signal in the infrared waveband and then generates an infrared image frame; and the image fusion processing unit performs image fusion, wherein specifically, the color image frame and the infrared image frame that are generated by the image sensor are fused, so as to generate a fused color image frame.

Description

Monocular camera, image processing system and image processing method

Technical field

This application relates to the field of imaging, especially monocular cameras, image processing systems and image processing methods.

Background technique

In the field of video security, color attributes play an important role in identifying objects. For the camera, when the working mode is daytime, the visible light information in the environment is rich, so the camera can work in the color mode; and when the working mode is night, there is less visible light information in the environment, and the camera is difficult to obtain high-quality color images.

Existing cameras can generate infrared image frames and color image frames at night, and merge the two to improve the display effect of the color image frames. However, the structure of this camera is more complicated and the cost is higher.

Summary of the invention

The present invention provides a monocular camera, an image processing system, and an image processing method in combination with various embodiments to solve one or more defects in the aforementioned prior art.

In a first aspect, an embodiment of the present invention provides a monocular camera, including: a lens for receiving light from the subject; a double-pass filter for filtering the light received by the lens, After filtering, the optical signal in the visible light band and the optical signal in the infrared band are retained; the image sensor is used to photoelectrically convert the optical signals of the two bands obtained after the filtering, and the optical signal in the visible light band generates color after photoelectric conversion Image frame, the optical signal in the infrared band is converted into an infrared image frame after photoelectric conversion; an image fusion processing unit is used to fuse the color image frame generated by the image sensor and the infrared image frame to generate a fused color Image frame.

This solution provides a monocular camera, and because the dual-pass filter is integrated into the camera that integrates the image, the image effect after the fusion is better. And because the single lens, single image sensor is used, cost is saved.

In a first possible implementation manner of the first aspect, the monocular camera further includes: an infrared supplement light lamp control circuit for controlling the infrared supplement light lamp to perform infrared supplement light on the object. Use this program to enhance the image effect at night.

In the second possible implementation manner of the first aspect: based on the first aspect or the first possible implementation manner of the first aspect, the image sensor is used to generate a first color image frame at a first moment, and to generate a first color image frame at a second moment. The first infrared image frame, the second color image frame is generated at the third time, the first time, the second time and the third time are adjacent in time, and the image fusion processing unit is specifically configured to: use the first color image Frame and the second color image frame, the predicted color image frame at the second moment is estimated; the predicted color image frame and the first infrared image frame are used for fusion to generate the fused color image frame. The program specifically introduces a specific implementation: how to use color image frames and infrared image frames to merge into a fused color image frame.

In a third possible implementation manner of the first aspect, the monocular camera further includes: an infrared supplement light lamp, which is used to receive the control of the infrared supplement light lamp control circuit to perform infrared supplement light on the object. Use this program to enhance the image effect at night.

In a fourth possible implementation manner of the first aspect, the image fusion processing unit specifically includes: a first image signal processor, configured to receive the color image frame from the image sensor, and compare the color image The frame performs image signal processing, and sends the processed color image frame to the digital signal processor; the second image signal processor is used to receive the infrared image frame from the image sensor, and image the infrared image frame Signal processing sends the processed infrared image frame to the digital signal processor; the digital signal processor is used to fuse the infrared image frame and the color image frame that have undergone image signal processing to generate the fused color image frame. This program introduces the specific structure of the image fusion processing unit.

In a fifth possible implementation manner of the first aspect: when the digital signal processor receives the color image frame, it sends a start-up control signal to the infrared supplemental light. This program introduces the timing of starting the infrared fill light.

In a fifth possible implementation manner of the first aspect: the operating wavelength range of the infrared fill light lamp is [840 nm, 1040 nm]. This solution introduces the wavelength range of the infrared fill light, which can avoid the impact on human eyes and reduce light pollution.

In a sixth possible implementation manner of the first aspect: the infrared fill light lamp is a 930 nm fill light lamp, and the wavelength range of the light emitted by the 930 nm fill light lamp is around 930 nm. This solution introduces the wavelength range of the infrared fill light, which can avoid the impact on human eyes and reduce light pollution.

In a second aspect, the present invention provides an embodiment of an image processing system, which includes the monocular camera in the first aspect or any possible implementation of the first aspect, and also includes a network and a server. The network is used to transmit the fused color image frames generated by the monocular camera; the storage server is used to store the fused color image frames received through the network. This solution introduces an image processing system based on an embodiment of a monocular camera, which can store the fused image frames.

The first possible implementation of the second aspect: the storage server or other device performs further processing on the stored image, for example: face recognition on the fused image frame; license plate recognition on the fused image frame .

In a third aspect, an image processing method is provided, which includes the following steps: filtering the light received by the lens, and retaining the optical signal in the visible light band and the light signal in the infrared band after filtering; The signal undergoes photoelectric conversion, the optical signal in the visible light band generates a color image frame after the photoelectric conversion, and the light signal in the infrared band generates an infrared image frame after the photoelectric conversion; the color image frame and the infrared image generated by the image sensor The frames are fused to generate a fused color image frame. This solution provides an image processing method, because the frame fusion image generated by the optical signals of the two wavelength bands obtained by filtering is used in the camera, so that the fusion image effect is better. Optionally, the cost can be saved by using a single lens and a single image sensor.

The third aspect also provides various possible implementation manners corresponding to the first aspect, and has corresponding technical effects.

In a fourth aspect, a computer program product is provided. The computer program product contains computer-readable code instructions. When the computer-readable code instructions are executed by a computer, the computer can configure the monocular camera so that the monocular camera can execute The method described in the third aspect and any of the various possible implementation manners of the third aspect.

In a fifth aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium contains computer program code instructions. When these computer program code instructions are executed by a computer, the computer The configuration of the monocular camera enables the monocular camera to perform the method described in the third aspect and any of the various possible implementation manners of the third aspect. The non-transitory computer-readable storage medium includes one or more of the group: Read-Only Memory (ROM), Programmable ROM (Programmable ROM), and Erasable PROM (Erasable PROM) , EPROM), flash memory (Flash Memory), electrical EPROM (Electrically EPROM, EEPROM) and hard drive (Hard drive).

Description of the drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings.

Fig. 1 is a structural diagram of an embodiment of a monocular camera of the present invention;

Figure 2 is a schematic diagram of an embodiment of a fused image of the present invention;

Fig. 3 is a structural diagram of an embodiment of an image processing system provided by the present invention.

Fig. 4 is a flowchart of an embodiment of an image processing method of the present invention.

Detailed ways

The terms "including" and "having" in the specification and claims of the present invention and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or devices is not limited to the listed steps or devices, but optionally includes steps or devices that are not listed, or optionally also includes Other steps or devices inherent to these processes, methods, products, or equipment. The terms "first", "second", "third", and "fourth" are used to distinguish different objects, rather than describing a specific order.

The embodiment of the present invention proposes a "dual-pass filter-single image sensor" camera. After the camera with the new architecture fuses infrared and color image frames, color image frames with good effects can be obtained at night. In order to cooperate with this structure, the embodiment of the present invention adopts a monocular (lens), which has a simple structure and can save cost.

In addition, in the field of video surveillance such as warehouse management, unattended, and safe cities, it is often necessary to shoot images/videos in low-light scenes. Due to insufficient light, information such as the outline of the subject is often partially missing. . If the 750nm infrared lamp is used for supplement light, although the image quality can be improved to a certain extent, because 750nm is still in the range that the human eye can perceive, it will cause visual stimulation to the human eye, which is the so-called light pollution.

In order to take as clear color photos as possible in low-illuminance scenes such as night, the embodiment of the present invention uses a 930nm fill light to perform infrared fill light on the subject, and the monocular camera uses a double-pass filter to perform infrared fill light on the light entering the lens. Filter, the filtered light only retains visible light of 380-650nm and infrared light of 930±15nm. Then it is processed by the chip on chip (system on chip, SOC) to generate infrared image frame and color image frame and merge the two frames. The image after fusion is clearer and has color and black and white information under low illumination, that is, low-light color Ability. Moreover, since the output light wavelength of the fill light used in the embodiment of the present invention is 930 nm, the human eye will not perceive the light of the fill light, and will not cause visual stimulation to the pedestrian being photographed.

It should be noted that in product practice, the 930nm mentioned in the embodiment of the present invention is not an accurate number, and there must be a certain error. Therefore, it actually refers to a wavelength range near 930nm, such as 930±100nm. In other words, the working wavelength of the infrared supplement light lamp appears within 840 nm-1040 nm, which can all belong to the protection scheme of the embodiment of the present invention.

The embodiment of the present invention will be described in detail below in conjunction with FIG. 1.

As shown in Figure 1, the camera includes: infrared fill light 12 (optional component), lens 13, double-pass filter 14, image sensor (sensor) 15, chip on chip (System on Chip, SOC) 16 and complement Light control circuit 17. The chip on chip specifically includes an image signal processor (ISP) 161, an image signal processor 162, a digital signal processor 163, and an encoder 164. According to different designs, the fill light 12 can also be independent of the camera, so it does not need to be part of the camera. In addition, the image signal processor 161, the image signal processor 162, the digital signal processor 163, and the encoder 164 may also be independent chips, which are not integrated in the SOC.

The lens 13 is used to collect light from the subject 11. The lens 13 is usually a lens group composed of one or more pieces of optical glass (or plastic), and may be composed of a concave lens, a convex lens, an M-type lens, or a combination of lenses. The lens 13 may be a spherical lens or an aspherical lens. As a monocular camera, the embodiment of the present invention has only one lens, so the structure is simple and the cost is low.

In the embodiment of the present invention, there is only one lens (lens 13) in the camera, so it is a monocular camera.

The double-pass filter 14 has a layer on the surface of the filter, and the light from the lens 13 is filtered by the coating, so as to select the optical signal of the desired radiation band. In the embodiment of the present invention, the light allowed by the double-pass filter 14 includes two wavelength bands: the wavelength is the visible light band (visible light band is understood differently in the industry, but generally the wavelength range is similar, for example, 380-650nm , It can also be 400-760nm, 380-780nm, etc.), and infrared light with a wavelength of 930±5nm (or 930±15nm). The double-pass filter 14 is not a prism. Compared with a prism, it has the advantage of reducing the light attenuation caused by scattering. According to calculations, the light energy intensity through the double-pass filter 14 will be higher than the light energy intensity after passing through the prism. More than 40%.

The sensor 15 receives the light signal filtered by the double-pass filter 14 and performs exposure to generate infrared image frames and color image frames. The sensor alternately exposes visible light and infrared light, thereby alternately generating infrared image frames and color image frames. Taking a video frame rate of 25 frames per second as an example, the sum of the total exposure time for obtaining an infrared image frame and a color image frame is 1/25 second. The average exposure time of a frame is 1/50 second. Taking into account that a longer exposure of visible light can generate better color image frames; and due to the existence of infrared fill light, even if the exposure time is slightly shorter, good infrared image frames can be generated. Therefore, in the embodiment of the present invention, the exposure time of the color image frame is longer than the exposure time of the infrared image frame. For example, the exposure time of a color image frame is 30ms, and the exposure time of an infrared image frame is 10ms.

The sensor is equipped with a control logic, and according to this logic, the visible light and the infrared light are continuously and alternately exposed. Thus, infrared image frames and color image frames are continuously generated. Infrared image frames can provide brightness information, and visible light images can provide color information and brightness information.

The image signal processor 161 receives the infrared image frame sent by the sensor 15, and performs image optimization on the infrared image frame, thereby optimizing the optical performance of the image. The infrared image frame reflects the spatial distribution of the target and background infrared radiation, and its radiance distribution mainly depends on the temperature and emissivity of the observed object. The content of image optimization can, for example, be one or more of the following: remove the bottom current noise, linearize the data, solve the non-linear problem of the data, remove the dead pixels, remove the noise, adjust the white balance, focus, and exposure , Adjust the sharpness of the image, color space conversion (convert to a different color space for processing), etc.

The image signal processor 162 receives the color image frame sent by the sensor 15. Image optimization for color image frames. So as to optimize the optical performance of the image. Compared with infrared images, visible light images provide more detailed information about the target or scene, which is more conducive to human observation. The content of image optimization can, for example, be one or more of the following: remove the bottom current noise, linearize the data, solve the non-linear problem of the data, remove the dead pixels, remove the noise, adjust the white balance, focus, and exposure , Adjust the sharpness of the image, color space conversion (convert to a different color space for processing), color enhancement of the image, and optimize the skin tone of the portrait.

In addition, the image signal processor 162 can also send a control signal to the supplement light control circuit 17 to control the operation of the infrared supplement light (for example: a 930 nm supplement light) 12. Specifically, the image processor 161 can send a control signal after receiving a color image frame. The image processor 161 is an internal device located in the SOC 16, so the control signal is transmitted through the pins of the on-chip chip 16 and then transmitted to the fill light control circuit 17 through the internal path of the camera cable. SOC16 is an image fusion processing unit.

The image signal processor 161 and the image signal processor 162 can also read the configuration of the sensor 15 to adjust image parameters such as contrast, saturation, and gain.

The digital processor 163 performs image fusion on the black and white frames processed by the image processor 161 and the color image frames processed by the image processor 162 to generate a fused color image frame.

Image fusion can use algorithms to fuse multiple image information of the same scene acquired by various image sensors working in different wavelength ranges or with different imaging mechanisms to generate a new image. The fused image will contain more sensitive information of the human visual system, which is more conducive to the observation of the human eye.

In the embodiment of the present invention, it is assumed that the exposure time length of the color image frame is 30 ms, and the exposure time length of the infrared image frame is 10 ms. Then, the time interval between the two color image frames before and after is 10ms. Using a motion estimation algorithm, two adjacent color image frames can be used to estimate the color image frame within the exposure time of the infrared image frame. Then the estimated color image frame and infrared image frame are fused.

Referring to FIG. 2, the sensor 15 alternately outputs color image frames and infrared image frames. Among these frames, the color image frame 21, the infrared image frame 22, and the color image frame 23 are three adjacent frames. The digital processor 163 uses the color image frame 21 and the color image frame 23 to estimate the color image frame 24, and the exposure time of the color image frame 24 is the same as that of the infrared image frame 22. Then, the digital processor 163 fuses the infrared image frame 22 and the color image frame 24 to obtain a fusion frame 25. Using this method to merge different image frames can continuously generate fused frames.

The encoder 164 is used for encoding into image frames that can be played. For example, the YUV data is generated into H.264 or H.265 image frames. The image frame can be played on the display or stored. Multiple image frames can be played continuously to form a dynamic video.

The fill light control circuit 17 is used to receive the control signal sent by the SOC and control the 930nm fill light. The 40nm fill light is triggered to emit 930nm infrared light to irradiate the subject 11, and the irradiation time length of the infrared light may be longer than the exposure time length of the infrared image frame.

The fill light control circuit 17 can intermittently control the turn-on of the 930nm fill light, which is turned on within the range of the exposure time of the infrared image frame, and turned off within the range of the color image frame exposure time. The SOC can intermittently send a control signal to the supplementary light control circuit 17, so that the supplementary light control circuit 17 intermittently controls the turn-on of the 930nm supplementary light.

It should be noted that the infrared supplement light 12 (an infrared supplement light lamp with a wavelength of 930 nm is an embodiment) is used to receive the control of the infrared supplement light control circuit 17 to perform infrared supplement light on the object. In the specific implementation process, the infrared supplement light 12 can be integrated into the camera to form an integrated device, or it can be sold separately from the camera. For the latter case, the camera needs to be equipped with a corresponding interface for Connect an external (independent) infrared supplement light to realize the control of the external infrared supplement light.

Referring to Fig. 3, the present invention also provides an embodiment of an image processing system. The image processing system includes a monocular camera 31 and a monocular camera 32, a network 33, and a server 34. Wherein, the monocular camera 31 and the monocular camera 32 are described. For the internal structure of the monocular camera 31 and the monocular camera 32, refer to the above introduction and FIG. 1, and only a brief description will be given below. The number of monocular cameras in the image processing system can be one or more, as shown in Fig. 3, taking two as an example; the number of servers 34 can be one or more, and Fig. 3 taking one as an example.

The monocular camera 31 (or the monocular camera 31) includes a lens 13, a double-pass filter 14, an image sensor 15 and an image fusion processing unit 16. Among them, the lens 13 is used to receive light from the subject; the double-pass filter 14 is used to filter the light received by the lens, and after filtering, the light signal in the visible light band and the light in the infrared wave band are retained. Signal; image sensor 15, used to photoelectrically convert the optical signals of the two bands obtained after the filtering, the optical signal in the visible light band is generated after photoelectric conversion to generate a color image frame, and the optical signal in the infrared band is generated after photoelectric conversion Infrared image frame; an image fusion processing unit 16 for fusing the color image frame generated by the image sensor and the infrared image frame to generate a fused color image frame.

The network 33 is used to provide the monocular camera 31 and the communication between the monocular camera 32 and the server 34. The network 33 may be a wired network, a wireless network, or a combination of a wired network and a wireless network. The network 33 may have network equipment such as routers and switches.

The server 34 is configured to receive the fused color image frame through the network 33, and store and/or analyze the above fused color image frame. Optionally, the server 34 (or other devices, such as a video analysis server, an image analysis server) may further process the images stored in the server 34. For example: Recognize the fused color image frame, identify these people as the personnel in the personnel list in the database based on the characteristics of the people appearing in the fused color image frame; perform license plate recognition on the vehicle, and identify the license plate number .

Referring to FIG. 4, the present invention also provides an embodiment of an image processing method, which can be executed by the above-mentioned monocular camera.

In step S41, the light received by the lens is filtered, and the optical signal in the visible light band and the light signal in the infrared band are retained after the filtering. This step can be performed by a double-pass filter or a prism.

Step S42, photoelectric conversion is performed on the optical signals of the two wavelength bands obtained by filtering, the optical signals in the visible light band are converted into a color image frame after the photoelectric conversion, and the optical signals in the infrared band are converted into an infrared image frame after the photoelectric conversion. This step can be performed by a sensor, which can process light signals in the visible light waveband and infrared waveband.

Step S43, fusing the color image frame and the infrared image frame generated by the image sensor to generate a fused color image frame. This step can be performed by an image fusion processing unit (for example, SOC).

Wherein, fusing the generated color image frame and the infrared image frame may specifically include: receiving the color image frame from the image sensor, performing image signal processing on the color image frame, and processing the processed color image frame. The color image frame is sent to the digital signal processor; the infrared image frame is received from the image sensor, image signal processing is performed on the infrared image frame, and the processed infrared image frame is sent to the digital signal processor; After the image signal processing, the infrared image frame and the color image frame are fused to generate a fused color image frame.

The following is a specific example, the low fusion algorithm is introduced.

Referring to Figure 2, a first color image frame (color image frame 21) is generated at the first time, a first infrared image frame (infrared image frame 22) is generated at the second time, and a second color image frame (color image frame) is generated at the third time. 23) The first time, the second time, and the third time are adjacent in time. The fusion of the generated color image frame and the infrared image frame specifically includes: using the first color image frame 21 and the second color image frame 23 to estimate the predicted color image frame 24 at the second moment; using the The predicted color image frame 24 and the first infrared image frame 22 are image fused to generate a fused color image frame.

Optionally, this embodiment of the present method further includes: when the digital signal processor receives the color image frame, it sends a start control signal to the 930nm fill light. After receiving the start control signal, the 930nm supplement light lamp emits 930nm infrared light so as to supplement the light of the object during the exposure period of the infrared image frame. When the exposure of the infrared image frame is completed, a control signal to turn off the 930nm fill light can be sent to stop the infrared fill light. When the digital signal processor receives the next color image frame, it sends a start control signal to the 930nm fill light again; when the exposure of the next infrared image frame is completed, it can again send a close control signal to the 930nm fill light , And so on.

The present invention also provides an embodiment of a computer program product. The computer program product contains computer-readable code instructions. When the computer-readable code instructions are executed by the computer, the computer can configure the monocular camera so that the monocular camera can be configured. Perform the method described in the third aspect and any of the various possible implementation manners of the third aspect.

The present invention also provides an embodiment of a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium contains computer program code instructions, which can be implemented when the computer program code instructions are executed by a computer. The configuration of the monocular camera by the computer enables the monocular camera to execute the method described in the third aspect and any of the various possible implementation manners of the third aspect. The non-transitory computer-readable storage medium includes one or more of the group: Read-Only Memory (ROM), Programmable ROM (Programmable ROM), and Erasable PROM (Erasable PROM) , EPROM), flash memory (Flash Memory), electrical EPROM (Electrically EPROM, EEPROM) and hard drive (Hard drive).

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, it can still be used to describe the technical solutions described in the foregoing embodiments. Modifications or equivalent replacements of some of the technical features are made; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A monocular camera including:

Lens for receiving light from the subject;

The double-pass filter is used to filter the light received by the lens, and after the filtering, the optical signal in the visible light band and the optical signal in the infrared band are retained;

The image sensor is used for photoelectric conversion of the optical signals in the two bands obtained after the filtering, wherein the optical signals in the visible light band undergo photoelectric conversion to generate a color image frame, and the optical signals in the infrared band are Generate infrared image frames after photoelectric conversion; and

The image fusion processing unit is used for fusing the color image frame and the infrared image frame generated by the image sensor to generate a fused color image frame.
The monocular camera according to claim 1, wherein the monocular camera further comprises:

The infrared supplement light control circuit is used to control the infrared supplement light to perform infrared supplement light on the subject.
The monocular camera according to claim 1 or 2, wherein the image sensor is used to generate a first color image frame at a first moment, a first infrared image frame at a second moment, and a second color image frame at a third moment, The first time, the second time, and the third time are adjacent in time, and the image fusion processing unit is specifically configured to:

The first color image frame and the second color image frame are used to estimate the predicted color image frame at the second moment; the predicted color image frame and the first infrared image frame image are fused to generate a fused color image frame.
The monocular camera according to claim 2 or 3, the monocular camera further comprising:

The infrared supplement light is used to receive the control of the infrared supplement light control circuit to perform infrared supplement light on the object.
The monocular camera according to any one of claims 1 to 4, wherein the image fusion processing unit specifically comprises:

The first image signal processor is configured to receive the color image frame from the image sensor, perform image signal processing on the color image frame, and send the processed color image frame to a digital signal processor;

The second image signal processor is configured to receive the infrared image frame from the image sensor, perform image signal processing on the infrared image frame, and send the processed infrared image frame to the digital signal processor;

The digital signal processor is used for fusing the infrared image frame and the color image frame that have undergone image signal processing to generate the fused color image frame.
The monocular camera according to claim 5, wherein:

The exposure time of the color image frame is longer than the exposure time of the infrared image frame.
The monocular camera according to claim 5, wherein the infrared supplement light control circuit is specifically used for:

When the digital signal processor receives the color image frame, it sends a start-up control signal to the infrared supplemental light.
The monocular camera according to any one of claims 2-7, the working wavelength range of the infrared fill light is [840nm, 1040nm].
An image processing system, comprising the monocular camera according to any one of claims 1-8, a network and a server;

The network is used to transmit the fused color image frames generated by the monocular camera;

The server is configured to receive the fused color image frame through the network, and store and/or analyze the fused color image frame.
An image processing method applied to a monocular camera, including:

Receiving light from the subject through a lens, and filtering the received light using a double-pass filter to obtain optical signals in the visible light band and infrared light signals;

The image sensor is used to perform photoelectric conversion on the optical signals of the two wavelength bands obtained by filtering, wherein the optical signal in the visible light band undergoes photoelectric conversion to generate a color image frame, and the optical signal in the infrared wave band is generated after photoelectric conversion. Infrared image frame;

An image fusion processing unit is used to fuse the color image frame and the infrared image frame generated after photoelectric conversion to generate a fused color image frame.
The method according to claim 10, before receiving the light from the subject, the method further comprises:

Send a control signal to the infrared supplement light lamp, and the control signal is used to control the infrared supplement light lamp to perform infrared supplement light on the object.
According to the method of claim 10 or 11, the process of using the image sensor to perform photoelectric conversion on the optical signals of the two wavelength bands obtained by filtering includes:

The first color image frame is generated at the first time, the first infrared image frame is generated at the second time, and the second color image frame is generated at the third time, wherein the first time, the second time, and the third time are relative in time. adjacent;

Using an image fusion processing unit to fuse the color image frame and the infrared image frame generated after photoelectric conversion includes:

Using the first color image frame and the second color image frame to estimate the predicted color image frame at the second moment;

The predicted color image frame and the first infrared image frame image are used for fusion to generate a fused color image frame.
The method according to claim 11 or 12, after sending the control signal to the infrared supplement light lamp, the method further comprises:

Turn on the infrared fill light to perform infrared fill light on the subject.
The method according to any one of claims 10-13, fusing the color image frame and the infrared image frame generated after photoelectric conversion specifically includes:

Performing image signal processing on the color image frame, and sending the processed color image frame to a digital signal processor;

Receiving the infrared image frame from the image sensor, performing image signal processing on the infrared image frame, and sending the processed infrared image frame to the digital signal processor;

It is used to fuse infrared image frames and color image frames that have undergone image signal processing to generate fused color image frames.
The method according to claim 14, further comprising:

When the digital signal processor receives the color image frame, it sends a start-up control signal to the infrared supplemental light.
According to the method of any one of claims 11-15, the operating wavelength range of the infrared fill light lamp is [840 nm, 1040 nm].