WO2021184362A1

WO2021184362A1 - Photographing device

Info

Publication number: WO2021184362A1
Application number: PCT/CN2020/080448
Authority: WO
Inventors: 黄进新; 汪鹏程; 刘军
Original assignee: 华为技术有限公司
Priority date: 2020-03-20
Filing date: 2020-03-20
Publication date: 2021-09-23
Also published as: CN113711584A; CN113711584B

Abstract

Disclosed are a photographing device. The photographing device comprises one color image sensor and one black-and-white image sensor; moreover, the resolution of the two image sensors is identical. The color image sensor has a neighboring pixel merging function. The color image sensor generates a low-resolution color image, the black-and-while image sensor generates a high-resolution grayscale image, and then, the color image and the grayscale image are merged to produce a target image. This allows an image of increased quality to be outputted in a low illumination environment, and reduces the working illuminance lower limit of the photographing device.

Description

A camera device

Technical field

The embodiments of the present application relate to the field of security monitoring, and in particular, to a camera device.

Background technique

Low illumination scene (low illumination scene/low light scene) refers to scenes with insufficient light, such as outdoors at night or indoors without sufficient lighting. In the field of security surveillance, in order to obtain high-definition and colorful images in low-illuminance scenes, cameras are often used in conjunction with visible light supplementary lights or infrared light supplementary lights. However, the use of visible light supplementary light is easy to cause light pollution and is not conducive to covert monitoring; although the use of infrared supplementary light has a clear image, it cannot record colors. In recent years, the industry has widely adopted a dual-light fusion architecture. Under this architecture, the camera uses two sensors to image infrared light and visible light separately, and then integrates the infrared image and the visible light image to improve the camera’s low illumination Imaging capabilities.

Specifically, a light splitting prism is set in the camera, and the light splitting prism divides the incident light into visible light and infrared light according to the frequency spectrum. Then, the camera inputs the aforementioned visible light and infrared light into two identical image sensors respectively. Among them, the image sensor inputting visible light outputs a color image, and the image sensor inputting infrared light outputs a grayscale image, and the size and resolution of the foregoing color image and grayscale image are the same. The camera then fuses the aforementioned color image and grayscale image to obtain a target image. The details and texture of the target image mainly come from the grayscale image, and the color information comes from the color image.

In the foregoing solution, when the ambient illuminance is lower than the lower limit of the imaging signal-to-noise ratio of the visible light image sensor, the color information of the color image is overwhelmed by noise, resulting in low color saturation of the final output fused image, or even a grayscale image. Therefore, under the current dual-light fusion architecture, it is very important to further reduce the lower limit of the camera's operating illuminance.

Summary of the invention

The embodiments of the present application provide a camera device, which is used to output a higher-quality image in a low-illuminance environment and reduce the lower limit of the operating illuminance of the camera device.

In the first aspect, an embodiment of the present application provides a camera device, which includes an optical component, a first image sensor, a second image sensor, and an image processor. The resolution of the first image sensor is equal to the resolution of the second image sensor. In addition, the first image sensor has a function of combining adjacent pixels. Wherein, the optical component is used to receive the incident optical signal, and separate the incident optical signal into a first optical signal and a second optical signal. The first image sensor is used to sense the first light signal to generate a third image, and combine every N adjacent pixels in the third image into one pixel to obtain a first image, and image information of the first image It includes first color information and first brightness information. The second image sensor is used to sense the second light signal to generate a second image, the image information of the second image includes first brightness information, and the resolution of the first image is smaller than the resolution of the second image. The image processor is configured to determine a target image based on the first image and the second image, and the color and brightness of the target image are determined by the image information of the first image and the image information of the second image; or, for A target image is generated based on the third image and the second image, and the color and brightness of the target image are determined by the image information of the third image and the image information of the second image.

In this embodiment, the aforementioned N is an integer greater than one. Among them, the function of merging adjacent pixels refers to the physical method of superimposing the charges of adjacent pixel units together as a pixel output signal. This function can make a smaller resolution image from a larger resolution image sensor, so it can increase the photosensitive area, improve the sensitivity of light sensing in the dark and the image output speed.

In the embodiment of the present application, since the first image sensor has the function of combining adjacent pixels (that is, the Binning function), the light input of each pixel of the low-resolution first image output by the first image sensor will be The increase can reduce the signal-to-noise ratio of the first image, thereby improving the color sensitivity of the first image, ensuring that the color of the first image is true and having higher brightness; while the second image sensor outputs a high-resolution second image The image has high definition and can present rich details and textures. The target image determined based on the foregoing two images can retain the advantages of the first image and the second image, and therefore, the camera device can be made to work in an environment with lower light intensity.

According to the first aspect, in the first implementation manner of the first aspect of the embodiments of the present application, the image processor is specifically configured to: adjust the resolution of the first image to be the same as the resolution of the second image to obtain A fourth image, where the fourth image carries the first color information and the first brightness information; the fourth image is fused with the second image to obtain the target image.

In this embodiment, since the resolution of the first image is smaller than the resolution of the second image, adjusting the resolution of the first image to be the same as the resolution of the second image can be understood as changing the resolution of the first image The resolution is increased to the resolution of the second image to obtain the fourth image. Since the first image has higher color sensitivity, true colors, and higher brightness, the fourth image also has true colors and higher brightness. Therefore, the target image determined based on the foregoing fourth image and the second image can retain the advantages of the fourth image and the second image, and therefore, the camera device can be made to work in a lower light intensity environment.

According to the first aspect or the first implementation manner of the first aspect, in a second implementation manner of the first aspect of the embodiments of the present application, the first optical signal includes visible light, and the second optical signal includes visible light and infrared light. The energy of the visible light in the first optical signal is greater than the energy of the visible light in the second optical signal, and the frequency band of the visible light in the first optical signal is the same as the frequency band of the visible light in the second optical signal.

In this embodiment, it is proposed that the processing of the aforementioned optical component on the incident light signal is not only frequency division, but also energy division. Wherein, the energy division refers to dividing the visible light in the incident light signal, which can also be understood as processing to make the energy of the visible light in the first light signal different from the energy of the visible light in the second light signal. Since the energy of the visible light in the first light signal is different from the energy of the visible light in the second light signal, it is beneficial for the two image sensors to have different brightness of the two images obtained when the light intensity is higher than the preset value (that is, during the day). . Determining the target image based on the aforementioned two images with different brightness is beneficial to improve the dynamic range.

According to the second implementation manner of the first aspect, in the third implementation manner of the first aspect of the embodiments of the present application, the first image sensor and the second image sensor are both color image sensors.

According to the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect of the embodiments of the present application, the imaging device further includes a visible light cut-off filter; the imaging device is also used when the light intensity is lower than the predetermined When the value is set, the visible light cut-off filter is activated, the visible light cut-off filter is located between the optical component and the second image sensor, and the visible light cut-off filter is used to filter the visible light in the second light signal. The second image sensor is specifically used to sense infrared light in the second light signal to generate the second image. The image processor is specifically configured to fuse the fourth image with the second image to obtain the target image.

In this embodiment, it is proposed to use a visible light filter to filter out visible light in the second light signal when the light intensity is low, so as to cut off the second image sensor from being affected by visible light when sensing infrared light. Although the second image sensor is a color image sensor, the second light signal only includes infrared light. Therefore, the second image only includes second brightness information and no color information. Since the aforementioned fourth image has more real colors than the third image, the target image obtained by fusing the fourth image and the second image has better quality than the image obtained by fusing the fourth image and the second image. It is beneficial for the camera device to work under lower light intensity.

According to the third implementation manner of the first aspect, in the fifth implementation manner of the first aspect of the embodiments of the present application, the imaging device further includes an infrared light cut-off filter; the imaging device is also used when the light intensity is higher than The infrared cut-off filter is activated at a preset value, the infrared cut-off filter is located between the optical component and the second image sensor, and the infrared cut-off filter is used to filter out the second optical signal. Infrared light. The second image sensor is specifically configured to sense visible light in the second light signal, and the image information of the second image further includes second color information. The image processor is specifically configured to fuse the third image with the second image to obtain the target image.

In this embodiment, it is proposed to use an infrared cut filter to filter the infrared light in the second light signal when the light intensity is high, so that the second image output by the second image sensor presents colors. In addition, it is also proposed that the aforementioned third image and the second image can be directly fused. Since the energy of the visible light sensed by the third image is different from the energy of the visible light sensed by the second image, the brightness and color presented by the third image are different from the brightness and color presented by the second image. The fusion of images can make the output target image a high dynamic range image.

According to any one of the first aspect, the first implementation manner of the first aspect to the fifth implementation manner of the first aspect, in the sixth implementation manner of the first aspect of the embodiments of the present application, the optical assembly It includes a lens and a dichroic prism, and the dichroic prism is located between the lens and the image sensor. The lens is used to receive the incident light signal. The light splitting prism is used for dividing the incident light signal received by the lens into the first light signal and the second light signal.

In this embodiment, it is proposed to use a light splitting prism to divide the incident light signal into a first light signal and a second light signal of different frequencies and different energies, which is beneficial to improve the dynamic range of the target image.

According to the sixth implementation manner of the first aspect, in the seventh implementation manner of the first aspect of the embodiments of the present application, the lens is an infrared confocal lens.

According to any one of the first aspect, the first implementation manner of the first aspect to the fifth implementation manner of the first aspect, in the eighth implementation manner of the first aspect of the embodiments of the present application, the imaging device Also includes infrared cut filter. The optical assembly includes a first lens and a second lens, and the first lens and the second lens are used to jointly receive the incident light signal. The focal length of the first lens is the same as the focal length of the second lens. The aperture of the first lens is larger than the aperture of the second lens. The second lens is an infrared confocal lens. The first lens is used to receive a part of the incident light signal and transmit the received light signal to the infrared cut-off filter; the infrared cut-off filter is used to filter out the light from the first lens The infrared light in the signal obtains the first optical signal, and transmits the first optical signal to the first image sensor; the second lens is used to receive the remaining part of the incident optical signal and receive the The optical signal is transmitted to the second image sensor as a second optical signal.

In this embodiment, it is proposed that binocular lenses with different apertures can be used to make the energy of the light signals illuminating different image sensors different. Since the larger the aperture, the greater the luminous flux. Therefore, the energy of the visible light in the first light signal output by the first lens is greater than the energy of the visible light in the second light signal output by the second lens. In addition, an infrared cut filter is also arranged between the first lens and the first image sensor, so that only visible light but no infrared light is included in the first light signal.

According to any one of the foregoing implementation manners, in the twelfth implementation manner of the first aspect of the embodiments of the present application, the resolution of the second image is equal to the resolution of the target image.

In the second aspect, an embodiment of the present application provides an image processor, which is connected to a memory in a camera device. The memory is used to store data or programs processed by the processor, such as the aforementioned first image, second image, third image, and fourth image; the image processor is used to call the program in the memory to perform The image, the second image, the third image, and the fourth image are subjected to image processing.

It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application.

FIG. 1A is a schematic diagram of an embodiment of a camera device in an embodiment of the application;

FIG. 1B is a schematic diagram of another embodiment of the camera device in the embodiment of the application;

FIG. 1C is a schematic diagram of another embodiment of the imaging device in the embodiment of the application;

2A is a schematic diagram of another embodiment of the camera device in the embodiment of the application;

2B is a schematic diagram of an embodiment of an image processing flow in an embodiment of the application;

FIG. 2C is a schematic diagram of another embodiment of the image processing flow in the embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, without having to use To describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in a sequence other than the content illustrated or described herein. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those clearly listed. Those steps or units may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or equipment.

For ease of understanding, the following first introduces the application scenarios of the camera device proposed in the embodiments of the present application:

The camera device proposed in the embodiment of the present application can be applied to camera in a low illumination/low light environment. Specifically, the low-illuminance environment refers to an environment where the light intensity is lower than a certain value, which is generally measured by the energy received by the visible light per unit area of the image sensor in the imaging device, and the unit is Lux (Lx). Generally, an illumination environment greater than 0 Lux and less than 1 Lux can be referred to as a low-illuminance environment. Specifically, the low-illuminance environment can be a dim outdoor street, for example, a street at night, or a street on a rainy day; it can also be an interior with only weak light, for example, a store, warehouse, etc., with only weak light. There is no limitation here.

In the aforementioned low-illuminance scene, because when the photosensitive area of the image sensor is fixed, there is an inverse correlation between the color sensitivity of the output image and the resolution of the image. Taking into account that the grayscale image generated by the image sensor sensing infrared light can provide higher spatial resolution, and the human visual system is more sensitive to brightness information and less sensitive to color spatial resolution. Therefore, the spatial resolution of some color images can be sacrificed in exchange for an increase in color sensitivity, thereby achieving the effect of reducing the lower limit of the operating illuminance of the camera device. It is also considered that some image sensors have a camera binning mode, which is often referred to as Binning mode for short. The image sensor with this pixel binning mode can physically superimpose the charges of multiple adjacent pixels together as a pixel output signal, so as to reduce the resolution and improve the sensitivity of light sensing in the dark. Therefore, the embodiment of the present application uses the characteristics of this image sensor to propose a camera device for reducing the lower limit of the operating illuminance of the camera device. For example, the lower limit of the original camera's working illumination requirement is 1Lux, and it is difficult to obtain a color image that can be accepted by the human eye if it is lower than 1Lux. After using the solution provided by the embodiment of the present invention, the lower limit of the operating illuminance of the camera device can be reduced to 0.1 Lux, or even 0.01 Lux, which is not specifically limited here.

The camera device proposed in the embodiment of the present application will be introduced below. As shown in FIG. Wherein, the optical component 103 is used to receive the incident signal and process the incident light signal into a first light signal and a second light signal, wherein the incident light signal is emitted by an object photographed by the imaging device 10. The optical component 103 is also used to control the first light signal to be directed to the first image sensor 101 and control the second light signal to be directed to the second image sensor 102. In addition, the resolution of the first image sensor 101 is equal to the resolution of the second image sensor 102, and the first image sensor 101 has a function of combining adjacent pixels, that is, the first image sensor has a pixel combining mode (ie, Binning mode).

Wherein, the first image sensor 101 is used to sense the first light signal to generate a third image, and combine every N adjacent pixels in the third image into one pixel to obtain the first image. The third image and the first image are both color images, and the first image has higher color sensitivity than the third image. Therefore, the first image can record richer colors and can appropriately increase brightness. The image information of the first image includes first color information and first brightness information. Wherein, the N is an integer greater than 1. At this time, the resolution of the first image is smaller than the resolution of the third image.

The second image sensor 102 is used to sense the second light signal to generate a second image, the second image is a color image or a grayscale image, and the image information of the second image includes second brightness information (when the second image is In the case of a color image, the second image also includes second color information). Since the resolution of the first image sensor 101 is equal to the resolution of the second image sensor 102, the resolution of the third image is equal to the resolution of the second image, and the resolution of the first image is smaller than that of the second image Resolution.

The image processor 104 is configured to determine a target image based on the first image and the second image, and the color and brightness of the target image are determined by the image information of the first image and the image information of the second image. Optionally, the image processor 104 is further configured to determine a target image based on the third image and the second image, and the color and brightness of the target image are determined by the image information of the third image and the image information of the second image. Sure. For example, the image processor 104 may be a system on chip (system on chip, SoC).

It should be understood that, since the first image contains first color information, that is, the first image can present colors, the first image sensor 101 is a color image sensor. Optionally, the second image sensor 102 may be a color image sensor or a black-and-white image sensor. For details, please refer to the related introduction of the embodiment corresponding to FIG. 2A below, and details are not repeated here.

Based on the foregoing embodiment, the process of determining the target image by the image processor 104 may include the following:

First, the image processor 104 adjusts the resolution of the first image to be the same as the resolution of the second image to obtain the fourth image. Then, the image processor 104 determines the target image based on the fourth image and the second image. Optionally, the resolution of the target image is equal to the resolution of the second image. In the foregoing process, although the resolution of the fourth image is different from the resolution of the first image, the color and brightness presented by the fourth image are derived from the first image. Therefore, the process of adjusting the aforementioned low-resolution first image to a high-resolution fourth image can not only retain the color and brightness presented by the first image, but also facilitate the determination of the target image based on the fourth image and the second image.

It should be understood that when the image processor 104 adjusts the resolution of the first image to be the same as the resolution of the second image, the image processor 104 may adopt an up-sampling algorithm or a super-resolution algorithm. There is no limit. In addition, the process of determining the target image by the image processor 104 based on the foregoing fourth image and second image mainly refers to an image fusing process. Image fusion (image fusion) refers to an image processing technology that can use a specific algorithm to combine two or more images into a new image. The combined image has the original image (that is, the two or more images before synthesis). ) Excellent characteristics. For example, characteristics such as brightness, clarity, and color.

Optionally, the camera device 10 further includes an image signal processor (image signal processor, ISP) chip (not shown), and the ISP chip is located between the aforementioned image sensor and the image processor 104. Optionally, the aforementioned ISP chip may be a two-way processing process, respectively processing the first image output by the first image sensor 101 and the second image output by the second image sensor 102, and combining the processed first image and the second image. The two images are sent to the aforementioned image processor 104 for subsequent processing. Optionally, the ISP chip can perform multiple ISP processing on the aforementioned first image and second image, for example, 3D noise reduction, demosaicing, brightness correction, and color correction. The aforementioned basic image processing can be adjusted according to actual application requirements. Specifically, the content of the aforementioned basic image processing is not limited here.

It should be noted that, in this embodiment and subsequent embodiments, the image output from the first image sensor 101 to the image before the resolution is adjusted are all referred to as the first image. Similarly, the image output from the second image sensor 102 to the image before the image fusion or combination is called the second image. That is to say, in practical applications, the original image output from the image sensor may go through a variety of processing procedures, but the embodiment of the present application does not limit the foregoing processing procedures. In other words, if the RAW image output by the image sensor is directly fused later, then the first image is in the RAW format; if the RGB image output by the ISP chip is fused, then the first image is in the RGB format; If the YUV image output by the ISP chip is fused, the first image is in the YUV format (in order to reduce redundancy, the subsequent embodiments only use images in the YUV format as an example for the introduction of the first and second images). In addition, the first image and the second image are not compressed and encoded, and the fused image can be encoded into an image format that is easy to be recognized by the human eye and occupies a smaller storage space, such as jpeg format (referred to as jpg format), bmp format , Tga format, png format and gif format.

Optionally, before fusing the foregoing fourth image and the second image, the ISP chip may also perform image correction on the foregoing fourth image and the second image. Specifically, the ISP chip can correct the coordinate system of the fourth image to the coordinate system of the second image, so that the scenes in the foregoing two images are aligned, which can also be understood as aligning the texture details of the foregoing two images. . Optionally, the ISP chip may use preset correction parameters to adjust the aforementioned two images, or may adaptively configure the correction parameters according to changes in the current temperature to complete image correction.

Optionally, the aforementioned first image sensor 101 or the second image sensor 102 may be a CCD image sensor composed of a charged coupled device (CCD), or may be a complementary metal oxide semiconductor (complementary metal oxide semiconductor, The CMOS image sensor composed of CMOS) may also be other types of image sensors, which is not specifically limited here.

In the foregoing embodiment, the first image sensor in the imaging device 10 adopts the Binning mode, and the spatial resolution of the first image output by the first image sensor is reduced in exchange for color sensitivity (ie, low-light sensitivity). Therefore, the camera device 10 can be made to work better in a low-illuminance environment. On this basis, the optical component 103 is specifically used to process the incident light signal so that the energy of the visible light in the first light signal is greater than the energy of the visible light in the second light signal. Wherein, the first light signal includes visible light, and the second light signal includes visible light and infrared light. It can also be understood that the optical component 103 performs frequency division and energy division on the incident light signal. Among them, frequency division refers to dividing the incident light signal according to different frequencies, for example, dividing the incident light signal into visible light and infrared light. The aforementioned energy is proportional to the square of the amplitude of the light wave, and the aforementioned visible light energy can also be understood as the intensity of visible light. Therefore, energy division can be understood as the use of lens coating and other physical structures to separate the visible light in the incident light signal into a first light signal and a second light signal, the intensity of the visible light in the first light signal and the second light signal The intensity of the visible light in the first optical signal is different, and the intensity ratio between the visible light in the first optical signal and the visible light in the second optical signal is fixed. It should be noted that the frequency band of visible light in the first optical signal is the same as the frequency band of visible light in the second optical signal; or, the frequency band of visible light in the first optical signal is the same as the possession of visible light in the second optical signal. Light of the same frequency band, for example, both the first optical signal and the second optical signal have green light.

In such an embodiment, since the energy of the visible light in the first light signal is different from the energy of the visible light in the second light signal, it is beneficial for the two image sensors to obtain the result when the light intensity is higher than the preset value (that is, during the day). The brightness of the two images is different. Determining the target image based on the aforementioned two images with different brightness is beneficial to improve the dynamic range. In addition, the ratio between the energy of the visible light in the first optical signal and the energy of the visible light in the second optical signal can be flexibly controlled according to actual application requirements. For example, the first optical signal and the second optical signal contain the same visible light frequency band, and the ratio of the energy of the visible light in the first optical signal to the energy of the visible light in the second optical signal can be kept at 9:1, or kept at 8. :2, or remain as 7:3, or remain as 6:4, or remain as 6.5:3.5, etc. The specifics are not limited here.

Further, the foregoing optical component 103 may adopt any one of the following implementation modes:

As shown in FIG. 1B, it is an implementation manner of the aforementioned optical assembly 103. The optical component 103 includes a dichroic prism 1031 and a lens 1032. The dichroic prism 1031 can also be called a beam splitter. It is an optical device that is coated with one or more films on the surface of the optical glass. The refraction and reflection of light split the incident light signal into two beams. In this embodiment, the dichroic prism 1031 is used to divide the incident optical signal into a first optical signal and a second optical signal. Wherein, the first light signal includes visible light, and the second light signal includes visible light and infrared light. Specifically, part of the visible light in the incident light signal is directed to the first image sensor 101 through the coating layer, and the other part of the visible light and all infrared light in the incident light signal is reflected at the coating layer and irradiated to the second image sensor 102 . It should be understood that if different optical components choose to use optical glass with different thicknesses and composition coating layers, then different optical components correspond to the energy of the visible light in the first optical signal and the energy of the visible light in the second optical signal. The ratio between the two will also be different. Optionally, the lens 1032 is an infrared confocal lens, and the infrared confocal lens is used to realize infrared confocal.

In addition, the optical component 103 is often used in conjunction with a filter. Specifically, a filter (e.g., filter 1051) may be provided between the dichroic prism 1031 and the first image sensor 101, or a filter (e.g., Filter 1052). Optionally, when the aforementioned filter 1051 is an infrared cut filter, it can prevent the first light signal entering the first image sensor 101 from being mixed with infrared light. Optionally, when the aforementioned filter 1052 is an infrared cut filter, the infrared light in the second optical signal can be filtered; when the aforementioned filter 1052 is a visible light cut filter, the second light can be filtered Visible light in the signal.

In this embodiment, the splitting prism is redesigned and the filter is combined to achieve frequency division (in the first and second optical signals, the infrared filter is used to cut off the light, avoiding infrared light in the first optical signal, and achieving Infrared light frequency band and visible light frequency band are divided) energy (the first and second light signals both include visible light of the same frequency band, and the energy of the two visible lights is different, which realizes the energy division), which can divide the visible light And infrared light is divided into two image sensors, and the ratio of visible light in the incident light signal into the two image sensors can be controlled at the same time. Therefore, the final output of the fused image can be increased when the light intensity is high (for example, during the day), and the dynamic range can be increased when the light intensity is low (for example, at night). Further make the color of the target image more natural.

As shown in FIG. 1C, it is another implementation of the aforementioned optical assembly 103.

The optical assembly 103 includes a first lens 1033 and a second lens 1034. The first lens 1033 is used to converge a part of the incident light signal so that the output light signal illuminates the first image sensor 101, and the second lens 1034 is used for The remaining part of the incident light signal is converged to make the output light signal illuminate the second image sensor 102. Wherein, the focal length of the first lens 1033 is the same as the focal length of the second lens 1034. Optionally, the aperture of the first lens 1033 is larger than the aperture of the second lens 1034. Therefore, the luminous flux of the first lens 1033 is greater than the luminous flux of the second lens 1034. Therefore, the energy of the visible light output by the first lens 1033 is greater than the energy of the visible light output by the second lens 1034. It can also be understood that the light signals actually passed by the first lens 1033 and the second lens 1034 are different. In addition, for the explanation of the energy of the visible light, please refer to the relevant introduction of the corresponding embodiment in FIG. 1B, which will not be repeated here.

In addition, a filter (for example, a filter 1051) is provided between the first lens 1033 and the first image sensor 101, and a filter may also be provided between the second lens 1034 and the second image sensor 102 (For example, filter 1052). Specifically, the aforementioned filter 1051 is an infrared cut filter for filtering infrared light in the optical signal from the first lens 1033, so that only visible light is included in the first optical signal sensed by the first image sensor 101 It does not contain infrared light. Optionally, when the aforementioned filter 1052 is an infrared cut filter, the infrared light in the second optical signal can be filtered; when the aforementioned filter 1052 is a visible light cut filter, the second light can be filtered Visible light in the signal. In addition, the second lens 1034 is an infrared confocal lens.

In this embodiment, two lenses with the same focal length but different aperture sizes are used to form a binocular lens to achieve energy distribution (different sensors receive different light energy). Since the size of the aperture determines the intensity of the light that can pass through the lens, the use of apertures of different sizes can control the energy of the visible light entering the two image sensors to be different. Combined with the aforementioned image sensor architecture using different filters under different light intensities, the final output of the fused image can be improved when the light intensity is high (for example, during the day), and when the light intensity is low ( For example, dark night) further makes the color of the target image more natural.

In practical applications, the aforementioned optical component 103 in FIG. 1A and the optical components involved in the following may adopt the implementation shown in FIG. 1B or the implementation shown in FIG. Not limited.

Specifically, as shown in FIG. 2A, the first image sensor and the second image sensor in the aforementioned camera device are both color image sensors. Optionally, the color image sensor may be a Bayer image sensor (Bayer image sensor) or a color image sensor in other formats. An infrared cut filter is arranged between the first image sensor and the optical component, and a dual optical filter is arranged between the second image sensor and the optical component. This dual optical filter is also called an IR-CUT automatic switching filter. The IR-CUT automatic switching filter is provided with a photosensitive device, or the IR-CUT automatic switching filter is connected to a photosensitive device, and the photosensitive device transmits the sensed light intensity to the imaging device. When a change in light intensity is detected, the camera device (specifically, a filter control chip in the camera device) can control the IR-CUT to automatically switch the filter automatically. For example, when the light intensity is greater than the preset value (for example, during the day), switch to the infrared cut filter, and when the light intensity is less than the preset value (for example, at night), switch to the visible light cut off filter. Optionally, the dual-optical filter can be replaced with an infrared cut-off filter, and the infrared cut-off filter is controlled by the camera device to enable and disable the infrared cut-off filter.

When the light intensity is lower than the preset value, it will be introduced in conjunction with Figure 2B.

The first image sensor is a high-resolution color image sensor. The first image sensor senses visible light in the first light signal to generate a third image, and combines every N adjacent pixels in the third image into one pixel to obtain a first image, where N is an integer greater than 1. Wherein, the third image is a high-resolution color image, and the first image is a low-resolution color image. Wherein, the low-resolution color image includes first color information and first brightness information, and the high-resolution color image includes third color information and third brightness information. The first color information is used to indicate the color of the low-resolution color image, and the first brightness information is used to indicate the brightness of the low-resolution color image. The third color information is used to indicate the color of the high-resolution color image, and the third brightness information is used to indicate the brightness of the high-resolution color image. This embodiment does not limit the specific forms of the first brightness information, the first color information, the third color information, and the third brightness information.

The dual optical filter provided between the second image sensor and the optical assembly is switched to a visible light cut-off filter, the second image sensor senses infrared light in the second optical signal and outputs a high-resolution grayscale image, That is, the aforementioned second image. Wherein, the high-resolution gray-scale image includes second brightness information, and the second brightness information is used to indicate the brightness of the high-resolution gray-scale image. The second brightness information can be represented by a brightness component Y. The embodiment of the present application does not limit the specific form of the second brightness information. It should be understood that although the second image sensor can record colors, the second image sensor only senses infrared light and no visible light, so the second image only presents brightness and cannot present colors. Therefore, the second image has only the luminance component Y and no chrominance component U/V.

It should be understood that the image directly generated by the aforementioned image sensor is in RAW format. According to the design of the image sensor, the RAW format is also divided into multiple types. For example, it can be Bayer RGGB, RYYB, RCCC, RCCB, RGBW, CMYW and other formats. . Using the ISP chip, you can convert various formats of RAW images into RGB format images. Using the ISP chip, you can also convert RAW format images into YUV format images, or HSV format images, or Lab format images, or CMY format images, or YCbCr format images. For example: the ISP chip first converts RAW format images into RGB format images, and then converts RGB format images into YUV format images.

The ISP chip in the camera device can also perform the aforementioned ISP processing on the aforementioned low-resolution color image and the aforementioned high-resolution grayscale image. For example, 3D noise reduction, demosaicing, brightness correction and color correction and other processing. Optionally, the ISP chip can also adjust the formats of the aforementioned low-resolution color images and high-resolution grayscale images, for example, adjusting the Bayer format to the YUV format, etc. The specifics are not limited here.

In addition, the image processor adopts an up-sampling algorithm or a super-resolution algorithm to adjust the aforementioned low-resolution color image to a high-resolution color image, and the high-resolution color image has the same resolution as the aforementioned high-resolution grayscale image. Rate. Then, the image processor fuses the aforementioned high-resolution color image and high-resolution grayscale image to obtain the target image.

For ease of understanding, take Fig. 2B as an example for introduction. The RGB format image 201 collected by the first image sensor is combined with adjacent pixels to obtain a low-resolution RGB format image 202. Then, the low-resolution RGB format image 202 is converted into a low-resolution YUV format image 203, and then the low-resolution YUV format image 203 is up-sampled to obtain a high-resolution YUV format image 204. Then, the high-resolution YUV format image 204 and the Y-component grayscale image 205 are fused to obtain a high-resolution YUV format image 206 (that is, the target image).

In this embodiment, the high-resolution color image and the high-resolution grayscale image that have been merged and then up-sampled are merged. Among them, high-resolution color images have high color sensitivity, and high-resolution grayscale images have high brightness. Therefore, it can ensure that the color of the output target image is true and the texture is clear, which is beneficial for the camera device to work in lower light. Intensity of the environment.

When the light intensity is higher than the preset value, it will be introduced in conjunction with Figure 2C.

The first image sensor senses the visible light in the first light signal and outputs a high-resolution color image (that is, the aforementioned third image).

The dual-optical filter provided between the second image sensor and the optical component is switched to an infrared cut-off filter, and the infrared cut-off filter is used to filter the infrared light in the second optical signal. Therefore, the second image sensor senses the visible light in the second light signal and outputs a high-resolution color image. At this time, the high-resolution color image (that is, the aforementioned second image) includes not only the second brightness information but also the second color information. It should be understood that when the format of the aforementioned image sensor is different, the format of the output image will also be different, which has been described in detail in the foregoing, and will not be repeated here.

The ISP chip in the camera device can also perform the aforementioned ISP processing on the aforementioned two high-resolution color images. For example, 3D noise reduction, demosaicing, brightness correction and color correction and other processing. Optionally, the ISP chip can also adjust the format of the two high-resolution color images, for example, adjust the Bayer format to the YUV format, etc. The specifics are not limited here.

The image processor fuses the aforementioned two high-resolution color images to obtain the target image. Since the energy of the visible light sensed by the aforementioned two images is different, the aforementioned two images have different brightness, and fusing the aforementioned two images to obtain the target image is beneficial to increase the dynamic range of the target image.

For ease of understanding, take Figure 2C as an example for introduction. The high-resolution RGB format image 211 collected by the first image sensor is converted into a high-resolution YUV format image 212. At the same time, the second image sensor converts the collected high-resolution RGB format image 213 into a high-resolution YUV format image 214. Among them, since the energy of the visible light sensed by the first image sensor and the second image sensor are different, the color and brightness of the high-resolution RGB format image 211 and the high-resolution RGB format image 213 are not completely the same. Similarly, the high-resolution YUV format image 212 and the high-resolution YUV format image 214 are not exactly the same. Then, the high-resolution YUV format image 212 and the high-resolution YUV format image 214 are merged to obtain a high-resolution YUV format image 215 (that is, the target image). The target image can have the advantages of the aforementioned two images, and the dynamic range is improved while the quality of the target image is improved.

It should be understood that although the foregoing FIG. 2B and FIG. 2C use YUV420 as examples, in actual applications, the format of the image can be adjusted according to specific requirements, which is not specifically limited here. In addition, although the aforementioned YUV format and the RGB format are different color coding methods, the change of the coding format will not affect the colors presented by the image.

Optionally, the ratio of visible light in the first optical signal to visible light in the second optical signal in this embodiment is 9:1. It can also be understood that the aforementioned optical components are split according to frequency spectrum and energy at the same time, wherein 90% of the visible light in the incident light signal is irradiated to the first image sensor, and 10% of the visible light and 100% of the infrared light in the incident light signal is irradiated to the first image sensor. The second image sensor. And combined with the filter to further adjust the light signal actually entering the two image sensors.

Optionally, the second image sensor in the aforementioned camera device may also be a black and white image sensor. The black-and-white image sensor may be a MONO format image sensor (Mono image sensor) or a black-and-white image sensor of other formats, which is not specifically limited here. At this time, when the light intensity is lower than the preset value, the second image sensor still outputs a high-resolution grayscale image. Therefore, the implementation of the remaining parts of the camera device may be the same as the implementation corresponding to FIG. 2A, and details are not repeated here. Such an embodiment can also improve the quality of the target image, so that the camera device works in an environment with lower light intensity.

The present invention also provides an image processing method, wherein a first image sensor generates a first image, and a second image sensor generates a second image, and the resolution of the first image is smaller than the resolution of the second image. The image processor performs steps of: adjusting the resolution of the first image to be the same as the resolution of the second image; and generating a target image based on the first image and the second image.

It should be understood that, in the embodiments of the present application, the same reference numerals in different drawings may be regarded as the same object. Unless otherwise specified, the explanations of the same reference numerals between the foregoing various drawings may be mutually cited. Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the above-described system, device, and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

For the convenience and conciseness of the description, the specific working process of the above-described system, device, and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the application has been described in detail with reference to the foregoing embodiments, it is still possible to modify the technical solutions recorded in the foregoing embodiments, or to Some of the technical features are equivalently replaced; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

A camera device, characterized in that it comprises:

An optical component for receiving an incident optical signal, and processing the incident optical signal into a first optical signal and a second optical signal;

The first image sensor has the function of combining adjacent pixels, and is used to sense the first light signal to generate a third image, and combine every N adjacent pixels in the third image into one pixel to obtain the first An image, the image information of the first image includes first color information and first brightness information, and the N is an integer greater than 1;

The second image sensor is used to sense the second light signal to generate a second image, the image information of the second image includes first brightness information, and the resolution of the first image sensor is equal to that of the second image sensor. Resolution, the resolution of the first image is smaller than the resolution of the second image;

An image processor, configured to generate a target image based on the first image and the second image, the color and brightness of the target image are determined by the image information of the first image and the image information of the second image; Or, for generating a target image based on the third image and the second image, the color and brightness of the target image are determined by the image information of the third image and the image information of the second image.
The imaging device according to claim 1, wherein the image processor is specifically configured to:

Adjusting the resolution of the first image to be the same as the resolution of the second image to obtain a fourth image, the fourth image carrying the first color information and the first brightness information;

The fourth image and the second image are merged to obtain the target image.
The imaging device according to claim 1 or 2, wherein the first optical signal includes visible light, the second optical signal includes visible light and infrared light, and the energy of the visible light in the first optical signal is greater than that of the visible light. For the energy of the visible light in the second optical signal, the frequency band of the visible light in the first optical signal is the same as the frequency band of the visible light in the second optical signal.
3. The imaging device according to claim 3, wherein the first image sensor and the second image sensor are both color image sensors.
The imaging device according to claim 4, wherein the imaging device further comprises a visible light cut-off filter;

The imaging device is further configured to activate the visible light cut-off filter when the light intensity is lower than a preset value, the visible light cut-off filter is located between the optical assembly and the second image sensor, and the The visible light cut-off filter is used to filter out visible light in the second optical signal;

The second image sensor is specifically configured to sense infrared light in the second light signal to generate the second image;

The image processor is specifically configured to fuse the fourth image with the second image to obtain the target image.
4. The imaging device according to claim 4, characterized in that the imaging device further comprises an infrared light cut-off filter;

The camera device is also used for activating the infrared light cut-off filter when the light intensity is higher than a preset value, and the infrared cut-off filter is located between the optical assembly and the second image sensor, so The infrared light cut-off filter is used to filter the infrared light in the second optical signal;

The second image sensor is specifically configured to sense visible light in the second light signal to generate the second image, and the image information of the second image further includes second color information;

The image processor is specifically configured to fuse the third image with the second image to obtain the target image.
The imaging device according to any one of claims 1 to 6, wherein the optical component comprises a lens and a dichroic prism, and the dichroic prism is located between the lens and the image sensor;

The lens is used to receive the incident light signal;

The dichroic prism is used to divide the incident light signal received by the lens into the first light signal and the second light signal.
8. The imaging device according to claim 7, wherein the lens is an infrared confocal lens.
The imaging device according to any one of claims 1 to 6, wherein the imaging device further comprises an infrared cut filter;

The optical assembly includes a first lens and a second lens, the first lens and the second lens are used to jointly receive the incident light signal, and the focal length of the first lens is the same as the focal length of the second lens , The aperture of the first lens is larger than the aperture of the second lens, the infrared cut filter is provided between the first lens and the first image sensor, and the second lens is infrared confocal Lens

The first lens is used to receive a part of the incident light signal and transmit the received light signal to the infrared cut filter;

The infrared cut filter is used to filter infrared light in the optical signal from the first lens to obtain the first optical signal, and transmit the first optical signal to the first image sensor ；

The second lens is used to receive the remaining part of the incident light signal, and transmit the received light signal as a second light signal to the second image sensor.
The camera device according to any one of claims 1 to 6, wherein the first image sensor is a Binning sensor.
The imaging device according to claim 1 or 2, wherein:

The format of the first image is YUV format;

The format of the second image is YUV format.