WO2022068598A1

WO2022068598A1 - Imaging method and apparatus

Info

Publication number: WO2022068598A1
Application number: PCT/CN2021/118697
Authority: WO
Inventors: 季军; 陈敏; 胡翔宇
Original assignee: 华为技术有限公司
Priority date: 2020-09-29
Filing date: 2021-09-16
Publication date: 2022-04-07
Also published as: CN114338962A; CN114338962B

Abstract

Provided in the present application are an imaging method and apparatus. The imaging method of the present application comprises: acquiring a reflected optical signal of an image captured object; separating the reflected optical signal into a first optical signal and a second optical signal by means of a light splitting unit, the splitting unit being used for spectral and energy splitting; acquiring a first image signal according to the first optical signal; and acquiring a second image signal according to the second optical signal, the resolution of the first image signal being higher than the resolution of the second image signal. In the present application, a high signal-to-noise ratio of a low-resolution image can be retained, and a high-resolution image can also be acquired, thereby obtaining an image having brighter colors and a higher definition.

Description

Imaging method and apparatus

This application claims the priority of the Chinese patent application with the application number 202011058209.5 and the application title "Imaging Method and Apparatus" filed with the China Patent Office on September 29, 2020, the entire contents of which are incorporated into this application by reference.

technical field

The present application relates to image processing technology, and in particular, to an imaging method and apparatus.

Background technique

When shooting at night, many cameras have problems such as poor image quality and blurred images, which cannot meet the needs of low-illumination scenes, especially at night. At present, there are some technologies for shooting low-light scenes on the market. They use infrared light to fill in the light, and then use a "spectral prism" to divide the light signal into the camera into two light signals below 750nm and above 750nm. The light signal is imaged by two image sensors, and then the two imaged results are fused into a single color image.

However, as the resolution of the sensor increases, the signal-to-noise ratio of the sensor decreases, resulting in poor imaging results in low-light scenes.

SUMMARY OF THE INVENTION

The present application provides an imaging method and device, which can not only retain the high signal-to-noise ratio of low-resolution images, but also acquire high-resolution images, so that images with more vivid colors and higher definition can be obtained.

In a first aspect, the present application provides an imaging method, which includes: acquiring a reflected light signal of a photographed object; separating the reflected light signal into a first light signal and a second light signal by a light splitting unit, and the light splitting unit is used for spectrum and separation energy; obtain the first image signal according to the first optical signal; obtain the second image signal according to the second light signal; in the first illumination scene, obtain the first target image according to the first image signal and the second image signal; In the second illumination scene, the second target image is obtained according to the first image signal; wherein, the resolution of the first image signal is higher than the resolution of the second image signal; the illumination intensity of the second illumination scene is greater than that of the first illumination scene strength.

The application can capture the reflected light signal of the photographed object through a light capture module (such as a lens), and the reflected light signal is consistent with the light irradiated on the lens, that is, the natural light is still natural light after reflection, and the infrared light is still infrared light after reflection , the reflection of light does not change the wavelength of the optical signal in it. Therefore, the reflected light signal in the present application may include natural light (also known as visible light) emitted by the sun, visible light emitted by fluorescent lamps, infrared light emitted by infrared supplementary lights, and the like.

The spectroscopic unit can use a spectroscopic prism, which can separate the visible light signal and the infrared light signal in the reflected light signal to achieve the purpose of spectrum splitting; A first component signal of spectral energy and a second component signal of 60% to 90% visible spectral energy. The first component signal and the infrared light signal constitute the first optical signal on the first optical path, and the second component signal constitutes the second optical signal on the second optical path. The first optical signal is input to the first image sensor, the second optical signal is input to the second image sensor, and the resolution of the first image sensor is higher than that of the second image sensor. The first image sensor converts the first optical signal into a first image signal (for example, the suffix is .raw image) through photoelectric conversion, and the second image sensor converts the second optical signal into a second image signal (such as the suffix name) through photoelectric conversion. for .raw images). As the resolution of the image sensor increases, the unit pixel area on the photosensitive surface of the image sensor decreases, which in turn reduces the number of photons that are photosensitive, resulting in a decrease in the signal-to-noise ratio. Especially in a low illuminance scene (the above-mentioned first illuminance scene), due to insufficient light input on the color light path, an increase in resolution will seriously affect the color effect. In the above method, two image sensors with large and small resolutions are used to form a combination of image sensors with large and small resolutions. The image sensor with small resolution is used for color path imaging and only receives visible light signals; the image sensor with large resolution receives both visible light signals and infrared signals. For light signals, large-resolution color imaging can be directly achieved during the day, and large-resolution infrared images (grayscale images) and small-resolution color images (chromaticity images) can be fused at night. Thus, large-resolution imaging can be supported day and night.

Optionally, in the first illumination scene, the infrared fill light is turned on to illuminate the object to be photographed, and the infrared filter is turned off, so that the first light signal includes the first component signal of the visible light signal and is reflected by the infrared fill light. The infrared light signal; the second light signal includes the second component signal of the visible light signal.

In the first illumination scene, the infrared fill light works and emits an infrared light signal, so the reflected light signal includes a visible light signal and an infrared light signal from the infrared fill light. The spectroscopic unit uses optical principles to separate the reflected light signal by spectrum and energy to obtain a first light signal and a second light signal, wherein the first light signal includes the first component signal of the visible light signal (for example, 10% of the visible light signal) and the infrared signal. The infrared light signal obtained by the reflection of the fill light, and the second light signal includes the second component signal of the visible light signal (for example, 90% of the visible light signal).

In this application, the first grayscale image can be obtained according to the first image signal; the second grayscale image and the chromaticity map can be obtained according to the second image signal; the grayscale fusion of the first grayscale image and the second grayscale image can be performed to obtain the Three grayscale images; the first target image is obtained by color fusion of the third grayscale image and the chromaticity map; wherein, the resolution of the first grayscale image is higher than that of the second grayscale image; the first grayscale image is higher than the resolution of the chromaticity diagram.

Since the first light signal includes part of the visible light signal and the infrared light signal, the first image signal converted from the first light signal can obtain a black and white first grayscale (luma) image. In this application, the first grayscale image is obtained according to the first image signal, which involves an image processing algorithm of ISP, that is, converting an original image in .raw format into an image visible to the human eye.

Since the second light signal only contains a part of the visible light signal, and the visible light signal also includes the infrared light signal (refer to the spectrum of natural light), in this application, the second image signal converted based on the second light signal is converted into grayscale and color The second grayscale (luma) image of black and white and the chromaticity (chroma) image of color are obtained by separation of degrees, and the image processing algorithm of ISP is also involved here, that is, the original image in .raw format is converted into an image visible to the human eye, and the Convert RGB format to YUV format, and separate the image in YUV format.

Since the resolution of the first image sensor is higher than that of the second image sensor, that is, the photosensitive surface of the first image sensor contains more pixels than the photosensitive surface of the second image sensor, the first image acquisition The resolution of the first image signal obtained by the unit is higher than the resolution of the second image signal obtained by the second image acquisition unit. Correspondingly, the resolution of the first grayscale (luma) image is higher than that of the second grayscale (luma) image, and the resolution of the first grayscale (luma) image is also higher than that of the chromaticity (chroma) image. resolution.

In this application, the first grayscale (luma) image and the second grayscale (luma) image are gray-scaled to obtain a third grayscale (luma) image. The high-resolution of the third grayscale (luma) image can be obtained. The same resolution as the first grayscale image above. Then, the third grayscale (luma) image and the chromaticity (chroma) image are color-fused to obtain a target image, and the resolution of the target image is also the same as that of the first grayscale image.

As the resolution of the image sensor increases, the unit pixel area on the photosensitive surface in the image sensor decreases, which in turn reduces the number of photons that are photosensitive, resulting in a decrease in the signal-to-noise ratio. Especially in a low illuminance scene (the above-mentioned first illuminance scene), due to insufficient light input on the color light path, an increase in resolution will seriously affect the color effect. In the above method, two image sensors with large and small resolutions are used to form a combination of image sensors with large and small resolutions. The image sensor with small resolution is used for color path imaging and only receives visible light signals; the image sensor with large resolution receives both visible light signals and infrared signals. For light signals, large-resolution color imaging can be directly achieved during the day, and large-resolution infrared images (grayscale images) and small-resolution color images (chromaticity images) can be fused at night. Thus, large-resolution imaging can be supported day and night.

In the present application, the reflected light signal is separated by a spectroscopic unit, and then the separated light signal is photoelectrically converted by two image sensors of high resolution and low resolution to obtain a corresponding image signal, and then converted into a corresponding image, With the help of two-step fusion processing (grayscale fusion and color fusion), the final target image is obtained. During the grayscale fusion, the first grayscale image also contains visible light signals, which can achieve the same spectrum registration and improve the fusion efficiency of the image. Both grayscale fusion and color fusion are fusions between two high-resolution and low-resolution images, which can not only retain the high signal-to-noise ratio of low-resolution images, but also acquire high-resolution images, so that more color can be obtained. Vivid, higher-definition images.

Optionally, under the second illumination, turn off the infrared fill light and turn on the infrared filter, so that the first light signal includes the first component signal of the visible light signal but does not include the infrared light signal; the second light signal includes visible light. the second component signal of the signal.

In the second illumination scenario, the infrared fill light in the imaging device does not work, and the infrared filter works, so the reflected light signal includes visible light signal and infrared light signal from nature. The light splitting unit in the imaging device uses optical principles to separate the reflected light signal to obtain a first light signal and a second light signal, wherein the first light signal includes the first component signal of the visible light signal (for example, 10% of the visible light signal) and the In the natural infrared light signal, the second light signal includes the second component signal of the visible light signal (for example, 90% of the visible light signal).

The infrared filter filters out the infrared light signal in the first light signal, so the first light signal reaching the first image acquisition unit only includes the first component signal of the visible light signal.

The first image sensor converts the first optical signal into an electrical signal to form a first image signal with an initial image prototype. The first image signal is an electrical signal converted by photoelectricity and constitutes, for example, a suffix named .raw image. Since the first light signal includes a visible light signal, the first image signal is a colored visible light image signal.

The second image sensor converts the second optical signal into an electrical signal to form a second image signal with an initial image prototype, and the second image signal is also a photoelectrically converted electrical signal, forming, for example, an image with a suffix of .raw. Since the second light signal only includes visible light signals, the second image signal is a colored visible light image signal.

Similarly, the resolution of the first image signal is higher than that of the second image signal. The image processing module obtains a target image according to the first image signal, and the resolution of the target image is the same as that of the first image signal and higher than that of the second image signal.

In order to reduce the amount of computation, the image processing unit may only use the second image signal to detect the illumination intensity, and it is not necessary to use the second image signal to perform image fusion. In the second illumination scene, a high-resolution image can be directly acquired, thereby improving the imaging effect of the image.

In a possible implementation, the method further includes: judging the current illumination scene according to the second light signal; when the illumination intensity corresponding to the second light signal is less than the first threshold, determining that the current illumination scene is the first illumination scene; when the illumination intensity When it is greater than or equal to the first threshold, it is determined that the current illumination scene is the second illumination scene; or, when the signal gain corresponding to the second light signal is greater than the second threshold, it is determined that the current illumination scene is the first illumination scene; or, when the signal gain When it is less than or equal to the second threshold, it is determined that the current illumination scene is the second illumination scene.

The above-mentioned first threshold and second threshold may be set in advance according to historical data or experience. The light intensity can be detected, for example, by a light sensor on the imaging device. The signal gain can be detected, for example, by an image sensor in the second image acquisition unit.

In a second aspect, the present application provides an imaging device, comprising: a light capturing module for acquiring a reflected light signal of a photographed object; a light splitting module for separating the reflected light signal into a first light signal and a a second optical signal, the spectroscopic unit is used for spectrum and energy division; an image acquisition module is used for acquiring a first image signal according to the first optical signal; acquiring a second image signal according to the second optical signal; wherein , the resolution of the first image signal is higher than the resolution of the second image signal.

In a possible implementation manner, the method further includes: an image processing module, configured to acquire a first target image according to the first image signal and the second image signal under a first illumination scene; In an illumination scene, a second target image is acquired according to the first image signal; wherein the illumination intensity of the second illumination scene is greater than the illumination intensity of the first illumination scene.

In a possible implementation manner, the light splitting unit is used to separate the visible light signal and the infrared light signal in the reflected light signal, so as to achieve the purpose of spectrum splitting; the light splitting unit is also used to separate the visible light signal The signal is divided into a first component signal that accounts for 10%-40% of the visible spectrum energy and a second component signal that accounts for 60%-90% of the visible spectrum energy.

In a possible implementation manner, under the first illumination, the image processing module is further configured to turn on an infrared fill light to illuminate the photographed object, and turn off the infrared filter, so that the first An optical signal includes a first component signal of a visible light signal and an infrared light signal reflected by the infrared fill light; the second optical signal includes a second component signal of the visible light signal.

In a possible implementation manner, under the second illumination, the image processing module is further configured to turn off the infrared fill light and turn on the infrared filter, so that the first light signal includes a visible light signal The first component signal of but does not include the infrared light signal; the second light signal includes the second component signal of the visible light signal.

In a possible implementation manner, the image acquisition module is specifically configured to input the first optical signal into a first image sensor to acquire the first image signal; input the second optical signal into the second image sensor Acquiring the second image signal; wherein the resolution of the first image sensor is higher than the resolution of the second image sensor.

In a possible implementation manner, the image processing module is specifically configured to obtain a first grayscale image according to the first image signal; obtain a second grayscale image and a chromaticity map according to the second image signal; Performing grayscale fusion on the first grayscale image and the second grayscale image to obtain a third grayscale image; performing color fusion on the third grayscale image and the chromaticity map to obtain the first target image; wherein, the resolution of the first grayscale image is higher than the resolution of the second grayscale image; the resolution of the first grayscale image is higher than the resolution of the chromaticity map.

In a possible implementation manner, the image processing module is further configured to determine the current lighting scene according to the second light signal; when the light intensity corresponding to the second light signal is less than a first threshold, determine the The current illumination scene is the first illumination scene; when the illumination intensity is greater than or equal to the first threshold, it is determined that the current illumination scene is the second illumination scene; or, when the second light signal corresponds to When the signal gain is greater than the second threshold, determine that the current lighting scene is the first lighting scene; or, when the signal gain is less than or equal to the second threshold, determine that the current lighting scene is the first lighting scene Two-light scene.

In a third aspect, the present application provides a terminal device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors , causing the one or more processors to implement the method according to any one of the above first aspects.

In a fourth aspect, the present application provides a computer-readable storage medium, which is characterized by comprising a computer program, and when the computer program is executed on a computer, causes the computer to perform any one of the above-mentioned first aspects. method.

In a fifth aspect, the present application provides a computer program, characterized in that, when the computer program is executed by a computer, it is used to execute the method according to any one of the above-mentioned first aspects.

Description of drawings

FIG. 1 is a flowchart of an embodiment of an imaging device of the present application;

FIG. 2 is an exemplary block diagram of a device embodiment of the present application;

FIG. 3 is a flowchart of a process 300 of an embodiment of the imaging method of the present application;

FIG. 4 is an exemplary frame diagram of an embodiment of an imaging method of the present application;

FIG. 5 is a schematic structural diagram of an embodiment of an imaging device of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the accompanying drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application. , not all examples. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms "first", "second", etc. in the description, embodiments and claims of the present application and the drawings are only used for the purpose of distinguishing and describing, and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implied order. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, eg, comprising a series of steps or elements. A method, system, product or device is not necessarily limited to those steps or units expressly listed, but may include other steps or units not expressly listed or inherent to the process, method, product or device.

It should be understood that, in this application, "at least one (item)" refers to one or more, and "a plurality" refers to two or more. "And/or" is used to describe the relationship between related objects, indicating that there can be three kinds of relationships, for example, "A and/or B" can mean: only A, only B, and both A and B exist , where A and B can be singular or plural. The character "/" generally indicates that the associated objects are an "or" relationship. "At least one item(s) below" or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a) of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, c can be single or multiple.

The following is an explanation of the terms involved in this application:

Illuminance: Illumination intensity refers to the energy of visible light received per unit area, referred to as illuminance, and the unit is Lux (Lux or lx). Illuminance can be used to indicate the degree to which an object is illuminated, that is, the ratio of the luminous flux obtained on the surface of the object to the illuminated area. For example, in summer, under direct sunlight, the illuminance can reach 60000lx～100000lx; outdoors without the sun, the illuminance can reach 1000lx～10000lx; bright indoors, the illuminance can reach 100lx～550lx; under the full moon at night, the illuminance can reach 0.2lx.

The present application involves two application scenarios, namely a first illumination scene and a second illumination scene, and the illumination of the first illumination scene is smaller than that of the second illumination scene. For example, the first illuminance scene may refer to no light at night, a dimly lit room, a dark corner with poor daily light, etc.; the second illuminance scene may refer to an outdoor in the daytime, an indoor with sufficient light during the day, and an indoor with sufficient light. or outdoor etc.

YUV: A color space format for pixels, Y is the luminance (luma) component of the pixel, representing the luminance or gray level intensity, and images based on the luminance component are black and white images; UV is the chroma (chroma) component of the pixel, representing the pixel Color, an image based on chrominance components is a color image.

Visible light: Also known as visible light signal. The part of the electromagnetic spectrum that the human eye can perceive has a wavelength between 400nm and 750nm, for example.

Infrared light: Also known as infrared light signal. An electromagnetic wave whose frequency is between microwave and visible light, and its wavelength is greater than 750nm, for example, between 760nm-1mm.

Prism: A polyhedron made of transparent materials (such as glass, crystal, etc.) used to split or disperse light. Prisms are widely used in spectroscopic instruments. For example, the "dispersive prism" that decomposes the composite light into the spectrum, the more commonly used is the equilateral prism; or the "total reflection prism" that changes the direction of the light to adjust its imaging position, the more commonly used are the periscope, binocular Right angle prisms in instruments such as telescopes.

Image sensor: uses the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. The photosensitive surface is divided into many small units, and each small unit corresponds to a pixel. . For example, the Bayer sensor, RGB color filters are arranged on the photosensitive surface, forming a mosaic color filter array (color filter array), 50% of this color filter array is green, which is used to sense green light , 25% is red for sensing red light, and 25% is blue for sensing blue light.

Signal-to-noise ratio: The ratio of the signal strength experienced by the sensor to the generated noise strength. Usually, when the size of the lens and sensor remain unchanged, the signal-to-noise ratio of the sensor is positively related to the area of a single pixel on the photosensitive surface. For example, a 1/1.8-inch sensor with 4 million pixels occupies 3um per pixel. If the resolution is increased to 8 million pixels, each pixel occupies 2um. Obviously, the area of the photosensitive surface corresponding to a single pixel is reduced, which weakens the light signal it receives, thereby reducing the signal-to-noise ratio.

Heterospectral image: Image of different spectral imaging. For example, the wavelengths of 400nm-750nm belong to visible light, and the wavelengths of 750nm-850nm belong to infrared light, and the imaging images of these two kinds of light are heterospectral images.

FIG. 1 is a flowchart of an embodiment of an imaging device of the present application. As shown in FIG. 1 , the imaging device may include: a light capturing unit 10 , a light splitting unit 20 , a first image capturing unit 30 , a second image capturing unit 40 and an image processing unit unit 50. Wherein, the output end of the light capturing unit 10 is connected to the input end of the light splitting unit 20; the two output ends of the light splitting unit 20 are respectively connected to the input end of the first image obtaining unit 30 and the input end of the second image obtaining unit 40; The output terminal of the image acquisition unit 30 and the output terminal of the second image acquisition unit 40 are respectively connected to one input terminal of the image processing unit 50 .

The light capturing unit 10 is used to capture the reflected light signal of the photographed object. For example, the light capturing unit 10 may be a camera or a lens of a camera, light (or light signal) irradiates on a photographed object, and is captured by the lens after being reflected by the photographed object. The aforementioned light may be emitted by any light source in the environment where the subject is located, which may include natural light (also known as visible light) emitted by the sun, visible light emitted by fluorescent lamps, and infrared light emitted by infrared fill lights. The light capturing unit 10 is responsible for capturing the reflected light signal, which is consistent with the aforementioned light, that is, natural light is still natural light after reflection, and infrared light is still infrared light after reflection, and the reflection of light does not change the wavelength of the light signal.

The light splitting unit 20 is configured to separate the reflected light signal into a first light signal and a second light information number, transmit the first light signal to the first image acquisition unit, and transmit the second light signal to the second image acquisition unit. The light splitting unit 20 can use a prism, and use the optical principle of the prism to decompose the reflected optical signal into two optical signals, so that the two optical signals can be independently processed in subsequent modules.

The first image acquisition unit 30 is configured to generate a first image signal according to the first optical signal, and transmit the first image signal to the image processing unit; the second image acquisition unit 40 is configured to generate a second image signal according to the second optical signal, and The second image signal is transmitted to the image processing unit; the resolution of the first image signal is higher than that of the second image signal. The first image acquisition unit 30 and the second image acquisition unit 40 respectively include an image sensor to convert the incident optical signal into an electrical signal to form an image signal with an initial image. The optical signal is hit on the photosensitive surface, and the photoelectric conversion of the received optical signal is performed by the photosensitive surface to obtain the corresponding electrical signal, and the electrical signals of all the pixels on the photosensitive surface form the image signal. Usually, the image format output by the Bayer sensor can be the original image inside the imaging device, and its suffix is .raw. The resolution of the first image signal is higher than that of the second image signal because the resolution of the image sensor in the first image acquisition unit 30 is higher than the resolution of the image sensor in the second image acquisition unit 40, that is, The photosensitive surface of the image sensor in the first image acquisition unit 30 contains more pixels than the photosensitive surface of the image sensor in the second image acquisition unit 40 contains more pixels.

The image processing unit 50 is configured to acquire a target image according to the first image signal and the second image signal. The image processing unit 50 may be any processor or processing chip with data processing capability and computing capability, that is, a software program running on the aforementioned processor or processing chip. For example, the image processing unit 50 may adopt a system on chip (SOC) integrated in an image signal processor (ISP), and the SOC is a complete system integrated on a single chip, which is necessary for all or part of the The technique of packet grouping by electronic circuits. A complete system generally includes a central processing unit, memory, and peripheral circuits.

Optionally, the imaging device may further include: a supplementary light unit 60 connected to the image processing unit 50 . The supplementary light unit 60 may be any device that provides infrared light signals, such as an infrared supplementary light lamp. The present application can control the working state of the supplementary light unit 60 through the image processing unit 50. For example, the supplementary light unit 60 can only work in the first illumination scene, in order to supplement the light intensity; In order to cope with the situation where the illumination changes greatly or changes frequently.

Optionally, the imaging device may further include: a filter unit 70 , the filter unit 70 is disposed between the output end of the spectroscopic unit 20 and the input end of the first image acquisition unit 30 . The filter unit 70 may be any device with the function of filtering out optical signals of a certain wavelength or wavelengths in a certain range, especially to filter out infrared light with a wavelength greater than 750 nm, such as an infrared light filter. In the present application, the filter unit 70 may only work in the second illumination scene, in order to filter out infrared light signals when the illumination is relatively high.

The image processing unit 50 is further configured to determine whether the current illumination scene is the first illumination scene or the second illumination scene according to the second image signal. For example, the illumination intensity is detected according to the second image signal, and when the illumination intensity is less than the first threshold, it is determined that the current illumination scene is the first illumination scene; or, when the illumination intensity is greater than or equal to the first threshold, it is determined that the current illumination scene is the second illumination scene. Illumination scene. For another example, the signal gain is detected according to the second image signal, and when the signal gain is greater than the second threshold, it is determined that the current illumination scene is the first illumination scene; or, when the signal gain is less than or equal to the second threshold, it is determined that the current illumination scene is the first illumination scene. Two-light scene. The above-mentioned first threshold and second threshold may be set in advance according to historical data or experience. The light intensity can be detected, for example, by a light sensor on the imaging device. The signal gain can be detected by an image sensor in the second image acquisition unit 40, for example.

When the imaging device is in the first illumination scene, the fill light unit 60 works, and the filter unit 70 stops working.

The reflected light signal acquired by the light capturing unit 10 may include a visible light signal and a first infrared light signal from an infrared fill light (eg, fill light unit 60 ). The light splitting unit 20 separates the reflected light signal into a first component signal including a visible light signal (for example, a 10% visible light signal), a first light signal including a first infrared light signal, and a second component signal including the visible light signal (for example, 90%). % of the visible light signal) of the second light signal.

The first image acquisition unit 30 obtains a first image signal according to the first optical signal, where the first image signal is an electrical signal converted by photoelectric conversion, and constitutes, for example, an original picture with a suffix named .raw. Since the first light signal includes an infrared light signal, the first image signal is a first grayscale image signal in black and white.

The second image acquisition unit 40 obtains a second image signal according to the second optical signal, and the second image signal is also a photoelectrically converted electrical signal, forming an original image with a suffix named .raw, for example. Since the second light signal only includes visible light signals, the second image signal is a colored visible light image signal.

The image processing unit 50 performs luminance and chrominance separation on the second image signal to obtain a second grayscale map and a chroma map, and obtains a first grayscale map based on the first image signal. Since the resolution of the image sensor in the first image acquisition unit 30 is higher than that of the image sensor in the second image acquisition unit 40 , the resolution of the first grayscale image is higher than that of the second grayscale (luma) The resolution of the map and the chroma map. The image processing unit 50 performs grayscale fusion of the first grayscale image and the second grayscale image signal to obtain a third grayscale (luma) image, the high resolution of the third grayscale (luma) image is the same as the above-mentioned first Grayscale images have the same resolution. The image processing unit 50 then performs color fusion of the third grayscale (luma) image and the chromaticity (chroma) image to obtain a target image, and the resolution of the target image is also the same as the resolution of the first grayscale image.

In the first illumination scene, during grayscale fusion, the first grayscale image also contains visible light signals, which can achieve the same spectrum registration and improve the fusion efficiency of the image, while the grayscale fusion and color fusion are both high resolution and low The fusion between two high-resolution images can not only retain the high signal-to-noise ratio of low-resolution images, but also obtain high-resolution images, so that more vivid and high-definition images can be obtained.

When the imaging device is in the second illumination scene, the supplementary light unit 60 stops working, and the filter unit 70 works.

The reflected light signals acquired by the light capturing unit 10 may include visible light signals and infrared light signals from nature. The light splitting unit 20 separates the reflected light signal into a first component signal including a visible light signal (for example, a 10% visible light signal), a first light signal including an infrared light signal from nature, and a second component signal including the visible light signal (for example, 90% of the visible light signal) of the second light signal.

The filter unit 70 filters out the infrared light signal in the first light signal, so the first light signal reaching the first image acquisition unit 30 only includes the first component signal of the visible light signal.

The first image acquisition unit 30 obtains a first image signal according to the first optical signal, where the first image signal is an electrical signal converted by photoelectric conversion, and constitutes, for example, an original picture with a suffix named .raw. Since the first light signal only includes visible light signals, the first image signal is a colored visible light image signal.

The second image acquisition unit 40 obtains a second image signal according to the second optical signal, and the second image signal is also a photoelectrically converted electrical signal, forming an original image with a suffix named .raw, for example. Since the second light signal only includes visible light signals, the second image signal is also a colored visible light image signal. In the present application, the second image signal is of low resolution. In order to reduce the amount of computation, only the second image signal can be used to detect the illumination intensity, and the image processing unit 50 does not need to use the second image signal to perform image fusion.

The image processing unit 50 obtains a target image based on the first image signal. The resolution of the target image is the same as the resolution of the first image signal and higher than the resolution of the second image signal.

In the second illumination scene, a high-resolution image can be directly acquired, thereby improving the imaging effect of the image.

The above imaging device can be applied to devices with shooting functions, such as smart phones, tablet computers, cameras, and surveillance cameras. Fig. 2 is an exemplary block diagram of an embodiment of a device of the present application, and Fig. 2 shows a schematic structural diagram when the device is a mobile phone.

As shown in FIG. 2, the mobile phone 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1 , Antenna 2, Mobile Communication Module 250, Wireless Communication Module 260, Audio Module 270, Speaker 270A, Receiver 270B, Microphone 270C, Headphone Interface 270D, Sensor Module 280, Key 290, Motor 291, Indicator 292, Camera 293, Display Screen 294, and a subscriber identification module (subscriber identification module, SIM) card interface 295 and so on. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, and ambient light. Sensor 280L, bone conduction sensor 280M, image sensor 280N, etc.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the mobile phone 200 . In other embodiments of the present application, the mobile phone 200 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

In cooperation with the camera 293 and the image sensor 280N, the processor 210 in this application can implement the functions of the image processing unit 50 in the imaging device shown in FIG. 1 .

The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in processor 210 is cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 210 . If the processor 210 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided, and the waiting time of the processor 210 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 210 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may contain multiple sets of I2C buses. The processor 210 can be respectively coupled to the touch sensor 280K, the charger, the flash, the camera 293 and the like through different I2C bus interfaces. For example, the processor 210 can couple the touch sensor 280K through the I2C interface, so that the processor 210 and the touch sensor 280K communicate with each other through the I2C bus interface, so as to realize the touch function of the mobile phone 200.

The I2S interface can be used for audio communication. In some embodiments, the processor 210 may contain multiple sets of I2S buses. The processor 210 may be coupled with the audio module 270 through an I2S bus to implement communication between the processor 210 and the audio module 270 . In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 may be coupled through a PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 210 with the wireless communication module 260 . For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 210 with peripheral devices such as the display screen 294 and the camera 293 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 210 communicates with the camera 293 through the CSI interface, so as to realize the shooting function of the mobile phone 200 . The processor 210 communicates with the display screen 294 through the DSI interface to realize the display function of the mobile phone 200 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 210 with the camera 293, the display screen 294, the wireless communication module 260, the audio module 270, the sensor module 280, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 230 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 230 can be used to connect a charger to charge the mobile phone 200, and can also be used to transmit data between the mobile phone 200 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other mobile phones, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration, and does not constitute a structural limitation of the mobile phone 200 . In other embodiments of the present application, the mobile phone 200 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 240 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from the wired charger through the USB interface 230 . In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through the wireless charging coil of the mobile phone 200 . While the charging management module 240 charges the battery 242 , it can also supply power to the mobile phone through the power management module 241 .

The power management module 241 is used to connect the battery 242 , the charging management module 240 and the processor 210 . The power management module 241 receives input from the battery 242 and/or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 241 may also be provided in the processor 210 . In other embodiments, the power management module 241 and the charging management module 240 may also be provided in the same device.

The wireless communication function of the mobile phone 200 can be realized by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in handset 200 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 250 may provide a wireless communication solution including 2G/3G/4G/5G, etc. applied on the mobile phone 200 . The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), and the like. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and then convert it into electromagnetic waves for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 250 may be provided in the processor 210 . In some embodiments, at least part of the functional modules of the mobile communication module 250 may be provided in the same device as at least part of the modules of the processor 210 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 270A, the receiver 270B, etc.), or displays images or videos through the display screen 294 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 210, and may be provided in the same device as the mobile communication module 250 or other functional modules.

The wireless communication module 260 can provide applications on the mobile phone 200 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2 , modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210 . The wireless communication module 260 can also receive the signal to be sent from the processor 210 , perform frequency modulation on the signal, amplify the signal, and then convert it into an electromagnetic wave for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the mobile phone 200 is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the mobile phone 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou navigation satellite system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi satellite system) -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The mobile phone 200 realizes the display function through the GPU, the display screen 294, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 294 is used to display images, videos, and the like. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, cell phone 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

The mobile phone 200 can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor.

The ISP is used to process the data fed back by the camera 293 . For example, when taking a photo, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 293 . In this application, the ISP can acquire the target image according to the first image signal and the second image signal. It can also be determined according to the second image signal whether the current illumination scene is the first illumination scene or the second illumination scene. Optionally, the ISP performs luminance and chromaticity separation on the second image signal to obtain a first grayscale (luma) map and a chrominance (chroma) map, and obtains a first grayscale map based on the first image signal, the first grayscale map The resolution is higher than that of the second grayscale image and the chromaticity image. The first grayscale (luma) image and the second grayscale image are grayscale-fused to obtain a third grayscale image, and the third grayscale image has the same high resolution as the first grayscale image. The third grayscale image and the chroma map are then color-fused to obtain a target image, and the resolution of the target image is also the same as the resolution of the first grayscale image.

Camera 293 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the mobile phone 200 may include one or N cameras 293 , where N is a positive integer greater than one.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the mobile phone 200 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point, and the like.

Video codecs are used to compress or decompress digital video. The handset 200 may support one or more video codecs. In this way, the mobile phone 200 can play or record videos in various encoding formats, such as: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information, and can continuously learn by itself. Applications such as intelligent cognition of the mobile phone 200 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 200 . The external memory card communicates with the processor 210 through the external memory interface 220 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 221 may be used to store computer executable program code, which includes instructions. The internal memory 221 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area can store data (such as audio data, phone book, etc.) created during the use of the mobile phone 200 and the like. In addition, the internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like. The processor 210 executes various functional applications and data processing of the mobile phone 200 by executing the instructions stored in the internal memory 221 and/or the instructions stored in the memory provided in the processor.

The mobile phone 200 can implement audio functions through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, and an application processor. Such as music playback, recording, etc.

The audio module 270 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 270 may also be used to encode and decode audio signals. In some embodiments, the audio module 270 may be provided in the processor 210 , or some functional modules of the audio module 270 may be provided in the processor 210 .

Speaker 270A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. Mobile phone 200 can listen to music through speaker 270A, or listen to hands-free calls.

The receiver 270B, also referred to as an "earpiece", is used to convert audio electrical signals into sound signals. When the mobile phone 200 receives a call or a voice message, the voice can be received by placing the receiver 270B close to the human ear.

The microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 270C through the human mouth, and input the sound signal into the microphone 270C. The mobile phone 200 may be provided with at least one microphone 270C. In other embodiments, the mobile phone 200 may be provided with two microphones 270C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the mobile phone 200 may further be provided with three, four or more microphones 270C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The headphone jack 270D is used to connect wired headphones. The earphone interface 270D may be a USB interface 230, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 280A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 280A may be provided on the display screen 294 . There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. .

The gyroscope sensor 280B can be used to determine the motion attitude of the mobile phone 200 .

Air pressure sensor 280C is used to measure air pressure.

Magnetic sensor 280D includes a Hall sensor.

The acceleration sensor 280E can detect the magnitude of the acceleration of the mobile phone 200 in various directions (generally three axes).

Distance sensor 280F for measuring distance.

Proximity light sensor 280G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes.

The ambient light sensor 280L is used to sense ambient light brightness.

The fingerprint sensor 280H is used to collect fingerprints.

The temperature sensor 280J is used to detect the temperature.

The touch sensor 280K is also called "touch device". The touch sensor 280K may be disposed on the display screen 294, and the touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 280K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 294 . In other embodiments, the touch sensor 280K may also be disposed on the surface of the mobile phone 200 , which is different from the location where the display screen 294 is located.

The bone conduction sensor 280M can acquire vibration signals.

The image sensor (sensor) 280N uses the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. The photosensitive surface is divided into many small units, and each small unit corresponds to a pixel. . For example, the Bayer sensor, RGB color filters are arranged on the photosensitive surface, forming a mosaic color filter array (color filter array), 50% of this color filter array is green, which is used to sense green light , 25% is red for sensing red light, and 25% is blue for sensing blue light.

The keys 290 include a power-on key, a volume key, and the like. Keys 290 may be mechanical keys. It can also be a touch key. The cell phone 200 can receive key input and generate key signal input related to user settings and function control of the cell phone 200 .

Motor 291 can generate vibrating cues. The motor 291 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback.

The indicator 292 can be an indicator light, which can be used to indicate the charging status, the change of power, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 295 is used to connect a SIM card. The SIM card can be contacted and separated from the mobile phone 200 by inserting into the SIM card interface 295 or pulling out from the SIM card interface 295 . The mobile phone 200 may support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 295 can support Nano SIM card, Micro SIM card, SIM card and so on. The mobile phone 200 interacts with the network through the SIM card to realize functions such as call and data communication. In some embodiments, the handset 200 employs an eSIM, ie: an embedded SIM card. The eSIM card can be embedded in the mobile phone 200 and cannot be separated from the mobile phone 200 .

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the device. In other embodiments of the present application, the device may include more or less components than shown, or some components may be combined, or some components may be split, or different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

FIG. 3 is a flowchart of a process 300 of an embodiment of an imaging method of the present application. The process 300 may be performed by the imaging device shown in FIG. 1 , and specifically, may be performed by a mobile phone, a tablet computer, a video camera or a camera, etc. including the imaging device. Process 300 is described as a series of steps or operations, and it should be understood that process 300 may be performed in various orders and/or concurrently, and is not limited to the order of execution shown in FIG. 3 . Process 300 may include:

Step 301: Acquire a reflected light signal of a photographed object.

The imaging device can capture the reflected light signal of the photographed object through the light capture unit (such as a lens) therein, and the reflected light signal is consistent with the aforementioned light, that is, after the natural light is reflected, it is still natural light, and after the infrared light is reflected, it is still infrared light. The reflection does not change the wavelength of the optical signal in it. Therefore, the reflected light signal in the present application may include natural light (also known as visible light) emitted by the sun, visible light emitted by fluorescent lamps, infrared light emitted by infrared supplementary lights, and the like.

Step 302: The reflected optical signal is separated into a first optical signal and a second optical signal by the optical splitting unit.

The beam splitting unit can use a beam splitter prism, and in this application, the beam splitter prism is used to separate the reflected light signal into a first optical path and a second optical path, wherein the first optical path can be allocated to 10% to 40% of the visible spectrum energy and greater than 80% The infrared spectral energy, the second optical path can be distributed to 60% to 90% of the visible spectral energy and less than 20% of the infrared spectral energy.

The optical signal (first optical signal) on the first optical path is input to the first image sensor, and the optical signal (second optical signal) on the second optical path is input to the second image sensor, and the resolution of the first image sensor is higher than that of the second image The resolution of the sensor. The first image sensor converts the first optical signal into a first image signal (for example, the suffix is .raw image) through photoelectric conversion, and the second image sensor converts the second optical signal into a second image signal (such as the suffix name) through photoelectric conversion. for .raw images). As the resolution of the image sensor increases, the unit pixel area on the photosensitive surface in the image sensor decreases, which in turn reduces the number of photons that are photosensitive, resulting in a decrease in the signal-to-noise ratio. Especially in a low illuminance scene (the above-mentioned first illuminance scene), due to insufficient light input on the color light path, an increase in resolution will seriously affect the color effect. In the above method, two image sensors with large and small resolutions are used to form a combination of image sensors with large and small resolutions. The image sensor with small resolution is used for color path imaging and only receives visible light signals; the image sensor with large resolution receives both visible light signals and infrared signals. For light signals, large-resolution color imaging can be directly achieved during the day, and large-resolution infrared images (grayscale images) and small-resolution color images (chromaticity images) can be fused at night. Thus, large-resolution imaging can be supported day and night.

In the first illumination scene, the supplementary light unit in the imaging device works and emits an infrared light signal, so the reflected light signal includes a visible light signal and the first infrared light signal from the infrared supplementary light. The spectroscopic unit in the imaging device uses optical principles to perform spectral and energy separation on the reflected light signal to obtain a first light signal and a second light signal, where the first light signal includes the first component signal of the visible light signal (for example, 10% of the visible light signal). ) and the infrared light signal reflected by the infrared fill light, the second light signal includes the second component signal of the visible light signal (for example, 90% of the visible light signal).

Step 303: Acquire a first image signal according to the first optical signal.

The first image acquisition unit in the imaging device converts the first optical signal into an electrical signal by using the working principle of the image sensor to form a first image signal with an initial image prototype. The first image signal is an electrical signal converted by photoelectricity, which constitutes For example, the suffix is .raw original image. Since the first light signal includes an infrared light signal, the first image signal is a first grayscale image signal in black and white.

The image processing unit in the imaging device obtains the first grayscale image according to the first image signal, which involves the image processing algorithm of the ISP, that is, converts the original image in the .raw format into an image visible to the human eye. The first grayscale image is black and white.

Step 304: Acquire a second image signal according to the second optical signal.

The second image acquisition unit in the imaging device converts the second optical signal into an electrical signal by using the working principle of the image sensor to form a second image signal with an initial image prototype, and the second image signal is also an electrical signal converted by photoelectricity, forming For example, the suffix is .raw original image. Since the second light signal only includes visible light signals, the second image signal is a colored visible light image signal.

The image processing unit in the imaging device separates the luminance and chromaticity of the second image signal to obtain a second grayscale (luma) image and a chrominance (chroma) image, which also involves the image processing algorithm of the ISP, that is, the .raw format. The original image is converted into an image visible to the human eye, while the RGB format is converted into YUV format, and the image in YUV format is separated. The second grayscale map (luma) is in black and white, and the chroma map is in color.

Because the resolution of the image sensor in the first image acquisition unit in the imaging device is higher than the resolution of the image sensor in the second image acquisition unit, that is, the pixels included in the photosensitive surface of the image sensor in the first image acquisition unit There are more pixels than the photosensitive surface of the image sensor in the second image acquisition unit. The resolution of the first image signal is higher than that of the second image signal, so the resolution of the first image signal obtained by the first image acquisition unit has a higher resolution. higher than the resolution of the second image signal obtained by the second image acquisition unit. Correspondingly, the resolution of the first grayscale image obtained in step 303 is higher than the resolution of the second grayscale (luma) image and the chromaticity (chroma) image obtained in step 304 .

Optionally, in the first illumination scene, the first target image is acquired according to the first image signal and the second image signal.

Optionally, in the second illumination scene, the second target image is acquired according to the first image signal.

The resolution of the first image signal is higher than the resolution of the second image signal; the illumination intensity of the second illumination scene is greater than the illumination intensity of the first illumination scene.

The image processing unit in the imaging device performs grayscale fusion of the first grayscale (luma) image and the second grayscale image to obtain a third grayscale image. The high resolution of the third grayscale image is the same as the above-mentioned first grayscale image. The resolution of the degree map is the same. The third grayscale image and the chroma map are then color-fused to obtain a target image, and the resolution of the target image is also the same as the resolution of the first grayscale image.

In the second illumination scene, the supplementary light unit in the imaging device does not work, and the filter unit works, so the reflected light signal includes the visible light signal and the second infrared light signal from nature. The light splitting unit in the imaging device uses optical principles to separate the reflected light signal to obtain a first light signal and a second light signal, wherein the first light signal includes the first component signal of the visible light signal (for example, 10% of the visible light signal) and the In the natural infrared light signal, the second light signal includes the second component signal of the visible light signal (for example, 90% of the visible light signal).

The filter unit in the imaging device filters out the second infrared light signal in the first light signal, so the first light signal reaching the first image acquisition unit only includes the first component signal of the visible light signal.

The first image acquisition unit in the imaging device converts the first optical signal into an electrical signal by using the working principle of the image sensor to form a first image signal with an initial image prototype. The first image signal is an electrical signal converted by photoelectricity, which constitutes For example, the suffix is .raw original image. Since the first light signal includes a visible light signal, the first image signal is a colored visible light image signal.

Similarly, the resolution of the first image signal is higher than that of the second image signal. The image processing unit in the imaging device obtains a target image according to the first image signal, and the resolution of the target image is the same as that of the first image signal and higher than that of the second image signal.

The above-mentioned first threshold and second threshold may be set in advance according to historical data or experience. The light intensity can be detected, for example, by a light sensor on the imaging device. The signal gain can be detected by an image sensor in the second image acquisition unit 40, for example.

A specific embodiment is used below to describe in detail the technical solution of the method embodiment shown in FIG. 3 .

The following is the configuration of the imaging device in this embodiment:

1. The light capture unit uses an F1.4 constant aperture lens, which is confocal in the wavelength range of 400nm to 940nm.

2. The fill light unit uses a set of 850nm band light emitting diode (LED) infrared fill lights, a total of 6 pieces. The fill light unit is built in the imaging device, and the imaging device controls the switch and brightness of the infrared fill light through the I2C bus.

3. The beam splitting unit uses an optical prism, which is composed of two isosceles right-angled triangle glasses. Based on the coating technology and the design of prism parameters, it can transmit all the infrared light signals and 10% of the visible light signals in the received reflected light signals. to the A direction, while refracting 90% of the visible light signal in the reflected light signal to the B direction. The A direction and the B direction form an included angle of 90 degrees.

4. The filter unit uses IR-CUT dual filters, which are installed in the A direction behind the optical prism. When the IR-CUT dual filter is in working state, it only allows the light signal with the wavelength of 400nm to 750nm to pass through. When the IR-CUT dual filter is in non-working state, all light signals can be passed through.

5. The first image acquisition unit uses a 1/1.2 target surface, a 4K Bayer sensor (hereinafter referred to as a 4K sensor), and the resolution of the sensor is 3840×2160.

6. The second image acquisition unit uses a 1/1.2 target surface, a 2K Bayer sensor (hereinafter referred to as a 2K sensor), and the resolution of the sensor is 1920×1080.

7. The image processing unit uses the SOC processor of the ARM+DSP architecture.

FIG. 4 is an exemplary frame diagram of an embodiment of an imaging method of the present application. As shown in FIG. 4 , the imaging method of this embodiment includes a day mode and a night mode, wherein the night mode corresponds to the first illumination mode, and the day mode corresponds to the second illumination mode Scenes.

Deploy the camera on the pole at a preset angle, the initial mode of the camera is day mode, and the exposure time of the 2K sensor is set to 10ms. When the SOC processor detects that the gain of the second image signal from the 2K sensor is greater than or equal to 30db, the camera switches to night mode; when the SOC processor detects that the gain of the second image signal from the 2K sensor is less than 30db, it continues to run in day mode.

In day mode:

(1) Turn off the LED infrared fill light in the camera. Start the IR-CUT dual filter work to filter the second infrared light signal in the A direction.

(2) The 4K sensor captures 10% of the visible light signal and converts it into the first image signal and transmits it to the SOC processor. At the same time, the 2K sensor captures 90% of the visible light signal and converts it into a second image signal, which is also transmitted to the SOC processor.

(3) The SOC processor opens an ISP pipe (pipe), and performs image processing on the first image signal through the ISP pipe, and finally outputs a high-resolution RGB image, and the RGB image is color.

(4) The SOC processor also detects the signal gain according to the second image signal to determine whether mode conversion is required.

In night mode:

(1) Turn on the LED infrared fill light in the camera, and the LED infrared fill light emits infrared light with a wavelength of 850 nm toward the shooting direction. Set the IR-CUT dual filter to not work, and the first infrared light signal in the A direction can pass through.

(2) The 4K sensor captures 10% of the visible light signal and the first infrared light signal and converts it into a first image signal, which is transmitted to the SOC processor. At the same time, the 2K sensor captures 90% of the visible light signal and converts it into a second image signal, which is also transmitted to the SOC processor.

(3) The SOC processor opens two ISP pipes, performs image processing on the first image signal through one ISP pipe, and converts the 4K RGB image signal into a high-resolution first grayscale image. At the same time, image processing is performed on the second image signal through another ISP pipe, and the 2K RGB signal is converted into a low-resolution RGB image.

(4) The SOC processor converts the low-resolution RGB image into YUV format to obtain a low-resolution luminance (luma) image (ie, Y channel) and a low-resolution chrominance (Choma) image (ie, UV channel).

(5) The SOC processor performs grayscale fusion of the low-resolution luminance (luma) image and the high-resolution first grayscale image to obtain a high-resolution luminance (luma) image.

(6) The SOC processor performs color fusion of a high-resolution luminance (luma) image and a low-resolution chrominance (Choma) image to obtain a high-resolution RGB image output.

(7) The SOC processor also detects the signal gain according to the second image signal to determine whether mode conversion is required.

FIG. 5 is a schematic structural diagram of an embodiment of an imaging apparatus of the present application. As shown in FIG. 5 , the apparatus may be applied to the terminal device in the above-mentioned embodiment. The imaging device of this embodiment may include: a light capturing module 1501 , a light splitting module 1502 , an image acquiring module 1503 and an image processing module 1504 . in,

The light capture module 1501 is used to acquire the reflected light signal of the photographed object; the light splitting module 1502 is used to separate the reflected light signal into a first light signal and a second light signal by a light splitting unit, and the light splitting unit is used for dividing spectrum and sub-energy; the image acquisition module 1503 is configured to acquire a first image signal according to the first optical signal; acquire a second image signal according to the second optical signal; wherein, the resolution of the first image signal is high on the resolution of the second image signal.

In a possible implementation manner, the image processing module 1504 is configured to acquire a first target image according to the first image signal and the second image signal in a first illumination scene; or, in a second illumination scene Next, a second target image is acquired according to the first image signal; wherein, the illumination intensity of the second illumination scene is greater than the illumination intensity of the first illumination scene.

In a possible implementation manner, under the first illumination, the image processing module 1504 is further configured to turn on an infrared fill light to illuminate the photographed object, and turn off the infrared filter, so that the The first optical signal includes a first component signal of a visible light signal and an infrared light signal reflected by the infrared fill light; the second optical signal includes a second component signal of the visible light signal.

In a possible implementation manner, under the second illumination, the image processing module 1504 is further configured to turn off the infrared fill light and turn on the infrared filter, so that the first light signal includes visible light The first component signal of the signal does not include the infrared light signal; the second light signal includes the second component signal of the visible light signal.

In a possible implementation manner, the image acquisition module 1503 is specifically configured to input the first optical signal into the first image sensor to acquire the first image signal; input the second optical signal into the second image The sensor acquires the second image signal; wherein the resolution of the first image sensor is higher than the resolution of the second image sensor.

In a possible implementation manner, the image processing module 1504 is specifically configured to obtain a first grayscale map according to the first image signal; obtain a second grayscale map and a chromaticity map according to the second image signal Perform grayscale fusion of the first grayscale image and the second grayscale image to obtain a third grayscale image; perform color fusion on the third grayscale image and the chromaticity map to obtain the first grayscale image The target image; wherein the resolution of the first grayscale image is higher than the resolution of the second grayscale image; the resolution of the first grayscale image is higher than the resolution of the chromaticity map.

In a possible implementation manner, the image processing module 1504 is further configured to determine the current lighting scene according to the second light signal; when the light intensity corresponding to the second light signal is less than the first threshold, determine the the current illumination scene is the first illumination scene; when the illumination intensity is greater than or equal to the first threshold, determine that the current illumination scene is the second illumination scene; or, when the second light signal When the corresponding signal gain is greater than the second threshold, determine that the current illumination scene is the first illumination scene; or, when the signal gain is less than or equal to the second threshold, determine that the current illumination scene is the first illumination scene The second illumination scene.

The apparatus of this embodiment can be used to execute the technical solution of the method embodiment shown in FIG. 3 , and its implementation principle and technical effect are similar, and details are not repeated here.

In the implementation process, each step of the above method embodiments may be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other Programming logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the methods disclosed in the embodiments of the present application may be directly embodied as executed by a hardware coding processor, or executed by a combination of hardware and software modules in the coding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

The memory mentioned in the above embodiments may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically programmable Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) ) and direct memory bus random access memory (direct rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution, and the computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

An imaging method, comprising:

Obtain the reflected light signal of the object being photographed;

The reflected light signal is separated into a first light signal and a second light signal by a light splitting unit, the light splitting unit is used for spectrum and energy;

obtaining a first image signal according to the first optical signal;

obtaining a second image signal according to the second optical signal;

Wherein, the resolution of the first image signal is higher than the resolution of the second image signal.
The method according to claim 1, wherein after acquiring the second image signal according to the second optical signal, the method further comprises:

In a first illumination scene, a first target image is acquired according to the first image signal and the second image signal; or, in a second illumination scene, a second target image is acquired according to the first image signal;

Wherein, the illumination intensity of the second illumination scene is greater than the illumination intensity of the first illumination scene.
The method according to claim 1 or 2, wherein the spectroscopic unit is used to separate the visible light signal and the infrared light signal in the reflected light signal, so as to achieve the purpose of spectrum splitting; the spectroscopic unit is further for dividing the visible light signal into a first component signal with 10%-40% visible spectrum energy and a second component signal with 60%-90% visible spectrum energy.
The method according to claim 3, wherein, under the first illuminance, before acquiring the reflected light signal of the photographed object, the method further comprises:

Turning on the infrared fill light to illuminate the object to be photographed, and turning off the infrared filter, so that the first light signal includes the first component signal and the infrared light signal reflected by the infrared fill light; The second optical signal includes the second component signal.
The method according to claim 3, wherein, under the second illuminance, before acquiring the reflected light signal of the photographed object, the method further comprises:

The infrared fill light is turned off, and the infrared filter is turned on, so that the first optical signal includes the first component signal but does not include the infrared light signal; the second optical signal includes the second component signal.
The method according to any one of claims 1-5, wherein the acquiring the first image signal according to the first optical signal comprises:

inputting the first optical signal into a first image sensor to obtain the first image signal;

The acquiring a second image signal according to the second optical signal includes:

inputting the second optical signal into a second image sensor to obtain the second image signal;

Wherein, the resolution of the first image sensor is higher than the resolution of the second image sensor.
The method according to any one of claims 1-6, wherein the acquiring the first target image according to the first image signal and the second image signal comprises:

obtaining a first grayscale image according to the first image signal;

obtaining a second grayscale map and a chromaticity map according to the second image signal;

performing gray fusion on the first grayscale image and the second grayscale image to obtain a third grayscale image;

performing color fusion on the third grayscale map and the chromaticity map to obtain the first target image;

The resolution of the first grayscale image is higher than the resolution of the second grayscale image; the resolution of the first grayscale image is higher than the resolution of the chromaticity map.
The method according to any one of claims 1-7, further comprising:

Determine the current lighting scene according to the second light signal;

When the illumination intensity corresponding to the second light signal is less than the first threshold, determine that the current illumination scene is the first illumination scene; when the illumination intensity is greater than or equal to the first threshold, determine the current illumination scene The illumination scene is the second illumination scene; or,

When the signal gain corresponding to the second optical signal is greater than the second threshold, it is determined that the current illumination scene is the first illumination scene; or, when the signal gain is less than or equal to the second threshold, it is determined that the current illumination scene is the first illumination scene; The current illumination scene is the second illumination scene.
An imaging device, characterized in that it includes:

The light capture module is used to obtain the reflected light signal of the photographed object;

a light splitting module, configured to separate the reflected light signal into a first light signal and a second light signal by a light splitting unit, where the light splitting unit is used for spectrum and energy;

an image acquisition module, configured to acquire a first image signal according to the first optical signal; acquire a second image signal according to the second optical signal;

Wherein, the resolution of the first image signal is higher than the resolution of the second image signal.
The device of claim 9, further comprising:

An image processing module, configured to obtain a first target image according to the first image signal and the second image signal under a first illumination scene; or, under a second illumination scene, obtain according to the first image signal the second target image;

Wherein, the illumination intensity of the second illumination scene is greater than the illumination intensity of the first illumination scene.
The device according to claim 9 or 10, wherein the spectroscopic unit is used to separate the visible light signal and the infrared light signal in the reflected light signal, so as to achieve the purpose of spectrum splitting; the spectroscopic unit is further for dividing the visible light signal into a first component signal with 10%-40% visible spectrum energy and a second component signal with 60%-90% visible spectrum energy.
The device according to claim 11, wherein, under the first illumination, the image processing module is further configured to turn on an infrared fill light to illuminate the photographed object, and turn off the infrared filter, so that all The first optical signal includes the first component signal and the infrared light signal reflected by the infrared fill light; the second optical signal includes the second component signal.
The device according to claim 11, wherein under the second illumination, the image processing module is further configured to turn off the infrared fill light and turn on the infrared filter, so that the first light signal includes The first component signal does not include the infrared light signal; the second light signal includes the second component signal.
The device according to any one of claims 9-13, wherein the image acquisition module is specifically configured to input the first optical signal into a first image sensor to acquire the first image signal; The second optical signal is input to a second image sensor to obtain the second image signal; wherein, the resolution of the first image sensor is higher than that of the second image sensor.
The device according to any one of claims 9-14, wherein the image processing module is specifically configured to obtain a first grayscale image according to the first image signal; obtain a grayscale image according to the second image signal a second grayscale image and a chromaticity map; perform grayscale fusion of the first grayscale image and the second grayscale image to obtain a third grayscale image; combine the third grayscale image and the chromaticity The first target image is obtained by color fusion of the images; wherein, the resolution of the first grayscale image is higher than that of the second grayscale image; the resolution of the first grayscale image is higher than that of the first grayscale image. The resolution of the chromaticity diagram.
The device according to any one of claims 9-15, wherein the image processing module is further configured to determine the current lighting scene according to the second light signal; when the lighting corresponding to the second light signal When the intensity is less than the first threshold, it is determined that the current illumination scene is the first illumination scene; when the illumination intensity is greater than or equal to the first threshold, the current illumination scene is determined to be the second illumination scene; Or, when the signal gain corresponding to the second optical signal is greater than a second threshold, it is determined that the current illumination scene is the first illumination scene; or, when the signal gain is less than or equal to the second threshold, It is determined that the current illumination scene is the second illumination scene.
A terminal device, characterized in that it includes:

one or more processors;

memory for storing one or more programs;

The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
A computer-readable storage medium, characterized by comprising a computer program, which, when executed on a computer, causes the computer to execute the method of any one of claims 1-8.
A computer program, characterized in that, when the computer program is executed by a computer, it is used to execute the method of any one of claims 1-8.