WO2023010874A1

WO2023010874A1 - Image photographing apparatus and image processing method

Info

Publication number: WO2023010874A1
Application number: PCT/CN2022/084916
Authority: WO
Inventors: 朱力于; 杨昆
Original assignee: 华为技术有限公司
Priority date: 2021-08-05
Filing date: 2022-04-01
Publication date: 2023-02-09
Also published as: CN115942071A

Abstract

An image photographing apparatus and an image processing method. In the present image photographing apparatus, after a visible light photographing structure photographs a target, a visible light image is generated, wherein the visible light image comprises a first dot matrix formed by dot matrix light. A non-visible light photographing structure photographs the target to generate a non-visible light image, wherein the non-visible light image comprises a second dot matrix formed by dot matrix light. A processor fuses the visible light image and an infrared light image according to the first dot matrix and the second dot matrix, and outputs a fused image. A visible light image and a non-visible light image are generated by different photographing structures, and the photographing structures are relatively independent, such that the design difficulty of the image photographing apparatus can be reduced. When a processor performs fusion, the fusion can be realized by means of dot matrices on the visible light image and the non-visible light image, such that the fusion difficulty can be reduced, thereby improving the image processing efficiency.

Description

An image capture device and image processing method

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202110898529.X and the application title "an image capturing device and image processing method" submitted to the China Patent Office on August 5, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the technical field of image processing, and in particular to an image capturing device and an image processing method.

Background technique

In the field of surveillance, there are many situations where surveillance images are not clear due to insufficient visible light illumination. In order to reduce the occurrence of unclear monitoring images, infrared supplementary light can be used to supplement light on the captured target. In addition to generating visible light images, infrared images can also be generated accordingly, and then infrared images and visible light images can be fused together. In order to obtain a clear surveillance image. On the one hand, infrared images can make up for the shortcomings of blurred images caused by insufficient visible light illumination. On the other hand, infrared images can cover more detailed information, so that the fusion of infrared images and visible light images can show more details.

At present, common shooting devices adopt monocular spectroscopic method to generate infrared images and visible light images respectively. That is to say, a lens is used to shoot the target, and then a dichroic prism is used behind the lens to separate the incident light into visible light and infrared light. After that, the visible light and infrared light pass through different transmission routes and are projected onto different photosensitive elements to form infrared light. Light image and visible light image. Afterwards, the infrared light image and the visible light image are fused to obtain a fused image.

The method of monocular light splitting needs to focus on infrared light and visible light at the same time, which poses a great challenge to the focusing performance of the lens, and also increases the difficulty of lens design, which is not easy to realize.

Contents of the invention

The present application provides an image capturing device and an image processing method, which are used to simplify the design difficulty of the image capturing device and reduce the difficulty of image fusion.

In a first aspect, an embodiment of the present application provides an image capturing device, which includes a processor and two different capturing structures. The two different shooting structures are the visible light shooting structure and the non-visible light shooting structure. These two shooting structures can shoot the target projected by the dot matrix light.

Wherein, after the visible light photographing structure photographs the target, it can generate a visible light image, and the visible light image includes a first dot matrix formed by dot matrix light. The non-visible light photographing structure photographs the target to generate a non-visible light image, and the non-visible light image includes a second dot matrix formed by dot matrix light.

The processor can fuse the visible light image and the infrared light image according to the first lattice and the second lattice, and output a fusion image.

Through the above device, the visible light image and the non-visible light image are generated by different photographing structures, and the photographing structures are relatively independent, which can reduce the design difficulty of the image capturing device. Since there are dot matrixes (such as the first dot matrix and the second dot matrix) formed by dot matrix light on both the visible light image and the non-visible light image, the processor can use the dot matrix on the visible light image and the non-visible light image when performing fusion The realization of fusion can reduce the difficulty of fusion and improve the efficiency of image processing.

In a possible implementation manner, the shooting time of the visible light shooting structure is the same as the shooting time of the non-visible light shooting structure. For example, the visible light shooting structure and the non-visible light shooting structure can simultaneously shoot the target and maintain the same exposure time.

Through the above device, the exposure time of the visible light imaging structure and the non-visible light imaging structure is the same, which can ensure that the generated visible light image and the content displayed in the non-visible light image are basically the same.

In a possible implementation manner, the non-visible light may be infrared light, for example, the wavelength of the non-visible light in this application may be greater than 750 nanometers (nm).

Through the above device, when the invisible light is infrared light, the generated invisible image is an infrared image, and the infrared image can contain more detailed information than the visible light image, and the fusion image generated by the subsequent fusion of the visible light image and the infrared image is also clearer.

In a possible implementation manner, the dot matrix light source capable of providing dot matrix light may be an external device, or may be a device built into the image capture device. The dot matrix light source can project dot matrix light to the target.

Through the above device, the setting method of the dot matrix light source is more flexible, and can be applied to different scenes.

In a possible implementation manner, when the processor fuses the visible light image and the non-visible light image, based on the point-to-point correspondence between the first lattice and the second lattice, the visible light image and the non-visible light image Perform registration; fuse the registered visible light image and non-visible light image to generate a fusion image.

Through the above device, the arrangement of light points in the dot matrix in the visible light image and the non-visible light image is relatively regular, and the registration can be realized more conveniently by using the dot matrix in the visible light image and the non-visible light image. The visible light image and the dot matrix in the non-visible light image after registration Non-visible light images can also be well fused, and the fusion method is relatively simple.

In a possible implementation manner, the first dot matrix and the second dot matrix include a plurality of light points, and the processor calculates the visible light When the image and the non-visible light image are registered, it is possible to first determine the pair of light points that have a corresponding relationship between the first lattice and the second lattice, and the pair of light points that have a corresponding relationship include a light point of the first lattice and a light point of the second For the dots of the dot matrix, the dots that have a corresponding relationship are the dots whose feature similarity is greater than the threshold in the first dot matrix and the second dot matrix. The light spots formed by the same beam of light in the array light; the sum, the processor then registers the visible light image and the non-visible light image according to the corresponding light spots.

By means of the above device, the registration is realized by utilizing the corresponding pairs of light spots, which can improve the accuracy of the registration and ensure the clarity of the subsequently generated fused image.

In a possible implementation, the visible light imaging structure includes a visible light lens and a visible light sensor; the visible light lens captures the visible light reflected by the shooting target, and projects the visible light onto the visible light sensor, that is, the optical lens only passes through the visible light. The visible light sensor senses visible light and generates a visible light image.

The non-visible light shooting structure includes a non-visible light lens and a non-visible light sensor; the non-visible light lens can capture the non-visible light reflected by the target, and project the non-visible light onto the non-visible light sensor. The non-visible light sensor senses non-visible light and generates non-visible light images.

Through the above device, the visible light shooting structure and the non-visible light shooting structure respectively include a lens and a sensor, which are relatively independent. There is no need for confocal between the visible light shooting structure and the non-visible light shooting structure, which can reduce the construction difficulty of the two shooting structures.

In a possible implementation, the visible light shooting structure further includes an infrared fill light; the infrared fill light can project infrared light to the target when the visible light shooting structure is shooting.

Through the above device, the infrared supplementary light can project infrared light to ensure the clarity of the infrared image generated by the non-visible light shooting structure.

In a possible implementation manner, the visible light lens or the non-visible light lens may be a fixed-focus lens or a zoom lens.

Through the above device, the type of the visible light lens or the non-visible light lens is relatively flexible, and there may be many different building methods for the visible light shooting structure and the non-visible light shooting structure.

In a possible implementation manner, the wavelength of the dot matrix light covers the visible light band and the infrared light band, so that the dot matrix can be formed in the visible light image and the non-visible light image at the same time.

In a possible implementation manner, the wavelength of the lattice light may also only cover the visible light band, such as greater than 560 nm and less than 750 nm. In this case, the visible light lens can be a lens that only passes visible light, and the non-visible light lens can be an image that can pass both visible light and non-visible light. In this way, a dot matrix can be formed in both the visible light image and the non-visible light image.

In a possible implementation manner, the projection duration of the dot matrix light is less than 80 milliseconds, and the projection time of the dot matrix light is short, which will not affect the shooting of subsequent image capture devices.

In the second aspect, the embodiment of the present application provides an image processing method, which can be executed by an image capturing device. For specific beneficial effects, please refer to the relevant description in the first aspect.

The visible light shooting structure and the non-visible light shooting structure in the image shooting device can shoot the target projected by the lattice light, the visible light shooting structure generates a visible light image, and the visible light image includes the first dot matrix formed by the lattice light. The non-visible light photographing structure generates a non-visible light image, and the non-visible light image includes a second lattice formed by lattice light.

The processor in the image capture device can fuse the visible light image and the non-visible light image according to the first dot matrix and the second dot matrix, and output a fusion image.

In a possible implementation manner, the shooting time of the visible light shooting structure is the same as that of the non-visible light shooting structure.

In a possible implementation manner, the non-visible light is infrared light.

In a possible implementation manner, when the processor fuses the visible light image and the non-visible light image according to the first lattice and the second lattice, and outputs the fused image, it may According to the corresponding relationship between light points and light points, the visible light image and the non-visible light image are registered; after that, the registered visible light image and non-visible light image are fused to generate a fusion image.

In a possible implementation manner, the first dot matrix and the second dot matrix include a plurality of light points, and the processor calculates the visible light When the image and the non-visible light image are registered, it is possible to first determine the pair of light points that have a corresponding relationship between the first lattice and the second lattice, and the pair of light points that have a corresponding relationship include a light point of the first lattice and a light point of the second For the light points of the lattice, the light points with a corresponding relationship are the light points with a feature similarity greater than the threshold in the first lattice and the second lattice; after that, the visible light image and the non-visible light image are matched according to the light points with the corresponding relationship. allow.

In a possible implementation manner, when the processor registers the visible light image and the non-visible light image according to the corresponding light points, it can obtain the affine transformation parameters according to the corresponding light points, and then use the affine The transformation parameters register the visible and non-visible images.

In a possible implementation, if the visible light shooting structure and the non-visible light shooting structure in the image shooting device subsequently shoot a target (during subsequent shooting, the target may no longer project dot matrix light), a new The visible light image and the new non-visible light image (dot matrix may not exist on the new visible light image and the new non-visible light image). The processor in the image capture device can register the new visible light image and the new infrared image according to the previously determined affine transformation parameters, and then fuse the registered visible light image and infrared light image to output a fused image .

Description of drawings

FIG. 1 is a schematic structural diagram of an image capturing device provided by the present application;

FIG. 2 is a schematic structural diagram of a visible light shooting structure provided by the present application;

Fig. 3 is a structural schematic diagram of a non-visible light shooting structure provided by the present application;

4A to 4B are schematic diagrams of a dot matrix light, a first dot matrix and a second dot matrix provided by the present application;

FIG. 5 is a schematic diagram of an image processing method provided by the present application;

FIG. 6 is a schematic diagram of various images generated during an image processing process provided by the present application.

Detailed ways

As shown in FIG. 1 , an embodiment of the present application provides an image capturing device, and the image capturing device 100 includes two different capturing structures and a processor 130 .

The two different shooting structures are capable of shooting the target, one of which is used to generate visible light images, and the other shooting structure is used to generate non-visible light images. For the convenience of description, the photographing structure for generating visible light images is referred to as the visible light photographing structure 110 , and the photographing structure for generating non-light images is referred to as the non-visible light photographing structure 120 .

The objects photographed by the visible light photographing structure 110 and the invisible light photographing structure 120 are the objects projected by the dot matrix light. In the embodiment of the present application, the dot matrix light includes multiple parallel light beams, and the multiple parallel light beams are arranged according to a specific rule or a specific pattern. For example, the multiple parallel light beams may be arranged in a matrix (that is, the distance between adjacent light beams is the same). For another example, the multiple parallel light beams may form a five-pointed star or a hexagon.

It should be noted that in the embodiment of the present application, the objects photographed by the visible light photographing structure 110 and the non-visible light photographing structure 120 are not limited to people, animals, plants or scenery, but also living places (such as office buildings, buildings, office areas) , landscapes, buildings, traffic roads, etc., all objects that can be photographed are applicable to this embodiment of the application.

Since the target is projected by the lattice light, the lattice light will also be reflected on the images generated by the visible light photographing structure 110 and the invisible light photographing structure 120 , and there will be some objects formed by the lattice light on the visible light image generated by the visible light photographing structure 110 . dot matrix. In order to distinguish the dot matrix on the visible light image from the dot matrix on the non-visible light image, the dot matrix on the visible light image is called the first dot matrix, and the dot matrix on the non-visible light image is called the second dot matrix.

After the visible light imaging structure 110 and the non-visible light imaging structure 120 photograph the target and generate the visible light image and the non-visible light image, the processor 130 can fuse the visible light image and the non-visible light image according to the first lattice and the second lattice, Generate a fused image.

Optionally, the image capture device 100 may also include a dot matrix light source 140 capable of generating dot matrix light. Coverage target. What needs to be explained here is that in addition to the target, the coverage of the dot matrix light can also include other things in the scene where the target is located. The coverage of the dot matrix light is related to the projection range of the dot matrix light source.

In the embodiment of the present application, when the processor 130 fuses the visible light image and the infrared light image, it can use the first dot matrix in the non-visible light image and the second dot matrix in the visible light image, the first dot matrix and the second dot matrix The existence of can effectively reduce the difficulty of image fusion, so that visible light images and infrared light images can achieve more accurate and effective image fusion, and the effect of fusion images is also better.

Each component in the image capture device 100 is further described below:

(1) Visible light imaging structure 110 .

As shown in FIG. 2 , the visible light shooting structure 110 includes a visible light lens 111 and a visible light sensor 112 . The visible light sensor 112 may be located at the light output side of the visible light lens 111 , that is, the visible light sensor 112 may be located at the light output side of the visible light lens 111 . In the embodiment of the present application, visible light refers to light with a wavelength greater than 580 nm to 750 nm.

The specific positions of the visible light lens 111 and the visible light sensor 112 are not limited in the embodiment of the present application, for example, the visible light lens 111 and the visible light sensor 112 may be arranged along the optical axis. For another example, the visible light shooting device may include a lens to change the propagation direction of the visible light collected by the visible light lens 111 , so that the visible light collected by the visible light lens 111 can be projected onto the visible light sensor 112 . For any arrangement of the visible light lens 111 and the visible light sensor 112, if the arrangement can achieve the effect that the visible light collected by the visible light lens 111 can be projected to the visible light sensor 112, then this arrangement is applicable to the embodiment of the present application.

The visible light lens 111 can capture the visible light reflected by the target (the visible light also includes the light reflected by the dot light projected on the target), gather the visible light reflected by the target, and project it onto the visible light sensor 112 . The visible light lens 111 may be a fixed-focus lens or a zoom lens.

The visible light sensor 112 can also be called an image sensor or a photosensitive element. The visible light sensor 112 can realize photoelectric conversion, sense the light projected on the visible light sensor 112 (here mainly refers to visible light), convert the light into a corresponding electrical signal, and then Visible light images are generated using electrical signals. The embodiment of the present application does not limit the type of the visible light sensor 112, and the visible light sensor 112 may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide semiconductor (complementary metal-oxide semiconductor, CMOS).

It should be noted that in the visible light shooting structure 110, the visible light lens 111 can only pass through visible light, and can filter the non-visible light reflected by the target, remove the non-visible light reflected by the target, and keep the visible light, that is, the visible light lens 111 can After the visible light reflected by the target is converged, the light projected onto the visible light sensor 112 is visible light. The embodiment of the present application does not limit the manner in which the visible light lens 111 filters non-visible light. For example, a filter is added to the visible light lens 111, and the filter can filter non-visible light, such as infrared light. For another example, the visible light lens 111 may be coated with a filter film capable of filtering non-visible light.

(2) The non-visible light photographing structure 120 .

As shown in FIG. 3 , the non-visible light shooting structure 120 includes a non-visible light lens 121 and a non-visible light sensor 122 . The non-visible light sensor 122 may be located at the light output side of the non-visible light lens 121 , that is, the non-visible light sensor 122 may be located at the light output side of the non-visible light lens 121 . In the embodiment of the present application, the non-visible light may include all light or part of light except visible light. For example, non-visible light may specifically refer to infrared light (light with a wavelength range greater than 0.75 microns and less than 1000 microns), and may also include infrared light and other non-visible light.

In the embodiment of the present application, the specific positions of the non-visible light lens 121 and the non-visible light sensor 122 are not limited. See the above description.

The non-visible light lens 121 can capture the non-visible light reflected by the target and the light reflected by the dot matrix light projected on the target, gather the non-visible light reflected by the target and the light reflected by the dot matrix light projected on the target, and project to the invisible light sensor 122. The non-visible light lens 121 may be a fixed-focus lens or a zoom lens.

The invisible light sensor 122 is similar to the visible light sensor 112 , and details can be referred to the foregoing description. It should be noted here that the non-visible light lens 121 can project the non-visible light and the light reflected by the dot light projected on the target to the non-visible light sensor 122 . The invisible light sensor 122 can sense the invisible light and the light reflected by the dot light projected on the target, convert the sensed invisible light into a corresponding electrical signal, and then use the electrical signal to generate an invisible light image.

Optionally, in order to make the shooting effect of the non-visible light shooting structure 120 better, as shown in FIG. 110 in. The infrared supplementary light 123 can emit infrared light, and when the invisible light shooting structure 120 is photographing a target, the infrared supplementary light 123 can be turned on to project the infrared light onto the target. In this way, the non-visible light lens 121 will capture a large amount of infrared light, and a large amount of infrared light will be converged by the non-visible light lens 121 and projected onto the non-visible light sensor 122. The non-visible light sensor 122 can realize photoelectric conversion, and the generated non-visible light image It is clearer and can cover more detailed information on the target. Of course, in some possible scenarios, the infrared supplementary light 123 can also be installed externally, such as being arranged outside the entire image capture device 100 .

In the embodiment of the present application, there are two possible implementations of the non-visible light photographing structure 120 and the visible light photographing structure 110 .

Method 1: The visible light lens 111 in the visible light shooting structure 110 is different from the non-visible light lens 121 of the non-visible light shooting structure 120 , and the non-visible light sensor 122 and the visible light sensor 112 may be the same.

In this implementation manner, the visible light lens 111 in the visible light shooting structure 110 can filter non-visible light and retain visible light. The non-visible light lens 121 is different from the visible light lens 111 in that the light that can be captured includes non-visible light. Optionally, the light that can be captured by the non-visible light lens 121 may also include visible light. It can be seen from the foregoing description about the visible light lens 111 that the visible light lens 111 can filter non-visible light, such as by setting a filter or coating the visible light lens 111 to achieve the effect of filtering non-visible light, then without setting a filter or on the lens A lens that is not coated with a filter film capable of filtering non-visible light can be used as the non-visible light lens 121 .

In this implementation, the invisible light sensor 122 and the visible light sensor 112 can be the same photosensitive element, and the photosensitive element can sense the light projected onto the photosensitive element. Here, the light projected onto the photosensitive element can be invisible light or visible light.

In this implementation, the wavelength range of the lattice light can only cover visible light, for example, the wavelength of the lattice light is greater than 560 nanometers and less than 750 nanometers, so that the lattice can be formed on both visible light images and non-visible light images. Optionally, the wavelength range of the lattice light may also cover non-visible light, that is, the wavelength of the lattice light is greater than 560 nanometers. In the case where the wavelength range of the dot matrix light covers non-visible light and visible light, the dot matrix can also be formed on the visible light image or the non-visible light image.

Method 2: The visible light sensor 112 in the visible light photographing structure 110 is different from the invisible sensor of the invisible light photographing structure 120 , and the invisible light lens 121 and the visible light lens 111 may be the same.

In this implementation manner, the visible light sensor 112 can only sense visible light projected on the visible light sensor 112 , but not non-visible light. The invisible light sensor 122 may only sense invisible light projected on the invisible light sensor 122 instead of visible light.

The non-visible light lens 121 and the visible light lens 111 may be the same lens, for example, the non-visible light lens 121 and the visible light lens 111 are lenses that do not filter non-visible light. The non-visible light lens 121 and the visible light lens 111 can be different lenses, for example, the non-visible light lens 121 can be a lens that does not filter non-visible light (such as a lens that is not provided with a filter or is not plated with a filter film), and the visible light lens 111 is a filter that does not filter non-visible light. A lens for visible light (such as a lens with a filter or a filter coating).

In this implementation, the wavelength range of the dot matrix light can cover visible light and non-visible light, for example, the wavelength of the dot matrix light is greater than 560 nanometers, so that the dot matrix can be formed on both visible light images and non-visible light images.

No matter which implementation is adopted, when the image capture device 100 captures a target, the visible light capture structure 110 and the non-visible light capture structure 120 can capture the object at the same time or at a similar time (a similar time refers to a time when the time difference is less than a threshold) The target is photographed to generate a visible light image and a non-visible light image at the same time or a similar time. The shooting time of the visible light shooting structure 110 and the non-visible light shooting structure 120 may be the same, that is, the same exposure time may be maintained. In this way, it can be guaranteed that the generated visible light image and the non-visible light image cover the same information.

(3). The processor 130.

The embodiment of the present application does not limit the specific type of the processor 130, the processor 130 is a central processing unit (central processing unit, CPU), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA), artificial intelligence (artificial intelligence, AI) chip, system on chip (system on chip, SoC) or complex programmable logic device (complex programmable logic device, CPLD), graphics processing unit (graphics processing unit, GPU )wait. Any processor capable of processing images and realizing image fusion is applicable to the embodiments of the present application.

In the embodiment of the present application, when the processor 130 fuses the visible light image and the non-visible light image, the first dot matrix in the visible light image and the second dot matrix in the non-visible light image can be used to realize the fusion of the visible light image and the non-visible light image. fusion.

Processor 130 includes two steps in the process of obtaining the fused image. The first step is to first register the visible light image and the non-visible light image, and the second step is to fuse the registered visible light image and the non-visible light image. , to obtain a fused image.

Step 1. Registration.

In the embodiment of the present application, the registration (registration) is to match and superimpose the visible light image and the non-visible light image. When performing registration, first extract features from the visible light image and the non-visible light image to obtain the feature points; find the corresponding feature point pairs by measuring the similarity of the characteristics; then obtain the affine transformation parameters through the corresponding feature point pairs ; Afterwards, the visible light image and the non-visible light image are aligned using the affine transformation parameters. Wherein, the affine transformation refers to a change between two two-dimensional spaces, and the affine transformation parameter is a parameter for realizing a change from one two-dimensional space to another two-dimensional space. In the embodiment of the present application, an image can represent a two-dimensional space. In this way, in the embodiment of the present application, the affine transformation parameters obtained by using the corresponding feature point pairs are the parameters for realizing the transformation from the visible light image to the non-visible light image, or Parameters for converting non-visible light images to visible light images.

The key to registration is to be able to find feature point pairs that have a corresponding relationship. Since there are lattices in both the visible light image and the non-visible light image in the embodiment of the present application, the processor 130 can use the first lattice and the second lattice to find the pairs of feature points that exist. Corresponding pairs of light spots. Afterwards, the processor 130 obtains affine transformation parameters using the light point.

When performing registration, the processor 130 may directly use the light points in the first dot matrix and the second dot matrix to find a corresponding light point pair. It is also possible to perform edge alignment on the visible light image and the non-visible light image first, that is, to first align the light spots located on the edge of the image among the light spots in the first dot matrix and the second dot matrix, and find the corresponding light dot pairs. Then use the remaining light points of the first lattice and the second lattice to find pairs of light points that have a corresponding relationship.

The processor 130 may analyze the light points in the first dot matrix and the light dots in the second dot matrix, and determine the light dot pairs corresponding to the first dot matrix and the second dot matrix. There may be one or more corresponding light point pairs in the first dot matrix and the second dot matrix. A corresponding light point pair includes two light points, one is a light point in the first dot matrix, and the other is a light dot in the second dot matrix. The reason why the two light points have a corresponding relationship is that the similarity degree of the image features of the two light points is greater than a threshold. That is to say, the corresponding light points are the light points whose feature similarity in the first lattice and the second lattice is greater than a threshold.

The processor 130 may use an image analysis algorithm to determine, from the light points in the first dot matrix and the light points in the second dot matrix, the dots in the first dot matrix and the second dot matrix that have a corresponding relationship. The image analysis algorithms here include but are not limited to scale-invariant feature transform (SIFT), OPB (oriented FAST and rotated BRIEF), accelerated robust features (speeded up robust features, SURF).

Taking the SIFT algorithm as an example, the SHIFT algorithm can analyze the light points in the first lattice and the light points in the second lattice, and find the distance between the light points in the first lattice and the light points in the second lattice. Correspondence, and then determine the pair of light points that have a correspondence.

After the processor 130 determines the corresponding light point pairs in the first dot matrix and the second dot matrix, use the coordinates of the light points in the first dot matrix in the light point pair in the visible light image and the light point centering in the second dot pair. The coordinates of the light points in the lattice in the non-visible light image obtain the affine transformation parameters.

After determining the affine transformation parameters, the processor 130 may use the affine transformation parameters to align the visible light image to the non-visible light image, or align the non-visible light image to the visible light image. So far, the registration of the visible light image and the non-visible light image is completed. After the registration is completed, the processor 130 may execute step 2---fusion.

Step two, fusion.

The embodiment of the present application does not limit the manner in which the processor 130 fuses the registered visible light image and the non-visible light image.

For example, when the processor 130 fuses the registered visible light image and the non-visible light image, a manner of fusing the high frequency component and the low frequency component separately may be adopted. Low-frequency components represent areas in the image where brightness or grayscale values change slowly, such as flat or main parts of the image. High-frequency components are portions of an image that change drastically, for example, edges (contours), noise, or detail portions of an image. The low-frequency components of the visible-light image and the non-visible-light image are fused at a set first ratio, for example, the first ratio may be a ratio in which the low-frequency component of the visible-light image dominates. The visible light image and the non-visible light image are fused at a set second ratio in the high-frequency domain, for example, the second ratio may be a ratio dominated by high-frequency components of the non-visible light image.

(4) Dot matrix light source 140 .

The dot-matrix light source 140 is a light source capable of emitting dot-matrix light, that is, the light source can emit multiple parallel light beams. In the embodiment of the present application, the wavelength of the dot matrix light emitted by the dot matrix light source 140 may be greater than 530 nanometers and less than 750 nanometers, that is, the dot matrix light may be visible light. For another example, the wavelength of the lattice light may also be greater than 750 nanometers, that is, in addition to visible light, infrared light may also be included in the lattice light.

Both the first lattice and the second lattice include a plurality of light points. The shape formed by the plurality of light spots is related to the shape of the lattice light. For example, as shown in Figure 4A, if multiple parallel light beams in the lattice light are arranged in the shape of a five-pointed star, then the shapes of the first dot matrix in the visible light image and the second dot matrix in the non-visible light image It may also be arranged in the shape of a five-pointed star. As shown in Figure 4B, if multiple parallel light beams in the lattice light are arranged in the shape of a matrix, then the shape of the first lattice in the visible light image and the second lattice in the non-visible light image may also be in the shape of a matrix shape arrangement.

The duration of the lattice light projected by the lattice light source 140 may be greater than the photographing time of the visible light photographing structure 110 and the invisible light photographing structure 120, that is, the duration may be equal to or greater than the exposure time of the visible light photographing structure and the invisible light photographing structure 120 Specifically, the duration of the dot-matrix light projected by the dot-matrix light source 140 is less than 80 milliseconds.

After describing the components of the image capturing device 100 provided in the embodiment of the present application, the image processing process of the image capturing device 100 will be described below. Referring to Figure 5, the method includes:

Step 501: After the image capturing device 100 moves or deflects, the visible light capturing structure 110 and the non-visible light capturing structure 120 can capture the same object. The visible light photographing structure 110 and the non-visible light photographing structure 120 can maintain the same exposure time when photographing.

In step 501, when the visible light photographing structure 110 and the invisible light photographing structure 120 photograph the same target, the dot matrix light source 140 may project dot matrix light to the target.

Step 502: The visible light imaging structure 110 generates a first visible light image, and the non-visible light imaging structure 120 generates a non-first visible light image. The first visible light image includes a first lattice formed by lattice light, and the first non-visible light image includes a second lattice formed by lattice light. As shown in FIG. 6 , the first group of images is a schematic diagram of the first visible light image and the first non-visible light image.

After that, the processor 130 first registers the first visible light image and the first non-visible light image (that is, steps 503 to 504 ), and then fuses the images (step 505 ).

Step 503: The processor 130 obtains the first visible light image and the first non-visible light image, and the processor 130 determines that there are corresponding light point pairs according to the first lattice and the second lattice, and obtains the first visible light image and the first non-visible light image Affine transformation parameters required for alignment.

Step 504: The processor 130 uses affine transformation parameters to align the first visible light image and the first non-visible light image. As shown in FIG. 6 , the second group of images is a schematic diagram after alignment of the first visible light image and the first non-visible light image.

Step 505: The processor 130 fuses the registered first visible light image and the first non-visible light image to obtain a first fused image. As shown in FIG. 6 , the last image is a schematic diagram after fusion of the first visible light image and the first non-visible light image.

Steps 501 to 505 are an image capture operation performed after the image capture device 100 moves or deflects. If the subsequent image capture device 100 does not move or deflect, the affine transformation parameters obtained in step 503 can be used to The second visible light image and the second non-visible light image generated during subsequent image capture operations are aligned. See step 506 to step 509 for details.

Step 506: The visible light photographing structure 110 and the non-visible light photographing structure 120 can photograph the same object. The visible light photographing structure 110 and the non-visible light photographing structure 120 can maintain the same exposure time when photographing.

In step 506, when the visible light photographing structure 110 and the non-visible light photographing structure 120 photograph the same target, the dot matrix light source 140 may project dot matrix light to the target.

Step 507: The visible light imaging structure 110 generates a second visible light image, and the non-visible light imaging structure 120 generates a non-second visible light image.

Step 508: The processor 130 uses affine transformation parameters to align the second visible light image and the second non-visible light image.

Step 509: The processor 130 fuses the registered second visible light image and the second non-visible light image to obtain a second fused image.

It can be seen from the above that after each movement or rotation of the image capturing device 100, affine transformation parameters required for registration can be obtained by taking one shot. In this way, the process of image fusion in the subsequent shooting process can be greatly simplified and the speed of image fusion can be accelerated.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

An image capture device, characterized in that the device comprises:

A visible light imaging structure, configured to photograph the target projected by the lattice light to generate a visible light image, the visible light image including the first dot matrix formed by the lattice light;

A non-visible light photographing structure, configured to photograph the target to generate a non-visible light image, the non-visible light image including a second lattice formed by the lattice light;

A processor, configured to fuse the visible light image and the non-visible light image according to the first lattice and the second lattice, and output a fusion image.
The device according to claim 1, wherein the shooting time of the visible light shooting structure is the same as the shooting time of the non-visible light shooting structure.
The device according to claim 1 or 2, characterized in that said non-visible light is infrared light.
The device according to any one of claims 1 to 3, wherein the image capturing device further comprises a dot matrix light source, and the dot matrix light source is used to project the dot matrix light to the target.
The device according to any one of claims 1-4, wherein the processor is specifically used for:

Registering the visible light image and the non-visible light image based on the corresponding relationship between light spots and light spots between the first lattice and the second lattice;

The registered visible light image and the non-visible light image are fused to generate the fused image.
The device according to claim 5, wherein the first dot matrix and the second dot matrix comprise a plurality of light points, and the processor is based on the first dot matrix and the second dot matrix The corresponding relationship between light spots and light spots between arrays, registering the visible light image and the non-visible light image, specifically for:

Determining a light point pair with corresponding relationship between the first lattice and the second lattice, where the corresponding light point pair includes one light point of the first lattice and one light point of the second dot The corresponding light points are the light points in the first lattice and the second lattice whose feature similarity is greater than a threshold;

The visible light image and the non-visible light image are registered according to the corresponding light points.
The device according to any one of claims 1 to 6, wherein the visible light imaging structure includes a visible light lens and a visible light sensor;

The visible light lens is used to capture the visible light reflected by the shooting target, and project the visible light onto the visible light sensor;

The visible light sensor is configured to sense the visible light and generate the visible light image;

The non-visible light shooting structure includes a non-visible light lens and a non-visible light sensor;

The non-visible light lens is used to capture the non-visible light reflected by the target, and project the non-visible light onto the non-visible light sensor;

The non-visible light sensor is configured to sense the non-visible light and generate the non-visible light image.
The device according to claim 7, wherein the non-visible light shooting structure further includes an infrared supplementary light;

The infrared supplementary light is used to project infrared light to the target when the non-visible light shooting structure is shooting.
The device according to claim 7 or 8, wherein the visible light lens or the non-visible light lens is a zoom lens.
The device according to any one of claims 1 to 9, characterized in that the wavelengths of the dot matrix light cover visible light bands and infrared light bands.
The device according to any one of claims 1-9, characterized in that the wavelength of the lattice light is greater than 560 nanometers and less than 750 nanometers.
The device according to any one of claims 1-11, characterized in that the projection duration of the dot matrix light is less than 80 milliseconds.
An image processing method, characterized in that the method comprises:

The visible light shooting structure in the image capturing device shoots the target projected by the lattice light to generate a visible light image, and the visible light image includes a first dot matrix formed by the lattice light;

The non-visible light photographing structure in the image capturing device photographs the target to generate a non-visible light image, and the non-visible light image includes a second lattice formed by the lattice light;

The processor in the image capture device fuses the visible light image and the non-visible light image according to the first lattice and the second lattice, and outputs a fusion image.
The method according to claim 13, wherein the processor in the image capture device fuses the visible light image and the non-visible light image according to the first lattice and the second lattice , output the fused image, including:

The processor in the image capture device registers the visible light image and the non-visible light image based on the corresponding relationship between light spots and light spots between the first dot matrix and the second dot matrix;

The processor in the image capture device fuses the registered visible light image and the non-visible light image to generate the fusion image.
The method according to claim 14, wherein the first dot matrix and the second dot matrix include a plurality of light spots, and the processor in the image capturing device is based on the first dot matrix and the second dot matrix. The corresponding relationship between light spots and light spots between the second dot matrix, registering the visible light image and the non-visible light image includes:

The processor in the image capture device determines a corresponding light point pair in the first dot matrix and the second dot matrix, and the corresponding light point pair includes one of the first dot matrix A light point and a light point of the second lattice, the light point having a corresponding relationship is a light point with a feature similarity degree greater than a threshold in the first lattice and the second lattice;

The processor in the image capture device registers the visible light image and the non-visible light image according to the corresponding light points.