WO2022100242A1

WO2022100242A1 - Image processing method and apparatus, electronic device, and computer-readable storage medium

Info

Publication number: WO2022100242A1
Application number: PCT/CN2021/116809
Authority: WO
Inventors: 林枝叶
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-11-12
Filing date: 2021-09-07
Publication date: 2022-05-19
Also published as: CN112258579A; CN112258579B

Abstract

An image processing method comprises: acquiring a first image to be processed and a second image to be processed, the first image being captured by a first camera, and the second image being captured by a second camera; performing pixel mapping on the second image on the basis of a pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image, wherein the pixel mapping relationship is determined based on a first camera response function of the first camera and a second camera response function of the second camera; and aligning the mapped image corresponding to the second image with the first image.

Description

Image processing method, apparatus, electronic device, and computer-readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application filed on November 12, 2020 with the application number 2020112608189 and the title of the invention is "image processing method, device, electronic device and computer-readable storage medium", the entire contents of which are Incorporated herein by reference.

technical field

This application relates to the technical field of image processing, and in particular, to an image processing method, apparatus, electronic device, and computer-readable storage medium, as well as a method, apparatus, electronic device, and computer-readable storage medium for a pixel mapping relationship of a binocular camera .

Background technique

Various electronic devices such as mobile phones and tablet computers have become indispensable tools in today's life. In order to meet people's needs to record beautiful moments, electronic device photography has become an important function. With the upgrading of electronic devices, more and more electronic devices are equipped with multiple cameras to meet people's increasing shooting needs.

Currently, in order to enhance the image quality of images captured by electronic devices, images captured by multiple cameras are often aligned and then fused to fuse information collected by multiple cameras, which can effectively enhance image quality. However, due to differences in information sources, images captured by different cameras may have similar image information structures but inconsistent gradients, resulting in poor image alignment accuracy and limited alignment effect.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and a pixel mapping relationship method, apparatus, electronic device, and computer-readable storage medium for a binocular camera, which can improve image alignment precision.

An image processing method, comprising:

Obtain the first image and the second image to be processed; the first image is captured by the first camera, and the second image is captured by the second camera;

Based on the pixel mapping relationship between the first camera and the second camera, pixel mapping is performed on the second image to obtain a mapped image corresponding to the second image; wherein the pixel mapping relationship is based on the first camera response function of the first camera and the second image. The second camera response function of the camera is determined;

Align the mapping image corresponding to the second image with the first image.

An image processing device, the device includes:

a to-be-processed image acquisition module, configured to acquire the to-be-processed first image and the second image; the first image is captured by the first camera, and the second image is captured by the second camera;

The pixel mapping processing module is configured to perform pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image; wherein the pixel mapping relationship is based on the first camera A camera response function is determined with the second camera response function of the second camera;

The image alignment processing module is used for aligning the mapped image corresponding to the second image with the first image.

An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Align the mapping image corresponding to the second image with the first image.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Align the mapping image corresponding to the second image with the first image.

The above image processing method, device, electronic device and storage medium, according to the pixel mapping relationship determined by the first camera response function of the first camera and the second camera response function of the second camera, the second image captured by the second camera is processed. Pixel mapping, aligning the obtained mapping image corresponding to the second image with the first image. In the image processing process, pixel mapping is performed on the second image by using the pixel mapping relationship determined by the first camera response function of the first camera and the second camera response function of the second camera, and the second image can be mapped by using the camera response function of the camera. Mapping to the pixel space of the first image can solve the problem of similar image information structure but inconsistent gradients, ensure the accuracy of image alignment, and improve the effect of image alignment.

A method for determining a pixel mapping relationship of a binocular camera, comprising:

Obtain a first calibration image group and a second calibration image group; the first calibration image group includes the first calibration image obtained by shooting the first camera in the binocular camera under the same scene and different exposure time conditions, and the second calibration image group includes a second calibration image obtained by shooting the second camera in the binocular camera under the same scene and different exposure time conditions;

determining a first camera response function corresponding to the first camera based on each first calibration image;

determining a second camera response function corresponding to the second camera based on each second calibration image;

The pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function and the second camera response function.

A device for determining a pixel mapping relationship of a binocular camera, the device comprising:

The calibration image group acquisition module is used to obtain the first calibration image group and the second calibration image group; the first calibration image group includes the first calibration image obtained by the first camera in the binocular camera shooting under the same scene and different exposure time conditions an image, the second calibration image group includes a second calibration image obtained by shooting the second camera in the binocular camera under the same scene and different exposure time conditions;

a first camera response function determination module, configured to determine a first camera response function corresponding to the first camera based on each first calibration image;

A second camera response function determination module, configured to determine a second camera response function corresponding to the second camera based on each second calibration image;

The pixel mapping relationship determination module is configured to determine the pixel mapping relationship between the first camera and the second camera according to the first camera response function and the second camera response function.

The above-mentioned method, device, electronic device and storage medium for determining the pixel mapping relationship of the binocular camera, according to the images captured by the binocular camera under the same scene and different exposure time conditions, respectively determine the first camera corresponding to the first camera in the binocular camera. The camera response function and the second camera response function corresponding to the second camera, and the pixel mapping relationship between the first camera and the second camera is determined based on the first camera response function and the second camera response function. The pixel mapping relationship is determined according to the first camera response function of the first camera and the second camera response function of the second camera. Through the pixel mapping relationship, the camera response function of the camera can be used to convert the second image captured by the second camera in the binocular camera. Mapping to the pixel space of the first image captured by the first camera can solve the problem of similar image information structure but inconsistent gradients, ensure the accuracy of image alignment, and improve the effect of image alignment.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Figure 1 is a schematic diagram of the analysis of RGB image imaging.

Figure 2 is a schematic diagram of the analysis of NIR image imaging.

FIG. 3 is an application environment diagram of an image processing method or a method for determining a pixel mapping relationship of a binocular camera in one embodiment.

FIG. 4 is a flowchart of an image processing method in one embodiment.

FIG. 5 is a flowchart of determining a first camera response function in one embodiment.

FIG. 6 is a flowchart of an image processing method in another embodiment.

FIG. 7 is a flowchart of camera calibration in one embodiment.

Figure 8 is a flow chart of calibration of CRF in one embodiment.

FIG. 9 is a schematic diagram of a camera response curve in one embodiment.

FIG. 10 is a schematic diagram of a camera response curve in another embodiment.

FIG. 11 is a schematic diagram of a camera response curve in yet another embodiment.

FIG. 12 is a flowchart of a method for determining a pixel mapping relationship of a binocular camera in one embodiment.

FIG. 13 is a structural block diagram of an image processing apparatus in an embodiment.

FIG. 14 is a structural block diagram of an apparatus for determining a pixel mapping relationship of a binocular camera in one embodiment.

Figure 15 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

At present, electronic devices usually use sensors with Red, Green and Blue filters to receive reflected light from objects and generate color images (RGB). The obtained color images conform to human visual perception, but are vulnerable to insufficient ambient light, fog bad weather effects. However, infrared images (NIR, Near Infra-red Spectrum), which describe the thermal radiation of objects, are more penetrating than RGB images in poor lighting, fog, and other bad weather, and have better detail than RGB images, but NIR images Color information is not available and the image resolution is low. Therefore, the electronic equipment is equipped with a visible light camera and an infrared camera at the same time, and the RGB image and the NIR image are obtained by shooting. Through the fusion of the information of the RGB image and the NIR image, it can not only be used for image quality enhancement, but also for object recognition in extremely dark scenes. , image denoising, high dynamic range (HDR, High-Dynamic Range), image dehazing, skin depigmentation, etc.

The information fusion between RGB images and NIR images includes two major processes, image alignment and image fusion. Alignment is the foundation and fusion is the fundamental. If the alignment error is large, it will cause artifacts such as ghosts, Problems such as ghosting; if the fusion effect is poor, there will be problems such as color distortion and white borders. For traditional RGB image and RGB image alignment tasks, feature point detection and matching are often used. These feature points include Harris corner points, FAST (Features From Accelerated Segment Test, accelerated segmentation test features) feature operator, SURF (Speeded Up) Robust Features, acceleration robust features) feature operator, SIFT (Scale-invariant features transform, scale-invariant feature transform) feature operator, etc., with rotation invariance and illumination invariance. However, these feature point-based image alignment techniques largely rely on the consistency of the image structure similar regions in the gradient size direction, especially the SIFT feature operator.

However, RGB images and NIR images have similar structures but inconsistent gradient directions between different objects in the same scene due to different information sources. As shown in Figure 1-2, Figure 1 is an RGB image, and Figure 2 is an NIR image. The green plant area represented by the two black boxes is darker in the RGB image, while the NIR image is brighter; the extremely dark area represented by the white box, RGB The image is darker than its surrounding area, while the NIR image is comparable to its surrounding area; the sky and other areas of the building, RGB image and NIR image brightness are comparable. The essential reason for this problem is that the RGB and NIR bands are different, and the transmittance to different objects is inconsistent. Based on the traditional feature point detection and alignment technology, such as using SIFT feature point detection and matching alignment technology to align RGB images and NIR images, the alignment accuracy is poor, the alignment effect is limited, and it cannot meet the needs of subsequent image fusion. .

Based on this, the present application proposes an image processing method, device, electronic device, and computer-readable storage medium that can improve the effect of image alignment, as well as a method, device, electronic device, and computer-readable storage for a pixel mapping relationship of a binocular camera The medium is specifically described by the following examples.

It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client, without departing from the scope of this application. Both the first client and the second client are clients, but they are not the same client.

FIG. 3 is a schematic diagram of an application environment of the image processing method in one embodiment. As shown in FIG. 3 , the application environment includes an electronic device 302. The electronic device 302 is equipped with multiple cameras. The electronic device 302 can shoot through the multiple cameras, and align and fuse the images captured by the multiple cameras to enhance the shooting. Image quality effect. Specifically, the electronic device 302 acquires the first image captured by the first camera and the second image captured by the second camera, and the electronic device 302 obtains the first camera response function of the first camera and the second camera of the second camera according to the first camera response function of the first camera and the second camera of the second camera. In response to the pixel mapping relationship determined by the function, pixel mapping is performed on the second image captured by the second camera, and the mapping image corresponding to the obtained second image is aligned with the first image. In addition, in other applications, the above image processing method can also be implemented by a server, that is, the server acquires the first image and the second image to be processed, such as acquiring the first image and the second image to be processed from a database, or The electronic device 302 directly sends the captured first image and the second image to be processed to the server through the network, so that the server performs image alignment processing.

On the other hand, FIG. 3 is a schematic diagram of an application environment of a method for determining a pixel mapping relationship of a binocular camera in one embodiment. Specifically, the electronic device 302 obtains a first calibration image group and a second calibration image group, and the first calibration image group includes a first calibration image obtained by shooting the first camera in the binocular camera under the conditions of the same scene and different exposure times, The second calibration image group includes second calibration images captured by the second camera in the binocular camera under the conditions of the same scene and different exposure times, and the electronic device 302 respectively determines the first camera response function corresponding to the first camera in the binocular camera A second camera response function corresponding to the second camera, and a pixel mapping relationship between the first camera and the second camera is determined based on the first camera response function and the second camera response function. In addition, in other applications, the above-mentioned method for determining the pixel mapping relationship of a binocular camera can also be implemented by a server, that is, the server obtains the first calibration image group and the second calibration image group, such as obtaining the first calibration image group and the second calibration image group from a database. The second calibration image group, or the electronic device 302 directly sends the captured first calibration image group and the second calibration image group to the server through the network, so that the server performs the processing of determining the pixel mapping relationship of the binocular camera.

The electronic device 302 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, portable wearable devices, etc.; the server can be implemented by an independent server or a server cluster composed of multiple servers.

FIG. 4 is a flowchart of an image processing method in one embodiment. The image processing method in this embodiment is described by taking the operation on the electronic device in FIG. 3 as an example. As shown in FIG. 4 , the image processing method includes process 402 to process 406 .

In process 402, a first image and a second image to be processed are acquired; the first image is captured by the first camera, and the second image is captured by the second camera.

Specifically, the first image and the second image need to be aligned and processed. Specifically, the images can be captured by two cameras for the same scene, wherein the first image is captured by the first camera, and the second image is captured by the second camera. . For example, the first image may be a color image captured by a visible light camera, and the second image may be an infrared image captured by an infrared camera.

In a specific application, the electronic device may be provided with a binocular camera, including a first camera and a second camera. For example, two rear cameras may be provided, and the two cameras can be used to shoot at the same time to obtain the first image to be processed. and the second image.

Process 404: Perform pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image; wherein the pixel mapping relationship is based on the first camera response function of the first camera The second camera response function of the second camera is determined.

The pixel mapping relationship reflects the pixel value of each pixel in the image captured by the first camera and the pixel value of each pixel in the image captured by the second camera when the first camera and the second camera shoot the same scene at the same time The mapping relationship between them, that is, through the pixel mapping relationship, the images captured by the first camera and the second camera can be mapped to the color space, such as mapping the image captured by the first camera to the color space corresponding to the image captured by the second camera. , in order to overcome the problem that the images captured by the first camera and the second camera have similar information structures but inconsistent gradients due to differences in information sources, resulting in poor image alignment accuracy.

The pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function of the first camera and the second camera response function of the second camera. Among them, the camera response function (Camera Response Function, CRF) is used to characterize the corresponding relationship between the brightness of the image captured by the camera and the illuminance (Radiance) in the real world. Generally, the brightness or illuminance observed in the real world is constant and will not change with different cameras, and there is a certain correspondence between the brightness of the image captured by the camera and the illuminance in the real world. Function description. Different cameras have different CRF curves, but it is established that the brightness of the image captured by the camera has a certain relationship with the illumination of the real world. By using the illumination of the real world as a bridge, the color gamut of different cameras can be mapped to the same space. , to overcome the problem that the image information obtained by different cameras has similar structures but inconsistent gradients. The camera response function can be pre-calibrated by the image captured by the camera. The mapped image is obtained by performing pixel mapping processing on the second image through the pixel mapping relationship between the first camera and the second camera. Specifically, the pixel mapping relationship between the first camera and the second camera can be used to respectively map the pixels in the second image. The pixel value of each pixel is updated to obtain a mapped image.

Specifically, after obtaining the first image and the second image to be processed, the electronic device obtains the pixel mapping relationship between the first camera and the second camera, and performs pixel mapping on the second image based on the pixel mapping relationship to obtain the second image. The image is mapped to the color space of the first image. Compared with the second image, the problem that the image information structure of the mapped image and the first image is similar but the gradient is inconsistent is solved. By aligning the mapped image and the first image, you can Make sure the images are aligned.

In process 406, the mapping image corresponding to the second image is aligned with the first image.

The mapped image corresponding to the second image is obtained by performing pixel mapping on the pixel mapping relationship between the first camera and the second camera, and the gradient of the mapped image and the first image is more consistent. Alignment, for example, aligning the mapped image and the first image through an alignment method of SIFT feature detection and matching, so as to accurately align the image captured by the first camera and the image captured by the second camera, thereby improving the effect of image alignment.

In the image processing method in this embodiment, pixel mapping is performed on the second image captured by the second camera according to the pixel mapping relationship determined by the first camera response function of the first camera and the second camera response function of the second camera. The obtained mapping image corresponding to the second image is aligned with the first image. In the image processing process, pixel mapping is performed on the second image by using the pixel mapping relationship determined by the first camera response function of the first camera and the second camera response function of the second camera, and the second image can be mapped by using the camera response function of the camera. Mapping to the pixel space of the first image can solve the problem of similar image information structure but inconsistent gradients, ensure the accuracy of image alignment, and improve the effect of image alignment.

In one embodiment, the image processing method further includes a process of determining a pixel mapping relationship based on the first camera response function of the first camera and the second camera response function of the second camera, specifically including: acquiring a first calibration image group and a second Calibration image group; the first calibration image group includes the first calibration image obtained by the first camera under the same scene and different exposure time conditions, and the second calibration image group includes the second camera under the same scene and different exposure time conditions shooting the obtained second calibration image; determining a first camera response function corresponding to the first camera based on each first calibration image; determining a second camera response function corresponding to the second camera based on each second calibration image; according to the first camera response function and the second camera response function to determine the pixel mapping relationship between the first camera and the second camera.

The first calibration image group includes a first calibration image captured by a first camera under the same scene and different exposure time conditions, and the second calibration image group includes a second calibration image captured under the same scene and different exposure time conditions. the second calibration image. That is, the images in the first calibration image group and the second calibration image group are both captured by the corresponding cameras for the same scene, and the exposure times of the first calibration images in the first calibration image group are different when they are captured. The exposure times of the respective second calibration images in the calibration image group are different during shooting. In specific implementation, the shooting scenes corresponding to the first calibration image group and the second calibration image group may be high dynamic range scenes including overexposed and overdark areas, so as to ensure that the determined pixel mapping relationship can be applied to high dynamic range scenes, Guarantees the applicable scope of the pixel mapping relationship. The number of the first calibration image and the second calibration image and the corresponding exposure time can be flexibly set according to actual needs. For example, the number of the first calibration image and the second calibration image can be 5, and the corresponding exposure time can be increased. , and the respective exposure times of the first calibration image and the second calibration image may be different. The exposure time can be adjusted by modifying the signal gain (gain value) and shutter speed (shutter value) of the electronic device.

Further, when the electronic device determines the pixel mapping relationship between the first camera and the second camera, that is, when the electronic device calibrates the pixel mapping relationship between the first camera and the second camera, the first camera and the second camera can be respectively The second camera performs self-calibration, determines the first camera response function corresponding to the first camera and the second camera response function corresponding to the second camera, and uses the first camera response function and the second camera response function to perform mutual calibration to obtain the first camera response function. The pixel mapping relationship between the camera and the second camera.

Specifically, after obtaining the first calibration image group and the second calibration image group, the electronic device determines the first camera response function corresponding to the first camera based on each first calibration image, and determines the corresponding second camera based on each second calibration image. The second camera response function. Specifically, the electronic device may first align the calibration images in the first calibration image group and the second calibration image group, for example, by using the median threshold bitmap alignment method to align the first calibration image group and the second calibration image group. Perform median threshold alignment on each of the calibration images, and determine the corresponding camera response function based on the first calibration image after median threshold alignment and the second calibration image after median threshold alignment. Specifically, based on the luminance channel image of each first calibration image and the luminance channel image of each second calibration image, the electronic device can obtain the first camera response function corresponding to the first camera and the first camera corresponding to the second camera through the Debevec algorithm. Two camera response functions. After obtaining the first camera response function and the second camera response function, the electronic device determines the pixel mapping relationship between the first camera and the second camera based on the first camera response function and the second camera response function. For example, the pixel value of the matching point between the first calibration image and the second calibration image, and the relative illuminance value determined by the matching point according to the response function of the first camera and the response function of the second camera, can be used to determine the difference between the first camera and the second camera. The illuminance mapping relationship between them is determined, and the pixel mapping relationship between the first camera and the second camera is determined based on the illuminance mapping relationship.

In this embodiment, self-calibration is performed by using the calibration images captured by the first camera and the second camera, respectively, to determine the first camera response function and the second camera response function, and according to the obtained first camera response function and second camera response function Perform mutual calibration to obtain the pixel mapping relationship between the first camera and the second camera. The pixel mapping relationship is determined based on the response function of the first camera and the response function of the second camera, and the color space of the image captured by the first camera and the image captured by the second camera can be mapped through the pixel mapping relationship, so as to ensure the gradient when the images are aligned Consistency can effectively improve the effect of image alignment.

In one embodiment, the first camera is a visible light camera; as shown in FIG. 5 , the process of determining the first camera response function, that is, determining the first camera response function corresponding to the first camera based on each first calibration image, includes the process 502 to process 508.

In process 502, target channel images corresponding to each of the first calibration images respectively corresponding to the target color channel are acquired.

Among them, visible light cameras can capture color images, such as RBG cameras, sensors including Red, Green, and Blue filters to receive reflected light from objects and generate RGB color images. The target color channel is the color channel that needs to construct the corresponding camera response function. The camera response function is related to the camera itself. The correspondence between the brightness of the image captured by different cameras and the illuminance in the real world is different, that is, different cameras correspond to different camera response functions, and the same camera is in different color channels, the camera response function The expressions of the corresponding function curves are also different. For example, for an RGB image captured by a visible light camera, which consists of three color channels, the corresponding camera response functions can be calibrated based on the R, G, and B channels, respectively. The camera response functions corresponding to each channel are different from each other, but The camera response function corresponding to each channel reflects the correspondence between the brightness of the image captured by the visible light camera and the illumination in the real world. The target channel image is an image of the first calibration image corresponding to the target color channel. If the first calibration image is an RGB image and the target color channel is an R channel, the target channel image may be an R channel image obtained by channel separation of the RGB image. The target color channel can be set according to actual needs.

In process 504, the first feature points corresponding to the same position in each target channel image in the same scene are determined.

Each first calibration image is captured based on the same scene. For the same position in the same scene, the first feature point corresponding to the position in each target channel image is determined, and the first feature point corresponding to each target channel image points to reality. The same location of the scene in the world, but with different exposure times for each target channel image. Specifically, the electronic device may determine the first feature point corresponding to the same position in the same scene from each target channel image.

In process 506, the channel luminance value of each first feature point corresponding to the target color channel is determined.

After obtaining the first feature points corresponding to each other in each target channel image, the electronic device further determines the channel luminance value of each first feature point corresponding to the target color channel. Specifically, the electronic device may determine that the first feature point corresponds to the channel pixel value of the target color channel, and obtain the channel luminance value of the first feature point corresponding to the target color channel based on the channel pixel value. When the target color channel is a single channel, the channel luminance value is equal to the channel pixel value.

In process 508, a first camera response function corresponding to the first camera is determined according to the channel luminance value of each first feature point corresponding to the target color channel.

After obtaining the channel brightness values of the first feature points, the first camera response function corresponding to the first camera is determined based on the channel brightness values. During specific implementation, the electronic device may obtain the first camera response function corresponding to the first camera by using the channel luminance value of each first feature point corresponding to the target color channel based on the Debevec algorithm.

In this embodiment, for the visible light camera, the first calibration image captured by the camera corresponds to the target channel image of the target color channel to perform camera response function calibration, and the camera response functions of the first camera corresponding to various channels can be determined according to actual needs.

In one embodiment, acquiring the target channel images corresponding to the target color channels of the first calibration images respectively includes: performing channel separation on the first calibration images to obtain separate channel images; obtaining images corresponding to the target color according to the separated channel images The target channel image for the channel.

Wherein, the separated channel image is an image corresponding to each color channel obtained after the first calibration image is subjected to channel separation processing, and the separated channel image corresponds to the color space in which the first calibration image is located. For example, after channel separation of RGB image, R channel image, G channel image and B channel image can be obtained; HSV (Hue-Saturation-Value, Hue-Saturation-Value) image can be obtained after channel separation to obtain H channel image, S channel image and V channel image.

Specifically, the target color channel can be set according to actual requirements. When acquiring the target channel images of the first calibration image corresponding to the target color channel respectively, the electronic device performs channel separation on the first calibration image to obtain each separated channel image. Based on the obtained Each of the separate channel images determines the target channel image corresponding to the target color channel. For example, the separation channel image corresponding to the target color channel can be selected from the separation channel images as the target channel image; when the target color channel includes all separation channels, all separation channel images can also be directly used as the target channel image to establish the first channel image. A camera corresponds to the camera response function of each color channel. In addition, each separate channel image can also be transformed to obtain the target channel image. For example, when the target color channel is the luminance channel, the luminance channel refers to the channel in the color space that represents the brightness of the image, that is, the target channel image is the luminance channel image, and the separate channel images corresponding to the first calibration image include the R channel image, When the G channel image and the B channel image are used, the mapping relationship between the R, G and B channels and the brightness channel can be used, such as Y=0.299*R+0.587*G+0.114*B, where Y is the brightness, combined with the R channel image, The brightness channel image is obtained from the G channel image and the B channel image, that is, the target channel image is obtained.

In this embodiment, according to each separated channel image obtained after channel separation is performed on the first calibration image, the corresponding required target channel image is quickly determined, so as to ensure the processing efficiency of the camera response function calibration.

In one embodiment, acquiring a target channel image corresponding to each first calibration image respectively corresponding to a target color channel includes: transforming the first calibration image into a target color space including the target color channel to obtain a target color space image; according to the target color The spatial image results in a target channel image corresponding to the target color channel.

The target color channel is preset according to actual requirements, and each color channel of the target color space includes the target color channel. By transforming the first calibration image to the target color space, the corresponding target color can be obtained according to the image corresponding to the target color space. The target channel image for the channel.

Specifically, when acquiring the target channel image, the electronic device transforms the color space of the first calibration image. For example, the target color space including the target color channel can be determined first, and the color space transformation is performed on the first calibration image. The calibration image is transformed to the target color space to obtain the target color space image in the target color space. The electronic device obtains a target channel image corresponding to the target color channel according to the target color space image. Specifically, the electronic device may perform channel separation on the target color space image, and obtain the target channel image from the separated channel image obtained by the channel separation.

In this embodiment, after performing color space transformation on the first calibration image, the target channel image is obtained according to the transformed result, and the camera responses of the first camera corresponding to various channels can be obtained based on the first calibration image through channel transformation processing. function.

In one embodiment, the second camera is an infrared camera; determining a second camera response function corresponding to the second camera based on each second calibration image includes: respectively determining the corresponding position in each second calibration image of the same position in the same scene second feature points; determining the pixel value of each second feature point; determining the second camera response function corresponding to the second camera according to the pixel value of each second feature point.

Among them, the working principle of the infrared camera is that the infrared light emits infrared rays to illuminate the object, and the infrared rays are diffusely reflected and received by the monitoring camera to form infrared images, such as NIR images. The second camera is an infrared camera, the second calibration image captured by the second camera is a single-channel image, and the pixel value of the second calibration image has the same value as its brightness value. Based on this, an algorithm may be directly determined based on the camera response function, for example, based on the Debevec algorithm, and the second camera response function corresponding to the second camera may be obtained according to the pixel value of the second calibration image.

Specifically, the second camera is an infrared camera. When calibrating the camera response function of the second camera, it is similar to the camera response function calibration process of the first camera. The electronic device determines that the same position in the same scene is located in each second calibration image. The corresponding second feature point. Each second calibration image is captured based on the same scene. For the same position in the same scene, the second feature point corresponding to the position in each second calibration image is determined, and the second feature point corresponding to each second calibration image is Point to the same location in the scene in the real world, but with different exposure times for each second calibration image. Specifically, the electronic device may determine the second feature point corresponding to the same position in the same scene from each of the second calibration images. After obtaining each second feature point, the electronic device obtains the pixel value corresponding to each second feature point, and based on the Debevec algorithm, obtains the second camera response function corresponding to the second camera through the pixel value of each second feature point. .

In this embodiment, for the infrared camera, channel conversion processing is not required, and the camera response function is directly calibrated by the pixel value of the second calibration image captured by the infrared camera, which can quickly determine the camera response function corresponding to the second camera.

In one embodiment, determining the pixel mapping relationship between the first camera and the second camera according to the first camera response function and the second camera response function includes: acquiring at least one pair of matching point pairs, where the matching point pair The first matching point extracted from a calibration image and the second matching point extracted from the second calibration image are obtained by feature matching; the second point pixel value; determine the first relative illuminance value according to the first point pixel value and the first camera response function; determine the second relative illuminance value according to the second point pixel value and the second camera response function; based on the first relative illuminance value determining an illuminance mapping relationship with the second relative illuminance value; and determining a pixel mapping relationship between the first camera and the second camera according to the illuminance mapping relationship.

The matching point pair is obtained by feature matching according to the first matching point and the second matching point, the first matching point is extracted from the first calibration image, and the second matching point is extracted from the second calibration image. Specifically, the first matching point and the second matching point can be extracted from the first calibration image and the second calibration image, respectively, and feature matching is performed on each of the obtained first matching points and each second matching point, such as constructing a feature matching result. Each matching point pair. The matching point pair includes a first matching point from the first calibration image and a second matching point from the first calibration image. In specific implementation, feature point detection algorithms, such as Fast, SUSA (Smallest Univalue Segment Assimilating Nucleus, minimum single value segment assimilation kernel), SIFT, SURF or LBP (Local Binary Pattern, local binary pattern) and other algorithms can be used to detect The first calibration image and the second calibration image are processed to obtain a first matching point and a second matching point.

Feature matching refers to matching the obtained first matching point and second matching point to determine the corresponding matching points in the first calibration image and the second calibration image, generally the same position in the shooting scene corresponds to the first calibration image and the second calibration image. The second calibrates the pixels in the image. Specifically, the BRIEF (Binary Robust Independent Elementary Features) algorithm, the Hamming distance algorithm, etc. can be used to perform feature matching on the first matching point and the second matching point, and a matching point pair can be constructed based on the feature matching result. , each matching point pair includes a first matching point and a second matching point that match each other, the first matching point comes from the first calibration image, and the second matching point comes from the second calibration image.

Among them, illuminance refers to the energy of visible light received per unit area. The image captured by the camera in the real world is the relative illuminance perceived by the camera, and the relative illuminance has a certain proportional relationship with the real illuminance in the real world. The camera response function of the camera reflects the relationship between the pixel value of the image captured by the camera and the relative illuminance value, that is, the corresponding relative illuminance value can be obtained through the pixel value of the image captured by the camera and the camera response function. According to the relative illuminance values corresponding to the two feature points in the matching point pair, the illuminance mapping relationship between the relative illuminance of the first camera and the second camera can be obtained, and based on the illuminance mapping relationship, the illuminance mapping relationship between the first camera and the second camera can be constructed. Pixel mapping relationship.

Specifically, when the first camera response function and the second camera response function are obtained, and the pixel mapping relationship between the first camera and the second camera is determined, the electronic device obtains at least one pair of matching points, and respectively determines the first pair of matching points. The pixel value of the first point of a matching point and the pixel value of the second point of the second matching point. After obtaining the pixel value of the first point and the pixel value of the second point, the electronic device determines the first relative illuminance value based on the pixel value of the first point and the first camera response function, and determines the second point based on the pixel value of the second point and the response function of the second camera. Relative illuminance value. The electronic device determines the illuminance mapping relationship according to each of the first relative illuminance values and the corresponding second relative illuminance values. For example, the electronic device can perform statistical analysis on each of the first relative illuminance values and the corresponding second relative illuminance values to obtain the first camera and the second relative illuminance value. The illuminance mapping relationship of the camera. The illuminance mapping relationship describes the corresponding relationship between the relative illuminance value corresponding to the image captured by the first camera and the relative illuminance value corresponding to the image captured by the second camera under the same scene. Further, the electronic device obtains the pixel mapping relationship between the first camera and the second camera based on the determined illuminance mapping relationship. The pixel mapping relationship describes the correspondence between the pixel value of the image captured by the first camera and the pixel value of the image captured by the second camera in the same scene. Based on the correspondence, the image captured by the first camera can be realized. Pixel mapping with the image captured by the second camera. In specific implementation, the pixel values of the images captured by the first camera can be traversed, the first relative illuminance value corresponding to each pixel value can be determined through the first camera response parameter, and the corresponding first relative illuminance value and the illuminance mapping relationship can be determined based on each first relative illuminance value and the illuminance mapping relationship. Two relative illuminance values, based on each second relative illuminance value and the response parameter of the second camera to determine the pixel value of the image captured by the second camera, based on the pixel value of the image captured by the first camera and the image captured by the second camera. The pixel value is constructed to obtain the pixel mapping relationship between the first camera and the second camera.

In this embodiment, the first matching point extracted from the first calibration image and the second matching point extracted from the second calibration image are obtained by performing feature matching on the pixel value corresponding to the matching point in the matching point pair to determine the first matching point. The illuminance mapping relationship between the camera and the second camera, and the pixel mapping relationship between the first camera and the second camera is obtained based on the illuminance mapping relationship, thereby realizing the mutual calibration of the first camera and the second camera. Through the pixel mapping relationship It can solve the problem of similar image information structure but inconsistent gradients, ensure the accuracy of image alignment, and improve the effect of image alignment.

In one embodiment, the first calibration image and the second calibration image include calibration targets with different regions in the same scene; according to the first camera response function and the second camera response function, the distance between the first camera and the second camera is determined The pixel mapping relationship includes: determining the pixel values of the first area corresponding to each area of the calibration target in the first calibration image; determining the pixel value of the second area corresponding to each area of the calibration target in the second calibration image ; Determine the pixel mapping relationship between the first camera and the second camera according to the corresponding relationship between the first region pixel value and the second region pixel value of the same region in the calibration target.

Wherein, the calibration target is preset in the same scene corresponding to the first camera and the second camera when shooting, the calibration target is divided into different areas, and each area may be provided with a corresponding color. The calibration target can be set according to actual needs, such as color card, gray scale card, etc. When the first camera and the second camera are shooting in the same scene, the calibration target in the scene will be captured at the same time, and the pixel mapping relationship between the first camera and the second camera can be calibrated based on the pixel values of each area of the calibration target. .

Specifically, the first camera response function and the second camera response function are obtained, the pixel mapping relationship between the first camera and the second camera is determined, and the first calibration image and the second calibration image include different regions in the same scene. The calibration target, that is, when both the first camera and the second camera capture the calibration target in the scene, the electronic device respectively determines the pixel values of the first area corresponding to each area of the calibration target in the first calibration image, and the pixel values in the second area respectively. In the calibration image, the pixel values of the second region corresponding to each region of the calibration target respectively. After obtaining the pixel value of the first area and the pixel value of the second area, the electronic device obtains the difference between the first camera and the second camera according to the corresponding relationship between the pixel value of the first area and the pixel value of the second area of the same area in the calibration target. pixel mapping relationship. Specifically, the calibration target is divided into multiple regions, and the electronic device can determine the correspondence between the pixel values of the first region in the first calibration image and the pixel values of the second region in the second calibration image for each region, The illuminance mapping relationship between the first camera and the second camera, for example, the illuminance mapping relationship is obtained according to the ratio of the pixel value of the first area and the pixel value of the second area, and the illuminance mapping relationship between the first camera and the second camera is determined based on the illuminance mapping relationship. Pixel mapping relationship.

In this embodiment, based on the correspondence between the pixel values of the first area in the first calibration image and the pixel values of the second area in the second calibration image in the same area of the calibration target in the same scene captured, determine The pixel value of the first area in the first calibration image, thereby realizing the mutual calibration of the first camera and the second camera, through the pixel mapping relationship, the problem of similar image information structure but inconsistent gradients can be solved to ensure the accuracy of image alignment, Thereby improving the effect of image alignment.

In one embodiment, each area in the calibration target has a preset corresponding solid color.

Among them, a solid color refers to a color or hue that is not mixed with other hues. Each area in the calibration target has a preset corresponding solid color, and the colors between each area can be the same or different. Each area in the calibration target has a corresponding solid color, which can ensure that the color in each area is pure and uniform, and can improve the accuracy of determining the pixel value of the area, thereby ensuring the accuracy of determining the pixel mapping relationship, which is conducive to improving the effect of image alignment. . In specific implementation, the calibration target can be a grayscale card, a color scale card, a color scale diagram, and the like.

In one embodiment, performing pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image includes: respectively determining the pixel points of each pixel in the second image. original pixel value; perform pixel value mapping on each original pixel value based on the pixel mapping relationship between the first camera and the second camera to obtain the mapped pixel value corresponding to each pixel in the second image; update the first pixel value based on each mapped pixel value; Two images are obtained, and a mapping image corresponding to the second image is obtained.

The original pixel value is the pixel value of the second image captured by the second camera without pixel mapping; the mapped pixel value is the pixel value obtained by pixel mapping the original pixel value through the pixel mapping relationship; the mapped image is based on the mapping The pixel value updates the second image, that is, the mapping result obtained after pixel mapping is performed on the second image through the pixel mapping relationship.

Specifically, after obtaining the first image and the second image to be processed, the electronic device determines the original pixel value of each pixel in the second image respectively. For example, the electronic device can traverse each pixel in the second image to obtain the corresponding pixel value of each pixel. the original pixel value. The electronic device further acquires the pixel mapping relationship between the first camera and the second camera, and performs pixel value mapping on each original pixel value based on the pixel mapping relationship, that is, maps each original pixel value to the first image according to the pixel mapping relationship. In the color space, the mapped pixel value of each pixel in the second image is obtained. The electronic device updates the second image based on the obtained mapped pixel values, and specifically may update the pixel values of the corresponding pixel points in the second image based on the mapped pixel values to generate a mapped image corresponding to the second image, so as to realize the conversion of the second image. The color space mapped to the first image overcomes the problem that the image information structure is similar but the gradients are inconsistent due to the difference in information sources, resulting in poor image alignment accuracy. Based on the mapping image corresponding to the second image, the first image is fused, Improved image fusion effect.

In one embodiment, aligning the mapped images corresponding to the first image and the second image includes: respectively performing distortion correction on the mapped images corresponding to the first image and the second image to obtain the first distortion corrected image and the second distortion corrected image. Correcting the images; respectively performing stereo correction on the first and second distortion-corrected images to obtain the first and second corrected images; and performing grid alignment on the first and second corrected images.

Among them, the distortion correction is used to correct the image distortion caused by the lens distortion phenomenon, and specifically includes correcting radial distortion, tangential distortion, and the like. Stereo correction is used to ensure that the image planes of the two cameras are parallel, and are corrected to be aligned in coplanar rows. At this time, the optical axes of the cameras are concentric and the image rows are aligned, which is beneficial to reduce the subsequent grid alignment search range. Since the entire image in the scene is not coplanar, there will be multiple planes. When the image is aligned on the entire image, the complete alignment cannot be ensured. Therefore, the grid alignment method is used to divide the image into multiple small grids. Alignment is performed in each of the divided grids to achieve the alignment effect.

Specifically, the electronic device obtains calibration parameters corresponding to the first camera and the second camera respectively, and performs distortion correction and stereo correction through the calibration parameters corresponding to the first camera and the second camera respectively. The calibration parameters may specifically be camera parameters obtained by pre-calibration of the two cameras, and specifically include internal parameters, external parameters, and distortion parameters. The electronic device performs distortion correction on the mapped images corresponding to the first image and the second image, respectively, to obtain the first distortion corrected image and the second distortion corrected image, so as to overcome the radial direction existing in the mapped images corresponding to the first image and the second image. Distortion, tangential distortion and other distortion problems, improve image quality. Further, the electronic device performs stereo-correction on the first distortion-corrected image and the second distortion-corrected image respectively. Specifically, the first distortion-corrected image and the second distortion-corrected image can be stereo-corrected based on the Bouguet correction principle to obtain the first corrected image and the second distortion corrected image. The second corrected image is such that the obtained first corrected image and the second corrected image are in parallel with the plane, the optical axis is perpendicular to the image plane, and the pole is in a wireless distance. The electronic device then performs grid alignment on the obtained first corrected image and the second corrected image to achieve alignment of the captured image of the first camera and the captured image of the second camera.

Among them, the Bouguet correction principle is to decompose the rotation and translation matrices solved by OPencv into rotation and translation matrices that rotate half of each of the left and right cameras. The principle of decomposition is to minimize the distortion caused by the reprojection of the left and right images, and to maximize the common area of the left and right views. . Specifically, when performing stereo correction based on the Bouguet correction principle, the rotation matrix of the right image plane relative to the left image plane is decomposed into two matrices R1 and Rr, which are used as composite rotation matrices of the left and right cameras. Rotate the left and right cameras by half to make the optical axes of the left and right cameras parallel. At this time, the imaging planes of the left and right cameras are parallel, but the baseline is not parallel to the imaging plane. The transformation matrix Rrect is constructed so that the baseline is parallel to the imaging plane, and the construction method is completed by the offset matrix T of the right camera relative to the left camera. The overall rotation matrix of the left and right cameras is obtained by multiplying the composite rotation matrix and the transformation matrix. The left and right camera coordinate systems are multiplied by their respective overall rotation matrices to make the main optical axes of the left and right cameras parallel, and the image plane is parallel to the baseline. Through the above two overall rotation matrices, the ideal parallel configuration of the stereo corrected image can be obtained. image.

In this embodiment, the pre-calibrated camera parameters of the camera are used to sequentially perform distortion correction and stereo correction on the mapped images corresponding to the first image and the second image, so as to overcome the distortion phenomenon captured by the camera and reduce the distortion of the original image; The planes of the images captured by the two cameras are parallel, the optical axis is perpendicular to the image plane, and the pole is in the wireless distance, and the grid alignment is performed on the first corrected image and the second corrected image obtained after correction to ensure the alignment of the images. Effect.

In one embodiment, performing grid alignment on the first corrected image and the second corrected image includes: performing grid division on the first corrected image and the second corrected image respectively to obtain each first grid corresponding to the first corrected image. grid and each second grid corresponding to the second corrected image; respectively perform grid feature point detection on each first grid and each second grid, and obtain the first grid feature points and the first grid feature points corresponding to the first grid The second grid feature points corresponding to the two grids; image transformation is performed on the first corrected image and the second corrected image based on each of the first grid feature points and the second grid feature points to align the first corrected image and the second corrected image. Correct the image.

The grid division is used to divide the image into multiple small grids, and align each of the small grids separately, so as to avoid the problem that the image cannot be aligned as a whole when there are multiple planes. Grid feature point detection is used to detect feature points in the grid to align the grid with the feature points.

Specifically, when grid alignment is performed on the first corrected image and the second corrected image, the electronic device respectively performs grid division on the first corrected image and the second corrected image to obtain each first grid corresponding to the first corrected image. Each second grid corresponding to the second corrected image. The grid division parameters can be set according to actual needs, for example, the first corrected image and the second corrected image can be divided into N*N grids respectively. After obtaining each grid, the electronic device performs grid feature point detection on each first grid and each second grid respectively. Specifically, the feature point detection can be performed by algorithms such as Fast, SUSA, SIFT, SURF or LBP to obtain the first grid. The first grid feature point corresponding to one grid and the second grid feature point corresponding to the second grid. The electronic device performs image transformation on the first corrected image and the second corrected image based on each of the first grid feature points and the second grid feature points, so as to align the first corrected image and the second corrected image. During specific implementation, the electronic device can align each of the first grids and the corresponding second grids, so that multiple grid pairs can be aligned in parallel, and each grid pair includes a first grid and a second grid that match each other. Two grids. Specifically, after obtaining the first grid feature points corresponding to the first grid and the second grid feature points corresponding to the second grid, feature matching is performed based on the first grid feature points and the second grid feature points to obtain The matching of the first grid and the second grid is realized, and a grid pair is obtained by constructing. For each grid pair, mismatch removal processing is performed, for example, the mismatched grid pair of the feature point matching pair can be removed by the RANSAC (Random Sample Consensus) algorithm. The electronic device further calculates the homography matrix of each grid pair, and performs perspective transformation on the first grid and the second grid in the grid pair based on the homography matrix, so as to realize the transformation of the first grid and the second grid in the grid pair. For the alignment of the second grid, the aligned first image and the aligned second image are obtained according to the alignment results of each grid pair.

In this embodiment, the image is divided into a plurality of small grids, and the small grids are aligned separately, so as to avoid the problem that the image cannot be aligned as a whole when there are multiple planes, and further improve the image alignment effect.

In one embodiment, after the stereoscopic correction is performed on the first distortion corrected image and the second distortion corrected image respectively to obtain the first corrected image and the second corrected image, the method further includes: according to the first corrected feature point and the second corrected feature point to construct a matching pair of feature points; the first corrected feature point is extracted from the first corrected image, and the second corrected feature point is extracted from the second corrected image; based on the offset between the corrected feature points in each feature point matching pair parameters, determine the projection parameters between the first corrected image and the second corrected image; perform projection alignment on the first corrected image and the second corrected image through the projection parameters to obtain the first projected aligned image and the second projected aligned image.

The first corrected feature point is extracted from the first corrected image, and the second corrected feature point is extracted from the second corrected image. Specifically, a feature point detection algorithm, such as Fast, SUSA, SIFT, SURF, or LBP, may be used to process the first corrected image and the first corrected image, respectively, to obtain the first corrected feature point and the second corrected feature point. A feature point matching pair is constructed based on the extracted first correction feature point and the second correction feature point. The feature point matching pair reflects the corresponding relationship between the correction feature points in the first correction image and the second correction image. The feature point matching pair can specifically be It is obtained by performing feature matching on the obtained first correction feature point and the second correction feature point, and constructing it based on the first correction feature point and the second correction feature point corresponding to the successful matching. That is, each feature point matching pair includes a first corrected feature point and a second corrected feature point that match each other, the first corrected feature point is from the first corrected image, and the second corrected feature point is from the second corrected image.

The offset parameter is used to characterize the alignment degree between the corrected feature points in the feature point matching pair. If the alignment degree of the corrected feature points in each feature point matching pair is high, then the alignment effect of the corresponding first image and the second image is high. Also higher. In a specific application, the offset parameter can be measured according to the distance between the corrected feature points in the feature point matching pair, such as the Euclidean distance. Projection parameters are used for image alignment. Specifically, projection mapping can be performed on two images through projection parameters to achieve image alignment.

To align the first corrected image and the second corrected image, projection mapping may be performed on the second corrected image or the first corrected image by using projection parameters, so as to project the second corrected image into the coordinate system of the first corrected image, or to map the second corrected image to the coordinate system of the first corrected image. The first corrected image is projected into the coordinate system of the second corrected image, so as to realize the projected alignment of the first image and the second image, and obtain the first projected aligned image and the second projected aligned image.

Specifically, after obtaining the first corrected image and the second corrected image, the electronic device constructs feature points according to the first corrected feature points extracted from the first corrected image and the second corrected feature points extracted from the second corrected image matching pairs. After the feature point matching pairs are constructed and obtained, the electronic device determines the offset parameters between the correction feature points in each feature point matching pair, for example, the distance between the correction feature points in each feature point matching pair can be calculated separately, and The image offset function is constructed according to the distances corresponding to each feature point matching pair, and the projection parameters are determined by solving the image offset function. After obtaining the projection parameters, the electronic device uses the projection parameters to perform projection alignment on the first corrected image and the second corrected image. Specifically, the electronic device may perform projection mapping on the first corrected image or the second corrected image by using the projection parameters, so as to realize the alignment of the first corrected image or the second corrected image. Alignment of a corrected image and a second corrected image. The projection parameters are determined according to the offset parameters between the corrected feature points in the feature point matching pair, and the projection parameters can be dynamically calibrated according to the scene captured by the image, which can reduce the influence of random errors and improve the effect of image alignment using the projection parameters. .

Further, performing grid alignment on the first corrected image and the second corrected image includes: performing grid alignment on the first projected aligned image and the second projected aligned image.

After obtaining the projection-aligned first projection-aligned image and the second projection-aligned image, perform grid alignment on the first projection-aligned image and the second projection-aligned image, so as to align the first projection-aligned image and the second projection-aligned image based on the grid alignment method The alignment images are aligned, thereby realizing alignment of the first image and the second image.

In this embodiment, during image alignment, a feature point matching pair is constructed by using the first correction feature point in the first correction image and the second correction feature point in the second correction image, so as to ensure that each correction feature in the feature point matching pair can be ensured. At the same time, the projection parameters are determined according to the offset parameters between the corrected feature points in the feature point matching pair, and the projection parameters can be dynamically calibrated according to the scene captured by the image, which reduces the influence of random errors and improves the utilization of the projection parameters. The parameter performs the effect of image alignment. The first projection-aligned image and the second projection-aligned image are then grid-aligned, so as to avoid the problem that the images cannot be aligned as a whole when there are multiple planes, and further improve the alignment effect of the images.

In one embodiment, an image processing method is provided, and the image processing method is applied in the process of aligning the RGB image captured by the RGB camera of the mobile phone and the NIR image captured by the NIR camera. Specifically, as shown in FIG. 6 , the first image is an RGB image captured by an RGB camera, and the second image is an NIR image captured by an NIR camera. After the RGB image and the NIR image are obtained, CRF is performed on the RGB image and the NIR image. The correction is to perform pixel mapping on the NIR image according to the pixel mapping relationship determined by the first camera response function of the RGB camera and the second camera response function of the NIR camera, and align the mapping image corresponding to the obtained NIR image with the RGB image. Further, the pre-calibrated camera parameters are used to perform distortion correction on the CRF-corrected RGB image and the CRF-corrected NIR image, respectively, to obtain the distortion-corrected RGB image and the distortion-corrected NIR image. Stereo-corrected the NIR images, respectively, to obtain a stereo-corrected RGB image and a stereo-corrected NIR image. For the stereo-corrected RGB image and the stereo-corrected NIR image, the grid is constructed in turn, SIFT features are extracted, feature matching and false matching are removed, the homography matrix and perspective transformation are calculated, and the aligned RGB image and the aligned NIR image are obtained.

Among them, camera calibration is used to calibrate the internal and external parameters and distortion parameters of the camera sensor. RGB cameras only need to calibrate internal parameters and distortion parameters, while NIR cameras need to calibrate external parameters in addition to internal parameters and distortion parameters. As shown in Figure 7, when calibrating the camera parameters, it is first necessary to obtain the calibration board image pair, RGB image and NIR image. The calibration board image is taken indoors, and the light intensity is weak. The whole process of shooting needs to be filled with light, and then the calibration board is detected and calibrated. At the corners of the board, the RGB camera and the NIR camera were calibrated by Zhang Zhengyou's calibration method, respectively, and the calibration parameters of the RGB camera and the NIR camera were obtained. The obtained calibration parameters can be stored for subsequent image correction processing.

Further, the camera of the camera is used to capture images, and generally needs to be calibrated before leaving the factory. The calibration of RGB camera and NIR camera can be achieved by single camera calibration. Single-camera calibration refers to determining the values of the internal and external parameters of a single camera. The internal parameters of a single camera may include f _x , f _y , c _x , and _cy , where f _x represents the unit pixel size of the focal length in the x-axis direction of the image coordinate system, and f _y represents the unit pixel size of the focal length in the y-axis direction of the image coordinate system Pixel size, c _x , _cy represent the coordinates of the principal point of the image plane, and the principal point is the intersection of the optical axis and the image plane. f _x =f/d _x , f _y =f/ _dy , where f is the focal length of a single camera, d _x represents the width of a pixel in the x-axis direction of the image coordinate system, and _dy represents the y-axis direction of the image coordinate system The width of one pixel. The image coordinate system is a coordinate system established based on the two-dimensional image captured by the camera, and is used to specify the position of the object in the captured image. The origin of the (x, y) coordinate system in the image coordinate system is located on the optical axis of the camera and the focal point (c _x , c _y ) of the imaging plane, and the unit is the unit of length, that is, meters, and (u, v) in the pixel coordinate system The origin of the coordinate system is in the upper left corner of the image, and the unit is the unit of quantity, ie units. (x, y) is used to characterize the perspective projection relationship of the object from the camera coordinate system to the image coordinate system, and (u, v) is used to characterize the pixel coordinates. The conversion relationship between (x, y) and (u, v) is as formula (1):

Perspective projection refers to a single-sided projection image that is closer to the visual effect by projecting a body onto the projection surface by the central projection method.

The external parameters of a single camera include a rotation matrix and a translation matrix that convert the coordinates in the world coordinate system to the coordinates in the camera coordinate system. The world coordinate system reaches the camera coordinate system through rigid body transformation, and the camera coordinate system reaches the image coordinate system through perspective projection transformation. Rigid body transformation refers to the rotation and translation of a geometric object when the object is not deformed in three-dimensional space, which is the rigid body transformation. The rigid body transformation is as formula (2),

X _c =RX+T,

Among them, X _c represents the camera coordinate system, X represents the world coordinate system, R represents the rotation matrix from the world coordinate system to the camera coordinate system, and T represents the translation matrix from the world coordinate system to the camera coordinate system. The distance between the origin of the world coordinate system and the origin of the camera coordinate system is jointly controlled by the components in the three axis directions of x, y, and z, and has three degrees of freedom. R is the sum of the effects of rotation around the X, Y, and Z axes respectively. . t _x represents the translation amount in the x-axis direction, ty represents the translation amount in the y-axis direction, and t _z represents the translation amount in the _z -axis direction.

The world coordinate system is the absolute coordinate system of the objective three-dimensional space, which can be established at any position. For example, for each calibration image, the world coordinate system can be established with the upper left corner of the calibration plate as the origin, the calibration plate plane as the XY plane, and the Z axis perpendicular to the calibration plate plane upward. The camera coordinate system takes the optical center of the camera as the origin of the coordinate system, takes the optical axis of the camera as the Z axis, and the X axis and the Y axis are respectively parallel to the X axis and the Y axis of the image coordinate system. The principal point of the image coordinate system is the intersection of the optical axis with the image plane. The image coordinate system takes the principal point as the origin. The pixel coordinate system means that the origin is defined at the upper left corner of the image plane.

The distortion parameters of the camera are determined according to the intrinsic and extrinsic parameters of the camera. In one embodiment, a brown polynomial can be used as the distortion model, and the brown model includes 5 parameters, among which, 3 radial distortion parameters and 2 tangential distortion parameters. In other embodiments, the block surface function fitting can also be performed to obtain the distortion parameters.

Further, before the CRF correction is performed on the RGB image and the NIR image, the process of calibrating the CRF is also included. Specifically, the purpose of calibrating CRF is to calculate the color mapping relationship between RGB image and NIR image. CRF calibration includes two processes of CRF self-calibration and mutual calibration, wherein self-calibration is used to calculate the relationship between real-world illuminance and RGB image or NIR image brightness, The mutual calibration is to find the pixel relationship between the RGB image and the NIR image according to the brightness and illuminance relationship obtained from the self-calibration. As shown in Figure 8, when calibrating the CRF and determining the pixel mapping relationship between the RGB camera and the NIR camera, the image pairs captured by the RGB camera and the NIR camera under different exposure time conditions are obtained, and the CRF auto-autonomous Calibration and mutual calibration to determine the pixel mapping relationship between the RGB camera and the NIR camera. Specifically, high dynamic range scenes (including over-exposure and over-dark areas) are selected, and 5 sets of images with different exposure times are taken with RGB cameras and NIR cameras respectively. RGB images are RGB_1~RGB_5, and NIR images are NIR_1~NIR_5. The exposure time is modified by the gain value (signal gain) and shutter value (shutter speed) of the mobile phone, and it is decreased in multiples of 2. The maximum exposure time of the RGB camera and the NIR camera can be inconsistent. The exposure times are (EV-2, EV-1 ,EV0,EV+1,EV+2). Align RGB_1~RGB_5 and NIR_1~NIR_5 respectively. The alignment method adopts median threshold bitmaps, and the new aligned images are RGB'_1~RGB'_5, and NIR'_1~NIR'_5.

Separate all RGB image channels into R, G, B channels, calculate their luminance channel, NIR image is a single channel and does not need to be separated. The camera response curve corresponding to the camera response function is obtained by the Debevec method for the RGB luminance channel and the NIR image respectively. FIG. 9 is a schematic diagram of a camera response curve in one embodiment. Among them, the abscissa is the image pixel value (0-255), and the ordinate is the relative illuminance value. Curve 1 is the camera response curve of the RGB luminance channel, and curve 2 is the camera response curve of the NIR image. There is a certain proportional relationship between the relative illuminance value and the real illuminance, and the camera response curve represents the relationship between the pixel value of the image and the relative illuminance value.

In addition, in addition to the camera response function of the RGB color space, the phase response function of other color spaces can also be constructed, such as the V channel of HSV, the R, G or B channels of RGB separation, etc., which can be adjusted according to actual needs. As shown in Figure 10, curve 3 is the camera response curve of the NIR image, curve 4 is the camera response curve of the R channel image in the RGB color space, curve 5 is the camera response curve of the B channel image in the RGB color space, curve 6 and curve 7 Overlays, respectively, are the camera response curves corresponding to the G1 channel image and the G2 channel image in the RGGB color space of the RAW image Bayer mode. As shown in FIG. 11 , the curve 8 is the camera response curve of the NIR image, and the curve 9 is the camera response curve corresponding to the V channel image in the HSV color space.

The camera response curve can only calculate the relationship between the image pixel value and the relative illuminance value, and it is necessary to obtain the relationship between the relative illuminance and the true illuminance through CRF mutual calibration. However, the true illuminance needs to be measured with an illuminometer. To simplify this problem, the relationship between the illuminance values in the response curves of the RGB camera and the NIR camera is calculated. In specific implementation, on the one hand, the matching points between the RGB image and the NIR image can be extracted to obtain the pixel value of the matching point and its relative illuminance value in the response area, and then the illuminance mapping relationship between the two can be obtained. On the other hand, a gray-scale card can be placed in the scene when the RGB camera and NIR camera capture images, and the area of the gray-scale card can be detected to obtain the pixel value of each area of the gray-scale card in the RGB image and the NIR image, and then the pixel value Divide to get the illuminance mapping relationship. The pixel mapping relationship between the RGB image and the NIR image pixel value is established based on the illuminance mapping relationship determined by the mutual calibration, and the pixel mapping relationship describes the corresponding relationship between the NIR brightness value and the brightness value of a certain RGB channel. Using CRF calibration and correction, the NIR image can be corrected to the luminance domain of the RGB image, which can solve the problem of similar structures but inconsistent gradients when the images acquire image information due to different sensors, and can improve the image alignment effect. The pixel mapping relationship can be stored in a table, and can be saved offline after calibration, and only needs to be calibrated once. When in use, it is only necessary to traverse the look-up table, which can effectively improve the efficiency of image processing.

Further, when performing CRF correction on RGB images and NIR images, using the CRF calibration result, that is, the pixel mapping relationship between the RGB camera and the NIR camera, to perform brightness mapping on the NIR image, it is necessary to quickly traverse each pixel point of the NIR image, through Find the mapping relationship between the pixel values of the RGB image and the NIR image, and get the new pixel value of each pixel in the NIR image.

Further, perform distortion correction and stereo correction on the CRF correction results. Specifically, the internal and external parameters and distortion parameters of the camera calibrated by the RGB camera and the NIR camera are used to perform distortion correction and stereo correction on the image, and the non-coplanar lines of the image are aligned. Coplanar line alignment, at this time, the camera optical axis is concentric, and the image lines are aligned, which is beneficial to reduce the grid alignment search range later.

After obtaining the results of stereo correction, considering that the entire image in the scene is not coplanar, there will be multiple planes, and the alignment method based on SIFT features on the entire image cannot be completely aligned. Therefore, the grid alignment method is used to divide the image into multiple small grids, and the alignment method based on SIFT features is used in the small grids to achieve the alignment effect. Specifically, the stereo-corrected RGB image and the stereo-corrected NIR image are divided into N*N grids, and each grid is traversed. SIFT feature point extraction, SIFT feature matching, RANSAC removal of mismatched points, calculation Homography matrix and perspective transformation to obtain aligned RGB images and aligned NIR images.

FIG. 12 is a flowchart of a method for determining a pixel mapping relationship of a binocular camera in one embodiment. The method for determining the pixel mapping relationship of the binocular camera in this embodiment is described by taking the electronic device in FIG. 3 as an example for description. As shown in FIG. 12 , the image processing method includes process 1202 to process 1208 .

Process 1202: Obtain a first calibration image group and a second calibration image group; the first calibration image group includes a first calibration image obtained by shooting the first camera in the binocular camera under the same scene and different exposure time conditions, and the second calibration image is obtained. The image group includes a second calibration image captured by a second camera in the binocular camera under the conditions of the same scene and different exposure times.

Wherein, the first calibration image group includes the first calibration image obtained by the first camera in the binocular camera under the conditions of the same scene and different exposure times, and the second calibration image group includes the second camera in the binocular camera in the The second calibration image obtained by shooting under the conditions of the same scene and different exposure times. That is, the images in the first calibration image group and the second calibration image group are both captured by the corresponding cameras for the same scene, and the exposure times of the first calibration images in the first calibration image group are different when they are captured. The exposure times of the respective second calibration images in the calibration image group are different during shooting. In specific implementation, the shooting scenes corresponding to the first calibration image group and the second calibration image group may be high dynamic range scenes including overexposed and overdark areas, so as to ensure that the determined pixel mapping relationship can be applied to high dynamic range scenes, Guarantees the applicable scope of the pixel mapping relationship. The number of the first calibration image and the second calibration image and the corresponding exposure time can be flexibly set according to actual needs. For example, the number of the first calibration image and the second calibration image can be 5, and the corresponding exposure time can be increased. , and the respective exposure times of the first calibration image and the second calibration image may be different. The exposure time can be adjusted by modifying the signal gain (gain value) and shutter speed (shutter value) of the electronic device. Further, the electronic device can first align each calibration image in the first calibration image group and the second calibration image group, such as aligning the first calibration image group and the second calibration image group by the alignment method of the median threshold bitmap. Perform median threshold alignment on each of the calibration images, and determine the corresponding camera response function based on the first calibration image after median threshold alignment and the second calibration image after median threshold alignment.

In process 1204, a first camera response function corresponding to the first camera is determined based on each first calibration image.

Among them, the camera response function is used to represent the corresponding relationship between the brightness of the image captured by the camera and the illumination in the real world. Generally, the brightness or illuminance observed in the real world is constant and will not change with different cameras, and there is a certain correspondence between the brightness of the image captured by the camera and the illuminance in the real world. Function description. Specifically, the electronic device may obtain the first camera response function corresponding to the first camera by using the Debevec algorithm based on the luminance channel image of each first calibration image.

In process 1206, a second camera response function corresponding to the second camera is determined based on each of the second calibration images.

Similar to obtaining the first camera response function corresponding to the first camera, the electronic device can obtain the second camera response function corresponding to the second camera through the Debevec algorithm based on the luminance channel image of each second calibration image.

In process 1208, the pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function and the second camera response function.

Specifically, after obtaining the first camera response function and the second camera response function, the electronic device determines the pixel mapping relationship between the first camera and the second camera based on the first camera response function and the second camera response function. For example, the pixel value of the matching point between the first calibration image and the second calibration image, and the relative illuminance value determined by the matching point according to the response function of the first camera and the response function of the second camera, can be used to determine the difference between the first camera and the second camera. The illuminance mapping relationship between them is determined, and the pixel mapping relationship between the first camera and the second camera is determined based on the illuminance mapping relationship.

The above-mentioned method for determining the pixel mapping relationship of a binocular camera is to determine the first camera response function corresponding to the first camera in the binocular camera and the corresponding second camera respectively according to the images captured by the binocular camera under the conditions of the same scene and different exposure times. and the pixel mapping relationship between the first camera and the second camera is determined based on the first camera response function and the second camera response function. The pixel mapping relationship is determined according to the first camera response function of the first camera and the second camera response function of the second camera. Through the pixel mapping relationship, the camera response function of the camera can be used to convert the second image captured by the second camera in the binocular camera. Mapping to the pixel space of the first image captured by the first camera can solve the problem of similar image information structure but inconsistent gradients, ensure the accuracy of image alignment, and improve the effect of image alignment.

In one embodiment, the first camera is a visible light camera; determining a first camera response function corresponding to the first camera based on each first calibration image includes: acquiring a target channel image of each first calibration image corresponding to a target color channel respectively; Determine the first feature point corresponding to each target channel image at the same position in the same scene; determine the channel brightness value of each first feature point corresponding to the target color channel; according to the channel brightness value corresponding to each first feature point of the target color channel , and determine the first camera response function corresponding to the first camera.

In one embodiment, determining the pixel mapping relationship between the first camera and the second camera according to the first camera response function and the second camera response function includes: acquiring at least one pair of matching point pairs, where the matching point pair The first matching point extracted from a calibration image and the second matching point extracted from the second calibration image are obtained by position matching; respectively determine the pixel value of the first matching point in the matching point pair and the pixel value of the second matching point. the second point pixel value; determine the first relative illuminance value according to the first point pixel value and the first camera response function; determine the second relative illuminance value according to the second point pixel value and the second camera response function; based on the first relative illuminance value determining an illuminance mapping relationship with the second relative illuminance value; and determining a pixel mapping relationship between the first camera and the second camera according to the illuminance mapping relationship.

It should be understood that although the respective processes in the flowcharts of FIGS. 4-8 and 12 are displayed in sequence according to the arrows, these processes are not necessarily executed sequentially in the sequence indicated by the arrows. Unless explicitly stated herein, there is no strict order in the execution of these processes, and these processes may be performed in other orders. Moreover, at least a part of the processes in FIGS. 4-8 and 12 may include multiple sub-processes or multiple stages. These sub-processes or stages are not necessarily executed at the same time, but may be executed at different times. These sub-processes are not necessarily completed at the same time. Alternatively, the order of execution of the stages is not necessarily sequential, but may be performed alternately or alternately with other processes or sub-processes of other processes or at least a portion of the stages.

FIG. 13 is a structural block diagram of an image processing apparatus 1300 according to an embodiment. As shown in FIG. 13, the image processing apparatus 1300 includes a to-be-processed image acquisition module 1302, a pixel mapping processing module 1304 and an image alignment processing module 1306, wherein:

The to-be-processed image acquisition module 1302 is used to acquire the to-be-processed first image and the second image; the first image is captured by the first camera, and the second image is captured by the second camera;

The pixel mapping processing module 1304 is configured to perform pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image; wherein the pixel mapping relationship is based on the pixel mapping relationship of the first camera. The first camera response function and the second camera response function of the second camera are determined;

The image alignment processing module 1306 is configured to align the mapped image corresponding to the second image with the first image.

In one embodiment, it further includes a calibration image group acquisition module, a first camera response function determination module, a second camera response function determination module, and a pixel mapping relationship determination module; wherein: a calibration image group acquisition module for acquiring the first calibration an image group and a second calibration image group; the first calibration image group includes the first calibration image obtained by the first camera in the same scene and under different exposure time conditions, and the second calibration image group includes the second camera in the same scene, a second calibration image obtained by shooting under different exposure time conditions; a first camera response function determination module, used for determining a first camera response function corresponding to the first camera based on each first calibration image; a second camera response function determination module, with for determining the second camera response function corresponding to the second camera based on each second calibration image; the pixel mapping relationship determination module is used for determining the relationship between the first camera and the second camera according to the first camera response function and the second camera response function pixel mapping relationship.

In one embodiment, the first camera is a visible light camera; the first camera response function determination module includes a target channel image acquisition module, a first feature point determination module, a channel brightness value determination module and a first camera response function acquisition module; wherein: The target channel image acquisition module is used to acquire the target channel images of each first calibration image corresponding to the target color channel respectively; the first feature point determination module is used to determine the same position in the same scene in each target channel image corresponding to the first channel image. a feature point; a channel luminance value determination module for determining a channel luminance value of each first feature point corresponding to a target color channel; a first camera response function obtaining module for determining a channel luminance value corresponding to each first feature point corresponding to the target color channel; The channel luminance value determines the first camera response function corresponding to the first camera.

In one embodiment, the target channel image acquisition module includes a channel separation module and a separated channel image processing module; wherein: the channel separation module is used to perform channel separation on the first calibration image to obtain each separated channel image; the separated channel image processing module , which is used to obtain the target channel image corresponding to the target color channel according to the separate channel images.

In one embodiment, the target channel image acquisition module includes a target color space image acquisition module and a target color space image processing module; wherein: the target color space image acquisition module is used to transform the first calibration image to a target including the target color channel The color space is used to obtain the target color space image; the target color space image processing module is used to obtain the target channel image corresponding to the target color channel according to the target color space image.

In one embodiment, the second camera is an infrared camera; the second camera response function determination module includes a second feature point determination module, a second feature point pixel determination module and a second feature point pixel processing module; wherein: the second feature point a determination module, used for respectively determining the second feature points corresponding to the same position in each second calibration image in the same scene; a second feature point pixel determination module, used for determining the pixel value of each second feature point; the second feature point The point pixel processing module is configured to determine the second camera response function corresponding to the second camera according to the pixel value of each second feature point.

In one embodiment, the pixel mapping relationship determination module includes a matching point pair acquisition module, a matching point pair pixel determination module, a relative illuminance value determination module, an illuminance mapping relationship determination module, and an illuminance mapping relationship processing module; wherein: the matching point pair acquisition module , used to obtain at least one pair of matching point pairs, the matching point pairs are obtained by performing feature matching on the first matching point extracted from the first calibration image and the second matching point extracted from the second calibration image; the matching point pair pixel is determined by a module for respectively determining the pixel value of the first point of the first matching point and the pixel value of the second point of the second matching point in the matching point pair; the relative illuminance value determination module is used for determining the pixel value of the first point and the first camera The response function determines the first relative illuminance value; the second relative illuminance value is determined according to the second point pixel value and the second camera response function; the illuminance mapping relationship determination module is used for determining the illuminance based on the first relative illuminance value and the second relative illuminance value A mapping relationship; an illuminance mapping relationship processing module, configured to determine the pixel mapping relationship between the first camera and the second camera according to the illuminance mapping relationship.

In one embodiment, the first calibration image and the second calibration image include calibration targets with different regions in the same scene; the pixel mapping relationship determination module includes a first region pixel determination module, a second region pixel determination module, and a region pixel analysis module module; wherein: the first area pixel determination module is used to determine the first area pixel values corresponding to each area of the calibration target in the first calibration image; the second area pixel determination module is used to determine the second calibration image. , the second area pixel values corresponding to each area of the calibration target respectively; the area pixel analysis module is used to determine the first area pixel value according to the corresponding relationship between the first area pixel value and the second area pixel value of the same area in the calibration target. The pixel mapping relationship between the camera and the second camera.

In one embodiment, the pixel mapping processing module 1304 includes an original pixel determination module, a mapped pixel acquisition module and an image update module; wherein: an original pixel determination module is used to determine the original pixel value of each pixel in the second image respectively; mapping The pixel obtaining module is used to perform pixel value mapping on each original pixel value based on the pixel mapping relationship between the first camera and the second camera, so as to obtain the mapped pixel value corresponding to each pixel point in the second image; the image updating module, using The second image is updated based on each mapped pixel value to obtain a mapped image corresponding to the second image.

In one embodiment, the image alignment processing module 1306 includes a distortion correction module, a stereo correction module, and a grid alignment module; wherein: the distortion correction module is configured to perform distortion correction on the mapped images corresponding to the first image and the second image, respectively, obtaining a first distortion corrected image and a second distortion corrected image; a stereo correction module for performing stereo correction on the first distortion corrected image and the second distortion corrected image respectively to obtain the first corrected image and the second corrected image; grid alignment The module is used for grid-aligning the first corrected image and the second corrected image.

In one embodiment, the grid alignment module includes a grid division module, a grid feature extraction module and an image transformation module; wherein: a grid division module is included for meshing the first corrected image and the second corrected image respectively Divide to obtain each first grid corresponding to the first corrected image and each second grid corresponding to the second corrected image; the grid feature extraction module is used to perform the first grid and each second grid respectively. Grid feature point detection, to obtain the first grid feature point corresponding to the first grid and the second grid feature point corresponding to the second grid; the image transformation module is used for each first grid feature point and second grid feature point based on The grid feature points perform image transformation on the first corrected image and the second corrected image to align the first corrected image and the second corrected image.

In one embodiment, it also includes a matching pair building module, a projection parameter determining module, and a projection alignment module; wherein: a matching pair building module is used to build a matching pair of feature points according to the first corrected feature point and the second corrected feature point; A correction feature point is extracted from the first correction image, and the second correction feature point is extracted from the second correction image; the projection parameter determination module is used to match the offset parameters between the correction feature points based on each feature point pair , determine the projection parameters between the first corrected image and the second corrected image; the projection alignment module is used to perform projection alignment on the first corrected image and the second corrected image through the projection parameters to obtain the first projected aligned image and the second projected Align the images; the grid alignment module is also used for grid alignment of the first projection alignment image and the second projection alignment image.

FIG. 14 is a structural block diagram of an apparatus 1400 for determining a pixel mapping relationship of a binocular camera according to an embodiment. As shown in FIG. 14 , the device 1400 for determining the pixel mapping relationship of the binocular camera includes:

The calibration image group obtaining module 1402 is used to obtain the first calibration image group and the second calibration image group; the first calibration image group includes the first calibration image obtained by the first camera in the binocular camera under the same scene and different exposure time conditions. Calibration images, the second calibration image group includes second calibration images obtained by shooting the second camera in the binocular camera under the same scene and different exposure time conditions;

a first camera response function determination module 1404, configured to determine a first camera response function corresponding to the first camera based on each first calibration image;

A second camera response function determining module 1406, configured to determine a second camera response function corresponding to the second camera based on each second calibration image;

The pixel mapping relationship determining module 1408 is configured to determine the pixel mapping relationship between the first camera and the second camera according to the first camera response function and the second camera response function.

In one embodiment, the first camera is a visible light camera; the first camera response function determination module 1404 includes a target channel image acquisition module, a first feature point determination module, a channel brightness value determination module, and a first camera response function acquisition module; wherein : The target channel image acquisition module is used to acquire the target channel images of the first calibration images corresponding to the target color channels respectively; the first feature point determination module is used to determine the corresponding position in each target channel image in the same scene. a first feature point; a channel luminance value determination module for determining the channel luminance value of each first feature point corresponding to the target color channel; a first camera response function obtaining module for determining the target color channel corresponding to each first feature point The channel brightness value of , determines the first camera response function corresponding to the first camera.

In one embodiment, the second camera is an infrared camera; the second camera response function determination module 1406 includes a second feature point determination module, a second feature point pixel determination module and a second feature point pixel processing module; wherein: the second feature point The point determination module is used to respectively determine the second feature points corresponding to the same position in each second calibration image in the same scene; the second feature point pixel determination module is used to determine the pixel value of each second feature point; the second The feature point pixel processing module is configured to determine the second camera response function corresponding to the second camera according to the pixel value of each second feature point.

In one embodiment, the pixel mapping relationship determination module 1408 includes a matching point pair acquisition module, a matching point pair pixel determination module, a relative illuminance value determination module, an illuminance mapping relationship determination module, and an illuminance mapping relationship processing module; wherein: the matching point pair acquisition The module is used to obtain at least one pair of matching point pairs, and the matching point pairs are obtained by performing feature matching on the first matching point extracted from the first calibration image and the second matching point extracted from the second calibration image; the matching point pair pixel The determining module is used to respectively determine the pixel value of the first point of the first matching point and the pixel value of the second point of the second matching point in the matching point pair; the relative illuminance value determination module is used to determine the pixel value of the first point and the first point pixel value The camera response function determines the first relative illuminance value; the second relative illuminance value is determined according to the second point pixel value and the second camera response function; the illuminance mapping relationship determination module is used for determining based on the first relative illuminance value and the second relative illuminance value The illuminance mapping relationship; the illuminance mapping relationship processing module is used to determine the pixel mapping relationship between the first camera and the second camera according to the illuminance mapping relationship.

In one embodiment, the first calibration image and the second calibration image include calibration targets with different regions in the same scene; the pixel mapping relationship determination module 1408 includes a first region pixel determination module, a second region pixel determination module and a region pixel An analysis module; wherein: a first area pixel determination module is used to determine the first area pixel values corresponding to each area of the calibration target in the first calibration image; the second area pixel determination module is used to determine the first area pixel value in the second calibration image In the image, the pixel values of the second area corresponding to each area of the calibration target respectively; the area pixel analysis module is used to determine the first area pixel value according to the corresponding relationship between the pixel value of the first area and the pixel value of the second area in the same area of the calibration target. Pixel mapping relationship between a camera and a second camera.

The division of each module in the image processing device or the device for determining the pixel mapping relationship of the binocular camera is only used for illustration. In other embodiments, the device for determining the pixel mapping relationship of the image processing device or the binocular camera can be divided into Different modules are used to complete all or part of the functions of the image processing device or the device for determining the pixel mapping relationship of the binocular camera.

For the specific limitation of the image processing apparatus, reference may be made to the limitation of the image processing method above, which will not be repeated here. For the specific limitation of the device for determining the pixel mapping relationship of the binocular camera, reference may be made to the definition of the method for determining the pixel mapping relationship of the binocular camera above, which will not be repeated here. Each module in the image processing device or the device for determining the pixel mapping relationship of the binocular camera can be implemented in whole or in part by software, hardware, and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

FIG. 15 is a schematic diagram of the internal structure of an electronic device in one embodiment. As shown in Figure 15, the electronic device includes a processor and a memory connected by a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include non-volatile storage media and internal memory. The nonvolatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided by the following embodiments or the pixel mapping relationship determination of a binocular camera. Internal memory provides a cached execution environment for operating system computer programs in non-volatile storage media. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales, a sales terminal), a vehicle-mounted computer, a wearable device, and the like.

The implementation of each module in the image processing apparatus or the apparatus for determining a pixel mapping relationship of a binocular camera provided in the embodiment of the present application may be in the form of a computer program. The computer program can be run on a terminal or server. The program modules constituted by the computer program can be stored on the memory of the electronic device. When the computer program is executed by the processor, the process of the method described in the embodiments of the present application is implemented.

Embodiments of the present application also provide a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when executed by one or more processors, cause the processors to perform the processes of the image processing method.

A computer program product containing instructions, when run on a computer, causes the computer to perform an image processing method.

Embodiments of the present application also provide a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to execute the pixel mapping relationship of the binocular camera The process of determining the method.

A computer program product containing instructions, when run on a computer, causes the computer to execute a method for determining a pixel mapping relationship of a binocular camera.

Any reference to a memory, storage, database, or other medium as used herein may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Memory Bus (Rambus) Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM).

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims

An image processing method, comprising:

acquiring a first image and a second image to be processed; the first image is captured by the first camera, and the second image is captured by the second camera;

Based on the pixel mapping relationship between the first camera and the second camera, pixel mapping is performed on the second image to obtain a mapped image corresponding to the second image; wherein the pixel mapping relationship is based on the The first camera response function of the first camera and the second camera response function of the second camera are determined; and

Align the mapping image corresponding to the second image with the first image.
The method according to claim 1, wherein the method further comprises:

Obtain a first calibration image group and a second calibration image group; the first calibration image group includes a first calibration image obtained by shooting the first camera under the same scene and different exposure time conditions, and the second calibration image The group includes a second calibration image captured by the second camera under the conditions of the same scene and different exposure times;

determining a first camera response function corresponding to the first camera based on each of the first calibration images;

determining a second camera response function corresponding to the second camera based on each of the second calibration images; and

A pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function and the second camera response function.
The method according to claim 2, wherein the first camera is a visible light camera; and the determining a first camera response function corresponding to the first camera based on each of the first calibration images comprises:

acquiring target channel images of each of the first calibration images corresponding to the target color channel respectively;

determining the first feature point corresponding to the same position in each of the target channel images in the same scene;

determining a channel luminance value of each of the first feature points corresponding to the target color channel; and

A first camera response function corresponding to the first camera is determined according to a channel luminance value of each of the first feature points corresponding to the target color channel.
The method according to claim 3, wherein the acquiring the target channel images of the first calibration images corresponding to the target color channels respectively comprises:

performing channel separation on the first calibration image to obtain separate channel images; and

A target channel image corresponding to the target color channel is obtained according to each of the separated channel images.
The method according to claim 3, wherein the acquiring the target channel images of the first calibration images corresponding to the target color channels respectively comprises:

transforming the first calibration image to a target color space including a target color channel to obtain a target color space image; and

A target channel image corresponding to the target color channel is obtained according to the target color space image.
The method according to claim 2, wherein the second camera is an infrared camera; and the determining a second camera response function corresponding to the second camera based on each of the second calibration images comprises:

respectively determining the second feature points corresponding to the same position in each of the second calibration images in the same scene;

determining the pixel value of each of the second feature points; and

A second camera response function corresponding to the second camera is determined according to the pixel value of each of the second feature points.
The method according to claim 2, wherein the pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function and the second camera response function ,include:

Obtain at least one pair of matching point pairs, and the matching point pairs are obtained by performing feature matching on the first matching point extracted from the first calibration image and the second matching point extracted from the second calibration image;

respectively determining the pixel value of the first point of the first matching point and the pixel value of the second point of the second matching point in the pair of matching points;

Determine a first relative illuminance value according to the first point pixel value and the first camera response function; determine a second relative illuminance value according to the second point pixel value and the second camera response function;

determining an illuminance mapping relationship based on the first relative illuminance value and the second relative illuminance value; and

The pixel mapping relationship between the first camera and the second camera is determined according to the illuminance mapping relationship.
The method according to claim 2, wherein the first calibration image and the second calibration image include calibration targets with different regions in the same scene; the first camera response function according to the first camera response function and the second camera response function to determine the pixel mapping relationship between the first camera and the second camera, including:

Determining in the first calibration image, the first region pixel values corresponding to each region of the calibration target respectively;

determining, in the second calibration image, pixel values of the second region corresponding to each region of the calibration target; and

The pixel mapping relationship between the first camera and the second camera is determined according to the corresponding relationship between the first region pixel value and the second region pixel value of the same region in the calibration target.
The method according to claim 8, wherein each area in the calibration target has a preset corresponding solid color.
The method according to claim 1, wherein the second image is obtained by performing pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera Corresponding mapping images, including:

respectively determining the original pixel value of each pixel in the second image;

Perform pixel value mapping on each of the original pixel values based on the pixel mapping relationship between the first camera and the second camera, to obtain mapped pixel values corresponding to each pixel in the second image; and

The second image is updated based on each of the mapped pixel values to obtain a mapped image corresponding to the second image.
The method according to any one of claims 1 to 10, wherein the aligning the mapping images corresponding to the first image and the second image comprises:

Performing distortion correction on the mapping images corresponding to the first image and the second image, respectively, to obtain a first distortion corrected image and a second distortion corrected image;

respectively performing stereo correction on the first distortion corrected image and the second distortion corrected image to obtain a first corrected image and a second corrected image; and

Grid-aligning the first corrected image and the second corrected image.
The method according to claim 11, wherein the grid alignment of the first corrected image and the second corrected image comprises:

Perform grid division on the first corrected image and the second corrected image, respectively, to obtain each first grid corresponding to the first corrected image and each second grid corresponding to the second corrected image;

Perform grid feature point detection on each of the first grids and each of the second grids, respectively, to obtain a first grid feature point corresponding to the first grid and a first grid corresponding to the second grid. two grid feature points; and

Image transformation is performed on the first corrected image and the second corrected image based on each of the first grid feature points and the second grid feature points, so as to align the first corrected image and the second corrected image Correct the image.
The method according to claim 11, wherein after the stereoscopic correction is performed on the first distortion corrected image and the second distortion corrected image respectively to obtain the first corrected image and the second corrected image, further include:

A feature point matching pair is constructed according to the first corrected feature point and the second corrected feature point; the first corrected feature point is extracted from the first corrected image, and the second corrected feature point is obtained from the second corrected image extracted;

determining a projection parameter between the first corrected image and the second corrected image based on the offset parameter between the corrected feature points in each of the feature point matching pairs;

Perform projection alignment on the first corrected image and the second corrected image by using the projection parameters to obtain a first projected aligned image and a second projected aligned image;

The grid alignment of the first corrected image and the second corrected image includes:

Grid-aligning the first projected alignment image and the second projected aligned image.
A method for determining a pixel mapping relationship of a binocular camera, comprising:

Obtain a first calibration image group and a second calibration image group; the first calibration image group includes the first calibration image obtained by shooting the first camera in the binocular camera under the same scene and different exposure time conditions, the second calibration image The calibration image group includes a second calibration image captured by a second camera in the binocular camera under the conditions of the same scene and different exposure times;

determining a first camera response function corresponding to the first camera based on each of the first calibration images;

determining a second camera response function corresponding to the second camera based on each of the second calibration images; and

A pixel mapping relationship between the first camera and the second camera is determined according to the first camera response function and the second camera response function.
The method according to claim 14, wherein the first camera is a visible light camera; and the determining a first camera response function corresponding to the first camera based on each of the first calibration images comprises:

acquiring target channel images of each of the first calibration images corresponding to the target color channel respectively;

determining the first feature point corresponding to each target channel image at the same position in the same scene;

determining a channel luminance value of each of the first feature points corresponding to the target color channel; and

A first camera response function corresponding to the first camera is determined according to a channel luminance value of each of the first feature points corresponding to the target color channel.
The method according to claim 15, wherein the acquiring the target channel images corresponding to the first calibration images respectively corresponding to the target color channel comprises:

performing channel separation on the first calibration image to obtain separate channel images; and

A target channel image corresponding to the target color channel is obtained according to each of the separated channel images.
The method according to claim 15, wherein the acquiring the target channel images corresponding to the first calibration images respectively corresponding to the target color channel comprises:

transforming the first calibration image to a target color space including a target color channel to obtain a target color space image; and

A target channel image corresponding to the target color channel is obtained according to the target color space image.
The method according to claim 14, wherein the second camera is an infrared camera; and the determining a second camera response function corresponding to the second camera based on each of the second calibration images comprises:

respectively determining the second feature points corresponding to the same position in each of the second calibration images in the same scene;

determining the pixel value of each of the second feature points; and

A second camera response function corresponding to the second camera is determined according to the pixel value of each of the second feature points.
The method according to any one of claims 14 to 18, wherein the determining the relationship between the first camera and the second camera according to the first camera response function and the second camera response function The pixel mapping relationship between, including:

Obtain at least one pair of matching point pairs, and the matching point pairs are obtained by performing position matching on the first matching point extracted from the first calibration image and the second matching point extracted from the second calibration image;

respectively determining the pixel value of the first point of the first matching point and the pixel value of the second point of the second matching point in the pair of matching points;

Determine a first relative illuminance value according to the first point pixel value and the first camera response function; determine a second relative illuminance value according to the second point pixel value and the second camera response function;

determining an illuminance mapping relationship based on the first relative illuminance value and the second relative illuminance value; and

The pixel mapping relationship between the first camera and the second camera is determined according to the illuminance mapping relationship.
The method according to any one of claims 14 to 18, wherein the first calibration image and the second calibration image include calibration targets with different regions in the same scene; The first camera response function and the second camera response function determine the pixel mapping relationship between the first camera and the second camera, including:

Determining in the first calibration image, the first region pixel values corresponding to each region of the calibration target respectively;

determining, in the second calibration image, pixel values of the second region corresponding to each region of the calibration target; and

The pixel mapping relationship between the first camera and the second camera is determined according to the corresponding relationship between the first region pixel value and the second region pixel value of the same region in the calibration target.
The method according to claim 20, wherein each area in the calibration target has a preset corresponding solid color.
An image processing device, comprising:

a to-be-processed image acquisition module, configured to acquire the to-be-processed first image and the second image; the first image is captured by the first camera, and the second image is captured by the second camera;

A pixel mapping processing module, configured to perform pixel mapping on the second image based on the pixel mapping relationship between the first camera and the second camera to obtain a mapped image corresponding to the second image; wherein the The pixel mapping relationship is determined based on the first camera response function of the first camera and the second camera response function of the second camera;

The image alignment processing module is used for aligning the mapping image corresponding to the second image and the first image.
A device for determining a pixel mapping relationship of a binocular camera, comprising:

The calibration image group acquisition module is used to obtain the first calibration image group and the second calibration image group; the first calibration image group includes the first camera in the binocular camera shooting under the same scene and different exposure time conditions. a calibration image, wherein the second calibration image group includes a second calibration image obtained by shooting a second camera in the binocular camera under the conditions of the same scene and different exposure times;

a first camera response function determination module, configured to determine a first camera response function corresponding to the first camera based on each of the first calibration images;

A second camera response function determination module, configured to determine a second camera response function corresponding to the second camera based on each of the second calibration images;

A pixel mapping relationship determination module, configured to determine a pixel mapping relationship between the first camera and the second camera according to the first camera response function and the second camera response function.
An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that, when the computer program is executed by the processor, the processor executes any one of claims 1 to 21. A step of the method.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.