WO2022267506A1

WO2022267506A1 - Image fusion method, electronic device, storage medium, and computer program product

Info

Publication number: WO2022267506A1
Application number: PCT/CN2022/077713
Authority: WO
Inventors: 乔晓磊; 丁大钧; 肖斌; 陈珂; 朱聪超
Original assignee: 荣耀终端有限公司
Priority date: 2021-06-23
Filing date: 2022-02-24
Publication date: 2022-12-29
Also published as: CN115514876A; CN115514876B

Abstract

The present application is applicable to the technical field of image processing, and provides an image fusion method, an electronic device, a computer readable storage medium, and a computer program product. The method comprises: obtaining at least two frames of first images having different exposure amounts and photographed by a first camera, and obtaining at least two frames of second images having different exposure amounts and photographed by a second camera, wherein the angle of field of view of the first camera is greater than the angle of field of view of the second camera, and the first images comprise the second images; fusing the at least two frames of the first images having different exposure amounts into a first high dynamic range image, and fusing the at least two frames of the second images having different exposure amounts into a second high dynamic range image; and fusing the first high dynamic range image and the second high dynamic range image to obtain a fused image. By means of this processing, some of image details that are lost due to the excessive exposure of an image of a small angle of field of view can be restored, and thus, the definition of the fused image is improved.

Description

Image fusion method, electronic device, storage medium and computer program product

This application claims the priority of the Chinese patent application submitted to the State Intellectual Property Office on June 23, 2021, with the application number 202110707247.7 and the application name "Image fusion method, electronic equipment, storage medium and computer program product", the entire content of which Incorporated in this application by reference.

technical field

The present application belongs to the technical field of image processing, and in particular relates to an image fusion method, electronic equipment, a computer-readable storage medium and a computer program product.

Background technique

Existing electronic devices are usually configured with multiple cameras, and the effect of captured images can be effectively improved by fusing images captured by the multiple cameras. However, the inventors found that when the small field of view image taken by the telephoto camera is fused to the large field of view image taken by the ordinary camera, some areas of the small field of view image will be overexposed due to backlighting and other reasons, and the image details will be lost. Affects the sharpness of the fused image.

Contents of the invention

In view of this, the embodiments of the present application provide an image fusion method, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the definition of a fused image.

In the first aspect, the embodiment of the present application provides an image fusion method, including:

Acquiring at least two frames of first images with different exposures captured by the first camera, and acquiring at least two frames of second images with different exposures captured by the second camera, wherein the angle of view of the first camera is larger than the The angle of view of the second camera, the first image includes the second image;

fusing the at least two frames of first images with different exposures into a first high dynamic range image, and fusing the at least two frames of second images with different exposures into a second high dynamic range image;

The first high dynamic range image and the second high dynamic range image are fused to obtain a fused image.

In the embodiment of the present application, at first, at least two frames of images with a large field of view with different exposures and at least two frames of images with a small field of view with different exposures are obtained; High dynamic range fusion processing, obtaining the first high dynamic range image corresponding to the large field of view image and the second high dynamic range image corresponding to the small field of view image; finally, combining the first high dynamic range image and the second high dynamic range image Fusion to obtain the fused image. The above process can restore some image details lost due to overexposure of the small field of view image by performing high dynamic range fusion processing on the small field of view image, thereby improving the clarity of the fused image.

In an embodiment of the present application, fusing the at least two frames of second images with different exposures into a second high dynamic range image may include:

According to the image features of each pixel in the first high dynamic range image, calculate the image features of each pixel in the second high dynamic range image from the weights of the second images;

According to the weights of the image features of each pixel in the second high dynamic range image from the respective second images, fusing the at least two frames of second images with different exposures into the second high dynamic range image range image.

Using the image features of the first high dynamic range image to guide the process of fusing the second images with HDR effects, so that the obtained second high dynamic range image and the first high dynamic range image have the same or similar brightness in the overlapping area and other image features, so that the image obtained after fusing the two high dynamic range images will also have the same or similar image features as the first high dynamic range image. This process can be mainly based on the image features of each pixel in the first high dynamic range image, and the image features of each pixel in the second high dynamic range image are calculated from the weights of each second image, and then calculated according to The weights of are used to fuse the respective second images into a second high dynamic range image.

In one embodiment of the present application, according to the image features of each pixel in the first high dynamic range image, the image features of each pixel in the second high dynamic range image are obtained from the respective The weights of the second image may include:

According to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, calculate and obtain each pixel in the first high dynamic range image The image features of each come from the weights of each of the first images;

According to the weights that the image features of each pixel in the first high dynamic range image come from each of the first images, it is calculated that the image features of each pixel in the second high dynamic range image come from weights for each of the second images.

According to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, the fusion weight of each first image can be calculated, and then calculated accordingly to obtain the first high dynamic range image. The image features of each pixel in the two high dynamic range images come from the weights of the second images respectively. When calculating the weight, specifically, a weight reuse method or an image feature multiplexing method may be used.

Further, the image features of each pixel in the second high dynamic range image are calculated according to the weights of the image features of each pixel in the first high dynamic range image from each of the first images. The weights from each of the second images may include:

For any target pixel in the second high dynamic range image, the image features of the corresponding pixel of the target pixel in the first high dynamic range image are respectively obtained from each of the first images The weights are determined as the weights of the image features of the target pixel points from the respective second images.

This is an implementation manner of weight reuse, that is, the fusion weight of the first high dynamic range image is reused in the fusion process of the second high dynamic range image. In the case that the image characteristics of each first image and each second image are similar, a second high dynamic range image having image characteristics similar to that of the first high dynamic range image can be obtained by using weight reuse.

For any target pixel in the second high dynamic range image, according to the image features of the corresponding pixel of the target pixel in the first high dynamic range image and the position of the target pixel in each of the The image features of the corresponding pixels in the second image are calculated to obtain the image features of the target pixels from the weights of each of the second images, wherein the image features of the target pixels and the target pixels Image features of corresponding pixel points in the first high dynamic range image are equal.

This is an implementation of image feature multiplexing, that is, the fusion weight of each second image is calculated by taking the same image feature of each corresponding pixel point in the two high dynamic range images as a known condition. Even if the image features of each first image and each second image are quite different, the second high dynamic range image with the same image features as the first high dynamic range image can be obtained by using image feature multiplexing.

Further, the at least two frames of second images with different exposures include a first exposure image and a second exposure image, the exposure of the first exposure image is greater than the exposure of the second exposure image; according to the target The image features of the corresponding pixels of the pixels in the first high dynamic range image and the image features of the corresponding pixels of the target pixels in each of the second images are calculated to obtain the image of the target pixels The features come from the weights of each of the second images, and may include:

The image features of the target pixel points are calculated according to the following formula from the weights of the first exposure image and the second exposure image:

A*X+B*Y=P

X+Y=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and A indicates that the target pixel The image feature of the corresponding pixel point in the first exposure image, B represents the image feature of the corresponding pixel point of the target pixel point in the second exposure image, and P represents the image feature of the target pixel point in the Image features of corresponding pixels in the first high dynamic range image.

In the above formula, A, B, and P are all known values, so the weight X and weight Y can be calculated.

Further, the at least two frames of second images with different exposures include a first exposure image, a second exposure image and a third exposure image, the exposure of the first exposure image is greater than the exposure of the second exposure image , the exposure amount of the second exposure image is greater than the exposure amount of the third exposure image; according to the image feature of the corresponding pixel point of the target pixel point in the first high dynamic range image and the target pixel point In the image features of the corresponding pixels in each of the second images, calculating the image features of the target pixels from the weights of each of the second images may include:

According to the weights of the image features of the corresponding pixels of the target pixel in the first high dynamic range image respectively from the first images, set the image features of the target pixel to come from the first high dynamic range image respectively. Any one of the weights of an exposure image, the second exposure image and the third exposure image is a set value;

The image features of the target pixel point are calculated according to the following formula from the weights of the first exposure image, the second exposure image and the third exposure image:

A*X+B*Y+C*Z＝P

X+Y+Z=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and Z indicates that the target pixel The image feature of the point comes from the weight of the third exposure image, A represents the image feature of the corresponding pixel point of the target pixel point in the first exposure image, and B represents the image feature of the target pixel point in the second exposure image. The image feature of the corresponding pixel in the exposure image, C represents the image feature of the corresponding pixel of the target pixel in the third exposure image, and P represents the image feature of the target pixel in the first high dynamic range image The image features of the corresponding pixels in .

In the above formula, A, B, C and P are all known values, but since there are three unknown quantities X, Y and Z, it is necessary to introduce additional restrictive conditions to calculate the values of X, Y and Z value. Specifically, according to the weights that the image features of the corresponding pixel points of the target pixel points in the first high dynamic range image come from the respective first images, the image features of the target pixel points come from the first exposure image, the second exposure image, and the second exposure image respectively. Any one of the weights of the exposure image and the third exposure image is a set value.

In an embodiment of the present application, acquiring at least two frames of first images with different exposures captured by the first camera may include:

Obtaining more than one frame of images with an exposure greater than a first reference exposure by shooting with the first camera, where the first reference exposure is the exposure corresponding to the preview stream of the first camera;

Obtaining more than one frame of images with an exposure less than the first reference exposure by shooting with the first camera;

Determining the images with the exposure of more than one frame greater than the first reference exposure and the images with the exposure of more than one frame less than the first reference exposure as the first images of the at least two frames with different exposures ;

Acquiring at least two frames of second images with different exposures captured by the second camera may include:

Obtaining more than one frame of images with an exposure greater than a second reference exposure through the second camera, where the second reference exposure is the exposure corresponding to the preview stream of the second camera;

Obtaining more than one frame of images with an exposure less than the second reference exposure by shooting with the second camera;

Determining the image whose exposure amount is greater than the second reference exposure amount for more than one frame and the image whose exposure amount is smaller than the second reference exposure amount for more than one frame as the second image of the at least two frames with different exposure amounts .

When the camera captures images, it will obtain the corresponding preview stream, that is, the data stream corresponding to the photo preview interface displayed on the display interface of the electronic device after the user turns on the camera. The exposure corresponding to the preview stream is generally the default value set by the camera. You can use the exposure amount corresponding to the preview stream as a benchmark, increase the exposure amount by a certain percentage on this benchmark, and obtain an image with more than one frame of high exposure, and reduce the exposure amount by a certain percentage on this benchmark, and obtain a frame by shooting Image above with less exposure.

In a second aspect, the embodiment of the present application provides an image fusion device, including:

An image acquisition module, configured to acquire at least two frames of first images with different exposures captured by the first camera, and acquire at least two frames of second images with different exposures captured by the second camera, wherein the first camera The field of view angle is greater than the field of view angle of the second camera, and the first image includes the second image;

A high dynamic range processing module, configured to fuse the at least two frames of first images with different exposures into a first high dynamic range image, and fuse the at least two frames of second images with different exposures into a second high dynamic range range image;

An image fusion module, configured to fuse the first high dynamic range image and the second high dynamic range image to obtain a fused image.

In one embodiment of the present application, the high dynamic range processing module may include:

A fusion weight calculation unit, configured to calculate, according to the image features of each pixel in the first high dynamic range image, that the image features of each pixel in the second high dynamic range image come from each of the first high dynamic range images. The weight of the second image;

A high dynamic range fusion unit, configured to combine the at least two frames of second images with different exposures according to the weights of the image features of each pixel in the second high dynamic range image from each of the second images fused into the second high dynamic range image.

In one embodiment of the present application, the fusion weight calculation unit may include:

The first fusion weight calculation subunit is used to calculate and obtain the first high dynamic range image according to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image. The image features of each pixel in a high dynamic range image respectively come from the weights of each of the first images;

The second fusion weight calculation subunit is used to calculate and obtain the weights in the second high dynamic range image according to the image features of each pixel in the first high dynamic range image respectively from the weights of the first images. The image features of each pixel point come from the weights of the second images respectively.

Further, the second fusion weight calculation subunit can be specifically configured to: for any target pixel in the second high dynamic range image, combine the target pixel in the first high dynamic range image The image features of the corresponding pixel points come from the weights of each of the first images, and the image features determined as the target pixel points come from the weights of each of the second images.

The third fusion weight calculation subunit is configured to, for any target pixel in the second high dynamic range image, according to the image features of the corresponding pixel of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the calculated image features of the target pixel are respectively derived from the weights of each of the second images, wherein the target pixel The image feature of the point is equal to the image feature of the corresponding pixel point of the target pixel point in the first high dynamic range image.

Further, the at least two frames of second images with different exposures include a first exposure image and a second exposure image, the exposure of the first exposure image is greater than the exposure of the second exposure image; the third The fusion weight calculation subunit can be specifically configured to: calculate the weights of the image features of the target pixel from the first exposure image and the second exposure image respectively according to the following formula:

A*X+B*Y=P

X+Y=1

Further, the at least two frames of second images with different exposures include a first exposure image, a second exposure image and a third exposure image, the exposure of the first exposure image is greater than the exposure of the second exposure image , the exposure amount of the second exposure image is greater than the exposure amount of the third exposure image; the third fusion weight calculation subunit may specifically include:

A weight setting subunit, configured to set the weight of the target pixel according to the weights of the image features of the corresponding pixels of the target pixel in the first high dynamic range image from each of the first images. Image features are respectively from any one of the weights of the first exposure image, the second exposure image and the third exposure image as a set value;

The formula calculation subunit is used to calculate the weights of the image features of the target pixel from the first exposure image, the second exposure image and the third exposure image respectively according to the following formula:

A*X+B*Y+C*Z＝P

X+Y+Z=1

In one embodiment of the present application, the image acquisition module may include:

The first image capturing unit is configured to capture more than one frame of images with an exposure greater than a first reference exposure through the first camera, and the first reference exposure is the exposure corresponding to the preview stream of the first camera ;

The second image capturing unit is configured to capture more than one frame of images with an exposure less than the first reference exposure through the first camera;

A first image determining unit, configured to determine the image with an exposure amount of more than one frame greater than the first reference exposure amount and the image with an exposure amount of more than one frame less than the first reference exposure amount as the at least two frame the first image with different exposures;

The third image capturing unit is configured to obtain more than one frame of images with an exposure greater than a second reference exposure through the second camera, and the second reference exposure is the exposure corresponding to the preview stream of the second camera ;

A fourth image capturing unit, configured to capture more than one frame of images with an exposure less than the second reference exposure through the second camera;

The second image determining unit is configured to determine the image whose exposure amount is greater than the second reference exposure amount for more than one frame and the image whose exposure amount is less than the second reference exposure amount for more than one frame as the at least two Frame a second image with a different exposure.

In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, , the electronic device implements the following image fusion method:

In one embodiment of the present application, the electronic device fuses the at least two frames of second images with different exposures into a second high dynamic range image, which may include:

In an embodiment of the present application, the electronic device calculates the image features of each pixel in the second high dynamic range image according to the image features of each pixel in the first high dynamic range image, respectively Weights from each of said second images may include:

Further, the electronic device calculates the weight of each pixel in the second high dynamic range image according to the image features of each pixel in the first high dynamic range image respectively from the weights of the first images. The image features of the points come from the weights of each of the second images, which may include:

Further, the at least two frames of second images with different exposures include a first exposure image and a second exposure image, and the exposure of the first exposure image is greater than the exposure of the second exposure image;

The electronic device calculates according to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images Obtaining the image features of the target pixel points respectively from the weights of each of the second images may include:

A*X+B*Y=P

X+Y=1

Further, the at least two frames of second images with different exposures include a first exposure image, a second exposure image and a third exposure image, the exposure of the first exposure image is greater than the exposure of the second exposure image , the exposure amount of the second exposure image is greater than the exposure amount of the third exposure image;

According to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the target is calculated and obtained. The image features of the pixels come from the weights of each of the second images, which may include:

A*X+B*Y+C*Z＝P

X+Y+Z=1

In an embodiment of the present application, the electronic device acquiring at least two frames of first images with different exposures captured by the first camera may include:

The electronic device acquires at least two frames of second images with different exposures captured by the second camera, which may include:

Determining the image whose exposure amount is greater than the second reference exposure amount for more than one frame and the image whose exposure amount is smaller than the second reference exposure amount for more than one frame as the second image of the at least two frames with different exposure amounts . In the fourth aspect, the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed, the image fusion as proposed in the first aspect of the embodiment of the present application is realized method.

In the fifth aspect, the embodiment of the present application provides a computer program product, which, when the computer program product runs on the electronic device, causes the electronic device to execute the image fusion method as proposed in the first aspect of the embodiment of the present application.

Description of drawings

FIG. 1 is a hardware structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 2 is a flow chart of an image fusion method provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of the shooting range of two cameras with different field of view angles adopted in the embodiment of the present application;

Fig. 4 is a schematic diagram of a large field of view image and a corresponding small field of view image provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of an operation principle of an image fusion method provided in an embodiment of the present application;

Fig. 6 is a schematic diagram of the effect of the long exposure frame of the main path, the short exposure frame of the main path and the high dynamic range image of the main path in Fig. 5;

Fig. 7 is a schematic diagram of the effect of the long exposure frame of the auxiliary road, the short exposure frame of the auxiliary road and the high dynamic range image of the auxiliary road in Fig. 5;

Fig. 8 is a schematic diagram of the effect of fusing the high dynamic range image of the main road in Fig. 6 and the high dynamic range image of the auxiliary road in Fig. 7;

FIG. 9 is a structural diagram of an image fusion device provided in an embodiment of the present application;

Fig. 10 is a schematic diagram of an electronic device provided by an embodiment of the present application.

detailed description

In the following description, for the purpose of illustration rather than limitation, specific details such as specific device structures and techniques are presented so as to thoroughly understand the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

The terms used in the following examples are for the purpose of describing particular examples only, and are not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also Expressions such as "one or more" are included unless the context clearly dictates otherwise. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and/or" describes the association relationship of associated objects, indicating that there may be three types of relationships; for example, A and/or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, wherein A and B may be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. In addition, the "plurality" mentioned in the embodiments of the present application should be interpreted as two or more.

An electronic device (such as a mobile phone) can usually be provided with multiple cameras with different viewing angles, such as a normal camera, a telephoto camera, and a wide-angle camera. In some specific photography occasions, multi-camera joint photography can be used to improve the quality of photos. Improve the clarity of the corresponding area of the image with a large field of view. In order to obtain more image details, people will also perform high-dynamic range (High-Dynamic Range, HDR) fusion processing on large-field-of-view images to obtain corresponding high-dynamic-range images, and then combine the high-dynamic-range images with Small field of view image fusion. However, in scenes such as backlighting, some areas of the small field of view image captured by the telephoto camera will be overexposed and the image details will be lost, which will affect the clarity of the fused image.

In view of the above problems, this application proposes an image fusion method, which performs high dynamic range fusion processing on the small field of view image, and then fuses with the high dynamic range image of the large field of view image to obtain the fused image. With this setting, partial image details lost due to overexposure of the small field of view image can be restored, thereby improving the clarity of the fused image. For specific implementations of the present application, please refer to the various embodiments described below.

The image fusion method proposed in this application can be applied to various electronic devices with at least two cameras with different field of view, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) equipment, notebook computer, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA), smart home equipment, etc. The specific type of equipment is not limited in any way.

Taking the electronic device as a mobile phone as an example, FIG. 1 shows a block diagram of a partial structure of the mobile phone provided by the embodiment of the present application. With reference to Fig. 1, mobile phone comprises: radio frequency (Radio Frequency, RF) circuit 101, memory 102, input unit 103, display unit 104, sensor 105, audio circuit 106, wireless fidelity (wireless fidelity, WiFi) module 107, processor 108 , power supply 109, common camera 110 and telephoto camera 111 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 1 does not constitute a limitation to the mobile phone, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

The following is a specific introduction to each component of the mobile phone in combination with Figure 1:

The RF circuit 101 can be used for sending and receiving information or receiving and sending signals during a call. In particular, after receiving the downlink information of the base station, it is processed by the processor 108; in addition, the designed uplink data is sent to the base station. Generally, an RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, and the like. In addition, the RF circuit 101 can also communicate with networks and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE)), email, Short Messaging Service (SMS), etc.

The memory 102 can be used to store software programs and modules, and the processor 108 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 102 . The memory 102 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating devices, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); Data created by the use of mobile phones (such as audio data, phonebook, etc.), etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

The input unit 103 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 103 may include a touch panel 1031 and other input devices 1032 . The touch panel 1031, also referred to as a touch screen, can collect touch operations of the user on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch panel 1031 or near the touch panel 1031). operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 1031 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the to the processor 108, and can receive and execute commands sent by the processor 108. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031 , the input unit 103 may also include other input devices 1032 . Specifically, other input devices 1032 may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.

The display unit 104 can be used to display information input by or provided to the user and various menus of the mobile phone. The display unit 104 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an organic light-emitting diode (Organic Light-Emitting Diode, OLED), or the like. Further, the touch panel 1031 may cover the display panel 1041, and when the touch panel 1031 detects a touch operation on or near it, it transmits to the processor 108 to determine the type of the touch event, and then the processor 108 according to the touch event The type provides a corresponding visual output on the display panel 1041 . Although in FIG. 1 , the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to form a mobile phone. Realize the input and output functions of the mobile phone.

The handset may also include at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and/or when the mobile phone is moved to the ear. or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is stationary, and can be used to identify the application of mobile phone posture (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. repeat.

The audio circuit 106, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 106 can transmit the electrical signal converted from the received audio data to the loudspeaker 1061, and the loudspeaker 1061 converts it into a sound signal output; After being received, it is converted into audio data, and then the audio data is processed by the output processor 108, and then sent to another mobile phone through the RF circuit 101, or the audio data is output to the memory 102 for further processing.

WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 107, which provides users with wireless broadband Internet access. Although FIG. 1 shows the WiFi module 107, it can be understood that it is not an essential component of the mobile phone, and can be completely omitted as required without changing the essence of the application.

The processor 108 is the control center of the mobile phone, and uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and/or modules stored in the memory 102, and calling data stored in the memory 102, execution Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole. Optionally, the processor 108 may include one or more processing units; preferably, the processor 108 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating devices, user interfaces and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 108 .

The mobile phone also includes a power supply 109 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 108 through a power management device, so that functions such as charging, discharging, and power consumption management can be realized through the power management device.

The mobile phone also includes at least two cameras with different viewing angles, for example, one of them is a normal camera 110, and the other is a telephoto camera 111. . Although not shown, the phone may also include other types of cameras such as infrared cameras, hyperspectral cameras, and wide-angle cameras. Optionally, the position of the camera on the mobile phone may be front or rear, which is not limited in this embodiment of the present application.

In addition, although not shown, the mobile phone may also include a bluetooth module, etc., which will not be repeated here.

Figure 2 shows a flow chart of an image fusion method provided by an embodiment of the present application, including:

201. The electronic device acquires at least two frames of first images with different exposures captured by the first camera, and acquires at least two frames of second images with different exposures captured by the second camera;

The electronic device has at least two cameras with different viewing angles, namely a first camera and a second camera, wherein the viewing angle of the first camera is larger than that of the second camera. The embodiment of the present application does not limit the specific types of the first camera and the second camera. For example, if the first camera is an ordinary camera, the second camera can be a telephoto camera; if the first camera is a wide-angle camera, the second camera can be It can be a normal camera or a telephoto camera, and so on. In order to achieve image fusion, the first camera and the second camera should be set on the same surface of the electronic device, and keep the same or similar shooting angles when taking images, so that the shooting range of the first camera covers the shooting range of the second camera , that is, the first image captured by the first camera includes the second image captured by the second camera. In this application document, the first image may be recorded as an image with a large viewing angle, and the second image may be recorded as an image with a small viewing angle.

The camera can control the exposure of the captured image by adjusting parameters such as aperture size and exposure time. In the embodiment of the present application, by adjusting the parameters of the first camera, at least two frames of the first image with different exposures can be obtained, and by adjusting the parameters of the second The parameters of the second camera are taken to obtain at least two frames of second images with different exposures.

In an implementation manner of the present application, acquiring at least two frames of first images with different exposures captured by the first camera may include:

(1) Obtaining more than one frame of images with an exposure greater than a first reference exposure by shooting with the first camera, where the first reference exposure is the exposure corresponding to the preview stream of the first camera;

(2) Obtaining more than one frame of images with an exposure amount less than the first reference exposure amount captured by the first camera;

(3) Determining the images with the exposure of more than one frame greater than the first reference exposure and the images with the exposure of more than one frame less than the first reference exposure as the at least two frames with different exposures first image.

When adjusting the exposure, the exposure corresponding to the preview stream of the first camera may be acquired as a reference value, that is, the first reference exposure. When the camera captures an image, it will obtain the corresponding preview stream, that is, the data stream corresponding to the photo preview interface of the electronic device after the user turns on the camera. The exposure corresponding to the preview stream is generally the default value set by the camera. Take the exposure corresponding to the preview stream as a benchmark, increase the exposure by a certain percentage on this benchmark, and shoot more than one frame with a larger exposure (which can be called a long exposure frame), and reduce the exposure by a certain percentage on this benchmark Exposure, more than one frame of images with less exposure (may be referred to as short exposure frames) is obtained by shooting. For example, assuming that the exposure corresponding to the preview stream of the first camera is M, you can obtain a long exposure frame with an exposure of 4M and a short exposure frame with an exposure of 1/4M by adjusting the camera parameters. The long-exposure frame and the short-exposure frame are two acquired first images with different exposures.

(1) Obtaining more than one frame of images with an exposure greater than a second reference exposure through the second camera, where the second reference exposure is the exposure corresponding to the preview stream of the second camera;

(2) Obtaining more than one frame of images with an exposure amount less than the second reference exposure amount captured by the second camera;

(3) Determining the images with the exposure of more than one frame greater than the second reference exposure and the images with the exposure of more than one frame less than the second reference exposure as the at least two frames with different exposures second image.

The same method as that of the first camera may be used to acquire at least two frames of second images with different exposures captured by the second camera. It should be noted that the number of the first image and the number of the second image may be the same or different; there is no corresponding size-limited relationship between the exposure amount of each frame of the first image and the exposure amount of each frame of the second image.

202. The electronic device fuses the at least two frames of first images with different exposures into a first high dynamic range image, and fuses the at least two frames of second images with different exposures into a second high dynamic range image;

Next, perform high dynamic range fusion processing on each frame of the first image to obtain a first high dynamic range image; and perform high dynamic range fusion processing on each frame of second image to obtain a second high dynamic range image. high dynamic range

(High-Dynamic Range, HDR) fusion processing refers to synthesizing a high dynamic range image based on multiple frames of low dynamic range images with different exposures, so as to obtain more image details and improve image clarity. Among them, the low dynamic range image with a large exposure is mainly used to restore the image details of the dark area of the scene, and the low dynamic range image with a small exposure is mainly used to restore the image details of the bright area of the scene.

The following is a brief introduction to the process of HDR fusion processing of images: First, obtain multiple frames of low dynamic range images with different exposures (images for the same scene at the same time), assuming a total of 3 long-exposure frames, medium-exposure frames and short-exposure frames Frame low dynamic range image, in which the exposure of long exposure frame > the exposure of medium exposure frame > the exposure of short exposure frame, the long exposure frame, medium exposure frame and short exposure frame have performed image registration and pixel alignment processing . Then, obtain the fusion weight map corresponding to the long exposure frame, the medium exposure frame and the short exposure frame respectively (the fusion weight map can be manually set according to the empirical value, including the fusion weight value corresponding to each pixel). Finally, multiply the long-exposure frame and its corresponding fusion weight map, multiply the medium-exposure frame and its corresponding fusion weight map, and multiply the short-exposure frame and its corresponding fusion weight map, and then multiply the three The results are superimposed to obtain the corresponding high dynamic range image. In addition, if the image is in the RGB domain, the above-mentioned HDR fusion processing is performed on the R, G, and B channels respectively; if the image is in the YUV domain, the above-mentioned HDR fusion processing is performed on the Y (brightness parameter) channel , the UV (chromaticity parameter) channel directly multiplexes the UV value of the exposure frame.

On the other hand, when the first high dynamic range image and the second high dynamic range image are fused, if there is a large difference in image features such as brightness of the two images, the obtained fused image will also have the same characteristics as the first high dynamic range image. The image features of the high dynamic range images differ greatly, and in conventional image fusion scenarios, it is generally expected that the obtained fused image has the same or similar image features as the first high dynamic range image. In order to solve this problem, the process of fusing the second images of each frame can be guided by the HDR effect according to the image characteristics such as the brightness of the first high dynamic range image, so that the obtained second high dynamic range image has the same quality as the first high dynamic range image. Image characteristics such as the same or similar brightness of the image. For a specific implementation manner of performing HDR effect guidance according to the first high dynamic range image, reference may be made to the description below.

In an implementation manner of the present application, fusing the at least two frames of second images with different exposures into a second high dynamic range image may include:

(1) According to the image features of each pixel in the first high dynamic range image, calculate the image features of each pixel in the second high dynamic range image from the weights of the second images respectively ;

(2) According to the weights of the image features of each pixel in the second high dynamic range image from the respective second images, fusing the at least two frames of second images with different exposures into the first Two high dynamic range images.

For the above step (1), after the first high dynamic range image is generated, the image characteristics (such as RGB value, brightness value, etc.) of each pixel are known, so it can be based on the image of each pixel The features are respectively calculated to obtain the image features of each pixel in the second high dynamic range image from the weights of each second image, that is, to calculate the weight of each second image for fusion. The basic principle is to make the second image obtained after fusion The high dynamic range image has the same or similar image features as the first high dynamic range image.

For the above step (2), after obtaining the image features of each pixel in the second high dynamic range image from the weights of each second image, the HDR fusion process can be performed on each second image to obtain the corresponding first Two high dynamic range images. Assuming that the second image contains I ₁ and I ₂ , the image feature of a pixel point Q in the second high dynamic range image to be generated comes from the weight of I ₁ is X, and the weight from I ₂ is Y, the pixel point The image feature of Q corresponding to the pixel point in _I1 is A, and the image feature of the corresponding pixel point in _I2 is B, then when _I1 and _I2 are fused into the second high dynamic range image, Q is at the The image feature in the second high dynamic range image is A*X+B*Y, and by analogy, the image feature of each pixel in the second high dynamic range image can be calculated.

In one implementation of the present application, according to the image features of each pixel in the first high dynamic range image, the image features of each pixel in the second high dynamic range image are calculated from the respective The weight of the second image may include:

(1) According to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, calculate and obtain each pixel in the first high dynamic range image The image features of pixels are respectively from the weights of each of the first images;

(2) According to the image features of each pixel in the first high dynamic range image respectively from the weights of the first images, calculate the image features of each pixel in the second high dynamic range image Weights from each of the second images respectively.

After obtaining the first high dynamic range image, according to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, the first image of each first image can be calculated The fusion weights, that is, the image features of each pixel in the first high dynamic range image respectively come from the weights of the first images. Specifically, it is assumed that each first image is a total of 3 frames of low dynamic range images including a long exposure frame, a medium exposure frame, and a short exposure frame, wherein the exposure amount of the long exposure frame>the exposure amount of the middle exposure frame>the exposure amount of the short exposure frame, Assuming that the image feature of a certain pixel point Q in the long exposure frame is A, the image feature in the medium exposure frame is B, and the image feature in the short exposure frame is C, then the first height can be obtained by using the following formulas respectively The image features of pixel Q in the dynamic range image come from the weight X of the long-exposure frame, the weight Y of the medium-exposure frame, and the weight Z of the short-exposure frame:

Wherein, σ can be set to 52 or other values based on experience, and the weights X, Y and Z calculated by using the above formula can be normalized so that X+Y+Z=1.

For each pixel in the first high dynamic range image, weights whose image features come from each first image can be calculated in the same manner as the above pixel Q.

In an implementation manner of the present application, according to the weights of the image features of each pixel in the first high dynamic range image from each of the first images, the weights in the second high dynamic range image are calculated and obtained. The image features of each pixel are respectively derived from the weights of the second images, which may include:

This is an implementation manner of weight reuse, that is, the fusion weight of the first high dynamic range image is reused in the fusion process of the second high dynamic range image. For example, assuming that each first image is a first long exposure frame, a first medium exposure frame and a first short exposure frame, and each second image is a second long exposure frame, a second medium exposure frame and a second short exposure frame, the first The image feature of a target pixel point Q in a high dynamic range image is P, the image feature of Q in the first long exposure frame is A, the image feature of Q in the first medium exposure frame is B, and the image feature of Q in the first short exposure frame is The image feature in the frame is C, the image feature in the second long exposure frame is D, the image feature in the second medium exposure frame is E, and the image feature in the second short exposure frame is F, then there is P =A*X+B*Y+C*Z, where X is the weight of the image feature of Q from the first long exposure frame, Y is the weight of the image feature of Q from the first middle exposure frame, and Z is the weight of Q Image features are derived from the weights of the first short-exposure frame. Then, X, Y, and Z can be directly used as the fusion weights of the second long exposure frame, the second medium exposure frame, and the second short exposure frame respectively, then the image feature S=D* of Q in the second high dynamic range image X+E*Y+F*Z. It should be noted that the image feature may be a feature of any channel in the RGB domain, or a feature of the Y channel (brightness) in the YUV domain. In the case that the image characteristics of each first image and each second image are similar, a second high dynamic range image having image characteristics similar to that of the first high dynamic range image can be obtained by using weight reuse.

In another implementation manner of the present application, according to the image features of each pixel in the first high dynamic range image, the image features of each pixel in the second high dynamic range image are calculated from The weights of each of the second images may include:

In some cases, the difference in image features between each first image and each second image is relatively large, in order to meet the requirement that the image features of the second high dynamic range image and the image features of the first high dynamic range image are the same or similar , it is inappropriate to use weight reuse to determine the fusion weight of each second image at this time. In view of this situation, image feature multiplexing can be adopted, that is, the fusion weight of each second image can be calculated by using the same or similar image feature of each corresponding pixel in the two high dynamic range images as a known condition.

Specifically, assuming that each second image includes a first exposure image and a second exposure image, wherein the exposure of the first exposure image is greater than the exposure of the second exposure image, the image of the target pixel can be calculated according to the following formula The features come from the weights of the first exposure image and the second exposure image respectively:

A*X+B*Y=P

X+Y=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and A indicates that the target pixel is in the first exposure image. The image feature of the corresponding pixel in the exposure image, B represents the image feature of the corresponding pixel of the target pixel in the second exposure image, and P represents the corresponding pixel of the target pixel in the first high dynamic range image The image feature of the point (or other values close to the image feature).

In the above formula, A, B, and P are all known values, so the weight X and weight Y can be calculated. For example, if the image feature of the corresponding pixel of the target pixel in the first high dynamic range image is 92, the image feature of the corresponding pixel of the target pixel in the first exposure image is 120, and the target pixel is 120 in the second exposure image. The image feature of the corresponding pixel in the image is 10, then the following formula can be obtained:

120*X+10*Y＝92

X+Y=1

By calculating this formula, the weight X=74.5% and the weight Y=25.5% can be obtained. Obviously, each pixel can be calculated in the same way to obtain the corresponding fusion weight. When the first exposure image and the second exposure image are fused, each pixel is fused according to its corresponding fusion weight to obtain the image A second high dynamic range image having the same characteristics as the first high dynamic range image.

In another implementation manner of the present application, it is assumed that each second image includes a first exposure image, a second exposure image, and a third exposure image, wherein the exposure amount of the first exposure image is greater than the exposure amount of the second exposure image, and the exposure amount of the second exposure image is The exposure amount of the second exposure image is greater than the exposure amount of the third exposure image; at this time, the image features of the target pixels can be calculated according to the following formula from the weights of the first exposure image, the second exposure image and the third exposure image respectively :

A*X+B*Y+C*Z＝P

X+Y+Z=1

Wherein, X represents that the image feature of the target pixel point comes from the weight of the first exposure image, Y represents that the image feature of the target pixel point comes from the weight of the second exposure image, and Z represents the image feature of the target pixel point Weights from the third exposure image, A represents the image feature of the corresponding pixel of the target pixel in the first exposure image, B represents the image feature of the corresponding pixel of the target pixel in the second exposure image , C represents the image feature of the corresponding pixel of the target pixel in the third exposure image, and P represents the image feature of the corresponding pixel of the target pixel in the first high dynamic range image.

For example, assuming that each first image includes a fourth exposure image, a fifth exposure image, and a sixth exposure image with sequentially decreasing exposure amounts, the image feature of the corresponding pixel of the target pixel in the fourth exposure image is 100 (fusion weight 50%), the image feature of the corresponding pixel of the target pixel in the fifth exposure image is 50 (fusion weight is 20%), and the image feature of the corresponding pixel of the target pixel in the fifth exposure image is 10 ( The fusion weight is 30%), then after fusion, the obtained image feature of the corresponding pixel in the first high dynamic range image is 100*50%+50*20%+10*30%=63. Assume again that the image feature of the corresponding pixel of the target pixel in the first exposure image is 120, the image feature of the corresponding pixel of the target pixel in the second exposure image is 60, and the image feature of the target pixel in the third exposure image The image feature corresponding to the pixel is 5, then according to the above formula, it can be obtained:

120*X+60*Y+5*Z＝63

X+Y+Z=1

At this time, any weight of X, Y, and Z can be set as a set value. For example, since the fusion weight corresponding to the target pixel point in the fourth exposure image is the largest (50%), it can be set that the target pixel point is in the first exposure image. The corresponding fusion weight X in the exposure image (the first exposure image corresponds to the fourth exposure image, both of which are images with the largest exposure) is 50%, then the values of the other two weights Y and Z can be calculated. It should be noted that the image features targeted by this calculation method can be the features of any channel in the RGB domain, or the features of the Y channel (brightness) in the YUV domain. As for the characteristics of the UV channel (chroma) in the YUV domain, the characteristics of the image whose exposure is centered can be directly used. For example, in the above example, the characteristics of the UV channel of the target pixel point can use the corresponding UV value in the second exposure image , as the UV value of the target pixel in the fused second high dynamic range image.

The above is a specific implementation manner of HDR effect guidance based on the first high dynamic range image. With this setting, the obtained second high dynamic range image can have the same or similar image characteristics as the first high dynamic range image, such as brightness, so as to meet the requirements of the specified image fusion scene.

203. The electronic device fuses the first high dynamic range image and the second high dynamic range image to obtain a fused image.

After the first high dynamic range image and the second high dynamic range image are obtained, the two high dynamic range images can be fused, specifically, the second high dynamic range image is fused to the first high dynamic range image and the first high dynamic range image. Regions corresponding to the two high dynamic range images, so as to obtain the fused image. The high dynamic range image fusion process mainly includes image registration and image feature superposition. For details, reference may be made to related content on image fusion in the prior art, which will not be repeated here.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

FIG. 3 is a schematic diagram of shooting ranges of two cameras with different field of view angles adopted in the embodiment of the present application. Wherein, the camera with a larger field of view is the first camera, and the camera with a smaller field of view is the second camera, and the first camera and the second camera are located on the same surface of an electronic device. The shooting ranges of the two cameras are shown respectively by two rectangular boxes in Fig. 3, obviously the shooting range of the first camera includes the shooting range of the second camera, so the image (first image) captured by the first camera includes the shooting range of the second camera The captured image (second image). In practical applications, the first camera can be a camera with a larger field of view such as a wide-angle camera or an ordinary camera, and the second camera can be a camera with a smaller field of view such as a telephoto camera. However, it should be understood that two different fields of view Any type of camera at any angle can be used as the first camera and the second camera in the embodiment of the present application.

Fig. 4 is a schematic diagram of an image with a large field of view and a corresponding image with a small field of view provided by an embodiment of the present application. In Figure 4, the image on the left is an image with a large field of view, that is, a first image captured by the first camera mentioned above; the image on the right is an image with a small field of view, that is, the image captured by the second camera mentioned above A second image is obtained. In the prior art, in order to improve the clarity of the distant scene area in the obtained image with a large field of view, the image with a small field of view is usually fused into the image with a large field of view. However, due to backlighting and other reasons, some areas of the image with a small field of view may be overexposed (such as the overexposed area marked in Figure 4), resulting in loss of image details, which will affect the clarity of the fused image.

In view of the above problems, the embodiment of the present application proposes an image fusion method, and a schematic diagram of its specific operation principle is shown in FIG. 5 . In Figure 5, after the user clicks to take a photo, the main road (that is, the shooting channel where the main camera is located, which can usually be an ordinary camera with a large field of view) will set different exposure parameters according to the preview stream of the main camera. The main road long exposure frame and the main road short exposure frame are obtained by shooting, and the exposure of the main road long exposure frame is greater than the exposure of the main road short exposure frame; then, perform HDR fusion on the main road long exposure frame and the main road short exposure frame Operation to obtain the high dynamic range image of the main road. Similar to the main road, the auxiliary road (that is, the shooting channel where the auxiliary camera is located, which can usually be a telephoto camera with a small field of view) will set different exposure parameters according to the preview stream of the auxiliary camera, and the long exposure of the auxiliary road will be obtained by shooting Frames and short-exposure frames of the auxiliary road, wherein the exposure of the long-exposure frame of the auxiliary road is greater than the exposure of the short-exposure frame of the auxiliary road; then, an HDR fusion operation is performed on the long-exposure frame of the auxiliary road and the short-exposure frame of the auxiliary road to obtain a high dynamic range image of the auxiliary road. Finally, the high dynamic range image of the main road and the high dynamic range image of the auxiliary road are fused to obtain the fused image.

Further, in the image fusion method shown in FIG. 5 , the HDR fusion process of the long-exposure frame of the auxiliary road and the short-exposure frame of the auxiliary road can be guided according to the image characteristics of the high dynamic range image of the main road (as indicated by the dotted line in FIG. 5 ). shown), so that the obtained auxiliary road high dynamic range image has the same or similar brightness and other image characteristics as the main road high dynamic range image. That is to say, the main road can guide the auxiliary road with HDR effects, and the specific guidance method can refer to the related content above.

In order to facilitate the description of the processing effect of the image fusion method shown in Figure 5, the actual captured image is introduced, as shown in Figure 6, which is the long exposure frame of the main road, the short exposure frame of the main road and the high dynamic range image of the main road in Figure 5 Effect diagram. In Figure 6, the long exposure frame of the main road and the short exposure frame of the main road are images of different exposures for the same balcony scene, and the long exposure frame of the main road is an image with a larger exposure, which shows that the overall brightness of the image is brighter; The short-exposure frame of the main road is an image with a small exposure, and it can be seen that the overall brightness of the image is relatively dark. After HDR fusion of the long-exposure frame of the main road and the short-exposure frame of the main road, the high dynamic range image of the main road is obtained. It can be seen that the brightness of the image is moderate, and the long-exposure frame of the main road and/or the short-exposure frame of the main road are restored to a certain extent missing image details.

As shown in Figure 7, it is a schematic diagram of the effect of the long exposure frame of the auxiliary road, the short exposure frame of the auxiliary road and the high dynamic range image of the auxiliary road in Figure 5. In Figure 7, the long exposure frame of the auxiliary road and the short exposure frame of the auxiliary road are images of different exposures for the same balcony scene (the same scene as in Figure 6), and the long exposure frame of the auxiliary road is an image with a larger exposure, which can be seen The overall brightness is brighter; the short-exposure frame of the auxiliary path is an image with a small exposure, so the overall brightness of the image is relatively dark. Since the field of view of the camera on the auxiliary road is smaller than that of the camera on the main road, the long exposure frame of the main road shown in Figure 6 contains the long exposure frame of the auxiliary road shown in Figure 7, and the short exposure frame of the main road shown in Figure 6 contains 7 shows the short-exposure frame of the auxiliary path. After HDR fusion of the long exposure frame and the short exposure frame of the auxiliary road, the high dynamic range image of the auxiliary road is obtained. It can be seen that the brightness of the image is moderate, and the image details lost in the long exposure frame of the auxiliary road and/or the short exposure frame of the auxiliary road are restored to a certain extent . In addition, through the HDR effect guidance of the main road to the auxiliary road, the obtained high dynamic range image of the auxiliary road in Figure 7 has image characteristics such as brightness that are the same or similar to the high dynamic range image of the main road in Figure 6 .

As shown in FIG. 8 , it is a schematic diagram of the effect of fusing the high dynamic range image of the main road in FIG. 6 and the high dynamic range image of the auxiliary road in FIG. 7 . Since the high dynamic range image of the main road and the high dynamic range image of the auxiliary road have the same or similar image characteristics, the influence of image fusion on image characteristics such as brightness and color can be reduced, so that the obtained fused image has the same or similar image characteristics as the main road image image features. The right side of Fig. 8 is a schematic diagram of the fused image, in which the area within the dotted frame is the target area of image fusion, and it can be seen that the target area has obtained a certain degree of sharpness improvement effect.

Corresponding to the image fusion method described in the above embodiments, FIG. 9 shows a structural block diagram of an image fusion device provided in the embodiment of the present application.

Referring to Figure 9, the device includes:

An image acquisition module 901, configured to acquire at least two frames of first images with different exposures captured by the first camera, and acquire at least two frames of second images with different exposures captured by the second camera, wherein the first The angle of view of the camera is larger than the angle of view of the second camera, and the first image includes the second image;

A high dynamic range processing module 902, configured to fuse the at least two frames of first images with different exposures into a first high dynamic range image, and fuse the at least two frames of second images with different exposures into a second high dynamic range image dynamic range images;

The image fusion module 903 is configured to fuse the first high dynamic range image and the second high dynamic range image to obtain a fused image.

A*X+B*Y=P

X+Y=1

A*X+B*Y+C*Z＝P

X+Y+Z=1

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, various image fusion methods as proposed in the present application are implemented.

The embodiment of the present application also provides a computer program product, which, when the computer program product runs on the electronic device, causes the electronic device to execute each image fusion method proposed in the present application.

Fig. 10 is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 10 , the electronic device 100 of this embodiment includes: at least one processor 1000 (only one is shown in FIG. 10 ), a processor, a memory 1001, and stored in the memory 1001 and can be processed in the at least one processor. A computer program 1002 running on the processor 1000, when the processor 1000 executes the computer program 1002, implements the steps in any of the above embodiments of the image fusion method.

The electronic device may include, but not limited to, a processor 1000 and a memory 1001 . Those skilled in the art can understand that FIG. 10 is only an example of the electronic device 100, and does not constitute a limitation to the electronic device 100. It may include more or less components than shown in the figure, or combine certain components, or different components. , for example, may also include input and output devices, network access devices, and so on.

The so-called processor 1000 may be a central processing unit (Central Processing Unit, CPU), and the processor 1000 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory 1001 may be an internal storage unit of the electronic device 100 in some embodiments, such as a hard disk or a memory of the electronic device 100. The memory 1001 may also be an external storage device of the electronic device 100 in other embodiments, such as a plug-in hard disk equipped on the electronic device 100, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Further, the memory 1001 may also include both an internal storage unit of the electronic device 100 and an external storage device. The memory 1001 is used to store operating devices, application programs, bootloader programs (BootLoader), data, and other programs, such as program codes of the computer programs. The memory 1001 can also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above device, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts that are not detailed or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in computer-readable storage media. When executed by a processor, the steps in the foregoing method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program codes to electronic equipment, recording media, computer memory, read-only memory (ROM, Read-Only Memory), random-access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. Such as U disk, mobile hard disk, magnetic disk or CD, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.

The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still implement the foregoing embodiments Modifications to the technical solutions described in the examples, or equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application, and should be included in the Within the protection scope of this application.

Claims

An image fusion method, characterized in that, comprising:

Acquiring at least two frames of first images with different exposures captured by the first camera, and acquiring at least two frames of second images with different exposures captured by the second camera, wherein the angle of view of the first camera is larger than the The angle of view of the second camera, the first image includes the second image;

fusing the at least two frames of first images with different exposures into a first high dynamic range image, and fusing the at least two frames of second images with different exposures into a second high dynamic range image;

The first high dynamic range image and the second high dynamic range image are fused to obtain a fused image.
The image fusion method according to claim 1, wherein said at least two frames of second images with different exposures are fused into a second high dynamic range image, comprising:

According to the image features of each pixel in the first high dynamic range image, calculate the image features of each pixel in the second high dynamic range image from the weights of the second images;

According to the weights of the image features of each pixel in the second high dynamic range image from the respective second images, fusing the at least two frames of second images with different exposures into the second high dynamic range image range image.
The image fusion method according to claim 2, wherein the image of each pixel in the second high dynamic range image is calculated according to the image features of each pixel in the first high dynamic range image The features come from the weights of each of the second images, including:

According to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, calculate and obtain each pixel in the first high dynamic range image The image features of each come from the weights of each of the first images;

According to the weights that the image features of each pixel in the first high dynamic range image come from each of the first images, it is calculated that the image features of each pixel in the second high dynamic range image come from weights for each of the second images.
The image fusion method according to claim 3, characterized in that, according to the weights of the image features of each pixel in the first high dynamic range image from each of the first images, the second The image features of each pixel in the high dynamic range image are respectively derived from the weights of the second images, including:

For any target pixel in the second high dynamic range image, the image features of the corresponding pixel of the target pixel in the first high dynamic range image are respectively obtained from each of the first images The weights are determined as the weights of the image features of the target pixel points from the respective second images.
The image fusion method according to claim 2, wherein the image of each pixel in the second high dynamic range image is calculated according to the image features of each pixel in the first high dynamic range image The features come from the weights of each of the second images, including:

For any target pixel in the second high dynamic range image, according to the image features of the corresponding pixel of the target pixel in the first high dynamic range image and the position of the target pixel in each of the The image features of the corresponding pixels in the second image are calculated to obtain the image features of the target pixels from the weights of each of the second images, wherein the image features of the target pixels and the target pixels Image features of corresponding pixel points in the first high dynamic range image are equal.
The image fusion method according to claim 5, wherein the at least two frames of second images with different exposures include a first exposure image and a second exposure image, and the exposure of the first exposure image is greater than the the exposure amount of the second exposure image;

According to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the target is calculated and obtained. The image features of the pixels are respectively derived from the weights of each of the second images, including:

The image features of the target pixel points are calculated according to the following formula from the weights of the first exposure image and the second exposure image:

A*X+B*Y=P

X+Y=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and A indicates that the target pixel The image feature of the corresponding pixel point in the first exposure image, B represents the image feature of the corresponding pixel point of the target pixel point in the second exposure image, and P represents the image feature of the target pixel point in the Image features of corresponding pixels in the first high dynamic range image.
The image fusion method according to claim 5, wherein the at least two frames of second images with different exposures include a first exposure image, a second exposure image and a third exposure image, and the first exposure image The exposure amount is greater than the exposure amount of the second exposure image, and the exposure amount of the second exposure image is greater than the exposure amount of the third exposure image;

According to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the target is calculated and obtained. The image features of the pixels are respectively derived from the weights of each of the second images, including:

According to the weights of the image features of the corresponding pixels of the target pixel in the first high dynamic range image respectively from the first images, set the image features of the target pixel to come from the first high dynamic range image respectively. Any one of the weights of an exposure image, the second exposure image and the third exposure image is a set value;

The image features of the target pixel point are calculated according to the following formula from the weights of the first exposure image, the second exposure image and the third exposure image:

A*X+B*Y+C*Z＝P

X+Y+Z=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and Z indicates that the target pixel The image feature of the point comes from the weight of the third exposure image, A represents the image feature of the corresponding pixel point of the target pixel point in the first exposure image, and B represents the image feature of the target pixel point in the second exposure image. The image feature of the corresponding pixel in the exposure image, C represents the image feature of the corresponding pixel of the target pixel in the third exposure image, and P represents the image feature of the target pixel in the first high dynamic range image The image features of the corresponding pixels in .
The image fusion method according to any one of claims 1 to 7, wherein obtaining at least two first images of different exposures captured by the first camera comprises:

Obtaining more than one frame of images with an exposure greater than a first reference exposure by shooting with the first camera, where the first reference exposure is the exposure corresponding to the preview stream of the first camera;

Obtaining more than one frame of images with an exposure less than the first reference exposure by shooting with the first camera;

Determining the images with the exposure of more than one frame greater than the first reference exposure and the images with the exposure of more than one frame less than the first reference exposure as the first images of the at least two frames with different exposures ;

Obtaining at least two frames of second images with different exposures captured by the second camera, including:

Obtaining more than one frame of images with an exposure greater than a second reference exposure through the second camera, where the second reference exposure is the exposure corresponding to the preview stream of the second camera;

Obtaining more than one frame of images with an exposure less than the second reference exposure by shooting with the second camera;

Determining the image whose exposure amount is greater than the second reference exposure amount for more than one frame and the image whose exposure amount is smaller than the second reference exposure amount for more than one frame as the second image of the at least two frames with different exposure amounts .
An electronic device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein when the processor executes the computer program, the electronic device realizes The following image fusion method:

Acquiring at least two frames of first images with different exposures captured by the first camera, and acquiring at least two frames of second images with different exposures captured by the second camera, wherein the angle of view of the first camera is larger than the The angle of view of the second camera, the first image includes the second image;

fusing the at least two frames of first images with different exposures into a first high dynamic range image, and fusing the at least two frames of second images with different exposures into a second high dynamic range image;

The first high dynamic range image and the second high dynamic range image are fused to obtain a fused image.
The electronic device according to claim 9, wherein merging the at least two frames of second images with different exposures into a second high dynamic range image comprises:

According to the image features of each pixel in the first high dynamic range image, calculate the image features of each pixel in the second high dynamic range image from the weights of the second images;

According to the weights of the image features of each pixel in the second high dynamic range image from the respective second images, fusing the at least two frames of second images with different exposures into the second high dynamic range image range image.
The electronic device according to claim 10, wherein the image feature of each pixel in the second high dynamic range image is calculated according to the image feature of each pixel in the first high dynamic range image The weights from each of the second images respectively include:

According to the image features of each pixel in the first high dynamic range image and the image high dynamic range fusion algorithm corresponding to the first high dynamic range image, calculate and obtain each pixel in the first high dynamic range image The image features of each come from the weights of each of the first images;

According to the weights that the image features of each pixel in the first high dynamic range image come from each of the first images, it is calculated that the image features of each pixel in the second high dynamic range image come from weights for each of the second images.
The electronic device according to claim 11, wherein the second high dynamic range image is calculated according to the weights of the image features of each pixel in the first high dynamic range image from each of the first images. The image features of each pixel in the dynamic range image are respectively derived from the weights of the second images, including:

For any target pixel in the second high dynamic range image, the image features of the corresponding pixel of the target pixel in the first high dynamic range image are respectively obtained from each of the first images The weights are determined as the weights of the image features of the target pixel points from the respective second images.
The electronic device according to claim 10, wherein the image feature of each pixel in the second high dynamic range image is calculated according to the image feature of each pixel in the first high dynamic range image The weights from each of the second images respectively include:

For any target pixel in the second high dynamic range image, according to the image features of the corresponding pixel of the target pixel in the first high dynamic range image and the position of the target pixel in each of the The image features of the corresponding pixels in the second image are calculated to obtain the image features of the target pixels from the weights of each of the second images, wherein the image features of the target pixels and the target pixels Image features of corresponding pixel points in the first high dynamic range image are equal.
The electronic device according to claim 13, wherein the at least two frames of second images with different exposures include a first exposure image and a second exposure image, and the exposure of the first exposure image is greater than that of the first exposure 2. The exposure amount of the exposure image;

According to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the target is calculated and obtained. The image features of the pixels are respectively derived from the weights of each of the second images, including:

The image features of the target pixel points are calculated according to the following formula from the weights of the first exposure image and the second exposure image:

A*X+B*Y=P

X+Y=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and A indicates that the target pixel The image feature of the corresponding pixel point in the first exposure image, B represents the image feature of the corresponding pixel point of the target pixel point in the second exposure image, and P represents the image feature of the target pixel point in the Image features of corresponding pixels in the first high dynamic range image.
The electronic device according to claim 13, wherein the at least two frames of second images with different exposures include a first exposure image, a second exposure image and a third exposure image, and the exposure of the first exposure image The exposure amount is greater than the exposure amount of the second exposure image, and the exposure amount of the second exposure image is greater than the exposure amount of the third exposure image;

According to the image features of the corresponding pixels of the target pixel in the first high dynamic range image and the image features of the corresponding pixels of the target pixel in each of the second images, the target is calculated and obtained. The image features of the pixels are respectively derived from the weights of each of the second images, including:

According to the weights of the image features of the corresponding pixels of the target pixel in the first high dynamic range image respectively from the first images, set the image features of the target pixel to come from the first high dynamic range image respectively. Any one of the weights of an exposure image, the second exposure image and the third exposure image is a set value;

According to the following formula, the image features of the target pixels are calculated from the weights of the first exposure image, the second exposure image and the third exposure image:

A*X+B*Y+C*Z＝P

X+Y+Z=1

Wherein, X indicates that the image feature of the target pixel comes from the weight of the first exposure image, Y indicates that the image feature of the target pixel comes from the weight of the second exposure image, and Z indicates that the target pixel The image feature of the point comes from the weight of the third exposure image, A represents the image feature of the corresponding pixel point of the target pixel point in the first exposure image, and B represents the image feature of the target pixel point in the second exposure image. The image feature of the corresponding pixel in the exposure image, C represents the image feature of the corresponding pixel of the target pixel in the third exposure image, and P represents the image feature of the target pixel in the first high dynamic range image The image features of the corresponding pixels in .
The electronic device according to any one of claims 9 to 15, wherein acquiring at least two frames of first images with different exposures captured by the first camera comprises:

Obtaining more than one frame of images with an exposure greater than a first reference exposure by shooting with the first camera, where the first reference exposure is the exposure corresponding to the preview stream of the first camera;

Obtaining more than one frame of images with an exposure less than the first reference exposure by shooting with the first camera;

Determining the images with the exposure of more than one frame greater than the first reference exposure and the images with the exposure of more than one frame less than the first reference exposure as the first images of the at least two frames with different exposures ;

Obtaining at least two frames of second images with different exposures captured by the second camera, including:

Obtaining more than one frame of images with an exposure greater than a second reference exposure through the second camera, where the second reference exposure is the exposure corresponding to the preview stream of the second camera;

Obtaining more than one frame of images with an exposure less than the second reference exposure by shooting with the second camera;

Determining the image whose exposure amount is greater than the second reference exposure amount for more than one frame and the image whose exposure amount is smaller than the second reference exposure amount for more than one frame as the second image of the at least two frames with different exposure amounts .
A computer-readable storage medium, the computer-readable storage medium stores a computer program, wherein when the computer program is executed, the image fusion method according to any one of claims 1 to 8 is realized.
A computer program product, characterized in that, when the computer program product is run on an electronic device, the electronic device is made to execute the image fusion method according to any one of claims 1 to 8.