CN113810590A

CN113810590A - Image processing method, electronic device, medium, and system

Info

Publication number: CN113810590A
Application number: CN202010535245.XA
Authority: CN
Inventors: 卢恩涛; 王军
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-06-12
Filing date: 2020-06-12
Publication date: 2021-12-17

Abstract

The application relates to an image processing method, an electronic device, a medium, and a system. The method comprises the following steps: the electronic equipment starts a first shooting mode; the electronic equipment collects multiple frames of first images to be processed through a first camera and multiple frames of second images to be processed through a second camera, wherein at least partial images of the multiple frames of first images to be processed collected by the first camera and the multiple frames of second images to be processed collected by the second camera are collected synchronously, and the exposure time is the same; the electronic equipment fuses a plurality of frames of first images to be processed and a plurality of frames of second images to be processed and outputs the fused images. The image processing method and the image processing device can be applied to the technical field of image processing, multi-frame images collected through the first camera and multi-frame images collected through the second camera are fused, the fused images of the shooting main body and the background are output, the image shooting quality is improved, and the user experience is improved.

Description

Image processing method, electronic device, medium, and system

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image processing method, an electronic device, a medium, and a system.

Background

In the technique of taking a picture by using an image taking apparatus, Depth of Field (DOF) directly determines the sharpness of one picture. Depth of field refers to the range of distance between the front and back of a subject measured at the front edge of a camera lens or other imager where a sharp image can be obtained. Depth of field is related to object distance, focal length, aperture size, etc. The smaller the object distance, the smaller the depth of field, so especially in close-range shooting, such as mobile phone front self-shooting, front multi-person group photo, and other scenes, the depth of the scene is often shallow because of the small object distance, which finally causes the problem that not only the background which is supposed to be embodied in the photo becomes blurred, but also the image of the person located slightly behind the lens is blurred.

In order to obtain a clear shooting effect, the size of the circle of confusion can be controlled to be always in a range which cannot be sensed by human eyes by controlling the distance between the image plane of the fixed-focus lens and the lens, so that the clear shooting effect is obtained. However, due to the adoption of the fixed focus lens, the zoom function cannot be realized, and when the motor automatic focusing cannot be realized, for different shooting scenes, the definition of a shot object needs to be adjusted, and the adjustment is realized only through the movement of a photographer, which causes inconvenience in operation of the photographer.

In another scheme for obtaining a clear shooting effect, an automatic focusing lens is used on a shooting device, the motor drives the automatic focusing lens to focus on a shot main body, so that the shot main body obtains the best definition, then the motor drives the automatic focusing lens to collect images with different focal lengths, and finally the images collected with different focusing points are fused to achieve the effect of expanding the depth of field. For an application scene such as a self-timer of a mobile phone, on one hand, the cost of the device is high because an automatic focusing lens is used for a front camera of the mobile phone. In addition, by adopting the method, a plurality of images with different focal lengths need to be acquired in the shooting process, the time consumption is long, the time for holding the equipment by a user and keeping a certain posture is long, the hand shake is easily caused, and the shooting quality is influenced. Even if a tripod is used, if the subject moves locally due to a long photographing time, a phenomenon (also referred to as a ghost phenomenon) in which pixels are superimposed or misaligned due to poor registration may occur.

Disclosure of Invention

The embodiment of the application provides an image processing method, electronic equipment, medium and system.

The technical scheme of this application passes through terminal equipment's main camera and gathers multiframe image to gather multiframe image through wide-angle camera, wherein, at least part of picture in the multiframe image that main camera gathered is synchronous with the multiframe image that wide-angle camera gathered and both exposure times are the same. Electronic equipment gathers multiframe image and the multiframe image that wide-angle camera gathered to main camera and fuses, and the shooting main part after the output fuses and the all comparatively clear image of background promote the image and shoot the quality, promote user experience.

In a first aspect, an embodiment of the present application provides an image processing method, which is used for an electronic device, where the electronic device includes a first camera and a second camera, and the method includes:

the electronic equipment starts a first shooting mode; the electronic equipment collects multiple frames of first images to be processed through a first camera and multiple frames of second images to be processed through a second camera, wherein at least partial images of the multiple frames of first images to be processed collected by the first camera and the multiple frames of second images to be processed collected by the second camera are collected synchronously, and the exposure time is the same; the electronic equipment fuses a plurality of frames of first images to be processed and a plurality of frames of second images to be processed and outputs the fused images.

In a possible implementation of the first aspect, the method further includes: the first camera comprises a main camera for automatic focusing, and the second camera comprises a wide-angle camera for fixed focusing.

In some embodiments, first camera can be the leading main camera of auto focus, the second camera can be the leading wide-angle camera of fixed focus, fuse the multiframe image that leading main camera gathered and the multiframe image that leading wide-angle camera gathered, with guarantee by the definition of shooing the main part through leading main camera, it is little to utilize the focus of leading wide-angle camera simultaneously, the characteristics that the depth of field is big, promote the definition of shooing the background, finally obtain the equal clear image of shooing main part and background.

In other embodiments, first camera can be the rearmounted main camera of auto focus, the second camera can be the rearmounted wide angle camera of fixed focus, fuse the multiframe image that rearmounted main camera gathered and the multiframe image that rearmounted wide angle camera gathered, with guarantee the definition of being shot the main part through rearmounted main camera, it is little to utilize the focus of rearmounted wide angle camera simultaneously, the characteristics that the depth of field is big, promote the definition of shooting the background, finally obtain the equal clear image in shooting main part and background.

In a possible implementation of the first aspect, the method further includes: the electronic equipment acquires shooting scene information; the electronic equipment determines an image processing mode according to the acquired shooting scene information; and the electronic equipment fuses a plurality of frames of first images to be processed and a plurality of frames of second images to be processed according to the determined image processing mode and outputs the fused images.

In a possible implementation of the first aspect, the method further includes: the electronic equipment determines the image processing mode according to the shooting scene information, and comprises the following steps:

the electronic equipment receives an image preview operation acted on the electronic equipment by a user;

the electronic equipment responds to image preview operation and obtains a preview image;

the electronic equipment determines that the shooting scene is a first scene when determining that the proportion of the overexposed area in the preview image is smaller than a first proportion threshold value and the proportion of the underexposed area is smaller than a second proportion threshold value;

the electronic device determines that the image processing mode is the first processing mode when determining that the shooting scene is the first scene.

That is, the technical solution of the present application can distinguish different shooting scenes, for example, for shooting scenes with different dynamic ranges, thereby adopting image processing modes corresponding to the shooting scenes.

In a possible implementation of the first aspect, the method further includes: the electronic equipment fuses a plurality of frames of first images to be processed and a plurality of frames of second images to be processed according to the determined image processing mode, and outputting the fused images comprises the following steps:

the method comprises the steps that under the condition that an image processing mode is determined to be a first processing mode, electronic equipment fuses multiple frames of first images to be processed collected by a first camera to obtain a first fused image; fusing a plurality of frames of second images to be processed, which are acquired by a second camera, to obtain a second fused image; the multi-frame first to-be-processed image acquired by the first camera comprises at least two frames of normal exposure images, and the multi-frame second to-be-processed image acquired by the second camera comprises at least two frames of normal exposure images;

the electronic equipment expands the edge of the first fused image based on the field angle of the second camera to obtain a first amplified image, and the resolution of the first amplified image is the same as that of the second fused image;

the electronic equipment extracts the feature points of the first amplified image and the second fused image, and performs feature point matching to obtain feature point pairs;

the electronic equipment obtains a homography matrix of the first amplified image and the second fused image based on the characteristic point pairs;

the electronic equipment obtains a mapping image corresponding to the second fusion image based on the second fusion image and the homography matrix;

the electronic equipment registers the mapping image of the second fusion image and the first amplified image to obtain a remapping image corresponding to the second fusion image;

and the electronic equipment converts the remapped image of the second fused image and the first amplified image into a YUV format, and fuses the remapped image of the second fused image after the format conversion and the first amplified image to obtain a fused image.

For example, the first scene is a non-high dynamic scene, in the non-high dynamic scene, multiple frames of normal exposure images simultaneously acquired by a main camera and a wide-angle camera of a terminal device are respectively registered and fused, then the images registered and fused corresponding to the main camera (referred to as main shooting preprocessing images for short) and the images registered and fused corresponding to the wide-angle camera (referred to as wide-angle preprocessing images for short) are subjected to feature point extraction and pairing, a coordinate mapping relation (such as a homography matrix) of feature point pairs corresponding to the main shooting preprocessing images and the wide-angle preprocessing images is calculated, the wide-angle preprocessing images are mapped according to the mapping relation to obtain mapped wide-angle preprocessing images, the images and the main shooting preprocessing images are subjected to histogram brightness correction and are subjected to local registration by adopting an optical flow method, then the two images subjected to local registration are input into a pre-trained fusion network for fusion, so as to obtain clear images of the shooting subject and the background.

and when the electronic equipment determines that the proportion of the overexposed area in the preview image is greater than or equal to a first proportion threshold value or the proportion of the underexposed area in the preview image is greater than or equal to a second proportion threshold value, determining that the shooting scene is a second scene.

The electronic device determines that the image processing mode is the second processing mode when determining that the shooting scene is the second scene.

under the condition that the image processing mode is determined to be the second processing mode, the electronic equipment fuses a plurality of frames of first images to be processed, which are collected by the first camera, so as to obtain a third fused image; fusing a plurality of frames of second images to be processed, which are acquired by a second camera, to obtain a fourth fused image; the first camera is used for acquiring a plurality of frames of first images to be processed, wherein the plurality of frames of first images to be processed acquired by the first camera comprise at least one frame of short exposure image, at least one frame of long exposure image and at least two frames of normal exposure images; the multi-frame second image to be processed collected by the second camera comprises at least two frames of normal exposure images;

the electronic equipment expands the edge of the third fused image based on the field angle of the second camera to obtain a second amplified image, and the resolution of the second amplified image is the same as that of the fourth fused image;

the electronic equipment extracts the feature points of the second amplified image and the fourth fused image, and performs feature point matching to obtain feature point pairs;

the electronic equipment obtains a homography matrix of the second amplified image and the fourth fused image based on the characteristic point pairs;

the electronic equipment obtains a mapping image corresponding to the fourth fusion image based on the fourth fusion image and the homography matrix;

the electronic equipment registers the mapping image of the fourth fusion image and the second amplified image to obtain a remapping image corresponding to the fourth fusion image;

the electronic equipment converts the remapped image of the fourth fusion image and the second amplified image into a YUV format, and fuses the remapped image of the fourth fusion image after the format conversion and the second amplified image to obtain a fused image;

the electronic equipment cuts the fused image based on the field angle of the second camera to obtain a cut fused image;

and the electronic equipment fuses at least one frame of short-exposure image, at least one frame of long-exposure image and the cut fused image in the multiple frames of images to be processed collected by the first camera to obtain a fused image.

For example, the second scene is a high dynamic scene, and in the high dynamic scene, the terminal device performs registration fusion on the normal exposure image acquired by the main camera, the at least one long exposure image and the at least one short exposure image, that is, the details of the bright area in the short exposure image are fused to an invisible area (an overexposed area) in the normal exposure image, so as to solve the overexposure problem in the normal exposure image. And moreover, the details of the dark area in the long exposure image are fused to the underexposed area in the normal exposure image, so that the loss of the details of the dark area in the normal exposure image is compensated, and the dynamic range of the image is improved.

In a possible implementation of the first aspect, the method further includes: the electronic equipment fuses at least one frame of short-exposure image, at least one frame of long-exposure image and the cut fused image in the multi-frame images to be processed collected by the first camera to obtain a fused image, and the method comprises the following steps:

the electronic equipment converts at least one frame of short-exposure image and at least one frame of long-exposure image in a plurality of frames of images to be processed, which are acquired by the first camera, into an RGB format;

and the electronic equipment fuses the at least one frame of short-exposure image and the at least one frame of long-exposure image after format conversion and the cut fused image to obtain a fused image.

In a second aspect, an embodiment of the present application provides an electronic device, including:

the starting module is used for starting a first shooting mode;

the image acquisition module comprises a first camera and a second camera and is used for acquiring a plurality of frames of first images to be processed through the first camera and acquiring a plurality of frames of second images to be processed through the second camera, wherein at least partial images of the plurality of frames of first images to be processed acquired by the first camera and the plurality of frames of second images to be processed acquired by the second camera are acquired synchronously, and the exposure time is the same;

and the image fusion module is used for fusing the multiple frames of first images to be processed and the multiple frames of second images to be processed and outputting fused images.

In a third aspect, the present application provides a computer-readable medium, on which instructions are stored, and when executed on a computer, the instructions cause the computer to perform the first aspect and any one of the various possible implementations of the first aspect.

In a fourth aspect, an embodiment of the present application provides a system, including:

a memory for storing instructions for execution by one or more processors of the system, an

The processor, which is one of the processors of the system, is configured to perform any one of the image processing methods of the first aspect and various possible implementations of the first aspect.

Drawings

FIG. 1(a) illustrates a scene diagram of a user taking a photograph using an electronic device, according to some embodiments of the present application;

FIG. 1(b) illustrates a viewing interface employed by an electronic device, in accordance with some embodiments of the present application;

FIG. 1(c) illustrates a photograph taken by an electronic device using the image processing methods provided herein, according to some embodiments of the present application;

FIG. 2 illustrates a block diagram of a hardware configuration of an electronic device, according to some embodiments of the present application;

fig. 3(a) illustrates a process of taking a picture by a mobile phone using the image processing method provided by the present application, according to some embodiments of the present application;

FIG. 3(b) illustrates a schematic diagram of an image gold tower, according to some embodiments of the present application;

fig. 4 illustrates another mobile phone taking a picture by using the image processing method provided in the present application according to some embodiments of the present application;

FIG. 5 illustrates a flow diagram of a method of image processing, according to some embodiments of the present application;

FIG. 6(a) illustrates an image from a user self-filming using a front-facing main camera of a cell phone, according to some embodiments of the present application;

FIG. 6(b) illustrates an image from a user taking a self-portrait using a front wide-angle camera of a cell phone, according to some embodiments of the present application;

FIG. 6(c) illustrates an image from a user taking a self-portrait via a cell phone capable of performing the image processing methods provided herein, in accordance with some embodiments of the present application;

FIG. 7 illustrates a block diagram of a hardware configuration of an electronic device, according to some embodiments of the present application;

fig. 8 illustrates a block diagram of a system on a chip (SoC), according to some embodiments of the present application.

Detailed Description

Illustrative embodiments of the present application include, but are not limited to, image processing methods, electronic devices, media, and systems.

The technical scheme of this application is to different application scenes, and terminal equipment passes through main camera and gathers multiframe image to gather multiframe image through wide-angle camera, wherein, at least part of picture in the multiframe image that main camera gathered is synchronous with the multiframe image that wide-angle camera gathered, and both exposure times are the same. Electronic equipment gathers multiframe image and the multiframe image that wide-angle camera gathered to main camera and fuses, and the shooting main part after the output fuses all comparatively clear images of background.

The technical scheme of the application can further distinguish different shooting scenes, for example, for the shooting scenes with different dynamic ranges, and therefore the image processing method corresponding to each shooting scene is adopted.

The dynamic range mainly refers to the brightness range of the photographic subject, i.e. the brightness span from the darkest point to the brightest point in the photographic subject. In a scene with a large dynamic range (namely, a high dynamic range), the difference between the brightest part and the darkest part is large, the contrast of the picture is high, and the layers are rich; in a scene with a small dynamic range (i.e., a scene with a non-high dynamic range), most objects have almost the same brightness, and the contrast of the picture is low.

In a non-high dynamic scene, respectively registering and fusing multiple frames of normal exposure images simultaneously acquired by a main camera and a wide-angle camera of terminal equipment, extracting and matching characteristic points of the images (called main shooting preprocessing images for short) registered and fused corresponding to the main camera and the images (called wide-angle preprocessing images for short) registered and fused corresponding to the wide-angle camera, calculating a coordinate mapping relation (such as a homography matrix) of the characteristic point pairs corresponding to the main shooting preprocessing images and the wide-angle preprocessing images, mapping the wide-angle preprocessing images according to the mapping relation to obtain mapped wide-angle preprocessing images, performing histogram brightness correction on the images and the main shooting preprocessing images, performing local registration by adopting an optical flow method, inputting the two images subjected to local registration into a pre-trained fusion network for fusion, so as to obtain clear images of the shooting subject and the background.

In the high dynamic scene, the difference from the non-high dynamic scene is that besides the above processing, the main camera also collects at least one long exposure image and at least one short exposure image, and the normal exposure image, the at least one long exposure image and the at least one short exposure image collected by the main camera are registered and fused together, that is, the details of the bright area in the short exposure image are fused to the invisible area (over-exposure area) in the normal exposure image, so as to solve the over-exposure problem in the normal exposure image. And moreover, the details of the dark area in the long exposure image are fused to the underexposed area in the normal exposure image, so that the loss of the details of the dark area in the normal exposure image is compensated, and the dynamic range of the image is improved.

Embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

FIG. 1(a) illustrates a scene diagram for image processing using an electronic device 100, according to some embodiments of the present application. The electronic apparatus 100 includes a front main camera 101 and a front wide-angle camera 103. The front main camera 101 is an Auto Focus (AF) lens, and the front wide-angle camera 103 is a Fixed Focus (FF) lens. When the user is using the electronic apparatus 100 to perform self-shooting, the electronic apparatus 100 activates the front main camera 101 and the front wide-angle camera 103 to shoot the user and the background in which the user is located. In the related art, when a user uses a single-phase mobile phone to perform self-shooting, generally, the shooting distance is short, so the scene depth is shallow, the shooting background is blurred, and an image with a clear shot main body and without losing the depth of field cannot be obtained. In the embodiment shown in fig. 1(a), the electronic device 100 may perform the image processing method provided in the embodiment of the present application. When a user uses the electronic device 100 to perform self-shooting, the electronic device 100 performs registration and fusion on an image shot by the front main camera 101 and an image shot by the front wide-angle camera 103 by using the image processing method provided by the embodiment of the application, so that the definition of a shot main body is ensured by the front main camera 101, meanwhile, the definition of a shot background is improved by using the characteristics of small focal length and large depth of field of the front wide-angle camera 103, and finally, a relatively clear image of the shot main body and the shot background is obtained.

Specifically, for example, in the embodiment shown in fig. 1(a), icons of a plurality of applications including an icon of a camera APP (Application) are included on the home screen of the electronic apparatus 100, when the user acts on the camera APP of the electronic apparatus 100. For example, when the user clicks the camera APP with a finger, or the user sends a command to the electronic device 100 to open the camera APP in a voice manner, the electronic device 100 detects a click operation for the camera APP or receives the command to open the camera APP, starts the front main camera 101 and the front wide-angle camera 103, and displays a viewing interface.

For example, in the embodiment shown in fig. 1(b), a viewfinder interface of the mobile phone 100 is shown, and a user can select a corresponding shooting mode by sliding a finger on a selection control of different shooting modes of the mobile phone 100 in the viewfinder interface, for example, the user can select to shoot a video, a normal photo, a photo in a "depth enhancement mode", or a photo in other modes, such as a panorama module, a slow motion mode, a delayed shooting, and the like. When the user slides the shooting selection control with the finger to select the "depth-of-field enhancement mode" of the mobile phone 100 for shooting, the mobile phone 100 starts the image processing method provided by the embodiment of the application to obtain a function of shooting an image with a clear main body and a clear background. The above "depth enhancement mode" is only an exemplary illustration of the present application, and the present application is not limited thereto, and the photographing mode may be other names, such as "self-photographing mode" and the like.

When the user clicks the photographing control on the interface, an image with better definition of the main body and the background as shown in fig. 1(c) is finally obtained.

It is understood that in order for the front main camera 101 and the front wide-angle camera 103 of the electronic apparatus 100 to have the same viewing range as much as possible, the front main camera 101 and the front wide-angle camera 103 need to be disposed on the same side of the electronic apparatus 100. The arrangement of the front main camera 101 and the front wide-angle camera 103 of the electronic apparatus 100 shown in fig. 1(a) is merely exemplary and not limiting. In other embodiments of the present application, the front main camera 101 and the front wide-angle camera 103 may be arranged in other predetermined manners, which is not limited in the present application.

It can be understood that the image processing method provided by the embodiment of the present application is not limited to registering and fusing the image captured by the front main camera 101 and the image captured by the front wide-angle camera 103 of the electronic device 100, and finally obtaining an image with a relatively clear captured subject and background. In some embodiments, the electronic device 100 may further include a rear main camera and a rear wide-angle camera, and when the user uses the electronic device 100 to take a picture, the electronic device 100 may further start the rear main camera and the rear wide-angle camera, and perform registration and fusion on an image taken by the rear main camera and an image taken by the rear wide-angle camera, so as to finally obtain an image in which both the subject and the background are clear.

Fig. 2 illustrates a block diagram of a hardware configuration of an electronic device 100, according to some embodiments of the present application. The electronic device 100 is capable of executing the image processing method provided by the embodiment of the application. In fig. 2, similar components have the same reference numerals. As shown in fig. 2, the electronic device 100 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a front main camera 101, a sensor module 190, an audio module 150, a front wide-angle camera 103, an interface module 160, a display screen 102, and the like.

The Processor 110 may include one or more Processing units, for example, a Processing module or a Processing circuit that may include a Central Processing Unit (CPU), an Image Signal Processor (ISP), a Digital Signal Processor (DSP), a Microprocessor (MCU), an Artificial Intelligence (AI) Processor, or a Programmable logic device (FPGA), etc. The different processing units may be separate devices or may be integrated into one or more processors. The processor 110 may be configured to perform registration and fusion on an image captured by the front-facing main camera 101 and an image captured by the front-facing wide-angle camera 103 by using the image processing method provided in the embodiment of the present application, so as to finally obtain an image with a relatively clear shooting subject and background. In some embodiments, the storage unit in processor 110 is cache 180. The memory 180 may store an operating system and an application program (such as an application program for shooting a video) required by at least one function, the memory 180 may further store an image shot by the front main camera 101 and an image shot by the front wide-angle camera 103, respectively perform multi-frame registration and fusion on a multi-frame image shot by the front main camera 101 and a multi-frame image shot by the front wide-angle camera 103 by using a multi-frame fusion algorithm, respectively perform feature point information extracted by using an accelerated Robust Features (SURF) algorithm on the image obtained by performing registration and fusion on the multi-frame image shot by the front main camera 101 and the multi-frame image obtained by the front wide-angle camera 103, and perform registration and fusion on the image obtained by performing registration and fusion on the multi-frame image shot by the front main camera 101 and the multi-frame image obtained by performing registration and fusion on the front wide-angle camera 103, Fused images, and the like.

The power module 140 may include a power supply, power management components, and the like. The power source may be a battery. The power management component is used for managing the charging of the power supply and the power supply of the power supply to other modules. The charging management module is used for receiving charging input from the charger; the power management module is used for connecting a power supply, the charging management module and the processor 110.

The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, a Low Noise Amplifier (LNA), and the like. The mobile communication module 130 may provide a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 100. The mobile communication module 130 may receive electromagnetic waves from the antenna, filter, amplify, etc. the received electromagnetic waves, and transmit the electromagnetic waves to the modem processor for demodulation. The mobile communication module 130 may also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the same device as at least some of the modules of the processor 110. The wireless communication technology may include global system for mobile communications (GSM), General Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), Wideband Code Division Multiple Access (WCDMA), time division code division multiple access (time-division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), Bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), Wireless Local Area Network (WLAN), Near Field Communication (NFC), Frequency Modulation (FM), Infrared (IR), and other technologies. The GNSS may include a Global Positioning System (GPS), a global navigation satellite system (GLONASS), a bei dou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and/or a Satellite Based Augmentation System (SBAS).

The wireless communication module 120 may include an antenna, and implement transceiving of electromagnetic waves via the antenna. The wireless communication module 120 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (blue tooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The electronic device 100 may communicate with networks and other devices via wireless communication techniques.

In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the electronic device 100 may also be located in the same module.

The display screen 102 includes a display panel. The display panel may be a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-OLED, a quantum dot light-emitting diode (QLED), or the like. For example, when the display screen 102 is used to display a self-portrait taken by a user using the electronic device 100, the electronic device 100 performs registration and fusion on images shot by the front main camera 101 and the front wide-angle camera 103 by using the image processing method provided in the embodiment of the present application, and both the shot subject and the background in the image have better definition.

The front main camera 101 includes an auto-focus lens for ensuring the definition of the main body of the shooting, and the front wide-angle camera 103 has a small focal length and a large depth of field for improving the definition of the background of the shooting.

The sensor module 190 may include a proximity light sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

The audio module 150 may convert a digital audio signal into an analog audio signal output or convert an analog audio input into a digital audio signal. The audio module 150 may also be used to encode and decode audio signals. In some embodiments, the audio module 150 may be disposed in the processor 110, or some functional modules of the audio module 150 may be disposed in the processor 110. In some embodiments, audio module 150 may include speakers, an earpiece, a microphone, and a headphone interface.

The interface module 160 includes an external memory interface, a Universal Serial Bus (USB) interface, a Subscriber Identity Module (SIM) card interface, and the like. The external memory interface may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface to implement a data storage function. The usb interface is used for the electronic device 100 to communicate with other mobile phones. The SIM card interface is used for communicating with a SIM card mounted to the electronic device 100, for example reading a telephone number stored in the SIM card, or writing a telephone number to the SIM card.

In some embodiments, the electronic device 100 further includes keys, motors, indicators, and the like. The keys may include a volume key, an on/off key, and the like. The motor is used to generate a vibration effect to the electronic device 100, for example, when the electronic device 100 is called by the user, so as to prompt the user to answer the incoming call of the electronic device 100. The indicators may include laser indicators, radio frequency indicators, LED indicators, and the like.

It is understood that the electronic device 100 may be any hardware device having a photographing function, such as a smart phone, a tablet computer, a camera, a notebook computer, and the like, which is not particularly limited in this application.

Furthermore, it is to be understood that the illustrated structure of the embodiment of the present invention does not specifically limit the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

With reference to fig. 2 and fig. 3(a), taking the electronic device 100 as a mobile phone, and taking an example that a user uses the mobile phone 100 to perform self-shooting in a non-high dynamic range scene, the process of obtaining an image with a clear shooting subject and background by the mobile phone 100 using the image processing method provided in the embodiment of the present application is described in detail below.

Specifically, as shown in fig. 3(a), the process of the mobile phone 100 capturing and outputting the captured picture in the non-high dynamic range scene includes the following steps:

in step 302, the front main camera 101 and the front wide-angle camera 103 of the mobile phone 100 simultaneously acquire multiple frames of images.

Specifically, in some embodiments, a user may click on the camera APP of the mobile phone 100 with a finger, the mobile phone 100 detects a click operation on the camera APP, the front main camera 101 and the front wide-angle camera 103 are started, and the front main camera 101 brings a self-shot user (shooting subject) into an optimal definition through auto-focusing or manual focusing by the user.

In the embodiment according to the present application, the mobile phone 100 needs to control the front main camera 101 and the front wide-angle camera 103 thereof to simultaneously acquire multiple frames of images.

The reason for synchronizing the operations of the main camera 101 and the front wide-angle camera 103 is that it takes a long time to acquire images of different focal points in the related art. In addition, in the case that the user holds the mobile phone to shoot, due to the fact that different focusing point images need to be collected, the mobile phone stays in the hand of the user for a long time, and if the user is inconvenient to hold the mobile phone, shooting experience of the mobile phone can be affected. Meanwhile, due to the fact that hands of a user shake or the movement of a shot object in a shooting scene with a tripod (for example, the positions of people in different shot images are different), relative local movement exists between the shot images under different focal lengths, ghost images exist in the images generated after the images shot under the different focal lengths are fused, and the finally shot images are poor in effect.

In the scheme, the mobile phone 100 simultaneously collects multi-frame images by controlling the front main camera 101 and the front wide-angle camera 103, so that the problem is solved, and the image collection time can be greatly reduced. Specifically, when the user clicks the photographing control, the mobile phone 100 may control the front main camera 101 and the front wide-angle camera 103 to synchronously photograph through an Image Signal Processor (ISP) in the processor, and continuously acquire at least two frames of images. For example, the front main camera 101 continuously captures 6 frames of images, which are image a1, image a2, image A3, image a4, image a5, and image a6 in chronological order of capture; accordingly, the front wide-angle camera 103 continuously captures 6 frames of images, which are image B1, image B2, image B3, image B4, image B5, and image B6 in order of capture time. It should be noted that the time when the front main camera 101 and the front wide-angle camera 103 capture images is the same, and the front main camera 101 and the front wide-angle camera 103 capture the same picture at the same time. Namely: image a1 and image B1 were acquired simultaneously, image a2 and image B2 were acquired simultaneously, image A3 and image B3 were acquired simultaneously, image a4 and image B4 were acquired simultaneously, image a5 and image B5 were acquired simultaneously, and image a6 and image B6 were acquired simultaneously.

In some embodiments, in order to improve the accuracy of subsequent registration of images acquired by the front main camera 101 and the front wide-angle camera 103, and ensure that the brightness difference between the image acquired by the front main camera 101 and the image acquired by the front wide-angle camera 103 is not more than 10%, the mobile phone 100 automatically performs Automatic Exposure compensation processing on a shot picture according to the brightness of a shot scene through Automatic Exposure Control (AEC) to obtain a normally exposed image, and controls the Exposure time of the image acquired by the front main camera 101 and the Exposure time of the front wide-angle camera 103 to be consistent when shooting the image. For example, the exposure times of the 6 frames of images (image a1 to image a6) captured by the front main camera 101 are all t0, the exposure times of the 6 frames of images (image B1 to image B6) captured by the front wide-angle camera 103 are all t1, and t0 and t1 are the same.

In some embodiments, the images captured by the front main camera 101 and the front wide-angle camera 103 are in RAW format, because the RAW format pictures are not subjected to any image processing, the original information captured by the lens can be retained, while a file of some Metadata (Metadata, such as shutter speed, aperture value, white balance, etc.) generated by the lens capture is recorded. Post-production of the image is facilitated, such as adjustment of white balance, exposure level, color contrast, and the like.

It is understood that in other embodiments, the images captured by the front main camera 101 and the front wide-angle camera 103 may also be images in other formats that can retain enough original information captured by the lens, and the present application is not limited thereto.

In step 304, the mobile phone 100 performs registration fusion on the multi-frame images respectively acquired by the two cameras. So as to obtain the picture with noise reduction and improvement of the shot picture details.

In some embodiments, in order to enable the final fusion effect of the images acquired by the front main camera 101 and the front wide-angle camera 103 to be better and the images to be clearer, the mobile phone 100 may first separately perform registration fusion on the multi-frame images acquired by the front main camera 101 and separately perform registration fusion on the multi-frame images acquired by the front wide-angle camera 103 by using a multi-frame fusion algorithm.

Specifically, the mobile phone 100 first registers the multi-frame images respectively acquired by the front main camera 101 and the front wide-angle camera 103 by an image registration method based on gray information, a transform domain, or a characteristic. For example, the mobile phone 100 performs preprocessing such as image segmentation and feature extraction on 6 frames of images (image a1 to image a6) acquired by the front main camera 101 by using a feature-based image registration method, matches features between the image a1 and the image a6, and establishes a registration mapping relationship between the images through the feature matching relationship. Then, the registered images are fused by using an image fusion algorithm such as a logic filtering method, a gray-scale weighted average method, a wavelet transform method or a bayesian method, so as to obtain a registered fused image (hereinafter, referred to as a main shooting preprocessing image for convenience of description) corresponding to the 6 frames of images collected by the front main camera 101 of the mobile phone 100. Similarly, the mobile phone 100 may perform registration fusion on the 6 frames of images (image B1 to image B6) acquired by the front wide-angle camera 103 by using the same method, to obtain corresponding registration fusion images (for convenience of description, hereinafter referred to as wide-angle pre-processed images). The main shooting pre-processing image and the wide-angle pre-processing image are images in an RGB format.

In step 306, the handset 100 zooms in on the wide-angle pre-processed image. In some embodiments, the zoom factor due to the front wide-angle camera 103 is less than the zoom factor of the front main camera 101. For example, the zoom factor of the front wide-angle camera 103 is 0.6X, and the zoom factor of the front main camera 101 is 1X. Therefore, when the front wide-angle camera 103 and the front main camera 101 capture the same image, the size of the object in the image obtained by the front wide-angle camera 103 is smaller than the size of the same object in the image obtained by the front main camera 101, and in order to make the sizes of the two images consistent, the wide-angle preprocessed image needs to be enlarged, and the image content of the wide-angle preprocessed image needs to be larger than the content of the main preprocessed image.

In some embodiments, since the field angle of the front wide-angle camera 103 is large, the viewing range of the front wide-angle camera 103 is larger whether it is the longitudinal direction or the lateral direction when the front wide-angle camera 103 and the front main camera 101 capture the same screen. In order to keep the resolutions of the images captured by the two images consistent, it is necessary to extend the edge of the main-shot pre-processed image, for example, to extend the periphery of the main-shot pre-processed image, and the pixel value corresponding to the pixel of the extended area is 0, so that the pixel numbers of the main-shot pre-processed image and the wide-angle pre-processed image are consistent.

In step 308, the mobile phone 100 performs global registration on the main-shot pre-processed image and the wide-angle pre-processed image. Specifically, the mobile phone 100 extracts and matches feature points of the main shot preprocessed image and the wide-angle preprocessed image to be registered by using a Speeded Up Robust Features (Surf) algorithm, and obtains coordinates of feature point pairs by pairing the feature points. For example, feature point pairing is performed on a plurality of feature points in the corresponding areas of nose, eyes, mouth, ears, and the like in the same face in the main-shooting pre-processed image and the wide-angle pre-processed image, and coordinates of each feature point pair are obtained. A Homography matrix (Homography matrix) from the wide-angle pre-processing image to the main shooting pre-processing image is calculated based on the obtained coordinates of the feature point pairs corresponding to the main shooting pre-processing image and the wide-angle pre-processing image through OpenCV (open source computer vision library), namely, the mapping relation from the coordinates of the feature points on the wide-angle pre-processing image to the coordinates of the feature points on the main shooting pre-processing image is obtained, and the wide-angle pre-processing image is mapped according to the Homography matrix, so that an image (hereinafter referred to as an M image for convenience of description) after the wide-angle pre-processing image is registered according to the main shooting pre-processing image is obtained. It is understood that the M image includes all information of the wide-angle preprocessed image.

In some embodiments, the mobile phone 100 may also obtain the feature point coordinates of the main shot pre-processed image and the wide-angle pre-processed image to be registered through other algorithms, for example: an SIFT feature point detection method, a Harris feature point detection method, a Susan feature point detection method, a stereo image matching method, other feature point detection methods and the like. And then calculating a homography matrix of a mapping relation between the coordinates of the characteristic points on the wide-angle preprocessed image and the coordinates of the corresponding characteristic points on the main shot preprocessed image by using a RANSAC (Random Sample Consensus) algorithm. The embodiments of the present application do not limit this.

In step 310, the mobile phone 100 performs histogram brightness correction on the main shot pre-processed image and the M image, and then performs local registration. So as to eliminate the phenomenon that the ghost image exists in the final fused image caused by the motion vector existing before the main shooting pre-processing image and the M image.

In some embodiments, the front main camera 101 and the front wide-angle camera 103 of the mobile phone 100 are installed at different positions, so that the two cameras capture the same picture with different fields of view. In order to reduce the registration accuracy degradation caused by the brightness difference between the two images, the mobile phone 100 performs histogram brightness correction on the main-shot pre-processed image and the wide-angle pre-processed image before starting local registration. The brightness histogram is a graph for representing the brightness distribution of an image, and shows the proportion of the image occupied by different brightness objects in the image. In one embodiment, the horizontal axis of the histogram of an image represents the brightness in the image, from left to right, gradually transitioning from full black to full white; the vertical axis represents the relative number of pixels in the image that are in this luminance range. And taking the main shooting pre-processing image as a reference image, and performing global brightness correction on the M image through a gray histogram matching algorithm to enable the brightness of the M image to be consistent with that of the main shooting pre-processing image.

The local registration is to calculate the offset of each pixel point of the M image and the main shooting pre-processed image, namely a motion vector, by the main shooting pre-processed image and the M image through an optical flow method, so as to perform final image fusion based on the motion vector. Fig. 3(b) illustrates a schematic diagram of an image pyramid, according to some embodiments of the present application. Wherein the resolution of the image of the lower layer (i +1 th layer) is greater than the resolution of the image of the upper layer (i th layer).

Specifically, in some embodiments, a Lucas-Kanade optical flow algorithm is adopted, a rough-to-fine strategy is adopted, motion estimation is performed on the locally registered main shooting preprocessed image and the M image from a low resolution, and then the obtained motion vector is transmitted to the image with a higher resolution to perform vector refinement processing. For example, in some embodiments, the resolution of the main shot pre-processed image and the M image is 1000 × 1000, and the main shot pre-processed image and the M image are compressed at three levels, for example, sequentially into an image at a first resolution 500 × 500, an image at a second resolution 250 × 250, and an image at a third resolution 125 × 125. Then, the Lucas-Kanade optical flow algorithm is used for sequentially solving the optical flow values of the main shooting preprocessed image and the M image from the image with the lowest resolution, and then the optical flow values of the main shooting preprocessed image and the M image with the original resolution are obtained by recursion until the image with the original resolution (namely the image with the resolution of 1000 x 1000) is obtained, so that the offset of each pixel point of the main shooting preprocessed image and the M image is obtained. Thus, according to the shift amount, a mapped image of the M image to the main-shooting pre-processed image can be obtained.

In step 312, the handset 100 fuses the locally registered main shot pre-processed image and the mapped image of the M image to the main shot pre-processed image (hereinafter referred to as M0 image for descriptive convenience).

In some embodiments, in order to obtain an image with a relatively clear shooting subject and background, the locally registered main shooting preprocessed image and the M0 image may be converted into YUV format, Y channel information (i.e., brightness information) of the locally registered main shooting preprocessed image and the M0 image is extracted, the brightness information of the locally registered main shooting preprocessed image and the brightness information of the M0 image are used as input, and a pre-trained fusion network is used for fusion to obtain a fused image, so that the shooting subject definition of the front main camera 101 and the background definition of the front wide-angle camera 103 are fused to one image, an image with a relatively clear shooting subject and a relatively clear background is obtained, the requirement of a user on the definition of a shot picture is met, and the user experience is improved.

The fusion network may be a neural network model trained with a large amount of locally registered main shot pre-processed image brightness information and brightness of the M0 image and a desired target image. The desired target image may be an image formed by extracting a subject from an image (in which the subject is clear) captured by the front main camera 101 and extracting a shooting background from an image (in which the shooting background is clear) captured by the front wide-angle camera 103 by using image processing software (such as Photoshop), and stitching the extracted subject and the shooting background. The Neural Network model may be various Neural Network models, such as a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Binary Neural Network (BNN), and the like. In the specific implementation, the number of layers of the neural network model, the number of nodes in each layer, and the connection parameters of two connected nodes (i.e., the weights on the connection line of the two nodes) can be preset according to actual requirements.

In step 314, the mobile phone 100 clips the fused image and outputs the clipped image. Before the main shooting pre-processing image and the wide-angle pre-processing image are registered and fused, the main shooting pre-processing image is subjected to edge expansion operation in order to keep the resolution of the images shot by the main shooting pre-processing image and the wide-angle pre-processing image consistent, so that after the fusion is completed, the fused image needs to be cut according to the field angle of the front main camera 101. In some embodiments, the handset 100 may display the cropped image on its display screen 102.

Because in the actual life, there are many high dynamic range scenes, for example outdoor backlight scene in fine day, night scene portrait scene etc. but to the fixed exposure's of single camera module under the condition, can only gather some information under the high dynamic scene to lead to bright area overexposure, or dark space luminance not enough problem. In order to solve the problem, referring to fig. 2 and fig. 4, taking an example that a user uses the mobile phone 100 to perform self-shooting in a high dynamic range scene, the image processing method provided by the embodiment of the present application is used for the mobile phone 100 to achieve self-shooting of the user, so as to describe in detail a process of obtaining an image with a clear shooting subject and a clear background. Compared with the embodiment shown in fig. 3(a), the embodiment shown in fig. 4 not only can make the subject and the background in the image taken by the mobile phone 100 clearer, but also can improve the dynamic range of the image taken by the mobile phone 100.

In the embodiment shown in fig. 4, when the user uses the mobile phone 100 to perform self-timer shooting, the mobile phone 100 previews the shot image and analyzes an overexposed part and an underexposed part in the preview image, so as to determine the current shooting scene. In some embodiments, the mobile phone 100 may obtain a luminance histogram of the preview image, and count overexposure and underexposure ratios of the luminance histogram, so as to determine the shooting scene of the mobile phone 100. For example, pixels in the preview image having a luminance above 250 may be determined to be overexposed pixels and pixels having a luminance less than 10 may be determined to be underexposed pixels. If the proportion of the overexposed pixels in the preview image is greater than or equal to a set threshold value, or the proportion of the underexposed pixels in the preview image is greater than or equal to another preset threshold value, determining that the shooting scene is a high-dynamic scene; and if the proportion of the overexposed pixels in the preview image is smaller than a set threshold value and the proportion of the underexposed pixels in the preview image is smaller than another preset threshold value, determining that the shot scene is a non-high dynamic scene. In one example, when the proportion of overexposure is less than 1% and the proportion of underexposure is less than 10% in the preview image, the current shooting scene is considered to be a non-highly dynamic scene; and when the overexposure proportion in the preview image is more than 1% or the underexposure proportion in the preview image is more than 10%, the current shooting scene is considered to be a high-dynamic scene.

It is to be understood that the above-mentioned values and ratios are only general examples for facilitating understanding of the embodiments of the present invention, and other threshold values may be set according to parameters of the mobile phone 100.

It can be understood that the image previewed by the mobile phone 100 means that the mobile phone 100 presents the image captured by the camera on the display screen, and is intended to prompt the user about the content of the image that can be captured by the camera currently, so that the user can adjust the corresponding viewing state, but at this time, formal photographing is not performed, and the viewing image is not saved in the storage medium of the mobile phone 100.

The front main camera 101 of the mobile phone 100 outputs a plurality of frames of different exposure frames through automatic exposure control, and performs fusion of the different exposure frames on the two-way fusion depth-of-field enhanced image shown in fig. 3(a), that is, the details of the bright area in the short exposure image are fused to the invisible area (overexposure area) in the normal exposure image, so as to solve the overexposure problem in the normal exposure image. And the details of the dark area in the long exposure image are fused to the underexposed area in the normal exposure image to make up for the loss of the details of the dark area in the normal exposure image.

Specifically, as shown in fig. 4, the process of the mobile phone 100 capturing and outputting the captured picture in the high dynamic range scene includes the following steps:

in step 402, the front main camera 101 and the front wide-angle camera 103 of the mobile phone 100 simultaneously acquire multiple frames of images, and the front main camera 101 also acquires a frame of long-exposure image and a frame of short-exposure image. The specific process is similar to step 302 in fig. 3(a), and the difference is only that in the embodiment shown in fig. 4, the mobile phone 100 issues different Exposure parameters to the front main camera 101 through Automatic Exposure Control (AEC), so that the front main camera 101 outputs multiple frames of images with different Exposure degrees.

For example, in some embodiments, when the user clicks the photographing control, the mobile phone 100 may control the front main camera 101 and the front wide-angle camera 103 to synchronously photograph through an Image Signal Processor (ISP), and the front main camera 101 continuously acquires 6 frames of normal exposure images and one frame of overexposed images (long exposure images) and one frame of underexposed images (short exposure images). Image A1, image A2, image A3, image A4, image A5, image A6, image A7, and image A8 in chronological order of acquisition; accordingly, the front wide-angle camera 103 continuously captures 6 frames of images, which are image B1, image B2, image B3, image B4, image B5, and image B6 in order of capture time. It is to be noted that since the exposure time of the short-exposure image is shorter than that of the normal-exposure image and the exposure time of the long-exposure image is longer than that of the normal-exposure image, the image a1 captured by the front main camera 101 is a short-exposure image, the images a2 to a7 are 6-frame normal-exposure images, and the image A8 is a long-exposure image. The front main camera 101 and the front wide-angle camera 103 capture the same picture at the same time, the time of the 6-frame normal exposure image (image a2 to image a7) captured by the front main camera 101 and the 6-frame image captured by the front wide-angle camera 103 are the same, and the exposure time of the 6-frame normal exposure image (image a2 to image a7) captured by the front main camera 101 and the 6-frame image captured by the front wide-angle camera 103 are kept the same. Namely: image a2 and image B1 were acquired simultaneously, image A3 and image B2 were acquired simultaneously, image a4 and image B3 were acquired simultaneously, image a5 and image B4 were acquired simultaneously, image a6 and image B5 were acquired simultaneously, and image a7 and image B6 were acquired simultaneously.

In step 404, the mobile phone 100 performs registration fusion on the multi-frame images respectively acquired by the two cameras. So as to obtain the picture with noise reduction and improvement of the shot picture details. The specific process is similar to step 304 shown in fig. 3(a), except that in the embodiment shown in fig. 4, the front main camera 101 performs registration fusion on multiple frames of normal exposure images, one frame of long exposure images and one frame of short exposure images acquired by the front main camera simultaneously, so as to fuse details of the long exposure images and the short exposure images into the normal exposure images of the front main camera 101. For example, 8 frames of images, i.e., 6 frames of normal exposure images (image a2 to image a7), a short exposure image a1, and a long exposure image A8, acquired by the front main camera 101 are registered and fused to obtain a map (hereinafter referred to as a main shot pre-processing image) including details of the normal exposure image, the long exposure image, and the short exposure image. The images (hereinafter referred to as wide-angle preprocessed images) obtained by fusing the images with 6 frames of images (images B1 to B6) acquired by the front wide-angle camera 103 are registered and fused.

In step 406, the handset 100 zooms in on the wide-angle pre-processed image. In some embodiments, the zoom factor due to the front wide-angle camera 103 is less than the zoom factor of the front main camera 101. Therefore, when the front wide-angle camera 103 and the front main camera 101 capture the same image, the size of the object in the image obtained by the front wide-angle camera 103 is smaller than the size of the same object in the image obtained by the front main camera 101, and in order to make the sizes of the two images consistent, the wide-angle preprocessed image needs to be enlarged, and the image content of the wide-angle preprocessed image needs to be larger than the content of the main preprocessed image.

In some embodiments, since the field angle of the front wide-angle camera 103 is large, the viewing range of the front wide-angle camera 103 is larger whether it is the longitudinal direction or the lateral direction when the front wide-angle camera 103 and the front main camera 101 capture the same screen. In order to keep the resolution of the images captured by the two cameras consistent, the main-shot pre-processed image needs to be edge-extended, for example, the periphery of the main-shot pre-processed image is extended by one turn.

In step 408, the mobile phone 100 performs global registration on the main-shot pre-processed image and the wide-angle pre-processed image. For example, the mobile phone 100 extracts and matches feature points of a main shot preprocessed image and a wide-angle preprocessed image to be registered respectively by using a Speeded Up Robust Features (Surf) algorithm, and obtains coordinates of feature point pairs by matching the feature points. A Homography matrix (Homography matrix) from the wide-angle pre-processing image to the main shooting pre-processing image is calculated based on the obtained coordinates of the feature point pairs corresponding to the main shooting pre-processing image and the wide-angle pre-processing image through OpenCV (open source computer vision library), namely, the mapping relation from the coordinates of the feature points on the wide-angle pre-processing image to the coordinates of the feature points on the main shooting pre-processing image is obtained, and the wide-angle pre-processing image is mapped according to the Homography matrix, so that an image (hereinafter referred to as an N image for convenience of description) after the wide-angle pre-processing image is registered according to the main shooting pre-processing image is obtained. It is understood that the N image includes all information of the wide-angle pre-processed image. The process is similar to the registration process in the embodiment shown in fig. 3(a), and for the detailed description, reference is made to the above, which is not repeated herein.

In step 410, the mobile phone 100 performs histogram brightness correction on the main shot pre-processed image and the N image, and then performs local registration. So as to eliminate the phenomenon that the ghost image exists in the final fused image caused by the motion vector existing before the main shooting pre-processing image and the N image. The process is similar to the process of local registration in the embodiment shown in fig. 3(a), and for detailed description, reference is made to the above, which is not repeated here.

In step 412, the handset 100 fuses the locally registered main shot pre-processed image and the mapped image of the N image to the main shot pre-processed image (hereinafter referred to as the N0 image for descriptive convenience).

In some embodiments, in order to obtain an image with a clearer shooting subject and a clearer background, the locally registered main shooting preprocessed image and the N0 image may be converted into YUV format, Y channel information (i.e., brightness information) of the locally registered main shooting preprocessed image and the N0 image is extracted, the brightness information of the locally registered main shooting preprocessed image and the brightness information of the N0 image are used as input, and a pre-trained fusion network is used for fusion to obtain a fused image, so that the shooting subject definition brought by the front main camera 101 and the background definition brought by the front wide-angle camera 103 are fused to one image, an image with clearer shooting subject and clearer background is obtained, the requirement of a user on the definition of the shot image is met, and the user experience is improved.

The fusion network may be a neural network model trained by using a large amount of locally registered main shot pre-processing image brightness information and brightness information of the N0 image and a desired target image. The desired target image may be an image formed by extracting a subject from an image (in which the subject is clear) captured by the front main camera 101 and extracting a shooting background from an image (in which the shooting background is clear) captured by the front wide-angle camera 103 by using image processing software (such as Photoshop), and stitching the extracted subject and the shooting background. The Neural Network model may be various Neural Network models, such as a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Binary Neural Network (BNN), and the like. In the specific implementation, the number of layers of the neural network model, the number of nodes in each layer, and the connection parameters of two connected nodes (i.e., the weights on the connection line of the two nodes) can be preset according to actual requirements.

In step 414, the handset 100 crops the fused image. Before the main shooting pre-processed image and the wide-angle pre-processed image are registered and fused, the main shooting pre-processed image is subjected to edge expansion operation in order to keep the resolution of the images shot by the main shooting pre-processed image and the wide-angle pre-processed image consistent, so that after the fusion is finished, the fused image also needs to be cut according to the field angle of the front main camera 101.

In step 416, the mobile phone 100 converts the long and short frames (short-exposure image and long-exposure image) in RAW format into RGB format through the ISP, so as to perform re-registration fusion with the fused image in RGB format.

In step 418, the mobile phone 100 performs global registration and fusion on the fused and cropped image (the image obtained by fusing the main shot pre-processed image and the N0 image obtained by mapping the N image to the main shot pre-processed image and cropping the fused image), the long-exposure image a7 acquired by the front main camera 101, and the short-exposure image a8 by using a high dynamic expansion algorithm, so as to improve the dynamic range of the image.

In some embodiments, in order to obtain an image with a clear shot subject and background and an improved dynamic range, the fused and cropped image, the long-exposure image A8 acquired by the front main camera 101, and the short-exposure image a1 may be globally registered by using a high dynamic expansion algorithm for the mobile phone 100 in a manner similar to that in step 412, so as to obtain a mapping image corresponding to the short-exposure image and a mapping image corresponding to the long-exposure image. Then converting the mapping image of the short-exposure image, the mapping image of the long-exposure image and all the registered main shooting preprocessing images into YUV format, extracting the mapping image of the short-exposure image, the mapping image of the long-exposure image and Y channel information (namely brightness information) of the main shooting preprocessing images after global registration, taking the brightness information as input, and fusing through a pre-trained fusion network to obtain fused images, thereby realizing the fusion of the shooting main body definition of the front main camera 101 and the background definition of the front wide-angle camera 103 onto one image to obtain images with clear shooting main body and background, improving the image with dynamic range, and meeting the requirements of users for obtaining images with clear shooting main body and shooting background in high dynamic range scene, and the user experience is improved.

The fusion network may be a neural network model trained by using a large amount of brightness information of the mapped image corresponding to the short-exposure image after global registration, brightness information of the mapped image corresponding to the long-exposure image, brightness information of the main shooting pre-processed image after global registration and a desired target image.

In step 420, the mobile phone 100 converts the registered and fused image with the dynamic range improved from the RGB format to the JPG format.

In step 422, in some embodiments, the registered and fused JPG formatted image with the dynamic range enhanced by the handset 100 is displayed by the display screen 102 of the handset 100 for viewing by the user.

In the above, taking the example that the user uses the mobile phone 100 to perform self-shooting in the non-high dynamic range scene and the user uses the mobile phone 100 to perform self-shooting in the high dynamic range scene as an example, the image processing method provided by the embodiment of the present application is used for the mobile phone 100 to achieve self-shooting of the user, so that a detailed description is given to the process of obtaining an image with a clear shooting main body and a clear background.

It is understood that, in other embodiments of the present application, the mobile phone 100 may also not distinguish shooting scenes, and the front main camera 101 and the front wide-angle camera 103 of the mobile phone 100 respectively capture images and perform processing by using any one of the above image processing modes.

In the embodiment shown in fig. 5, the image processing method implemented by the mobile phone 100 has both high dynamic processing modes and non-high dynamic processing modes, and can determine a shooting scene and select an appropriate image processing mode, where the specific process includes the following steps:

in step 502, the cellular phone 100 activates the front main camera 101 and the front wide-angle camera 103. Specifically, in some embodiments, the user may click on the camera APP of the mobile phone 100 with a finger, the mobile phone 100 detects a click operation on the camera APP, and the front main camera 101 and the front wide-angle camera 103 are started. The front main camera 101 allows a user (subject) who is taking a picture to be in the best definition by auto-focusing or manual focusing by the user.

In step 504, the mobile phone 100 determines a shooting scene. After the front main camera 101 and the front wide-angle camera 103 are started, a user can see a view frame displayed on the display screen 102 of the mobile phone 100, image preview is performed through the view frame, the mobile phone 100 analyzes an overexposed part and an underexposed part in a preview image, if the ratio of the overexposed area and the underexposed area in the preview image is greater than a certain ratio threshold value, it is determined that a self-shooting scene where the user is currently located is a high-dynamic scene, and conversely, it is determined that the self-shooting scene where the user is currently located is a non-high-dynamic scene. In some embodiments, the mobile phone 100 may obtain a luminance histogram of the preview image, and count overexposure and underexposure ratios of the luminance histogram, so as to determine the shooting scene of the mobile phone 100. For example, pixels in the preview image having a luminance above 250 may be determined to be overexposed pixels and pixels having a luminance less than 10 may be determined to be underexposed pixels. If the proportion of the overexposed pixels in the preview image is greater than or equal to a set threshold value, or the proportion of the underexposed pixels in the preview image is greater than or equal to another preset threshold value, determining that the shooting scene is a high-dynamic scene; and if the proportion of the overexposed pixels in the preview image is smaller than a set threshold value and the proportion of the underexposed pixels in the preview image is smaller than another preset threshold value, determining that the shot scene is a non-high dynamic scene. In one example, when the proportion of overexposure is less than 1% and the proportion of underexposure is less than 10% in the preview image, the current shooting scene is considered to be a non-highly dynamic scene; and when the overexposure proportion in the preview image is more than 1% or the underexposure proportion in the preview image is more than 10%, the current shooting scene is considered to be a high-dynamic scene.

In the case that the mobile phone 100 determines that the current shooting scene is a non-high-dynamic scene, in order to ensure the definition of the shooting subject and the shooting background in the shot image, the mobile phone 100 may perform the following operations:

A. the front main camera 101 and the front wide-angle camera 103 of the mobile phone simultaneously acquire multi-frame images (506 a). Specifically, when the user clicks the photographing control, the mobile phone 100 may control the front main camera 101 and the front wide-angle camera 103 to shoot synchronously through an Image Signal Processor (ISP), and continuously acquire at least two frames of images (for example, one of the multi-frame images acquired by the front main camera 101 is an image shown in fig. 6(a), and one of the multi-frame images acquired by the front wide-angle camera 103 is an image shown in fig. 6 (b)). In some embodiments, in order to improve the accuracy of subsequent registration of images acquired by the front main camera 101 and the front wide-angle camera 103, and ensure that the brightness difference between the image acquired by the front main camera 101 and the image acquired by the front wide-angle camera 103 is not more than 10%, the mobile phone 100 automatically performs Automatic Exposure compensation processing on a shot picture according to the brightness of a shot scene through Automatic Exposure Control (AEC) to obtain a normally exposed image, and controls the Exposure time of the image acquired by the front main camera 101 and the Exposure time of the front wide-angle camera 103 to be consistent when shooting the image. The images acquired by the front main camera 101 and the front wide-angle camera 103 are in a RAW format.

B. The mobile phone 100 respectively performs registration fusion on the multi-frame images respectively acquired by the two cameras (508a), so as to obtain a registration fusion image (for convenience of description, hereinafter referred to as a main shooting pre-processing image) corresponding to the multi-frame image acquired by the front main camera 101 of the mobile phone 100, and a registration fusion image (for convenience of description, hereinafter referred to as a wide-angle pre-processing image) corresponding to the multi-frame image acquired by the front wide-angle camera 103 of the mobile phone 100. Therefore, the image with the noise reduction function and the shooting picture detail improvement function can be obtained, and the precision of subsequent image processing and the quality of the finally obtained fusion image are improved. The specific process is similar to the process of step 304 in the embodiment shown in fig. 3(a), and the detailed description is please refer to the above, which is not repeated herein.

C. The handset 100 globally registers the main-shot pre-processed image and the wide-angle pre-processed image (510 a). In some embodiments, the handset 100 needs to enlarge the wide-angle pre-processed image and expand the main-shot pre-processed image before global registration of the main-shot pre-processed image and the wide-angle pre-processed image. Extracting and matching the feature points through a Speeded Up Robust Features (Surf) algorithm, and obtaining the coordinates of the feature point pairs through matching the feature points. And then calculating a Homography matrix (Homography matrix) from the wide-angle preprocessed image to the main shooting preprocessed image based on the coordinates of the characteristic point pairs, and mapping the wide-angle preprocessed image according to the Homography matrix to obtain an image (for convenience of description, hereinafter referred to as an M image) obtained by registering the wide-angle preprocessed image according to the main shooting preprocessed image. The process of global registration is similar to the process from step 306 to step 308 in the embodiment shown in fig. 3(a), and the detailed description is please refer to the above, which is not repeated herein.

D. The mobile phone 100 performs histogram brightness correction on the main shot pre-processed image and the M image, and then performs local registration (512a) to eliminate the phenomenon that a ghost image exists in the final fused image due to the motion vector existing before the main shot pre-processed image and the M image. For the definition of the M image, please refer to the text description in the embodiment shown in fig. 3 (a). The process of local registration is similar to the process of step 310 in the embodiment shown in fig. 3(a), and the detailed description is please refer to the above, which is not repeated herein.

E. The cell phone 100 fuses the locally registered main shot pre-processed image and the mapped image of the M image to the main shot pre-processed image (M0 image) (514 a). The process of fusion is similar to the process of step 312 in the embodiment shown in fig. 3(a), and the detailed description is please refer to the above, which is not repeated herein.

F. And (516a) cutting and outputting. That is, in some embodiments, the mobile phone 100 crops the fused image according to the field angle of the front main camera 101, and displays the cropped image (e.g., the image shown in fig. 6 (c)) on its display screen 102 for the user to view.

In order to ensure the clarity of the subject and the background in the captured image and to improve the dynamic range of the image captured by the mobile phone 100, when the mobile phone 100 determines that the current capturing scene is a high dynamic scene, the mobile phone 100 may perform the following operations:

A. the front main camera 101 and the front wide-angle camera 103 of the mobile phone 100 simultaneously acquire multiple frames of images, and the front main camera 101 also acquires a frame of long-exposure image and a frame of short-exposure image (506 b). The specific process is similar to step 402 in fig. 4, and is not described herein again.

B. The mobile phone 100 respectively performs registration fusion on the multi-frame images respectively acquired by the two cameras (508 b). To fuse the details of the long-exposure image and the short-exposure image into the normal-exposure image of the front main camera 101. A picture (hereinafter referred to as a main shooting preprocessing picture) including details of a plurality of frames of normal exposure pictures, a frame of long exposure pictures and a frame of short exposure pictures collected by the front main camera 101 and a picture (hereinafter referred to as a wide-angle preprocessing picture) corresponding to the plurality of frames of normal exposure pictures collected by the front wide-angle camera 103 and having improved picture details are obtained. Compared with the scheme that the front main camera 101 and the front wide-angle camera 103 respectively collect only one frame of image for registration and fusion, the scheme can improve the image processing precision by performing subsequent registration and fusion on the result image obtained after the registration and fusion of the multi-frame images collected by the two cameras. The specific process is similar to step 404 in fig. 4, and is not described herein again.

It can be understood that, in some embodiments, the mobile phone 100 may further collect a multi-frame long exposure image and a multi-frame short exposure image, and fuse the multi-frame long exposure image, the multi-frame short exposure image, and the multi-frame normal exposure image to fuse details of the long exposure image and the short exposure image onto the normal exposure image, so as to solve an overexposure problem in the normal exposure image, compensate for loss of details of a dark area in the normal exposure image, and improve a dynamic range of the image.

C. The handset 100 globally registers the main-shot pre-processed image and the wide-angle pre-processed image (510 b). And obtaining an image (for convenience of description, hereinafter referred to as an N image) of the wide-angle preprocessed image after registration according to the main shot preprocessed image. In some embodiments, the handset 100 requires the wide-angle pre-processed image to be enlarged and the main-shot pre-processed image to be edge-enlarged before global registration of the main-shot pre-processed image and the wide-angle pre-processed image. The process of global registration is similar to the process from step 406 to step 408 in the embodiment shown in fig. 4, and the detailed description is please refer to the above, which is not repeated herein.

D. The mobile phone 100 performs histogram brightness correction on the main shot pre-processed image and the N image, and then performs local registration (512 b). For the definition of the N images, please refer to the above description of the text in the embodiment shown in fig. 4. The process of local registration is similar to the process of step 410 in the embodiment shown in fig. 4, and the detailed description is please refer to the above, which is not repeated herein.

E. The cell phone 100 fuses the locally registered main shot pre-processed image and the mapped image of the N image to the main shot pre-processed image (514 b). The process of fusion is similar to that of step 412 in the embodiment shown in fig. 4, and the detailed description is please refer to the above, which is not repeated herein.

F. And (516b) cutting. That is, in some embodiments, the mobile phone 100 clips the fused image according to the field angle of the front main camera 101.

G. The mobile phone 100 performs global registration and fusion on the fused and cut image, the long-exposure image and the short-exposure image collected by the front main camera 101 by using a high dynamic expansion algorithm (518 b). To improve the dynamic range of the image. In some embodiments, the long and short frames (short exposure image and long exposure image) may be converted to RGB format by the ISP before registration, fusion. The specific process is similar to the process of step 418 in the embodiment shown in fig. 4, and please refer to the above description for detailed description, which is not repeated herein.

H. And outputting (520 b). That is, the cell phone 100 displays the fused image on its display screen 102 for viewing by the user.

Fig. 7 provides a schematic structural diagram of an electronic device 700, according to some embodiments of the present application, as shown in fig. 7, including:

a starting module 702, configured to start a first shooting mode;

the image acquisition module 704 comprises a first camera and a second camera, and is used for acquiring a plurality of frames of first images to be processed through the first camera and acquiring a plurality of frames of second images to be processed through the second camera, wherein at least part of images of the plurality of frames of first images to be processed acquired by the first camera and at least part of images of the plurality of frames of second images to be processed acquired by the second camera are acquired synchronously, and the exposure time is the same;

and the image fusion module 706 is configured to fuse multiple frames of the first image to be processed and multiple frames of the second image to be processed, and output a fused image.

Fig. 8 shows a block diagram of a System on Chip (SoC) 800, according to an embodiment of the present application. In fig. 8, like parts have the same reference numerals. In addition, the dashed box is an optional feature of more advanced socs. In fig. 8, the SoC800 includes: an interconnection unit 850; a system agent unit 870; a bus controller unit 880; an integrated memory controller unit 840; a set or one or more coprocessors 820 which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a Static Random Access Memory (SRAM) unit 830; a Direct Memory Access (DMA) unit 860. In one embodiment, coprocessor 820 includes a special-Purpose processor, such as a network or communication processor, compression engine, graphics processor General Purpose Computing (GPGPU), high-throughput MIC processor, embedded processor, or the like.

Embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of these implementations. Embodiments of the application may be implemented as computer programs or program code executing on programmable systems comprising at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.

Program code may be applied to input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this Application, a Processing system includes any system having a processor such as a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code can also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described in this application are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy diskettes, optical disks, Read-Only memories (CD-ROMs), magneto-optical disks, Read-Only memories (ROMs), Random Access Memories (RAMs), Erasable Programmable Read-Only memories (EPROMs), Electrically Erasable Programmable Read-Only memories (EEPROMs), magnetic or optical cards, flash Memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet to transmit information in an electrical, optical, acoustical or other form of propagated signals. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

In the drawings, some features of the structures or methods may be shown in a particular arrangement and/or order. However, it is to be understood that such specific arrangement and/or ordering may not be required. Rather, in some embodiments, the features may be arranged in a manner and/or order different from that shown in the illustrative figures. In addition, the inclusion of a structural or methodical feature in a particular figure is not meant to imply that such feature is required in all embodiments, and in some embodiments, may not be included or may be combined with other features.

It should be noted that, in the embodiments of the apparatuses in the present application, each unit/module is a logical unit/module, and physically, one logical unit/module may be one physical unit/module, or may be a part of one physical unit/module, and may also be implemented by a combination of multiple physical units/modules, where the physical implementation manner of the logical unit/module itself is not the most important, and the combination of the functions implemented by the logical unit/module is the key to solve the technical problem provided by the present application. Furthermore, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units/modules which are not so closely related to solve the technical problems presented in the present application, which does not indicate that no other units/modules exist in the above-mentioned device embodiments.

It is noted that, in the examples and descriptions of this patent, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, same element in a process, method, article, or apparatus that comprises the element.

While the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image processing method is used for an electronic device, the electronic device comprises a first camera and a second camera, and the method is characterized by comprising the following steps:

the electronic equipment starts a first shooting mode;

the electronic equipment collects multiple frames of first images to be processed through the first camera and multiple frames of second images to be processed through the second camera, wherein at least partial images of the multiple frames of first images to be processed collected by the first camera and the multiple frames of second images to be processed collected by the second camera are collected synchronously, and the exposure time is the same;

and the electronic equipment fuses a plurality of frames of the first image to be processed and a plurality of frames of the second image to be processed and outputs the fused images.

2. The method of claim 1, wherein the first camera comprises an auto-focus primary camera and the second camera comprises a fixed-focus wide-angle camera.

3. The method according to claim 1 or 2, comprising:

the electronic equipment acquires shooting scene information;

the electronic equipment determines an image processing mode according to the acquired shooting scene information;

and the electronic equipment fuses a plurality of frames of the first image to be processed and a plurality of frames of the second image to be processed according to the determined image processing mode, and outputs the fused images.

4. The method of claim 3, wherein the electronic device determining an image processing mode from the capture scene information comprises:

the electronic equipment responds to the image preview operation to obtain a preview image;

the electronic equipment determines that the image processing mode is the first processing mode under the condition that the shooting scene is determined to be the first scene.

5. The method according to claim 4, wherein the electronic device fuses a plurality of frames of the first image to be processed and a plurality of frames of the second image to be processed according to the determined image processing mode, and outputting the fused images comprises:

the electronic equipment fuses a plurality of frames of first images to be processed collected by the first camera under the condition that the image processing mode is determined to be a first processing mode, so as to obtain a first fused image; fusing the plurality of frames of the second images to be processed collected by the second camera to obtain a second fused image; the multi-frame first to-be-processed image collected by the first camera comprises at least two frames of normal exposure images, and the multi-frame second to-be-processed image collected by the second camera comprises at least two frames of normal exposure images;

the electronic equipment expands the edge of the first fused image based on the field angle of the second camera to obtain a first enlarged image, and the first enlarged image and the second fused image have the same resolution;

the electronic equipment registers the mapping image of the second fusion image and the first amplified image to obtain a re-mapping image corresponding to the second fusion image;

6. The method of claim 3, wherein the electronic device determining an image processing mode from the capture scene information comprises:

the electronic equipment determines that the shooting scene is a second scene when determining that the proportion of the overexposed area in the preview image is greater than or equal to a first proportion threshold value or the proportion of the underexposed area is greater than or equal to a second proportion threshold value;

and the electronic equipment determines that the image processing mode is the second processing mode under the condition that the shooting scene is determined to be the second scene.

7. The method according to claim 6, wherein the electronic device fuses a plurality of frames of the first image to be processed and a plurality of frames of the second image to be processed according to the determined image processing mode, and outputting the fused images comprises:

the electronic equipment fuses a plurality of frames of the first image to be processed collected by the first camera under the condition that the image processing mode is determined to be a second processing mode, so as to obtain a third fused image; fusing a plurality of frames of the second images to be processed collected by the second camera to obtain a fourth fused image; the first to-be-processed images collected by the first camera comprise at least one short exposure image, at least one long exposure image and at least two normal exposure images; the second camera collects multiple frames of images to be processed, and the multiple frames of images to be processed comprise at least two frames of normal exposure images;

the electronic equipment expands the edge of the third fused image based on the field angle of the second camera to obtain a second enlarged image, and the resolution of the second enlarged image is the same as that of the fourth fused image;

the electronic equipment registers the mapping image of the fourth fusion image and the second amplified image to obtain a re-mapping image corresponding to the fourth fusion image;

the electronic equipment converts the remapped image of the fourth fused image and the second amplified image into a YUV format, and fuses the remapped image of the fourth fused image after the format conversion and the second amplified image to obtain a fused image;

8. The method according to claim 7, wherein the electronic device fuses at least one frame of short-exposure image, at least one frame of long-exposure image and the cropped fused image in the plurality of frames of images to be processed collected by the first camera to obtain a fused image, and the method comprises:

and the electronic equipment fuses the at least one frame of short-exposure image and the at least one frame of long-exposure image after format conversion and the cut fusion image to obtain a fused image.

9. An electronic device, comprising:

the starting module is used for starting a first shooting mode;

the image acquisition module comprises a first camera and a second camera, and is used for acquiring a plurality of frames of first images to be processed through the first camera and acquiring a plurality of frames of second images to be processed through the second camera, wherein at least part of images of the plurality of frames of first images to be processed acquired by the first camera and the plurality of frames of second images to be processed acquired by the second camera are acquired synchronously, and the exposure time is the same;

and the image fusion module is used for fusing the plurality of frames of the first image to be processed and the plurality of frames of the second image to be processed and outputting the fused images.

10. A computer-readable medium having stored thereon instructions which, when executed on a computer, cause the computer to perform the image processing method of any one of claims 1 to 8.

11. A system, comprising:

a memory for storing instructions for execution by one or more processors of the electronic device, an

A processor, being one of processors of an electronic device, for performing the image processing method of any one of claims 1 to 8.