WO2023130922A1

WO2023130922A1 - Image processing method and electronic device

Info

Publication number: WO2023130922A1
Application number: PCT/CN2022/138808
Authority: WO
Inventors: 肖斌; 乔晓磊; 朱聪超; 王宇; 邵涛
Original assignee: 荣耀终端有限公司
Priority date: 2022-01-10
Filing date: 2022-12-13
Publication date: 2023-07-13
Also published as: CN115550570A; CN115550570B

Abstract

An image processing method and an electronic device, which relate to the field of image processing. The image processing method is applied to the electronic device. The electronic device comprises a first camera module and a second camera module. The second camera module is a near-infrared camera module or an infrared camera module. The image processing method comprises: displaying a first interface, the first interface comprising a first control; detecting a first operation on the first control; in response to the first operation, obtaining N frames of a first image and M frames of a second image, wherein the first image is an image collected by the first camera module, the second image is an image collected by the second camera module, and N and M are both positive integers greater than or equal to 1; obtaining a target image on the basis of the N frames of the first image and the M frames of the second image; and storing the target image. On the basis of the technical scheme of the present application, image enhancement can be carried out on an image obtained by a main camera module in the electronic device, and the image quality is improved.

Description

Image processing method and electronic device

This application claims the priority of the Chinese patent application with the application number 202210023611.2 and the application name "Image Processing Method and Electronic Equipment" submitted to the State Intellectual Property Office on January 10, 2022, the entire contents of which are incorporated in this application by reference.

technical field

The present application relates to the field of image processing, and in particular, relates to an image processing method and electronic equipment.

Background technique

With the rapid development and wide application of multimedia technology and network technology, people use a lot of image information in their daily life and production activities. In some shooting scenes, for example, shooting scenes with poor lighting conditions, such as night scenes or dense fog scenes, due to the poor light conditions of the shooting scene, the amount of light entering the electronic device is less, causing the camera module of the main camera to There is a problem of partial image detail information loss in the acquired image; in order to improve the image quality, image enhancement processing can usually be used; image enhancement processing is a method for enhancing useful information in the image and improving the visual effect of the image.

Therefore, how to enhance the image acquired by the camera module of the main camera and improve the image quality has become an urgent problem to be solved.

Contents of the invention

The present application provides an image processing method and electronic equipment, which can perform image enhancement on images acquired by a camera module of a main camera to improve image quality.

In a first aspect, an image processing method is provided, which is applied to an electronic device, and the electronic device includes a first camera module and a second camera module, and the second camera module is a near-infrared camera module or an infrared camera module, the image processing method includes:

displaying a first interface, where the first interface includes a first control;

detecting a first operation on the first control;

In response to the first operation, N frames of first images and M frames of second images are acquired, the first images are images collected by the first camera module, and the second images are images collected by the second camera module The images collected by the group, N and M are both positive integers greater than or equal to 1;

Obtaining a target image based on the N frames of the first image and the M frames of the second image;

saving said target image;

Wherein, the target image is obtained based on the N frames of the first image and the M frames of the second image, including:

performing first image processing on the N frames of the first image to obtain N frames of the third image, the image quality of the N frames of the third image is higher than the image quality of the N frames of the first image;

Carrying out the second image processing to the second image of the M frames to obtain the fourth image of the M frames, the image quality of the fourth image of the M frames is higher than the image quality of the second image of the M frames;

Based on the semantically segmented image, the third image of the N frames and the fourth image of the M frame are fused to obtain a fused image, and the semantically segmented image is based on any frame image in the first image of the N frames or the The detailed information of the fused image is better than the detailed information of the N frames of the first image;

Performing third image processing on the fused image to obtain a target image.

Optionally, the first camera module may be a visible light camera module, or the first camera module may be other camera modules capable of obtaining visible light; this application does not make any limitation on the first camera module.

Optionally, the first camera module may include a first lens, a first lens and an image sensor, and the spectral range through which the first lens can pass is visible light (400nm-700nm).

It should be understood that the first lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of visible light bands to pass through.

Exemplarily, the second camera module may include a second lens, a second lens and an image sensor, and the spectral range that the second lens can pass is near-infrared light (700nm˜1100nm).

It should be understood that the second lens may refer to a filter lens; the second lens may be used to absorb light of certain specific wavelength bands and allow light of near-infrared wavelength bands to pass through.

In an embodiment of the present application, the electronic device may include a first camera module and a second camera module, wherein the second camera module is a near-infrared camera module or an infrared camera module (for example, the acquired spectral range is 700nm～1100nm); the first image is collected by the first camera module, and the second image is collected by the second camera module; since the image information included in the second image (for example, a near-infrared image) is in the first image ( For example, a visible light image) cannot be obtained; similarly, the image information included in the third image cannot be obtained by the fourth image; therefore, by comparing the third image (for example, a visible light image) with the fourth image (for example, near-infrared light image) for fusion processing, the multi-spectral information fusion of near-infrared light image information and visible light image information can be realized, so that the fused image includes more detailed information; therefore, through the embodiment of the present application provided The image processing method can perform image enhancement on the image acquired by the camera module of the main camera, enhance the detailed information in the image, and improve the image quality.

It should be understood that the image quality of the N frames of the third image is higher than the image quality of the N frames of the first image may mean that the noise in the N frames of the third image is less than the noise in the N frames of the first image; or, by evaluating the image quality The algorithm evaluates N frames of the third image and N frames of the first image, and the evaluation result obtained is that the image quality of the N frames of the third image is higher than that of the N frames of the first image, etc., which is not limited in this application.

It should also be understood that the image quality of the M frames of the fourth image is higher than the image quality of the M frames of the second image may mean that the noise in the M frames of the fourth image is less than the noise in the M frames of the second image; or, through the image quality The evaluation algorithm evaluates the M frames of the fourth image and the M frames of the second image, and the obtained evaluation result is that the image quality of the M frames of the fourth image is higher than that of the M frames of the second image, etc. This application makes no limitation on this.

It should also be understood that the detailed information of the fused image is better than the detailed information of the N frames of the first image may mean that the detailed information in the fused image is more than the detailed information in any one of the first images in the N frames of the first image; or, the fused The detail information of the image being better than the detail information of the N frames of first images may mean that the definition of the fused image is better than that of any first image in the N frames of first images. For example, the detail information may include edge information, texture information, etc. of the object to be photographed.

In the embodiment of the present application, the third image of N frames and the fourth image of M frames can be fused based on the semantically segmented image to obtain a fused image; the fused local image information can be determined by introducing the semantically segmented image into the fusion process For example, the local image information in the third image of N frames and the fourth image of M frames can be selected through the semantic segmentation image for fusion processing, so that the local detail information of the fused image can be increased. In addition, by performing fusion processing on the third image (for example, visible light image) and the fourth image (for example, near-infrared light image), the multi-spectral information fusion of near-infrared light image information and visible light image information can be realized, so that the fusion The final image includes more detail information, which can enhance the detail information in the image.

Optionally, the M frames of the fourth image are obtained by performing second image processing on the second image collected by the near-infrared camera module or the infrared camera module; therefore, the M frames of the fourth image include the reflection information of the near-infrared light of the photographed object ; Since the near-infrared light has a high reflectivity to the green scene, the details of the green scene captured by the near-infrared camera module or the infrared module are more; the green scene image can be selected from the fourth image by semantically segmenting the image Regions are fused, so that the details of the green scene in the dark and light areas of the image can be enhanced.

Optionally, the fourth image of the M frames is obtained by performing second image processing on the second image collected by the near-infrared camera module or the infrared camera module; since the spectral range that the near-infrared camera module or the infrared camera module can obtain is Near-infrared light, the wavelength of the spectrum of near-infrared light is longer, so the diffraction ability of near-infrared light is stronger; for shooting scenes of cloud and fog or shooting scenes of distant objects, the near-infrared camera module or infrared camera module collects The image has a stronger sense of transparency, that is, the image includes more detailed information of distant objects (for example, texture information of distant mountains); it is possible to select a distant image area from the fourth image by semantically segmenting the image, The fusion processing is carried out with the nearby image area selected from the third image through the semantic segmentation image, so as to enhance the detailed information in the fusion image.

With reference to the first aspect, in some implementation manners of the first aspect, performing the second image processing on the M frames of the second image to obtain the M frames of the fourth image includes:

performing black level correction processing and/or phase defect correction processing on the M frames of second images to obtain M frames of fifth images;

Taking any third image of the N frames of third images as a reference, performing a first registration process on the M frames of fifth images to obtain the N frames of fourth images.

Optionally, the global registration process may refer to taking the third image of the first frame as a reference, and mapping the whole of each fourth image in the M frames of fourth images to the third image of the first frame.

Optionally, black level correction (black level correction, BLC) is used to correct the black level. The black level refers to the video signal level without a line of bright output on a calibrated display device. Phase defect pixel correction (PDPC) can include phase point correction (phase defect correction, PDC) and bad pixel correction (bad pixel correction, BPC); wherein, the bad pixels in BPC are randomly positioned bright or dark The number of points is relatively small, and BPC can be realized by filtering algorithm; compared with ordinary pixel points, phase points are bad points with fixed positions, and the number is relatively large; PDC needs to remove phase points through the known phase point list.

Optionally, the second image processing may also include but not limited to:

Automatic white balance processing (Automatic white balance, AWB), lens shading correction (Lens Shading Correction, LSC), etc.

Among them, the automatic white balance processing is used to make the white color can be restored to white by the camera at any color temperature; due to the influence of color temperature, white paper will be yellowish at low color temperature, and blue at high color temperature; the purpose of white balance is to So that the white object is R=G=B at any color temperature and appears white. Lens shading correction is used to eliminate the inconsistency between the color and brightness of the surrounding image and the center of the image caused by the lens optical system.

It should be understood that the second image processing may include black level correction, phase bad pixel correction, and other Raw domain image processing algorithms; the above-mentioned examples describe other Raw domain image processing algorithms with automatic white balance processing and lens shading correction. Other Raw domain image processing algorithms are not subject to any limitation.

With reference to the first aspect, in some implementation manners of the first aspect, the first step is performed on the M frames of fifth images based on any third image of the N frames of third images. Registration processing, to obtain the fourth image of the N frames, including:

Taking any third image of the N frames of third images as a reference, performing the first registration processing and up-sampling processing on the M frames of fifth images to obtain the N frames of fourth images.

In the embodiment of the present application, the resolution of the fourth image can be adjusted to be the same as that of the third image; thus, it is convenient to perform fusion processing on N frames of the third image and M frames of the fourth image.

Using any one of the third images in the N frames of third images as a reference, performing a first registration process on the M frames of the fifth image to obtain M frames of first registration images;

Using the arbitrary third image as a reference, perform a second registration process on the M frames of the first registration image to obtain the M frames of the fourth image.

With reference to the first aspect, in some implementation manners of the first aspect, the first registration is performed on the M frames of the fifth image based on any one frame of the third image in the N frames of third images Process to obtain the first registration image of M frames, including:

Performing the first registration processing and upsampling processing on the M frames of the fifth image based on any one of the third images in the N frames of third images to obtain the M frames of the first registration image .

With reference to the first aspect, in some implementation manners of the first aspect, the first registration processing is global registration processing.

With reference to the first aspect, in some implementation manners of the first aspect, the second registration processing is local registration processing.

With reference to the first aspect, in some implementation manners of the first aspect, performing first image processing on the N frames of first images to obtain N frames of third images includes:

Perform black level correction processing and/or phase defect correction processing on the N frames of first images to obtain the N frames of third images.

Optionally, the first image processing may also include but not limited to:

It should be understood that the first image processing may include black level correction, phase bad pixel correction, and other Raw domain image processing algorithms; the above-mentioned examples describe other Raw domain image processing algorithms with automatic white balance processing and lens shading correction. Other Raw domain image processing algorithms are not subject to any limitation.

With reference to the first aspect, in some implementation manners of the first aspect, the electronic device further includes an infrared flash lamp, and the image processing method further includes:

In a dark light scene, turn on the infrared flashlight, the dark light scene refers to a shooting scene in which the amount of light entering the electronic device is less than a preset threshold;

In response to the first operation, acquiring N frames of the first image and M frames of the second image includes:

When the infrared flashlight is turned on, the N frames of the first image and the M frames of the second image are acquired.

With reference to the first aspect, in some implementations of the first aspect, the first interface includes a second control; the turning on the infrared flash in a dark scene includes:

detecting a second operation on the second control;

The infrared flashlight is turned on in response to the second operation.

In the embodiment of the present application, the infrared flashlight in the electronic device can be turned on; since the electronic device can include the first camera module and the second module, when the infrared flashlight is turned on, the reflected light of the object increases, so that The amount of light entering the second camera module increases; thereby increasing the detail information of the second image acquired through the second camera module; the first camera module and the second camera module are collected by the image processing method of the embodiment of the application The image fusion processing can be performed on the image acquired by the camera module of the main camera, and the detailed information in the image can be improved. In addition, the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.

With reference to the first aspect, in some implementation manners of the first aspect, performing fusion processing on the N frames of the third image and the M frames of the fourth image based on the semantically segmented image to obtain a fusion image, including :

Based on the semantically segmented image, the third image of N frames and the fourth image of M frames are fused by an image processing model to obtain the fused image, and the image processing model is a pre-trained neural network.

With reference to the first aspect, in some implementation manners of the first aspect, the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.

Optionally, the first operation may refer to a click operation on a control indicating to take a photo in the photo taking interface.

With reference to the first aspect, in some implementation manners of the first aspect, the first interface refers to a video recording interface, and the first control refers to a control for instructing video recording.

Optionally, the first operation may refer to a click operation on a control indicating to record a video in the video recording interface.

With reference to the first aspect, in some implementation manners of the first aspect, the first interface refers to a video call interface, and the first control refers to a control for instructing a video call.

Optionally, the first operation may refer to a click operation on a control indicating a video call in the video call interface.

It should be understood that the above-mentioned example illustrates the first operation as a click operation; the first operation may also include a voice instruction operation, or other operations instructing the electronic device to take a photo or make a video call; make any restrictions.

In a second aspect, an electronic device is provided, the electronic device includes one or more processors, memory, a first camera module and a second camera module; the second camera module is a near-infrared camera module Or an infrared camera module, the memory is coupled with the one or more processors, the memory is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors call the computer instructions to cause the device to perform:

detecting a first operation on the first control;

Save the target image; where,

The obtaining the target image based on the N frames of the first image and the M frames of the second image includes:

performing second image processing on the M frames of second images to obtain M frames of fourth images, where the image quality of the M frames of the fourth images is higher than the image quality of the M frames of the second images;

Performing third image processing on the fused image to obtain a target image.

Optionally, the first camera module may include a first lens, a first lens and an image sensor, and the spectral range through which the first lens can pass includes visible light (400nm-700nm).

With reference to the second aspect, in some implementation manners of the second aspect, the one or more processors call the computer instructions so that the electronic device executes:

Optionally, the second image processing may also include but not limited to:

With reference to the second aspect, in some implementation manners of the second aspect, the first registration process is a global registration process.

With reference to the second aspect, in some implementation manners of the second aspect, the second registration processing is local registration processing.

Optionally, the first image processing may also include but not limited to:

With reference to the second aspect, in some implementation manners of the second aspect, the electronic device includes an infrared flashlight, and the one or more processors call the computer instructions to make the electronic device execute:

In the case of turning on the infrared flashlight, the N frames of the first image and the M frames of the second image are acquired.

With reference to the second aspect, in some implementation manners of the second aspect, the first interface includes a second control, and the one or more processors call the computer instructions to make the electronic device execute:

detecting a second operation on the second control;

The infrared flashlight is turned on in response to the second operation.

With reference to the second aspect, in some implementation manners of the second aspect, the semantically segmented image is obtained by processing the third image in the first frame of the N frames of third images by using a semantic segmentation algorithm.

With reference to the second aspect, in some implementation manners of the second aspect, the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.

With reference to the second aspect, in some implementation manners of the second aspect, the first interface refers to a video recording interface, and the first control refers to a control for instructing video recording.

With reference to the second aspect, in some implementation manners of the second aspect, the first interface refers to a video call interface, and the first control refers to a control for instructing a video call.

In a third aspect, an electronic device is provided, including a module/unit for executing the first aspect or any image processing method in the first aspect.

According to a fourth aspect, an electronic device is provided, and the electronic device includes: one or more processors, a memory, a first camera module, and a second camera module; the memory is coupled to the one or more processors , the memory is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors call the computer instructions to make the electronic device perform the first aspect or any of the first aspects An image processing method.

In a fifth aspect, a chip system is provided, the chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call a computer instruction so that the electronic device executes the first aspect Or any image processing method in the first aspect.

In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores computer program code, and when the computer program code is run by an electronic device, the electronic device executes the first aspect or the first Any image processing method in the aspect.

In a seventh aspect, a computer program product is provided, the computer program product comprising: computer program code, when the computer program code is run by an electronic device, the electronic device is made to execute the first aspect or any one of the first aspects. An image processing method.

In addition, in the embodiment of the present application, since the spectral range that the second camera module can acquire is near-infrared light, the infrared light image collected by the second camera module is a grayscale image, and the grayscale image is used to represent The actual value of brightness; since the spectral range that the first camera module can obtain is visible light, the brightness value in the visible light image collected by the first camera module is discontinuous, and it is usually necessary to predict the discontinuous brightness value; When the infrared light image (true value of brightness) is used as a guide to demosaic the visible light image, it can effectively reduce the false texture appearing in the image.

Description of drawings

FIG. 1 is a schematic diagram of a hardware system applicable to an electronic device of the present application;

Fig. 2 is a schematic diagram of a software system applicable to the electronic device of the present application;

FIG. 3 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

FIG. 4 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

Fig. 5 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 6 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Fig. 7 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of the first registration process and the up-sampling process provided by the embodiment of the present application;

FIG. 9 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 10 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Fig. 11 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Fig. 12 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Fig. 13 is a schematic diagram showing the effect of the image processing method provided by the embodiment of the present application;

Fig. 14 is a schematic diagram of a graphical user interface applicable to the embodiment of the present application;

Fig. 15 is a schematic diagram of an optical path of a shooting scene applicable to an embodiment of the present application;

Fig. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 17 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In the embodiments of the present application, the following terms "first", "second", "third", and "fourth" are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating Indicates the number of technical characteristics.

In order to facilitate the understanding of the embodiments of the present application, firstly a brief description is given of related concepts involved in the embodiments of the present application.

1. Near infrared light (near infrared, NIR)

Near-infrared light refers to electromagnetic waves between visible light and mid-infrared light; the near-infrared light region can be divided into two regions: near-infrared short-wave (780nm-1100nm) and near-infrared long-wave (1100nm-2526nm).

2. Main camera module

The main camera module refers to a camera module that receives visible light in a spectral range; for example, the sensor included in the main camera module receives a spectral range of 400nm to 700nm.

3. Near-infrared camera module

A near-infrared camera module refers to a camera module that receives near-infrared light in a spectral range; for example, a sensor included in a near-infrared camera module receives a spectral range of 700 nm to 1100 nm.

4. High-frequency information of images

The high-frequency information of an image refers to the region where the gray value changes drastically in the image; for example, the high-frequency information in the image includes edge information, texture information, etc. of an object.

5. Low-frequency information of images

The low-frequency information of the image refers to the area where the gray value changes slowly in the image; for an image, the part except the high-frequency information is low-frequency information; for example, the low-frequency information of the image can include the content information within the edge of the object.

6. The detail layer of the image

The detail layer of the image includes high-frequency information of the image; for example, the detail layer of the image includes edge information, texture information, etc. of the object.

7. The base layer of the image

The base layer of the image includes the low-frequency information of the image; for an image, the part except the detail layer is the base layer; for example, the base layer of the image includes the content information within the edge of the object.

8. Image registration

Image registration refers to the process of matching and superimposing two or more images acquired at different times, different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.).

9. Lighting Value (LV)

The brightness value is used to estimate the ambient brightness, and its specific calculation formula is as follows:

Among them, Exposure is the exposure time; Aperture is the aperture size; Iso is the sensitivity; Luma is the average value of Y in the XYZ space of the image.

10. Color correction matrix (color correction matrix, CCM)

A color correction matrix is used to calibrate the accuracy of colors other than white.

11. Three dimension look up table (Three dimension look up table, 3D LUT)

Three-dimensional lookup tables are widely used in image processing; for example, lookup tables can be used for image color correction, image enhancement, or image gamma correction; for example, LUTs can be loaded in image signal processors, and the original image can be processed according to the LUT table Processing, realize the pixel value mapping of the original image frame and change the color style of the image, so as to achieve different image effects.

12. Global tone mapping (GTM)

Global tone mapping is used to solve the problem of uneven distribution of gray values in high dynamic images.

13. Gamma processing

Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.

14. Neural network

A neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer, To extract the features of the local receptive field, the local receptive field can be an area composed of several neural units.

15. Back propagation algorithm

The neural network can use the error back propagation (back propagation, BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, passing the input signal forward until the output will generate an error loss, and updating the parameters in the initial neural network model by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

The image processing method and the electronic device in the embodiment of the present application will be described below with reference to the accompanying drawings.

Fig. 1 shows a hardware system applicable to the electronic equipment of this application.

The electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, a vehicle electronic device, an augmented reality (augmented reality, AR) device, a virtual reality (virtual reality, VR) device, a notebook computer, a super mobile personal computer ( ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), projector, etc., the embodiment of the present application does not impose any limitation on the specific type of the electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It should be noted that the structure shown in FIG. 1 does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG. 2 , or the electronic device 100 may include a combination of some of the components shown in FIG. 2 , or , the electronic device 100 may include subcomponents of some of the components shown in FIG. 1 . The components shown in FIG. 1 can be realized in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor) , ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, neural network processor (neural-network processing unit, NPU). Wherein, different processing units may be independent devices or integrated devices. The controller can generate operation control signals according to the instruction opcode and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

Exemplarily, the processor 110 may be configured to execute the image processing method of the embodiment of the present application; for example, display a first interface, the first interface includes a first control; detect a first operation on the first control; respond to the first Operation, acquire N frames of the first image and M frames of the second image, the first image is the image collected by the first camera module, the second image is the image collected by the second camera module, and both N and M are greater than or equal to 1 A positive integer; based on the first image of N frames and the second image of M frames, the target image is obtained; the target image is saved; wherein, based on the first image of N frames and the second image of M frames, the target image is obtained, including: the first image of N frames Carrying out the first image processing on an image to obtain the third image of N frames, the image quality of the third image of N frames is higher than the image quality of the first image of N frames; performing the second image processing on the second image of M frames to obtain the first image of M frames Four images, the image quality of the fourth image of the M frame is higher than the image quality of the second image of the M frame; based on the semantic segmentation image, the third image of the N frame and the fourth image of the M frame are fused to obtain a fusion image, a semantic segmentation image It is obtained based on any frame image in the first image of N frames or any frame image in the second image of M frames, and the detailed information of the fused image is better than the detailed information of the first image of N frames; the third image processing is performed on the fused image , to get the target image.

The connection relationship between the modules shown in FIG. 1 is only a schematic illustration, and does not constitute a limitation on the connection relationship between the modules of the electronic device 100 . Optionally, each module of the electronic device 100 may also adopt a combination of various connection modes in the foregoing embodiments.

The wireless communication function of the electronic device 100 may be realized by components such as the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, and a baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The electronic device 100 can realize the display function through the GPU, the display screen 194 and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

Display 194 may be used to display images or video.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 , and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can optimize the algorithm of image noise, brightness and color, and ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard red green blue (red green blue, RGB), YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3 and MPEG4.

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x-axis, y-axis and z-axis) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used in scenarios such as navigation and somatosensory games.

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally x-axis, y-axis and z-axis). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100 as an input parameter for application programs such as horizontal and vertical screen switching and pedometer.

The distance sensor 180F is used to measure distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, for example, in a shooting scene, the electronic device 100 can use the distance sensor 180F for distance measurement to achieve fast focusing.

The ambient light sensor 180L is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing the application lock, taking pictures, and answering incoming calls.

The touch sensor 180K is also referred to as a touch device. The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch screen. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor 180K may transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display screen 194 . In some other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 and disposed at a different position from the display screen 194 .

The hardware system of the electronic device 100 has been described in detail above, and the software system of the image electronic device 100 will be introduced below.

Fig. 2 is a schematic diagram of the software system of the device provided by the embodiment of the present application.

As shown in FIG. 2 , the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

The application layer 210 may include a camera; optionally, the application program 210 may also include application programs such as a gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer 220 provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer; the application framework layer may include some predefined functions.

For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera equipment. Wherein, the camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an interface for accessing the camera.

The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can include the camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call the algorithms in the camera algorithm library.

For example, a library of camera algorithms may include software algorithms for image processing.

The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver; a digital signal processor driver, a graphics processor driver, or a central processing unit driver.

The hardware layer 250 may include camera devices as well as other hardware devices.

For example, the hardware layer 250 includes a camera device, a digital signal processor, a graphics processor or a central processing unit; for example, the camera device may include an image signal processor, and the image signal processor may be used for image processing.

At present, the spectral range obtained by the main camera module on the terminal device is visible light (400nm~700nm); Due to the poor light conditions of the shooting scene and the small amount of light entering the electronic device, there is a problem that some image detail information is lost in the image obtained by the camera module of the main camera.

In view of this, an embodiment of the present application provides an image processing method, which is applied to an electronic device; the electronic device may include a main camera module and a near-infrared camera module, wherein the spectral range that the main camera module can obtain is Including visible light (400nm-700nm); the spectral range that the near-infrared camera module can obtain is near-infrared light (700nm-1100nm); because the image collected by the near-infrared camera module includes the reflection information of the subject to near-infrared light; The image collected by the main camera module is fused with the image collected by the near-infrared camera module, which can realize the multi-spectral information fusion of the image information of near-infrared light and the image information of visible light, so that the fused image includes more details information; therefore, through the image processing method provided in the embodiment of the present application, the detailed information in the image can be enhanced.

The application scenario of the image processing method provided by the embodiment of the present application will be illustrated below with reference to FIG. 3 .

Exemplarily, the image processing method in the embodiment of the present application can be applied to the field of photography (for example, single-view photography, dual-view photography, etc.), recording video field, video call field or other image processing fields; A dual-camera module is used, and the dual-camera module includes a camera module that can obtain visible light and a camera module that can obtain near-infrared light (for example, a near-infrared camera module, or an infrared camera module); Visible light images and near-infrared light images are processed and fused to obtain images with enhanced image quality; image processing methods in the embodiments of the present application can enhance detailed information in the images and improve image quality.

In one example, as shown in FIG. 3 , when the embodiment of the present application is applied to shooting scenery (for example, cloud and fog scenes) under sunlight, since the spectral range that the near-infrared camera module can obtain is near-infrared light, which is similar to the spectral range of visible light The wavelength of the spectrum that can be obtained by the near-infrared camera module is longer, so the diffraction ability is stronger. It is shown as the image obtained by the image processing method provided by the embodiment of the present application after the image is collected by the main camera module and the near-infrared camera module; the image shown in Figure 3 is rich in detail information and can clearly display the details of the mountains Information; through the image processing method provided by the embodiment of the present application, the image acquired by the main camera module can be enhanced to enhance the detailed information in the image.

Exemplarily, the terminal device shown in FIG. 3 may include a first camera module, a second camera module, and an infrared flashlight; wherein, the spectral range that the first camera module can acquire is visible light (400nm-700nm); the second The spectral range that the camera module can obtain is near-infrared light (700nm-1100nm).

In one example, when the embodiment of the present application is applied to a scene including a green scene to take pictures, for a dark area with less incoming light, since the near-infrared light has a high reflectivity to the green scene, the camera module of the main camera and the The green scene captured by the near-infrared camera module has more detail information, which can enhance the detail information of the green scene in the dark area of the image.

In one example, when the embodiment of the present application is applied to night scene portrait shooting, the infrared flashlight in the electronic device can be turned on. For example, the portrait can include the subject's face, eyes, nose, mouth, ears, eyebrows, etc.; because the electronic device It includes the main camera module and the near-infrared camera module. When the infrared flashlight is turned on, the reflected light of the subject increases, which increases the amount of light entering the near-infrared camera module; thus making the photos taken by the near-infrared camera module The detailed information of the portrait is increased, and the images collected by the main camera module and the near-infrared camera module are fused through the image processing method of the embodiment of the application, and the image acquired by the main camera module can be image enhanced to improve the image quality. details in . In addition, the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.

Optionally, the electronic device can turn off the near-infrared camera module when detecting food or a portrait.

For example, a food shooting scene may include multiple foods, and the near-infrared camera module may collect images of some of the foods in the multiple foods; for example, the multiple foods may be peaches, apples, or watermelons, etc., and the near-infrared camera module may collect Images of peaches and apples are collected, and images of watermelons are not collected.

Optionally, the near-infrared camera module can display prompt information, prompting whether to enable the near-infrared camera module; the near-infrared camera module can only be enabled to collect images after the user authorizes the near-infrared camera module to be activated.

In one example, the image processing method of the present application can be applied to a folding screen terminal device; for example, the folding screen terminal device can include an outer screen and an inner screen; the angle between the outer screen and the inner screen of the folding screen terminal device When the temperature is zero, the preview image can be displayed on the external screen, as shown in (a) in Figure 4; when the angle between the external screen and the internal screen of the folding screen terminal device is an acute angle, the preview image can be displayed on the external screen , as shown in (b) in Figure 4; when the angle between the outer screen and the inner screen of the folding screen terminal device is an obtuse angle, one side of the inner screen can display a preview image, and the other side can display a preview image for Indicate the control for shooting, as shown in (c) in Figure 4; when the angle between the outer screen and the inner screen of the folding screen terminal device is 180 degrees, the preview image can be displayed on the inner screen, as shown in Figure 4 As shown in (d); the above preview image may be obtained by processing the collected image through the image processing method provided in the embodiment of the present application. Exemplarily, the folding screen terminal device shown in FIG. 4 may include a first camera module, a second camera module, and an infrared flashlight; wherein, the spectral range that the first camera module can obtain is visible light (400nm-700nm); The spectral range that the second camera module can acquire is near-infrared light (700nm-1100nm).

It should be understood that the foregoing is an illustration of an application scenario, and does not limit the application scenario of the present application in any way.

The image processing method provided by the embodiment of the present application will be described in detail below with reference to FIG. 5 and FIG. 15 .

FIG. 5 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1; the method 200 includes step S201 to step S205, which will be described in detail below respectively.

It should be understood that the image processing method shown in FIG. 5 is applied to an electronic device, and the electronic device includes a first camera module and a second camera module, and the spectrum acquired by the second camera module is a near-infrared camera module or an infrared camera module. group (for example, the acquired spectral range is 700nm ~ 1100nm).

Optionally, the first camera module may be a visible light camera module (for example, the acquired spectral range is 400nm-700nm), or the first camera module may be other camera modules capable of acquiring visible light.

Step S201, displaying a first interface, where the first interface includes a first control.

Optionally, the first interface may refer to the photographing interface of the electronic device, and the first control may refer to a control in the photographing interface for instructing photographing, as shown in FIG. 3 or FIG. 4 .

Optionally, the first interface may refer to a video recording interface of the electronic device, and the first control may refer to a control in the video recording interface for instructing to record a video.

Optionally, the first interface may refer to a video call interface of the electronic device, and the first control may refer to a control on the video call interface used to indicate a video call.

Step S202, detecting a first operation on the first control.

Step S203, in response to the first operation, acquire N frames of the first image and M frames of the second image.

Wherein, the N frames of the first image can be images collected by the first camera module, and the M frames of the second image are images collected by the second camera module; the second camera module is a near-infrared camera module or an infrared camera module. A camera module (for example, the acquired spectrum ranges from 700nm to 1100nm); N and M are positive integers greater than 1.

Optionally, the first image and the second image may refer to images in Raw color space.

For example, the first image may refer to an RGB image in a Raw color space; the second image may refer to an NIR image in a Raw color space.

Step S204, based on the N frames of the first image and the M frames of the second image, the target image is obtained.

Wherein, based on the first image of N frames and the second image of M frames, obtaining the target image may include the following steps:

Carrying out the first image processing on the first image of N frames to obtain the third image of N frames, the image quality of the third image of N frames is higher than the image quality of the first image of N frames; performing the second image processing on the second image of M frames, The fourth image of M frames is obtained, and the image quality of the fourth image of M frames is higher than the image quality of the second image of M frames; based on the semantic segmentation image, the third image of N frames and the fourth image of M frames are fused to obtain The fusion image, the semantic segmentation image is obtained based on any frame image in the first image of N frames or any frame image in the second image of M frames, and the detailed information of the fused image is better than the detailed information of the first image of N frames; The third image processing is performed on the fused image to obtain the target image.

It should be understood that the image quality of the N frames of the third image is higher than the image quality of the N frames of the first image may mean that the noise in the N frames of the third image is less than the noise in the N frames of the first image; or, by evaluating the image quality The algorithm evaluates N frames of the third image and N frames of the first image, and the evaluation result obtained is that the image quality of the N frames of the third image is higher than that of the N frames of the first image, etc., which is not limited in this application. Evaluation of image quality may include, for example, evaluation of aspects such as exposure, sharpness, color, texture, noise, anti-shake, flash, focus, and/or artifacts.

It should also be understood that the detailed information of the fused image is better than the detailed information of the N frames of the first image may mean that the detailed information in the fused image is more than the detailed information in any one of the first images in the N frames of the first image; or, the fused The detail information of the image being better than the detail information of the N frames of first images may mean that the definition of the fused image is better than that of any first image in the N frames of first images. It can also be other situations, which are not limited in this application. For example, the detail information may include edge information and texture information of the subject (for example, hair edges, face details, clothes folds, edges of each tree of a large number of trees, branches and leaves of green plants, etc.).

For example, the fourth image of the M frames is obtained by performing second image processing on the second image collected by the near-infrared camera module or the infrared camera module; therefore, the fourth image of the M frames includes the reflection information of the object to the near-infrared light; The near-infrared light has a high reflectivity to the green scene, so the details of the green scene captured by the near-infrared camera module or the infrared module are more detailed; the image area of the green scene can be selected from the fourth image by semantically segmenting the image. Fusion processing, so as to enhance the detail information of the green scene in the dark and light area of the image.

For example, the fourth image of the M frames is obtained by performing second image processing on the second image collected by the near-infrared camera module or the infrared camera module; light, the near-infrared light spectrum has a longer wavelength, so the near-infrared light has a stronger diffraction ability; for cloud and fog shooting scenes or shooting scenes of distant objects, the images collected by the near-infrared camera module or infrared camera module The sense of transparency is stronger, that is, the image includes more detailed information of distant objects (for example, the texture information of distant mountains); it is possible to select a distant image area from the fourth image through semantic segmentation, and through The semantic segmentation image selects the nearby image area from the third image for fusion processing, so as to enhance the detailed information in the fusion image.

Optionally, the N frames of the first image may refer to N frames of Raw images (for example, RGGB images) collected by the first camera module; the first image processing of the N frames of the first image may obtain the N frames of the third image; Wherein, the first image processing may include black level correction processing and/or phase bad point correction processing.

Among them, black level correction (black level correction, BLC) is used to correct the black level, and the black level refers to the video signal level without a line of bright output on a calibrated display device. Phase defect pixel correction (PDPC) can include phase point correction (phase defect correction, PDC) and bad pixel correction (bad pixel correction, BPC); wherein, the bad pixels in BPC are randomly positioned bright or dark The number of points is relatively small, and BPC can be realized by filtering algorithm; compared with ordinary pixel points, phase points are bad points with fixed positions, and the number is relatively large; PDC needs to remove phase points through the known phase point list.

Optionally, the first image processing may also include but not limited to:

Optionally, performing second image processing on M frames of second images to obtain M frames of fourth images, including:

Perform black level correction processing and/or phase bad point correction processing on the second image of the M frames to obtain the fifth image of the M frame; based on any third image of the third image of the N frames, the fifth image of the M frame The images are subjected to a first registration process to obtain N frames of fourth images.

Optionally, the second image processing may also include but not limited to:

In one example, the first camera module is a visible light camera module, and the second camera module is a near-infrared camera module or an infrared camera module; N frames of RGB Raw images can be collected through the first camera module, and N frames of RGB Raw images can be collected through the second camera module. The two-camera module can collect M frames of NIR Raw images; perform black level correction processing and/or phase bad point correction processing on N frames of RGB Raw images to obtain N frames of processed RGB Raw images; perform M frames of NIR Raw images Black level correction processing and/or phase dead point correction processing can obtain M frame processed NIR Raw images; since the two camera modules are not set at the same position, for the same shooting scene, the first camera module and the second camera The existence of the module has a certain baseline distance; therefore, the NIR Raw image after the M frame processing can be globally registered with any frame of the RGB Raw image after the N frame processing, and the M frame registered image can be obtained. The image; based on the semantic segmentation image, the RGB Raw image processed by N frames is fused with the registered image of M frames to obtain a fused image. Optionally, for specific steps, refer to the subsequent FIG. 9 .

Optionally, the global registration processing can refer to the global registration processing of the NIR Raw image after the M frame processing on the basis of any frame image in the RGB Raw image after the N frame processing; or, the global processing also It can refer to the global registration process performed on the RGB Raw image after N frame processing based on any frame of image in the NIR Raw image after M frame processing.

Optionally, in order to facilitate the fusion process, the image resolution of the M frames of the fourth image can be adjusted to be the same as the image resolution of the N frames of the third image; for example, by using any one of the N frames of the third image The three images are used as a reference, and the images after the registration of the M frames are subjected to up-sampling or down-sampling processing to obtain the fourth image of the M frames; or, the third image of any one of the fourth images of the M frames can be used as a reference. The processed RGB Raw image of N frames is subjected to up-sampling processing or down-sampling processing to obtain a third image of N frames.

Optionally, based on any one of the third images in the N frames of the third image, perform up-sampling or down-sampling processing on the registered images of the M frames to obtain the fourth image of the M frames. For specific procedures, please refer to the subsequent Image 8 is shown.

Perform black level correction processing and/or phase bad point correction processing on the second image of the M frames to obtain the fifth image of the M frame; based on any third image of the third image of the N frames, the fifth image of the M frame The first registration process is performed on the images to obtain M frames of the first registration image; taking any third image of the frame as a reference, the second registration process is performed on the M frames of the first registration image to obtain the M frames of the fourth image.

In one example, the first camera module can be a visible light camera module, and the second camera module can be a near-infrared camera module or an infrared camera module; N frames of RGB Raw images can be collected through the first camera module, and N frames of RGB Raw images can be collected through the first camera module. The second camera module can collect M frames of NIR Raw images; perform black level correction processing and/or phase bad point correction processing on N frames of RGB Raw images to obtain N frames of processed RGB Raw images; M frames of NIR Raw images Perform black level correction processing and/or phase dead point correction processing to obtain M frame processed NIR Raw images; since the two camera modules are not set at the same position, for the same shooting scene, the first camera module and the second camera module The existence of the camera module has a certain baseline distance; therefore, the NIR Raw image after the M frame processing can be globally registered on the basis of any frame of the RGB Raw image after the N frame processing, and the first frame of the M frame can be obtained. Register the image; further, any frame of image in the N frames of processed RGB Raw images can be used as a reference. Perform local registration processing on the M frames of the first registration image to obtain the second registration image; based on the semantic segmentation image, perform fusion processing on the processed RGB Raw image of the N frames and the second registration image to obtain the fusion image. Optionally, for specific steps, refer to the following figure 10 .

Optionally, local registration processing is further performed on the basis of the global registration processing, so that the local details in the M frames of the first registration image are subjected to image registration processing again; the local detail information of the fused image can be improved.

Optionally, in order to facilitate the fusion process, the image resolution of the M frames of the fourth image can be adjusted to be the same as the image resolution of the N frames of the third image; for example, by using any frame of the N frames of the third image The three images are used as a reference, and the images after the registration of the M frames are subjected to up-sampling or down-sampling processing to obtain the fourth image of the M frames; or, the third image of any one of the fourth images of the M frames can be used as a reference. The processed RGB Raw image of N frames is subjected to up-sampling processing or down-sampling processing to obtain a third image of N frames.

Optionally, based on the semantically segmented image, the third image of N frames and the fourth image of M frames are fused to obtain a fused image, including:

Based on the semantically segmented image, the third image of N frames and the fourth image of M frames are fused by an image processing model to obtain a fused image.

Exemplarily, the image processing model is a pre-trained neural network.

For example, a large amount of sample data and a loss function can be used to iteratively update the parameters of the neural network through the backpropagation algorithm to obtain an image processing model.

Step S205, saving the target image.

Optionally, a third image processing may be performed on the fused image to obtain a target image; the target image may refer to an image displayed on a display screen of an electronic device.

Exemplarily, the fused image may refer to an image in RGB color space; the target image may refer to an image sent by an electronic device to be displayed on a screen; the third image processing may include but not limited to: RGB domain image algorithm, or YUV domain image algorithm ; Optionally, refer to step S308 and step S309 shown in FIG. 6 .

Optionally, the electronic device may also include an infrared flashlight; in a dark scene, the infrared flashlight may be turned on; when the infrared flashlight is turned on, N frames of the first image and M frames of the second image may be acquired; wherein, the dark light The scene refers to a shooting scene in which the amount of light entering the electronic device is less than a preset threshold.

It should be understood that for dark scenes, the amount of light entering the electronic device is relatively small; after the electronic device turns on the infrared flashlight, it can increase the reflected light acquired by the second camera module, thereby increasing the amount of light entering the second camera module; so that the second The sharpness of the second image collected by the camera module increases; because the sharpness of the second image increases, the sharpness of the third image processed by the first image increases; because the sharpness of the third image increases, the image processing The sharpness of the fused image processed by the model on the third image and the fourth image is increased.

Optionally, since the brightness value of the electronic device is larger, it means that the amount of light entering the electronic device is more; the brightness value of the electronic device can be used to determine the amount of light entering the electronic device. When the brightness value of the electronic device is less than the preset brightness threshold, Then the electronic device turns on the infrared flashlight.

Among them, the brightness value is used to estimate the ambient brightness, and its specific calculation formula is as follows:

Optionally, the first interface of the electronic device may further include a second control; in a dark scene, the electronic device detects a second operation on the second control; in response to the second operation, the electronic device may turn on an infrared flash.

In an embodiment of the present application, the electronic device may include a first camera module and a second camera module, wherein the second camera module is a near-infrared camera module or an infrared camera module; for example, the available The spectral range is near-infrared light (700nm-1100nm); the first image is collected by the first camera module, and the second image is collected by the second camera module; since the image information included in the second image (for example, near-infrared image) It cannot be obtained in the first image (for example, visible light image); similarly, the image information included in the third image cannot be obtained in the fourth image; therefore, by combining the third image (for example, visible light image) with The fourth image (for example, a near-infrared light image) is fused, which can realize the multi-spectral information fusion of the image information of the near-infrared light and the image information of the visible light, so that the fused image includes more detailed information; therefore, by The image processing method provided in the embodiment of the present application can enhance detailed information in an image.

FIG. 6 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1 ; the image processing method includes step S301 to step S309 , and step S301 to step S309 will be described in detail below.

It should be understood that the image processing method shown in FIG. 6 can be applied to the electronic device shown in FIG. 1 , the electronic device includes a first camera module and a second camera module; the spectral range that the first camera module can obtain is Visible light (400nm-700nm); the spectral range that the second camera module can obtain is near-infrared light (700nm-1100nm).

Step S301. Obtain a first Raw image (for example, an example of the first image) through the first camera module.

Exemplarily, the first camera module may include a first lens, a first lens and an image sensor, and the spectral range through which the first lens can pass is visible light (400nm˜700nm).

Optionally, multiple frames of first Raw images (for example, N frames of first images) may be acquired in step 301 .

Step S302, acquiring a second Raw image (an example of the second image) through the second camera module.

It should be noted that, in the embodiment of the present application, the second Raw image collected by the second camera module may refer to a single-channel image; the second Raw image is used to represent the intensity information of photons superimposed together; for example, the second Raw images can be grayscale images in a single channel.

Optionally, the second Raw images acquired in step 302 may refer to multiple frames of second Raw images (for example, M frames of second images).

Optionally, the above step S301 and step S302 may be performed synchronously; that is, the first camera module and the second camera module may output frames synchronously, and obtain the first Raw image and the second Raw image respectively.

Step S303, black level correction and phase bad point correction.

Exemplarily, a third Raw image (an example of a third image) is obtained by performing black level correction and phase defect correction on the first Raw image.

Among them, black level correction (black level correction, BLC) is used to correct the black level, and the black level refers to the video signal level without a line of bright output on a calibrated display device. Phase defect pixel correction (phase defect pixel correction, PDPC) can include phase point correction (phase defect correction, PDC) and bad pixel correction (bad pixel correction, BPC); bad pixels in BPC are bright or dark spots at random positions, The number is relatively small; BPC can be implemented through a filtering algorithm, and PDC needs to perform phase points through a known phase point list.

Optionally, the black level correction and phase dead point correction are used as examples above; other image processing algorithms can also be performed on the first Raw image; for example, automatic white balance processing (Automatic white balance) can also be performed on the first Raw image , AWB) or Lens Shading Correction (Lens Shading Correction, LSC), etc.

Among them, the automatic white balance processing is used to make the white color can be restored to white by the camera at any color temperature; due to the influence of color temperature, white paper will be yellowish at low color temperature, and blue at high color temperature; the purpose of white balance is to So that the white object is R=G=B at any color temperature and appears white. Lens shading correction is used to eliminate the inconsistency between the color and brightness of the surrounding image and the center of the image caused by the lens optical system. It should be understood that the automatic white balance processing and lens shading correction are used as examples to describe other image processing algorithms, and this application does not make any limitation on other image processing algorithms.

Step S304, black level correction and phase bad point correction.

Exemplarily, a fourth Raw image (an example of a fifth image) is obtained by performing black level correction and phase defect correction on the second Raw image.

Optionally, step S303 and step S304 may not have timing requirements, or step S303 and step S304 may also be executed simultaneously.

Step S305, acquiring a semantically segmented image.

Exemplarily, a semantic segmentation algorithm may be used to process the first frame and the third Raw image to obtain a semantic segmentation image.

Optionally, the semantic segmentation algorithm may include a multi-instance segmentation algorithm; labels of various regions in the image may be output through the semantic segmentation algorithm. In the embodiment of the present application, a semantically segmented image can be obtained, and the semantically segmented image is used for the fusion processing of the image processing model in step S307; by introducing the semantically segmented image in the fusion process, some image regions can be selected from different images for Fusion processing, so as to increase the local detail information of the fusion image.

Exemplarily, a semantic segmentation algorithm may be used to process the fourth Raw image of the first frame to obtain a semantic segmentation image.

Optionally, in the embodiment of the present application, the semantic segmentation image can be obtained through the fourth Raw image, that is, the near-infrared image; since the near-infrared image has a better description ability for details, the semantic segmentation image detail information can be obtained through the near-infrared image richer.

Step S306, preprocessing.

Exemplarily, preprocessing is performed on the third Raw image, the fourth Raw image and the semantic segmentation image.

Optionally, the preprocessing may include performing upsampling processing and registration processing on the fourth Raw image. For example, the fourth Raw image may be upsampled and registered based on the third Raw image of the first frame to obtain the fifth image.

Optionally, the preprocessing may also include feature splicing processing; the feature splicing processing refers to the processing of superimposing the channel numbers of images.

It should be understood that since the resolution of the fourth Raw image is smaller than that of the third Raw image, it is necessary to perform upsampling processing on the fourth Raw image so that the resolution of the fourth Raw image is the same as that of the third Raw image; in addition, since the fourth Raw image The image is collected by the second camera module, and the third Raw image is collected by the first camera module; since the first camera module and the second camera module are respectively arranged in different positions in the electronic device, the first There is a certain baseline distance between the camera module and the second camera module, that is, there is a certain parallax between the image collected by the first camera module and the image collected by the second camera module. The images are processed for registration.

It should be understood that the above is an example of upsampling processing; if the resolution of the fourth Raw image is greater than the resolution of the third Raw image, the preprocessing process may include downsampling processing and registration processing; if the resolution of the fourth Raw image is The resolution is equal to the resolution of the third Raw image, and registration processing may be included in the preprocessing process; this embodiment of the present application does not make any limitation on this.

Exemplarily, a preprocessing process including upsampling and registration processing is used for illustration; the preprocessing in step S306 will be described in detail below in conjunction with FIG. 8 .

It should be understood that the fourth Raw image refers to the Raw image after performing black level correction and phase defect correction on the second Raw image, and the second Raw image refers to the Raw image collected by the second camera module; the third Raw image is Refers to the Raw image after performing black level correction and phase defect correction on the first Raw image, and the first Raw image refers to the Raw image collected by the first camera module.

Step S320, acquiring a fourth Raw image.

For example, the resolution size of the fourth Raw image is 7M.

Step S330, acquiring a third Raw image.

For example, the resolution size of the third Raw image is 10M.

Step S340, registration processing.

For example, registration processing is performed on the fourth Raw image with the third Raw image as a reference.

Exemplarily, the registration process in step S330 can be used to acquire the same pixels from the fourth Raw image as in the third Raw image; for example, the third Raw image is 10M, the fourth Raw image is 7M, and the third Raw image is The same pixels are 80% in the fourth Raw image; the registration process can be used to obtain 80% of the same pixels as the 10M third Raw image from the 7M fourth Raw image. Optionally, registration processing may be performed on multiple frames of fourth Raw images with the first frame of Raw images in the multiple frames of third Raw images as a reference.

Step S350, correction.

For example, correction processing is performed on the fourth Raw image to obtain a fifth Raw image.

Exemplarily, step S340 is used to perform up-sampling processing on the pixels in the fourth Raw image obtained in step S330 that are the same as those in the third Raw image to obtain a fifth Raw image with the same resolution and size as the fourth Raw image.

Optionally, image conversion may be performed on the fourth Raw image through an image transformation matrix (for example, a homography matrix), so that some pixels in the fourth Raw image are mapped to an image of the same size as the third Raw image; wherein, A homography matrix refers to a mapping between two planar projections of an image. Exemplarily, the black area in the third Raw image of 10M as shown in FIG. 8 represents an empty pixel.

Optionally, the preprocessing may further include performing feature extraction and splicing (contact) on the third Raw image, the fifth Raw image and the semantically segmented image.

It should be understood that the feature splicing process refers to the process of superimposing the channel numbers of images.

For example, assuming that the fifth Raw image is a 3-channel image, the third Raw image is a 3-channel image, and the semantic segmentation image is a single-channel image, then a 7-channel image is obtained after feature extraction and stitching.

Step S307, image processing model.

Exemplarily, the preprocessed image is input into the image processing model to obtain an output RGB image (an example of a fused image).

In one example, N frames of the third Raw image (an example of the third image), semantically segmented images, and M frames of the fifth Raw image (an example of the fourth image) can be input to the image processing model after preprocessing. Fusion processing; specific steps are shown in FIG. 9 .

In one example, N frames of the third Raw image (an example of the third image), the semantic segmentation image and the sixth Raw image (an example of the fourth image) of N frames can be input to the image processing model for fusion processing after preprocessing , wherein the sixth Raw image is a single-frame image, which is a single-frame image after fusion of the fifth Raw image of multiple frames based on the third Raw image in FIG. 10 ; the specific steps are shown in FIG. 10 .

It can be understood that the fifth Raw image/sixth Raw image in FIG. 6 and FIG. 7 is represented as the fifth Raw image or the sixth Raw image.

Optionally, the image processing model is a pre-trained neural network; the image processing model can be used for fusion processing and demosaic processing; for example, the image processing model can perform fusion processing and demosaic processing on the input Raw image based on the semantic segmentation image, Get the RGB image.

Optionally, the image processing model is a pre-trained neural network.

Optionally, the image processing model can be used for fusion processing and demosaic processing; for example, the image processing model can perform fusion processing and demosaic processing on the input Raw image based on the semantically segmented image to obtain an RGB image.

Optionally, the image processing model can also be used for denoising processing; since the noise of the image mainly comes from Poisson distribution and Gaussian distribution, when denoising processing is performed through multiple frames of images, when the multi-frame images are superimposed and averaged, the Gaussian distribution can be is approximately 0; therefore, denoising through multiple frames of images can improve the denoising effect of the image.

Exemplarily, in the embodiment of the present application, the images input into the image processing model may include N frames of the third Raw image (visible light image) and M frames of the fifth Raw image (near-infrared light image); or, the N frames of the first Raw image Three Raw images (visible light image) and the sixth Raw image (near-infrared light image); when performing demosaic processing, since the infrared light image is a grayscale image, it represents the true value of brightness; the visible light image is a Raw image in Bayer format , the luminance value in the Raw image in the Bayer format is discontinuous, and the brightness of the discontinuous area is usually predicted by an interpolation method; therefore, the Raw image in the Bayer format can be guided to demosaic by the real value of the luminance in the near-infrared image, so that It can effectively reduce the pseudo texture appearing in the image.

In the embodiment of the present application, the spectral range of the near-infrared image is 700nm to 1100nm, and the spectral range of the visible light image is 400nm to 700nm, since the image information included in the near-infrared image cannot be obtained in the visible light image; therefore, by The Raw image (visible light image) collected by the first camera module and the Raw image (near-infrared light image) collected by the second camera module are processed and then fused, so that the image information of near-infrared light and the image information of visible light can be realized. The fusion of multi-spectral information makes the fused image include more detailed information.

Optionally, the S307 can also output a Raw image (Raw color space), and then convert the Raw image into an RGB image (RGB color space) through other steps.

Step S308, RGB image processing.

Exemplarily, RGB domain algorithm processing is performed on the RGB image.

Optionally, RGB domain algorithm processing may include but not limited to:

Color correction matrix processing, or three-dimensional lookup table processing, etc.

Among them, the color correction matrix (color correction matrix, CCM) is used to calibrate the accuracy of colors other than white. The three-dimensional look-up table (Look Up Table, LUT) is widely used in image processing; for example, the look-up table can be used for image color correction, image enhancement or image gamma correction, etc.; for example, the LUT can be loaded in the image signal processor, according to the LUT The table can perform image processing on the original image, and realize the color style of the original image mapped to other images, so as to achieve different image effects.

Optionally, RGB image processing may be performed according to the semantically segmented image; for example, brightness processing may be performed on different regions in the RGB image according to the semantically segmented image.

It should be noted that the color correction matrix processing and three-dimensional lookup table processing are used as examples for illustration; this application does not make any limitation on RGB image processing.

Step S309, YUV image processing.

Exemplarily, the RGB image is converted into the YUV domain and processed by an algorithm in the YUV domain to obtain the target image.

Optionally, YUV domain algorithm processing may include but not limited to:

Global tone mapping processing or gamma processing, etc.

Among them, Global tone mapping (Global tone mapping, GTM) is used to solve the problem of uneven gray value distribution of high dynamic images. Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.

It should be noted that the global tone mapping processing and gamma processing are used as examples for illustration above; this application does not make any limitation on YUV image processing.

Optionally, part or all of step S307, step S308 and step S309 may be executed in an image processing model.

It should be understood that, when the electronic device is in a non-dark scene, the image processing method provided by the embodiment of the present application may be executed through the above steps S301 to S309.

Optionally, the electronic device may also include an infrared flashlight; when the electronic device is in a dark scene, that is, when the amount of light entering the electronic device is less than a preset threshold (for example, it can be judged according to the brightness value), the electronic device may execute Step S310 shown in Figure 7 turns on the infrared flashlight; after the infrared flashlight is turned on, the first Raw image is acquired by the first camera module, the second Raw image is acquired by the second camera module, and step S311 as shown in Figure 7 is executed Go to step S319; it should be understood that steps S310 to S319 are applicable to the relevant descriptions of steps S301 to S309, and will not be repeated here.

Among them, Exposure is the exposure time; Aperture is the aperture size; Iso is the sensitivity; Luma is the average value of Y in the XYZ space of the image. For example, in a dark scene, the electronic device can focus first after detecting the shooting instruction, and perform scene detection synchronously; after recognizing the dark scene and completing the focus, the infrared flashlight can be turned on. After the infrared flashlight is turned on, the first Raw The frame of the image and the second Raw image can be synchronized.

It should be understood that the amount of light entering the electronic device is relatively small for dark scenes; after the electronic device turns on the infrared flashlight, it can increase the reflected light acquired by the second camera module, thereby increasing the light entering of the second camera module; making the second camera The sharpness of the second Raw image collected by the module increases; due to the increased sharpness of the second Raw image, the sharpness of the fourth Raw image obtained through the second Raw image increases; due to the increased sharpness of the fourth Raw image, This increases the clarity of the fused image.

In an embodiment of the present application, the electronic device may include a first camera module and a second camera module, wherein the spectral range that the first camera module can acquire includes visible light (400nm-700nm); the second camera module The spectral range that can be obtained is near-infrared light (700nm～1100nm); the first image is collected by the first camera module, and the second image is collected by the second camera module; since the second image (for example, near-infrared image) includes The image information of the first image cannot be obtained in the first image (for example, visible light image); similarly, the image information included in the third image cannot be obtained in the fourth image; therefore, by analyzing the third image (for example, visible light image) and the fourth image (for example, a near-infrared light image) are fused, which can realize the multi-spectral information fusion of the image information of the near-infrared light and the image information of the visible light, so that the fused image includes more detailed information; Therefore, the detailed information in the image can be enhanced through the image processing method provided in the embodiment of the present application.

Step S306 to step S307 shown in FIG. 6 will be described in detail below with reference to FIG. 9 and FIG. 10 .

Implementation method one

In one example, image processing may be performed on the multi-frame first Raw images collected by the first camera module and the multi-frame second Raw images collected by the second camera module, so as to obtain an image with enhanced detail information; wherein, the image processing It may include but not limited to: noise reduction processing, demosaic processing or fusion processing.

FIG. 9 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1; the image processing method includes steps S401 to S406, and the steps S401 to S406 will be described in detail below.

Step S401, acquiring multiple frames of a third Raw image (an example of a third image).

Exemplarily, multiple frames of first Raw images can be obtained through the first camera module (400nm-700nm), and multiple frames of first Raw images can be subjected to black level correction and phase defect correction processing to obtain multiple frames of third Raw images .

Step S402, acquiring multiple frames of the fifth Raw image (an example of the fourth image).

Exemplarily, multiple frames of second Raw images can be acquired through the second camera module (700nm-1100nm); black level correction and phase defect correction processing are performed on multiple frames of second Raw images to obtain multiple frames of fourth Raw images ; Using the third Raw image as a reference, perform registration processing on the fourth Raw image to obtain a fifth Raw image.

Step S403, acquiring a semantically segmented image.

Exemplarily, a semantically segmented image can be obtained through a semantically segmented algorithm.

Optionally, the first frame of the third Raw image among the multiple frames of the third Raw image may be processed by a semantic segmentation algorithm to obtain the semantic segmentation image.

Optionally, the first frame of the fourth Raw image among the multiple frames of the fourth Raw image may be processed according to the semantic segmentation algorithm to obtain the semantic segmentation image.

In the embodiment of the present application, based on the semantically segmented image, fusion processing can be performed on the third Raw image of multiple frames and the fifth Raw image of multiple frames to obtain a fused image; by introducing the semantically segmented image in the fusion process, the local part of the fusion can be determined Image information; for example, partial image information in multiple frames of the third Raw image and multiple frames of the fifth Raw image may be selected for fusion processing through semantically segmented images, so as to increase the local detail information of the fused image. In addition, by performing fusion processing on the third Raw image (for example, a visible light image) and the fifth Raw image (for example, a near-infrared light image), multispectral information fusion of near-infrared light image information and visible light image information can be realized, The fused image includes more detailed information, and the detailed information in the image can be enhanced.

Step S404, feature splicing processing.

Exemplarily, the multi-frame fifth Raw image, the multi-frame third Raw image and the semantic segmentation image are subjected to feature splicing processing to obtain multi-channel image features.

For example, assuming that the fifth aw image is a single-channel image, the third Raw image is a three-channel image, and the semantic segmentation image is a single-channel image, then image features of five channels can be obtained after feature splicing.

Step S405, image processing model.

Exemplarily, multi-channel image features are input to an image processing model for fusion processing.

Exemplarily, the image processing model can be used to fuse images in the Raw color space; the image processing model is a pre-trained neural network.

Optionally, the image processing model can also be used for denoising processing and demosaic processing; for example, the image processing model can perform denoising processing and demosaicing on multiple frames of the third Raw image and multiple frames of the fifth Raw image based on the semantic segmentation image processing and fusion processing.

It should be understood that since the noise of the image mainly comes from the Poisson distribution and the Gaussian distribution, when performing denoising processing through multiple frames of images, the Gaussian distribution can be approximately 0 when multiple frames of images are superimposed and averaged; therefore, denoising through multiple frames of images Noise processing can improve the image denoising effect.

In the embodiment of the present application, the fifth Raw image of multiple frames is an infrared light image; the near-infrared light image is a single-channel grayscale image, and the grayscale image is used to represent the true value of brightness; the third Raw image of multiple frames It is a visible light image; the brightness value in the visible light image is discontinuous, and it is usually necessary to predict the discontinuous brightness value; when the near-infrared light image (the real value of brightness) is used as a guide to demosaic the visible light image, it can effectively reduce Pseudo-textures appearing in images.

Step S406, RGB image.

Exemplarily, the image processing model outputs an RGB image (an example of a fused image).

In the embodiment of the present application, the spectral range that the first camera module can acquire is visible light from 400nm to 700nm, and the spectral range that the second camera module can acquire is near-infrared light from 700nm to 1100nm; Information cannot be obtained from visible light images; therefore, by processing the Raw images (visible light images) collected by the first camera module and the Raw images (near-infrared light images) collected by the second camera module, and then performing fusion processing, it is possible to achieve The multi-spectral information fusion of the image information of near-infrared light and the image information of visible light makes the fused image include more detailed information; that is, through the image processing method provided in the embodiment of the present application, the main camera module can obtain Image enhancement is performed on the image to enhance the detail information in the image and improve the image quality.

Implementation method two

In an example, multi-frame noise reduction, super-resolution processing, or local registration processing (for example, an example of the second registration processing) can be performed on the fifth Raw image of multiple frames according to the third Raw image to obtain the sixth Raw image (for example, the sixth Raw image of a single frame); the sixth Raw image is compared with the fifth Raw image, the sixth Raw image may refer to a Raw image of noise-free local registration; the sixth Raw image is processed by an image processing model , The multi-frame third Raw image is fused with the semantically segmented image.

It should be understood that in the first implementation, the fourth Raw image is globally registered by using the method shown in FIG. 8 as a reference to obtain the fifth Raw image; The image is used as a reference, and local registration is further performed on the fifth Raw image, and the local registration processing can enhance the detail information in the fifth Raw image, so that the local detail information of the fused image is enhanced.

FIG. 10 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1 ; the image processing method includes step S501 to step S510 , and step S501 to step S510 will be described in detail below.

Step S501. Acquire multiple frames of fifth Raw images (an example of the first registration image).

Exemplarily, multiple frames of second Raw images can be acquired through the second camera module; black level correction and phase defect correction processing are performed on multiple frames of second Raw images to obtain multiple frames of fourth Raw images; The image is used as a reference to perform registration processing on the fourth Raw image to obtain a fifth Raw image.

For example, the second camera module may include a second lens, a second lens and an image sensor, and the spectral range that the second lens can pass is near-infrared light (700nm˜1100nm).

Step S502, acquiring a third Raw image (an example of the third image).

Optionally, the third Raw image may refer to the first frame of Raw image in the multiple frames of the third Raw image.

Exemplarily, multiple frames of first Raw images may be acquired by the first camera module, and black level correction and phase defect correction processing may be performed on the multiple frames of the first Raw images to obtain multiple frames of third Raw images.

Step S503, feature splicing processing.

Exemplarily, feature splicing processing is performed on the fifth Raw image and the third Raw image in multiple frames to obtain image features.

It should be understood that the feature splicing processing in S503 is the same as that in S404 and will not be repeated here.

Step S504, image processing.

For example, perform image processing on image features.

Exemplarily, the image processing may include but not limited to: one or more of multi-frame noise reduction, super-resolution processing, local registration processing, or fusion processing.

Step S505, obtaining the sixth Raw image (an example of the fourth image).

It should be understood that, compared with the fifth Raw image, the sixth Raw image may refer to a Raw image that is locally registered without noise.

Step S506, acquiring multiple frames of third Raw images.

For example, the first camera module may include a first lens, a first lens and an image sensor, and the spectral range through which the first lens can pass is visible light (400nm-700nm).

Step S507, acquiring a semantically segmented image.

Optionally, the semantic segmentation algorithm may include a multi-instance segmentation algorithm; labels of various regions in the image may be output through the semantic segmentation algorithm.

Step S508, feature splicing processing.

Exemplarily, feature stitching is performed on the sixth Raw image of a single frame, the third Raw image of multiple frames, and the semantically segmented image to obtain multi-channel image features.

For example, assuming that the sixth Raw image is a single-channel image, the third Raw image is a 3-channel image, and the semantic segmentation image is a single-channel image, then a 5-channel image is obtained after feature extraction and splicing.

Step S509, image processing model.

Optionally, the image processing model can be used for denoising processing, demosaic processing, and fusion processing; for example, the image processing model can perform denoising processing, Demosaic processing and fusion processing.

Step S510, RGB image.

In the embodiment of the present application, the electronic device includes a first camera module and a second camera module; the spectral range that the first camera module can obtain is visible light from 400nm to 700nm, and the spectral range that the second camera module can obtain It is near-infrared light of 700nm to 1100nm; since the image information included in the near-infrared image cannot be obtained in the visible light image; therefore, the Raw image (visible light image) collected by the first camera module and the image information collected by the second camera module Raw images (near-infrared light images) are processed and then fused, which can realize the multi-spectral information fusion of near-infrared light image information and visible light image information, so that the fused image includes more detailed information; that is, through this application The image processing method provided in the embodiment can perform image enhancement on the image acquired by the camera module of the main camera, enhance the detailed information in the image, and improve the image quality.

Optionally, in the embodiment of the present application, the image processing method shown in FIG. 5 or FIG. 6 can be used to perform fusion processing on the images collected by the first camera module and the second camera module; in addition, in order to enhance the fusion processing For local detail information in the image, the image processing method shown in Figure 11 can also be used to fuse the images collected by the first camera module and the second camera module; for example, the RGB image collected by the first camera can be processed by the second camera module The NIR image collected by two cameras; the local area is selected in the detail layer of the NIR image through semantic segmentation image, and the local area in the detail layer of the NIR image is fused with the detail layer of the RGB image and the basic layer of the RGB image, so as to realize Selectively enhance the details of local regions in visible light images.

FIG. 11 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1; the image processing method includes step S601 to step S619, and step S601 to step S619 will be described in detail below.

Step S601, acquiring a first Raw image.

Exemplarily, the first Raw image can be acquired through the first camera module; for example, the first camera module can include a first lens, a first lens and an image sensor, and the spectral range that the first lens can pass is visible light (400nm ~ 700nm).

Step S602, noise reduction processing.

Exemplarily, noise reduction processing is performed on the first Raw image to obtain the first Raw image after noise reduction processing.

Exemplarily, the noise information in the image can be effectively reduced by performing noise reduction processing on the first Raw image, so that when the fusion processing is performed on the first Raw image subsequently, the image quality of the fusion-processed image can be improved.

Optionally, step S603 may be performed after step S601 is performed.

Step S603, demosaic processing.

Exemplarily, demosaic processing is performed on the first Raw image after the noise reduction processing.

It should be understood that the above steps S602 and S603 are used as examples for illustration; the first Raw image may also be converted into an RGB image in other ways; this application does not make any limitation thereto.

Step S604, RGB image.

Exemplarily, demosaic processing is performed on the first Raw image after the noise reduction processing to obtain an RGB image.

Step S605, extract the V channel image from the HSV image.

Exemplarily, the RGB image is converted into the HSV color space to obtain the HSV image; and the V channel image of the HSV image is extracted.

In the embodiment of the present application, in order to obtain the luminance channel corresponding to the RGB image, the RGB image may be converted to other color spaces, so as to obtain the luminance channel corresponding to the RGB image.

Optionally, the above-mentioned HSV image color space is used as an example; it may also be a YUV color space, or other color spaces capable of obtaining brightness channels of an image.

Step S606, filter processing.

For example, the V channel image is processed through an edge-preserving smoothing filter.

Exemplarily, the edge-preserving smoothing filter may include, but not limited to: a guided filter, a bilateral filter, and a least-squares filter.

It should be understood that, in the embodiment of the present application, when the V-channel image is processed by an edge-preserving smoothing filter, edge information in the image can be effectively preserved during the filtering process.

Step S607, the first detail layer image.

For example, the image of the V channel is processed by an edge-preserving smoothing filter to obtain the first detail layer image.

Exemplarily, the detail layer of the image includes high-frequency information of the image; for example, the detail layer of the image includes edge information, texture information, and the like of the object.

Step S608, the first base layer image.

Exemplarily, the image of the V channel is processed by an edge-preserving smoothing filter to obtain the image of the first base layer.

Exemplarily, the base layer of the image includes low-frequency information of the image; for an image, the part except the detail layer is the base layer; for example, the base layer of the image includes content information within the edge of the object.

Step S609, acquiring a second Raw image.

Exemplarily, the second Raw image can be acquired through the second camera module; for example, the second camera module can include a second lens, a second lens and an image sensor, and the spectral range that the second lens can pass is near-infrared light (700nm ~ 1100nm).

Step S610, noise reduction processing.

Exemplarily, noise reduction processing is performed on the second Raw image to obtain the second Raw image after noise reduction processing.

Exemplarily, the noise information in the image can be effectively reduced by performing noise reduction processing on the second Raw image, so that when the fusion processing is performed on the second Raw image later, the image quality of the fusion-processed image is improved.

Optionally, step S612 may be performed after step S609 is performed.

It should be understood that the above step S610 is used as an example for illustration; the second Raw image may also be converted into an NIR image in other ways; this application does not make any limitation thereto.

Step S611, NIR image.

Exemplarily, conversion processing is performed on the second Raw image to obtain an NIR image.

Step S612, filter processing.

Exemplarily, the NIR image is processed through an edge-preserving smoothing filter.

Step S613, the second detail layer image.

Exemplarily, the NIR image is processed by an edge-preserving smoothing filter to obtain a second detail layer image.

Step S614, the second base layer image.

Exemplarily, the NIR image is processed by an edge-preserving smoothing filter to obtain a second base layer image.

Step S615, acquiring a semantically segmented image.

In the embodiment of the present application, local area information in the second level of detail image can be obtained based on the semantically segmented image; the partial image information in the second level of detail image is fused with the first level of detail image to achieve selectivity The detail enhancement of the local area in the image.

Step S616, multiply.

Exemplarily, the second detail level image is multiplied by the semantic segmentation image to obtain detail level information in the NIR image.

For example, multiplying the second level-of-detail image by the semantic segmentation image may refer to multiplying the second level-of-detail image by pixel values of corresponding pixels in the semantic segmentation image.

Exemplarily, the second detail layer image includes high-frequency information in the NIR image; local detail information in the image can be selectively enhanced according to the semantic segmentation image.

It should be understood that the second detail layer includes all the detailed information in the NIR image; when the visible light image captures the scene, some image detail information may be lost for the scene that is far away from the electronic device; therefore, the image can be segmented through semantics Multiply with the second detail layer to select the local area in the second detail layer image, and fuse the local area in the second detail layer image with the first detail layer image, so as to realize the optional local area in the image Details are enhanced.

Step S617, fusion processing.

Exemplarily, fusion processing is performed on the detail layer information of the NIR image, the first detail layer image, and the first base layer image.

Exemplarily, layer-of-detail information of the NIR image is superimposed on the first layer-of-detail image; and the first layer-of-detail image is superimposed on the first base layer image.

Step S618, HSV image.

Exemplarily, the HSV image is obtained after fusion processing.

Optionally, the above HSV image may also be in other color spaces; for example, it may be an image in HSL color space.

Optionally, the above-mentioned HSV image may also be an image in a YUV color space; or, an image in another color space from which a brightness channel can be extracted.

Step S619, RGB image.

Exemplarily, the fused HSV image is converted into an RGB color space to obtain a fused RGB image.

In the embodiment of the present application, the RGB image collected by the first camera module and the NIR image collected by the second camera module are filtered through an edge-preserving smoothing filter, and the first detail layer image and the first detail layer image included in the RGB image are respectively obtained. The first base layer image; the second detail layer image and the second base layer image included in the NIR image. Since the spectral range of near-infrared light is wider than that of visible light, more image details can be obtained in the near-infrared image ; Therefore, the local area in the second layer of detail image can be selected by semantically segmenting the image, and the local area in the second layer of detail image is fused with the first layer of detail image and the first base layer image, thereby realizing optional Detail enhancement for local regions in visible light images.

Optionally, in the embodiment of the present application, the image processing method shown in FIG. 5, FIG. 6 or FIG. 11 can be used to perform fusion processing on the images collected by the first camera module and the second camera module; In the embodiment of the application, in order to reduce the ghosting problem in the image after fusion processing, the image processing method shown in Figure 12 can also be used to perform fusion processing on the images collected by the first camera module and the second camera module; for example , by fusing the similar image information in the RGB image and the NIR image, effectively avoiding the ghosting problem in the image after fusion processing. For example, through the image fusion processing of the low-frequency information in the RGB image and the NIR image, the low-frequency information of the image is enhanced; by converting the RGB image to the YUV color space, the high-frequency information and low-frequency information of the Y channel are enhanced. Superposition is performed so that the high-frequency information part of the image is also enhanced.

Fig. 12 is a schematic diagram of an image processing method provided by an embodiment of the present application. The image processing method can be executed by the electronic device shown in FIG. 1; the image processing method includes step S701 to step S715, and step S701 to step S715 will be described in detail below.

Step S701, acquiring a first Raw image.

Exemplarily, the first Raw image can be acquired through the first camera module; for example, the first camera module can include a first lens, a first lens and an image sensor, and the spectral range through which the first lens can pass is visible light (400nm ~700nm).

Step S702, noise reduction processing.

Optionally, step S703 may be performed after step S701 is performed.

Step S703, demosaic processing, to obtain the first RGB image.

Exemplarily, demosaic processing is performed on the first Raw image after the noise reduction processing to obtain the first RGB image.

It should be understood that the above steps S702 and S703 are used as examples; the first Raw image may also be converted into an RGB image in other ways; this application does not make any limitation thereto.

Step S704: Process the first RGB image through a Gaussian low-pass filter to obtain low-frequency information in the first RGB image.

Exemplarily, the first RGB image is processed through a Gaussian low-pass filter to filter out high-frequency detail features of the first RGB image to obtain low-frequency information of the first RGB image.

It should be understood that the low-frequency information of an image refers to the area where the gray value changes slowly in the image; for an image, the part except the high-frequency information is low-frequency information; for example, the low-frequency information of the image can include the content within the edge of the object information.

Step S705, acquiring a second Raw image.

Step S706, noise reduction processing.

Exemplarily, noise reduction processing is performed on the second Raw image.

Optionally, step S708 may be performed after step S705 is performed.

Step S707, NIR image.

It should be understood that the above step S706 is used as an example for illustration; the second Raw image may also be converted into an NIR image in other ways; this application does not make any limitation thereto.

Step S708, performing registration processing on the NIR image to obtain a registered NIR image.

Exemplarily, a registration process is performed on the NIR image with the first RGB image as a reference to obtain a registered NIR image.

It should be understood that since the first camera module and the second camera module are respectively arranged at different positions in the electronic device, there is a certain baseline distance between the first camera module and the second camera module, that is, through the first camera module There is a certain parallax between the image collected by the module and the image collected by the second camera module, and the images collected by the two need to be registered.

Step S709, merging models.

Exemplarily, the low-frequency information of the first RGB image and the registered NIR image are input to the fusion model for processing.

Step S710, outputting the second RGB image.

For example, the fused model outputs a second RGB image.

Exemplarily, the fusion model may refer to a neural network pre-trained by a large amount of sample data; the fusion model is used to perform fusion processing on input images and output a fusion processed image.

It should be understood that a large amount of image low-frequency information is included in the NIR image; since the spectral ranges corresponding to the NIR image and the first RGB image are different, fusion of the low-frequency information of the NIR image and the first RGB image can make the low-frequency information of the NIR image significantly affect the second RGB image. The low-frequency information of an RGB image is supplemented; thereby the low-frequency information in the first RGB image is enhanced. In addition, by fusing similar image information, that is, fusing the low-frequency information included in the NIR image with the low-frequency information of the first RGB image, ghost images appearing in the fused image can be effectively reduced.

Step S711, extracting the image of the Y channel in the YUV image.

Exemplarily, the second RGB image is converted into a YUV color space to obtain a first YUV image; and an image of a Y channel in the first YUV image is extracted.

In the embodiment of the present application, in order to obtain the luminance channel corresponding to the RGB image, the RGB image may be converted into the YUV color space to obtain the first YUV image, and the Y channel of the first YUV image may be extracted.

It should be understood that the YUV color space is used as an example for illustration above; the RGB image can also be converted to other color spaces capable of extracting brightness channels, which is not limited in this application.

Step S712, converting the first RGB image to YUV.

Exemplarily, the first RGB image is converted into a YUV color space to obtain a second YUV image.

Step S713, process the Y channel.

For example, the Y channel of the second YUV image is processed.

Exemplarily, Gaussian blur is performed on the Y channel of the second YUV image to obtain a blurred image of the Y channel (blur image of the Y channel); the blurred image of the Y channel includes the low-frequency information of the Y channel; the image of the Y channel is subtracted The blurred image of the Y channel, and the high frequency information of the Y channel is obtained.

It should be understood that the high-frequency information of the image refers to the region in the image where the gray value changes rapidly; for example, the high-frequency information in the image includes edge information, texture information, etc. of the object.

Optionally, different gain coefficients can be determined according to different camera modes selected by the user; the Y channel of the second YUV image can be processed according to the following formula:

Processed Y channel=(Y channel image−Y channel blurred image)*gain factor.

Step S714, adding Y channels.

Exemplarily, the Y channel of the first YUV image is added to the Y channel of the processed second YUV image to obtain the processed YUV image.

It should be understood that in step S710, the low-frequency information of the NIR image and the first RGB image are fused to obtain the second RGB image; in steps S713 and S714, the high-frequency information part of the image needs to be processed.

Step S715, obtaining a third RGB image.

Exemplarily, the processed YUV image is converted into RGB color space to obtain a third RGB image.

In the embodiment of the present application, the RGB image is collected by the first camera module, the NIR image is collected by the second camera module, and the similar image information in the RGB image and the NIR image is fused to effectively avoid the post-fusion processing. There are ghosting problems in the image; for example, through image fusion processing of the low-frequency information in the RGB image and the NIR image, the low-frequency information of the image is enhanced; by converting the RGB image to the YUV color space, the high-frequency information of the Y channel Superimposed with the image enhanced by the low-frequency information part, the high-frequency information part of the image is also enhanced; because the similar image information in the RGB image and the NIR image is fused, it can effectively reduce the image after fusion processing. Ghosts that appear in .

FIG. 13 is a schematic diagram of the effect of the image processing method provided by the embodiment of the present application.

As shown in Figure 13, (a) in Figure 13 is the output image obtained by the existing main camera camera module; (b) in Figure 13 is the output image obtained by the image processing method provided by the embodiment of the present application ; The image shown in (a) in Figure 13 shows that the detailed information in the mountains is severely distorted; compared with the output image shown in (a) in Figure 13, the image shown in (b) in Figure 13 The detailed information of the displayed output image is relatively rich, and can clearly display the detailed information of mountains; through the image processing method provided in the embodiment of the present application, the image obtained by the main camera camera module can be image enhanced to improve the detailed information in the image.

In an example, in a dark scene, the user can turn on the infrared flashlight in the electronic device; collect images through the main camera module and the near-infrared camera module, and use the image processing method provided by the embodiment of the application to process the collected images. The image is processed to output the processed image or video.

FIG. 14 shows a graphical user interface (graphical user interface, GUI) of an electronic device.

The GUI shown in (a) in FIG. 14 is the desktop 810 of the electronic device; when the electronic device detects that the user clicks the operation of the camera application (application, APP) icon 820 on the desktop 810, the camera application can be started, and the display is as follows: Another GUI shown in (b) in Figure 14; the GUI shown in (b) in Figure 14 can be the display interface of the camera APP in the camera mode, and the GUI can include a shooting interface 830; Including a viewfinder frame 831 and controls; for example, the shooting interface 830 may include a control 832 for instructing shooting and a control 833 for instructing to turn on an infrared flash; in the preview state, a preview image may be displayed in real time in the viewfinder 831; wherein , the preview state can mean that before the user turns on the camera and does not press the photo/record button, the preview image can be displayed in the viewfinder in real time.

After the electronic device detects that the user clicks the operation of the control 833 indicating to turn on the infrared flashlight, the shooting interface shown in (c) in Figure 14 is displayed; The camera module collects images, processes the collected images through the image processing method provided in the embodiment of the present application, and outputs processed images with enhanced image quality.

Fig. 15 is a schematic diagram of an optical path of a shooting scene applicable to an embodiment of the present application.

As shown in Figure 15, the electronic device also includes an infrared flashlight; in a dark scene, the electronic device can turn on the infrared flashlight; when the infrared flashlight is turned on, the lighting in the shooting environment can include street lights and infrared flashlights; The light in the shooting environment is reflected, so that the electronic device obtains an image of the shooting object.

In the embodiment of the present application, when the infrared flashlight is turned on, the reflected light of the shooting object increases, so that the amount of incoming light of the near-infrared camera module in the electronic device increases; thereby making the details of the image captured by the near-infrared camera module The information is increased, and the images collected by the main camera module and the near-infrared camera module are fused through the image processing method of the embodiment of the present application, so that the image acquired by the main camera module can be image enhanced, and the details in the image can be improved. information. In addition, the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.

It should be understood that the above illustrations are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the illustrated specific values or specific scenarios. Those skilled in the art can obviously make various equivalent modifications or changes based on the above illustrations given, and such modifications or changes also fall within the scope of the embodiments of the present application.

The image processing method provided by the embodiment of the present application is described in detail above with reference to FIG. 1 to FIG. 15 ; the device embodiment of the present application will be described in detail below in conjunction with FIG. 16 and FIG. 17 . It should be understood that the devices in the embodiments of the present application can execute the various methods in the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

FIG. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 900 includes a display module 910 and a processing module 920 . The electronic device includes a first camera module and a second camera module, and the second camera module is a near-infrared camera module or an infrared camera module.

Wherein, the display module 910 is used for displaying a first interface, and the first interface includes a first control; the processing module 920 is used for detecting a first operation on the first control; in response to the first Operation, acquiring N frames of first images and M frames of second images, the first images are images collected by the first camera module, the second images are images collected by the second camera module, N and M are both positive integers greater than or equal to 1; based on the N frames of the first image and the M frames of the second image, the target image is obtained; the target image is saved; wherein the processing module 920 is specifically used for:

Performing first image processing on the N frames of the first image to obtain N frames of the third image, the image quality of the N frames of the third image is higher than the image quality of the N frames of the first image; for the M frames of the first image The second image is processed for the second image to obtain the fourth image of M frames, the image quality of the fourth image of the M frames is higher than the image quality of the second image of the M frames; based on the semantic segmentation image, the third image of the N frames is The image and the fourth image of the M frames are fused to obtain a fused image, and the semantically segmented image is obtained based on any frame image in the first image of the N frames or any frame image in the second image of the M frames Yes, the detailed information of the fused image is better than the detailed information of the N frames of the first image; the third image processing is performed on the fused image to obtain the target image.

Optionally, as an embodiment, the processing module 920 is specifically configured to:

Optionally, as an embodiment, the first registration processing is global registration processing.

Optionally, as an embodiment, the second registration processing is local registration processing.

Optionally, as an embodiment, the electronic device further includes an infrared flash lamp, and the processing module 920 is further configured to:

Optionally, as an embodiment, the first interface includes a second control; the processing module 920 is specifically configured to:

detecting a second operation on the second control;

The infrared flashlight is turned on in response to the second operation.

Optionally, as an embodiment, the semantically segmented image is obtained by processing the third image in the first frame of the N frames of third images by using a semantic segmentation algorithm.

Optionally, as an embodiment, the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.

Optionally, as an embodiment, the first interface refers to a video recording interface, and the first control refers to a control for instructing video recording.

Optionally, as an embodiment, the first interface refers to a video call interface, and the first control refers to a control for instructing a video call.

It should be noted that the above-mentioned electronic device 900 is embodied in the form of functional modules. The term "module" here may be implemented in the form of software and/or hardware, which is not specifically limited.

For example, a "module" may be a software program, a hardware circuit or a combination of both to realize the above functions. The hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.

Therefore, the units of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

FIG. 17 shows a schematic structural diagram of an electronic device provided by the present application. The dotted line in FIG. 17 indicates that this unit or this module is optional; the electronic device 1000 can be used to implement the methods described in the foregoing method embodiments.

The electronic device 1000 includes one or more processors 1001, and the one or more processors 1001 can support the electronic device 1000 to implement the image processing method in the method embodiment. Processor 1001 may be a general purpose processor or a special purpose processor. For example, the processor 1001 may be a central processing unit (central processing unit, CPU), a digital signal processor (digital signal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices such as discrete gates, transistor logic devices, or discrete hardware components.

The processor 1001 can be used to control the electronic device 1000, execute software programs, and process data of the software programs. The electronic device 1000 may further include a communication unit 1005, configured to implement signal input (reception) and output (send).

For example, the electronic device 1000 can be a chip, and the communication unit 1005 can be an input and/or output circuit of the chip, or the communication unit 1005 can be a communication interface of the chip, and the chip can be used as a component of a terminal device or other electronic devices .

For another example, the electronic device 1000 may be a terminal device, and the communication unit 1005 may be a transceiver of the terminal device, or the communication unit 1005 may be a transceiver circuit of the terminal device.

The electronic device 1000 may include one or more memories 1002, on which a program 1004 is stored, and the program 1004 may be run by the processor 1001 to generate an instruction 1003, so that the processor 1001 executes the image processing described in the above method embodiment according to the instruction 1003 method.

Optionally, data may also be stored in the memory 1002 .

Optionally, the processor 1001 may also read data stored in the memory 1002, the data may be stored in the same storage address as the program 1004, and the data may also be stored in a different storage address from the program 1004.

The processor 1001 and the memory 1002 may be set separately, or may be integrated together, for example, integrated on a system-on-chip (system on chip, SOC) of a terminal device.

Exemplarily, the memory 1002 can be used to store the related program 1004 of the image processing method provided in the embodiment of the present application, and the processor 1001 can be used to call the related program 1004 of the image processing method stored in the memory 1002 when performing image processing, Execute the image processing method of the embodiment of the present application; for example, display a first interface, the first interface includes a first control; detect a first operation on the first control; in response to the first operation, acquire N frames of the first image and M Frame the second image, the first image is the image collected by the first camera module, the second image is the image collected by the second camera module, N and M are both positive integers greater than or equal to 1; based on N frames of the first image and M frames of the second image to obtain the target image; save the target image; wherein, based on the N frames of the first image and the M frames of the second image, the target image is obtained, including: performing the first image processing on the N frames of the first image to obtain N The third image of the frame, the image quality of the third image of the N frame is higher than the image quality of the first image of the N frame; the second image processing is performed on the second image of the M frame, and the fourth image of the M frame is obtained, and the image of the fourth image of the M frame The quality is higher than the image quality of the second image of the M frame; based on the semantic segmentation image, the third image of the N frame and the fourth image of the M frame are fused to obtain a fusion image, and the semantic segmentation image is based on any one of the first images of the N frame The detailed information of the fused image is better than the detailed information of the N frames of the first image obtained from any one frame of the image or the M frames of the second image; the third image processing is performed on the fused image to obtain the target image.

The present application also provides a computer program product, which implements the method of any method embodiment in the present application when the computer program product is executed by the processor 1001 .

The computer program product can be stored in the memory 1002, such as the program 1004, and the program 1004 is finally converted into an executable object file that can be executed by the processor 1001 through processes such as preprocessing, compiling, assembling and linking.

The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the image processing method described in any method embodiment in the present application is implemented. The computer program may be a high-level language program or an executable object program.

The computer-readable storage medium is, for example, the memory 1002 . The memory 1002 may be a volatile memory or a nonvolatile memory, or, the memory 1002 may include both a volatile memory and a nonvolatile memory. Among them, the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM ) and direct memory bus random access memory (direct rambus RAM, DR RAM).

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the embodiments of the electronic equipment described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

It should be understood that in various embodiments of the present application, the sequence numbers of the processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, rather than by the embodiments of the present application. The implementation process constitutes any limitation.

In addition, the term "and/or" in this article is only an association relationship describing associated objects, indicating that there may be three relationships, for example, A and/or B, which may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In a word, the above description is only a preferred embodiment of the technical solution of the application, and is not intended to limit the protection scope of the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

An image processing method, characterized in that it is applied to an electronic device, the electronic device includes a first camera module and a second camera module, the second camera module is a near-infrared camera module or an infrared camera module , the image processing method includes:

displaying a first interface, where the first interface includes a first control;

detecting a first operation on the first control;

In response to the first operation, N frames of first images and M frames of second images are acquired, the first images are images collected by the first camera module, and the second images are images collected by the second camera module The images collected by the group, N and M are both positive integers greater than or equal to 1;

Obtaining a target image based on the N frames of the first image and the M frames of the second image;

Save the target image; where,

The obtaining the target image based on the N frames of the first image and the M frames of the second image includes:

performing first image processing on the N frames of the first image to obtain N frames of the third image, the image quality of the N frames of the third image is higher than the image quality of the N frames of the first image;

performing second image processing on the M frames of second images to obtain M frames of fourth images, where the image quality of the M frames of the fourth images is higher than the image quality of the M frames of the second images;

Based on the semantically segmented image, the third image of the N frames and the fourth image of the M frame are fused to obtain a fused image, and the semantically segmented image is based on any frame image in the first image of the N frames or the The detailed information of the fused image is better than the detailed information of the N frames of the first image;

performing third image processing on the fused image to obtain the target image.
The image processing method according to claim 1, wherein said performing second image processing on said M frames of second images to obtain M frames of fourth images comprises:

performing black level correction processing and/or phase defect correction processing on the M frames of second images to obtain M frames of fifth images;

Taking any third image of the N frames of third images as a reference, performing a first registration process on the M frames of fifth images to obtain the N frames of fourth images.
The image processing method according to claim 2, wherein the first registration process is performed on the M frames of the fifth image based on any one frame of the third image in the N frames of third images , to obtain the fourth image of the N frames, including:

Taking any third image of the N frames of third images as a reference, performing the first registration processing and up-sampling processing on the M frames of fifth images to obtain the N frames of fourth images.
The image processing method according to claim 1, wherein said performing second image processing on said M frames of second images to obtain M frames of fourth images comprises:

performing black level correction processing and/or phase defect correction processing on the M frames of second images to obtain M frames of fifth images;

Using any one of the third images in the N frames of third images as a reference, performing a first registration process on the M frames of the fifth image to obtain M frames of first registration images;

Using the arbitrary third image as a reference, perform a second registration process on the M frames of the first registration image to obtain the M frames of the fourth image.
The image processing method according to claim 4, wherein the first registration process is performed on the M frames of the fifth image based on any one frame of the third image in the N frames of third images , to obtain M frames of the first registration image, including:

Performing the first registration processing and upsampling processing on the M frames of the fifth image based on any one of the third images in the N frames of third images to obtain the M frames of the first registration image .
The image processing method according to any one of claims 2 to 5, wherein the first registration process is a global registration process.
The image processing method according to claim 4 or 5, wherein the second registration process is local registration process.
The image processing method according to any one of claims 1 to 7, wherein said performing first image processing on said N frames of first images to obtain N frames of third images comprises:

Perform black level correction processing and/or phase defect correction processing on the N frames of first images to obtain the N frames of third images.
The image processing method according to any one of claims 1 to 8, wherein the electronic device further comprises an infrared flash lamp, and the image processing method further comprises:

In a dark light scene, turn on the infrared flashlight, the dark light scene refers to a shooting scene in which the amount of light entering the electronic device is less than a preset threshold;

In response to the first operation, acquiring N frames of the first image and M frames of the second image includes:

When the infrared flashlight is turned on, the N frames of the first image and the M frames of the second image are acquired.
The image processing method according to claim 9, wherein the first interface includes a second control; and turning on the infrared flashlight in a dark scene includes:

detecting a second operation on the second control;

The infrared flashlight is turned on in response to the second operation.
The image processing method according to any one of claims 1 to 10, wherein, based on the semantically segmented image, fusion processing is performed on the N frames of the third image and the M frames of the fourth image, Get the fused image, including:

Based on the semantically segmented image, the third image of N frames and the fourth image of M frames are fused by an image processing model to obtain the fused image, and the image processing model is a pre-trained neural network.
The image processing method according to any one of claims 1 to 11, wherein the semantically segmented image is obtained by processing the third image of the first frame of the N frames of third images through a semantic segmentation algorithm of.
The image processing method according to any one of claims 1 to 12, wherein the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.
The image processing method according to any one of claims 1 to 12, wherein the first interface refers to a video recording interface, and the first control refers to a control for instructing video recording.
The image processing method according to any one of claims 1 to 12, wherein the first interface refers to a video call interface, and the first control refers to a control for instructing a video call.
An electronic device, characterized in that it comprises:

one or more processors and memory;

The memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to make the The electronic device executes the image processing method according to any one of claims 1 to 15.
A system on a chip, characterized in that the system on a chip is applied to an electronic device, and the system on a chip includes one or more processors, the processors are used to invoke computer instructions so that the electronic device executes the electronic device as claimed in claims 1 to 10. The image processing method described in any one of 15.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is made to execute any one of claims 1 to 15. image processing method.
A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is executed by a processor, the processor is made to perform the image processing described in any one of claims 1 to 15 method.