CN115473973A

CN115473973A - Image processing method, image processing device, electronic equipment and storage medium

Info

Publication number: CN115473973A
Application number: CN202210730519.XA
Authority: CN
Inventors: 陈泓至; 胡璇
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2022-06-24
Filing date: 2022-06-24
Publication date: 2022-12-13

Abstract

The application discloses an image processing method, an image processing device, electronic equipment and a storage medium, and belongs to the technical field of images. The method comprises the following steps: acquiring a first original image and M second original images, wherein M is a positive integer; determining target noise data of the image sensor according to the M second original images; based on the target noise data, carrying out noise reduction processing on the first original image to obtain a target image; when the first original image is obtained, the light transmittance of the light transmittance control layer is larger than 0; when a second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

Description

Image processing method, image processing device, electronic equipment and storage medium

Technical Field

The present application belongs to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

Generally, in a scene of shooting an image at night, an electronic device may perform noise reduction processing on a RAW (RAW) image output by an image sensor of the electronic device based on predetermined noise of the image sensor through an Artificial Intelligence (AI) noise reduction technology to reduce the noise in the RAW image, so that the electronic device may obtain a high-quality image from the RAW image after the noise reduction processing.

However, since the noise of the image sensor may change due to other factors (for example, temperature change, etc.), the effect of the electronic device performing noise reduction processing on the RAW image by using the AI noise reduction technique may be reduced, and thus, a large amount of noise may still exist in the RAW image after the noise reduction processing.

Thus, the quality of the image taken by the electronic device is poor.

Disclosure of Invention

An object of the embodiments of the present application is to provide an image processing method, an image processing apparatus, an electronic device, and a storage medium, which can improve the quality of an image captured by the electronic device.

In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, where the electronic device includes: a light transmittance control layer disposed between a lens and an image sensor of an electronic device, the image processing method comprising: acquiring a first original image and M second original images, wherein M is a positive integer; determining target noise data of the image sensor according to the M second original images; based on the target noise data, carrying out noise reduction processing on the first original image to obtain a target original image; when the first original image is obtained, the light transmittance of the light transmittance control layer is greater than 0; when a second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

In a second aspect, an embodiment of the present application provides an image processing apparatus, which is applied to an electronic device, where the electronic device includes: a light transmittance control layer disposed between a lens of the electronic device and the image sensor; the image processing apparatus includes: the device comprises an acquisition module, a determination module and a processing module. The acquisition module is used for acquiring a first original image and M second original images, wherein M is a positive integer. And the determining module is used for determining the target noise data of the image sensor according to the M second original images. The processing module is used for carrying out noise reduction processing on the first original image based on the target noise data to obtain a target original image; wherein, when the first original image is obtained, the light transmittance of the light transmittance control layer is greater than 0; when a second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

In a third aspect, embodiments of the present application provide an electronic device, which includes a processor and a memory, where the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method according to the first aspect.

In the embodiment of the application, the electronic device may obtain the first original image and the M second original images to determine target noise data of the image sensor, and perform noise reduction processing on the first original image according to the target noise data, so as to obtain the target image. In the scheme, the electronic equipment can acquire the target noise data in real time according to the M second original images and perform noise reduction processing on the first original image according to the target noise data to obtain the target image, so that the problem that when the electronic equipment performs noise reduction processing on the shot image, the image quality shot by the electronic equipment is poor due to the fact that the image sensor cannot determine accurate background noise in real time and the electronic equipment cannot accurately remove noise is avoided.

Drawings

Fig. 1 is a flowchart of an image processing method provided in an embodiment of the present application;

fig. 2 is a schematic view of an example of a light transmittance control layer provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of an example of an image processing method according to an embodiment of the present disclosure;

fig. 4 is a second schematic diagram of an example of an image processing method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an example of an electronic device according to an embodiment of the present disclosure;

fig. 6 is a second schematic diagram of an example of an electronic device according to an embodiment of the present application;

fig. 7 is a second schematic view illustrating an example of a light transmittance control layer according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 9 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present application;

fig. 10 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any inventive effort, shall fall within the scope of protection of the present application.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application are capable of operation in sequences other than those illustrated or described herein, and that the terms "first," "second," etc. are generally used in a generic sense and do not limit the number of terms, e.g., a first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

Terms related to the embodiments of the present application will be described below.

1. Image Sensor (Sensor)

The image sensor is the core of the camera and is also the most critical technology in the camera. The Sensor can be divided into two types, one is a widely used Charge-coupled Device (CCD) element; another type is a Complementary Metal Oxide Semiconductor (CMOS) device. In contrast to conventional cameras, conventional cameras use "film" as their carrier for recording information, whereas the "film" of a digital camera is its imaging photosensitive element, which is the "film" of the digital camera that is not to be replaced and is integral with the camera.

Currently, CMOS devices, which are semiconductors capable of recording light changes in digital cameras, like CCDs, are mainly used. The CMOS process includes sensing light signal with great amount of photosensitive diodes, converting the sensed light signal into electric signal, forming digital signal matrix, image processing with image signal processor and compressing and storing.

A CMOS Camera Module (CMOS Camera Module) is a Camera Module mainly used in a mobile phone at present, and mainly includes a Lens (Lens), a Voice Coil Motor (Voice Coil Motor), an infrared Filter (IR Filter), an image sensor (CMOS), a Digital Signal Processor (DSP), and a Flexible Printed Circuit (FPC).

The working process of the CCD Camera Module is that the voice coil motor drives the lens to reach a position with accurate focusing, external light passes through the lens, is filtered by the infrared filter and irradiates on a photosensitive diode (pixel) of the image sensor, the photosensitive diode converts sensed optical signals into electric signals, a digital signal matrix (namely an image) is formed through the amplifying circuit and the digital/analog conversion circuit, and the digital signal matrix is processed by the DSP and compressed and stored.

2. Camera lens (lens)

The camera lens is the most important component in the camera because its quality directly affects the quality of the captured image. The lens can be divided into two categories of zooming and fixed focus. The zoom lens has variable focal length and variable visual angle, namely, the zoom lens can be pushed and pulled; the fixed focus lens is a lens whose focal length cannot be changed to only one focal length or only one viewing angle.

3. Fixed Pattern Noise (Fixed-Pattern Noise, FPN)

FPN, a fixed noise, is usually present in the image sensor and appears as a fixed vertical stripe (Column FPN) or a horizontal stripe (Row FPN) when severe, and is commonly referred to as VFPN and HFPN by the sensor supplier, and generally depends on factors such as the difference in Column outputs caused by different matches of Column comparators or the horizontal stripe caused by coupling effects between different rows; and a bad power supply system, unmatched clock source signals and module layer design may also cause FPN problems, and the noise often varies in severity with different temperatures of the electronic equipment and different sensor Analog gains (sensor Analog gain), and cannot be processed by a noise reduction algorithm. When the electronic device is used for taking a picture in an extremely night environment, the FPN brings a very poor impression due to a weak image signal.

4. RAW map

The RAW image is a RAW data image in which an optical signal captured by a CMOS or CCD image sensor is converted into a digital signal.

The image processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.

At present, with the development of electronic equipment, the shooting functions of cameras in the electronic equipment are increasing, and the shooting effect of the electronic equipment in a night scene is poor, one of the reasons is that the size of a photodiode in the electronic equipment is too small, so that the light energy collected by the photodiode in an environment with low light is less, and when a rear end (such as an ISP) performs conversion processing on an optical signal, a lot of noise is introduced, so as to ensure that the brightness of a shot image meets the requirements of human eyes of a user.

In the related art, in order to solve the above problem, noise may be removed through a denoising algorithm, which may be divided into time domain denoising and spatial domain denoising, where the typical methods of time domain denoising are multi-frame denoising and wavelet denoising, and the purpose is to remove noise difference of a single-pixel signal in different time periods. Typical methods of spatial noise reduction are filter noise reduction, such as median filtering, gaussian filtering, etc. However, these noise reduction means are usually processed at the ISP backend, and the effect is not good. The recent popular AI noise reduction is to perform noise reduction at the front end, which is better than the traditional time domain and space domain noise reduction effect. The precondition of AI noise reduction is that the background noise needs to be accurately calculated, but the background noise state of the image sensor changes with factors such as the camera opening time, the environment temperature, the mobile phone temperature and the like, so that the AI noise reduction cannot achieve the best effect. In addition, fixed pattern noise of the image sensor cannot be removed by a noise reduction algorithm at present.

In the embodiment of the application, the electronic device may obtain the first original image and the M second original images to determine target noise data of the image sensor, and perform noise reduction processing on the first original image according to the target noise data, so as to obtain the target image. In the scheme, the electronic equipment can acquire the target noise data in real time according to the M pieces of second original images and perform noise reduction processing on the first original image according to the target noise data to obtain the target image, so that the problem that when the electronic equipment performs noise reduction processing on the shot image, the image quality shot by the electronic equipment is poor due to the fact that the image sensor cannot determine accurate background noise in real time and the electronic equipment cannot accurately remove the noise is avoided.

An embodiment of the present application provides an image processing method, and fig. 1 shows a flowchart of an image processing method provided in an embodiment of the present application, where the method may be applied to an electronic device, where the electronic device includes: and the light transmittance control layer is arranged between the lens and the image sensor of the electronic equipment. As shown in fig. 1, an image processing method provided in an embodiment of the present application may include steps 201 to 203 described below.

Step 201, the electronic device acquires a first original image and M second original images, where M is a positive integer.

In the embodiment of the application, when the first original image is obtained, the light transmittance of the light transmittance control layer is greater than 0; when the second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

In this embodiment, the material of the light transmittance control layer is an electrochromic material. In the embodiment of the application, after the light passes through the light transmittance control layer by the lens, the light is projected on the image sensor, so that the image sensor can convert the optical signal of the light into an original image of a digital signal, and the electronic device can acquire the first original image and M second original images through the image sensor.

Alternatively, in the embodiment of the present application, the electrochromic material may be a redox reaction material or a dispersed liquid crystal material.

Alternatively, in the embodiment of the present application, the light transmittance control layer may be composed of at least one layer of electrochromic material.

Optionally, in this embodiment of the application, the electronic device may change the light transmittance of the light transmittance control layer by changing the voltage at two ends of the light transmittance control layer.

Illustratively, the electronic device may change the light transmittance of the electrochromic material by a voltage, for example, the light transmittance of the electrochromic material is 0 in the case where no voltage is applied across the electrochromic material, and the light transmittance of the electrochromic material is 100% in the case where a voltage is applied across the electrochromic material for a second voltage (e.g., 2.8V).

Optionally, in this embodiment of the application, the electronic device may display an interface of a target application according to a click input of a user to an identifier (for example, an application icon) of the target application, and start a camera of the electronic device, and the electronic device may control light transmittance of the transmittance control layer to be first light transmittance according to a first input of the user to a target control in the interface of the target application, so that the electronic device may obtain a first original image through the image sensor according to the click input of the user to a shooting control in the interface of the target application; and the electronic equipment can control the light transmittance of the light transmittance control layer to be a second light transmittance according to a second input of the user to the target control in the interface of the target application, so that the electronic equipment can acquire M second original images through the image sensor according to the click input of the user to the shooting control in the interface of the target application.

Optionally, in this embodiment of the present application, the first input may be any one of the following: click input, long-press input, sliding input and preset track input; or a physical key combination (e.g., power key and volume key). The method can be determined according to actual use requirements, and the embodiment of the application is not limited. The second input may be any one of: click input, long-press input, sliding input and preset track input; or a physical key combination (e.g., power key and volume key). The method can be determined according to actual use requirements, and the embodiment of the application is not limited.

Optionally, in this embodiment of the application, the first original image may be one original image or multiple original images.

It should be noted that the first original image and the second original image are both images captured under the same environment.

Optionally, in this embodiment of the application, the sizes of the first original image and the second original image may be the same or different.

Optionally, in this embodiment of the present application, the step 201 may be specifically implemented by the following steps 201a and 201 b.

Step 201a, the electronic device controls the light transmittance of the light transmittance control layer to be a first light transmittance, and acquires a first original image through the image sensor.

In this embodiment, the electronic device may apply a preset voltage (e.g., 2.8V) to two ends of the light transmittance control layer, so that the light transmittance of the light transmittance control layer is 100%, and then obtain the first original image through the first light transmittance.

Step 201b, the electronic device controls the light transmittance of the light transmittance control layer to be 0, and M second original images are acquired through the image sensor.

In this embodiment, the electronic device may stop applying the preset voltage (e.g., 2.8V) to the two ends of the light transmittance control layer, so that the light transmittance of the light transmittance control layer is 0, and then obtain M second original images through the second light transmittance (i.e., the light transmittance is 0).

For example, taking an electrochromic material as a dispersed liquid crystal material as an example, as shown in fig. 2 (a), when a voltage of 2.8V (i.e., a first voltage) is applied across an electronic device, the dispersed liquid crystal material 10 may adjust small droplets in a liquid crystal interlayer according to the voltage, so that the refractive indexes of the substrates are relatively close, that is, when light is incident, light may pass through the droplets to present a transparent state, at this time, the electronic device may collect a first original image through a current light transmittance (i.e., the first light transmittance), as shown in fig. 2 (B), when a voltage is not applied across the electronic device (i.e., the second voltage is 0), the small droplets in the liquid crystal interlayer in the dispersed liquid crystal material 10 are in a disordered state, when light is incident, the refractive index of the light is greatly different from the refractive index of the substrate, light may be scattered when the light passes through the droplets, the light transmitting unit may present an opaque state (i.e., the light cannot enter into a plurality of photosensitive pixels), at this time, the electronic device may collect three second original images through the current light transmittance (i.e., the second light transmittance).

In this application embodiment, the electronic device can change the light transmittance of the light transmittance control layer by applying a voltage to the two ends of the light transmittance control layer to obtain the first original image and M second original images with different light transmittances, so that the electronic device can determine the true noise data of the image taken by the electronic device in the current environment through the first original image and M second original images, and the accuracy of processing the noise data in the image by the electronic device is improved.

Optionally, in this embodiment of the application, the "acquiring the M second original images by the image sensor" in the step 201 may specifically be implemented by the following step 201 e.

Step 201e, in case that the target noise data includes fixed pattern noise data, the electronic device collects M second original images using the first exposure parameters.

In an embodiment of the present application, the first exposure parameter is an exposure parameter when the image sensor acquires the first original image.

In this embodiment, after the electronic device controls the transmittance of the light transmittance control layer to be the second transmittance, the electronic device may acquire M second original images using the same exposure parameters as the first original image, so that the electronic device may compare the M second original images with the first original image to determine fixed pattern noise data in the first original image.

Optionally, in an embodiment of the present application, the fixed pattern noise data includes at least one of: row pattern noise data or column pattern noise data.

Optionally, in an embodiment of the present application, the exposure parameter may include at least one of: shutter speed, aperture value (e.g., F2.8), sensitivity (e.g., ISO 200), and exposure gain.

Optionally, in this embodiment of the application, after the electronic device controls the transmittance of the light transmittance control layer to be the second transmittance, the electronic device may acquire M second original images by using the same gain (i.e., gain) as the first original image.

In the embodiment of the application, the electronic equipment can adopt the exposure parameters same as the first original image to acquire M pieces of second original images, so that the electronic equipment can determine the fixed pattern noise data in the first original image according to the M pieces of second original images and the first original image, and then the electronic equipment can process the fixed pattern noise data, thereby avoiding that the electronic equipment cannot detect the fixed pattern noise, further being incapable of processing the fixed pattern noise and improving the image quality shot by the electronic equipment.

Step 202, the electronic device determines target noise data of the image sensor according to the M second original images.

Optionally, in an embodiment of the present application, the target noise data includes at least one of: the noise data and the fixed pattern noise data are read out.

Alternatively, in this embodiment of the application, the step 202 may be specifically implemented by the following steps 202a to 202 c.

Step 202a, the electronic device calculates K difference matrices according to the M second original images.

In this embodiment, the electronic device may perform processing according to the pixel values of the M second original images to obtain K difference matrices.

Optionally, the pixel values include at least one of: pixel brightness value, pixel saturation value, number of pixels, and pixel color temperature value.

Specifically, the electronic device may obtain pixel values of M second original images, and perform pairwise subtraction on the pixel values of the M second original images to obtain K difference matrices, K = (M-1) M/2.

Exemplarily, taking M second original images as 3 images as an example, and taking one pixel point in one second original image as an example, it is assumed that pixel values of pixel points corresponding to the 3 second original images obtained by the electronic device are 64, 58, and 75, respectively; the electronic device may perform a difference operation on the pixel value of the first image and the pixel value of the second image, that is, 64 minus 58, to obtain a first difference matrix (6), then the electronic device may perform a difference operation on the pixel value of the first image and the pixel value of the third image, that is, 64 minus 75, to obtain a second difference matrix (-11), and then the electronic device may perform a difference operation on the pixel value of the second image and the pixel value of the third image, that is, 58 minus 75, to obtain a third difference matrix (-17), and then store the three difference matrices (that is, the first difference matrix, the second difference matrix, and the third difference matrix).

Step 202b, the electronic device calculates the standard deviations of the K difference value matrixes respectively to obtain K standard deviations.

In the embodiment of the application, the electronic device may respond to the discrete degrees of the K difference matrices, that is, the noise data difference between the M second original images, by calculating the standard deviation of the K difference matrices.

Step 202c, the electronic device calculates the average value of the K standard deviations to obtain the read noise data.

Optionally, in an embodiment of the present application, the readout noise data includes at least one of: additive noise data, multiplicative noise data, stationary noise data, or non-stationary noise data.

In the embodiment of the application, the electronic device can obtain the real read-out noise data in the current environment according to the M second original images, so that the electronic device can process the shot image according to the read-out noise data, and the accuracy of processing the image by the electronic device is improved.

Alternatively, in this embodiment of the application, the step 202 may be specifically implemented by the following step 202d and step 202 e.

Step 202d, the electronic device calculates to obtain an average pixel matrix according to the M second original images.

In this embodiment, the electronic device may average each pixel position in the M second original images to obtain an average pixel matrix of the M second original images.

Step 202e, the electronic device calculates the row standard deviation and the column standard deviation of the average pixel matrix to obtain the target fixed pattern noise data.

In an embodiment of the application, the electronic device may determine the vertical fixed pattern noise data and the horizontal column fixed pattern noise data in the M second original images by calculating a row standard deviation and a column standard deviation of the average pixel matrix.

In the embodiment of the application, the electronic equipment can obtain the real target fixed graph noise data in the current environment according to the M second original images, so that the electronic equipment can process the shot images according to the target fixed graph noise data, and the accuracy of processing the images by the electronic equipment is improved.

And 203, the electronic equipment performs noise reduction processing on the first original image based on the target noise data to obtain a target image.

In the embodiment of the application, after the electronic device obtains the target noise data, the electronic device may input the target noise data and the first original image into a preset model, and then obtain the target image.

It should be noted that the target image is a digital signal image of an image captured by a camera in a shooting preview interface by a user.

Specifically, the electronic device may input the calculated target noise data and the first original image into the AI-NR algorithm model for processing to obtain the target image.

Optionally, in this embodiment of the present application, after the electronic device performs noise reduction processing on the first original Image and M second original images, the electronic device may input the first original Image after the noise reduction processing into an Image Signal Processor (ISP), so as to obtain a target Image.

Optionally, in this embodiment of the application, after obtaining the target image, the electronic device may store the obtained target image in a target application (e.g., an album application); alternatively, the electronic device may display the target image on a capture preview interface.

Alternatively, in this embodiment of the application, the step 203 may be specifically implemented by the following steps 203a and 203 b.

And 203a, the electronic equipment corrects the initial noise calibration curve based on the read noise data and the photon noise data to obtain an actual noise calibration curve.

Note that the above-described photon noise data is noise generated by the photosensitive element itself in the image sensor.

Specifically, the photon noise is noise generated due to a variation in the number of photons reaching the image sensor, resulting in a deviation of an actual situation from a theoretical situation. .

In the embodiment of the application, the electronic device can acquire the color card image through the color card, so that the initial read-out noise of the color card image is determined according to the color card image, and the electronic device can obtain the total noise through a noise algorithm according to the initial read-out noise and the photon noise, wherein the specific algorithm is as follows:

N ² ＝P ² +R ² (formula one)

The method comprises the steps that N is total noise, P is photon noise, R is initial read noise, then, under each sensitivity (ISO value), the electronic equipment can obtain an initial noise calibration curve under each sensitivity according to the total noise and signal value of a color card image, and then the electronic equipment can correct the initial noise calibration curve through read noise data and photon noise data obtained through M pieces of second original images to obtain an actual noise calibration curve. The initial readout noise is calculated in the same manner as in steps 202a-202c, and the photon noise is calculated by squaring the pixel values.

It can be understood that since the photon noise data is noise generated by the photosensitive element itself in the image sensor, which does not change due to a change in the photographing environment, the electronic device corrects the initial noise calibration curve by acquiring actual readout noise under different environments to obtain an actual noise calibration curve.

For example, as shown in fig. 3, the electronic apparatus may capture a color patch image at different sensitivities to obtain initial readout noise and photon noise at the different sensitivities, so that the electronic apparatus may determine an initial noise calibration curve at the different sensitivities according to the initial readout noise and photon noise at the different sensitivities, where the horizontal axis in fig. 3 represents the luminance average value of each color patch and the vertical axis in fig. 3 represents the square of the total noise.

Optionally, in this embodiment of the present application, the electronic device may obtain a slope and an intercept of the initial noise calibration curve at different sensitivities according to the initial noise calibration curve at different sensitivities, where the slope corresponds to the photon noise and the intercept corresponds to the initial readout noise.

For example, for the slopes at different sensitivities, as shown in (a) in fig. 4, the electronic device may fit the slopes at different sensitivities to obtain a fitted slope curve, in which the horizontal axis in (a) in fig. 4 represents the sensitivity (ISO value) and the vertical axis in (a) in fig. 4 represents the slope.

Further illustratively, for the intercepts at different sensitivities, as shown in (B) in fig. 4, the electronic device may fit the intercepts at different sensitivities to obtain a fitted intercept curve, where the horizontal axis in (B) in fig. 4 represents the sensitivity (ISO value) and the vertical axis in (B) in fig. 4 represents the intercept.

It should be noted that, because it is relatively troublesome to record the initial noise calibration curve and the memory occupied is relatively large, the electronic device may obtain the slope and the intercept under different ISO values according to the noise calibration curve under each ISO value, so as to characterize the initial noise calibration curve by the slope and the intercept.

And step 203b, the electronic device performs noise reduction processing on the first original image based on the actual noise calibration curve.

In the embodiment of the present application, the read noise data obtained in step 203a actually corresponds to the actual intercept of a certain ISO value noise reduction curve in fig. 3, so that a more accurate noise reduction curve can be obtained.

In the embodiment of the application, the electronic device can obtain the actual noise calibration curve according to the read noise data and the photon noise data, so that the first original image is subjected to noise reduction according to the actual noise calibration curve, namely, the electronic device can determine the actual noise calibration curve in real time according to the current environment, so that the electronic device performs noise reduction on the first original image according to the actual noise calibration curve, and thus, the quality of images shot by the electronic device is improved.

Alternatively, in this embodiment of the application, the step 203 may be specifically implemented by the step 203d described below.

And step 203d, the electronic equipment performs noise reduction processing on the first original image based on the target fixed pattern noise data.

In this embodiment, the electronic device may subtract the fixed image noise data from the first original image according to the target fixed image noise, so as to obtain the first original image without the fixed image noise.

It is understood that since the first original image and the M second original images are both captured under the same environment, the fixed image noise data in the M second original images and the fixed image noise data on the first original image should be consistent, so subtracting the target fixed image noise determined by the M second original images from the first original image can also remove the fixed image noise in the first original image.

In the embodiment of the application, the electronic device can determine the noise of the target fixed image in real time according to the current environment, so that the electronic device performs noise reduction processing on the first original image according to the noise of the target fixed image, and thus, the quality of the image shot by the electronic device is improved.

The embodiment of the application provides an image processing method, and electronic equipment can determine target noise data of an image sensor by acquiring a first original image and M second original images, and perform noise reduction processing on the first original image according to the target noise data to obtain a target image. In the scheme, the electronic equipment can acquire the target noise data in real time according to the M pieces of second original images and perform noise reduction processing on the first original image according to the target noise data to obtain the target image, so that the problem that when the electronic equipment performs noise reduction processing on the shot image, the electronic equipment cannot accurately remove noise due to the fact that an image sensor cannot determine accurate background noise in real time, and the quality of the image shot by the electronic equipment is poor is avoided, and the quality of the image shot by the electronic equipment is improved.

Optionally, in this embodiment of the application, before "acquiring M second original images by using the image sensor" in step 201, the image processing method provided in this embodiment of the application further includes step 301 described below.

In the case where the targeted noise data includes read-out noise data, the electronic device adjusts the first exposure parameter to a second exposure parameter, step 301.

In the embodiment of the present application, the first exposure parameter is an exposure parameter when the image sensor acquires a first original image, and the second exposure parameter is an exposure parameter when the image sensor acquires M second original images; and the parameter value of the first exposure parameter is larger than the parameter value of the second exposure parameter.

In the embodiment of the application, after the electronic device obtains the first original image, the electronic device can adjust the light transmittance of the electrochromic material to be the second light transmittance, and perform M times of shooting on the shot object by adopting the second exposure parameter to obtain M second original images.

It should be noted that the smaller the value of the second exposure parameter is, the more accurate the target noise data obtained based on the M second original images is.

In the embodiment of the application, the electronic device can shoot the M second original images by adopting exposure parameters different from those of the first original image, so that the background noise data in the M second original images is determined, then the first original image is processed according to the background noise data, a target image with a good noise reduction effect is obtained, and the image quality shot by the electronic device is improved.

An embodiment of the present application provides an electronic device, and fig. 5 shows a schematic structural diagram of the electronic device provided in the embodiment of the present application. As shown in fig. 5, an electronic device provided in an embodiment of the present application includes: and a light transmittance control layer 10 disposed between a lens 11 and an image sensor 12 of the electronic device.

Optionally, in this embodiment of the application, the light transmittance control layer includes at least one light transmitting unit, and each light transmitting unit in the at least one light transmitting unit is disposed corresponding to the microlens layer; when a voltage is applied to the light transmittance control layer, the light transmittance of at least one light transmitting unit in the light transmittance control layer is changed, so that the light entering amount of the microlens layer is changed.

Optionally, in this embodiment of the present application, the lens may be any one of the following: a long-focus lens, a short-focus lens, a fixed-focus lens, an ultra-wide-angle lens, a macro lens and the like.

Optionally, in this embodiment of the application, with reference to fig. 5, as shown in fig. 6, the electronic device further includes: a bracket 13 and a housing 14; the bracket 13 is disposed on two sides of the lens 11 for supporting the lens 11, the bracket 13 is connected to the housing 14, and the image sensor 12 is disposed on the housing 14.

Optionally, in this embodiment of the application, the image sensor may be connected to the housing through a flexible printed circuit board.

Optionally, the number of the light transmission control layers is at least two, and the at least two light transmission control layers are stacked.

For example, as shown in fig. 7 (a), the light transmittance control layer 10 may cover the microlens layer 16 completely in a row-covering manner, so that the amount of light entering the microlens layer varies.

Further illustratively, as shown in fig. 7 (B), the light transmittance control layer 10 may cover the microlens layer 16 completely in a column-covering manner, thereby varying the amount of light entering the microlens layer.

Further illustratively, as shown in (C) of fig. 7, the light transmittance control layer 10 may cover the microlens layer 16 completely in a column-covering or row-covering manner, thereby varying the amount of light entering the microlens layer.

As another example, as shown in fig. 7 (D), the light transmittance control layer 10 may cover the microlens layer 16 completely in a block covering manner, thereby varying the amount of light entering the microlens layer.

The embodiment of the application provides an electronic equipment, and electronic equipment can include camera lens, image sensor and set up the luminousness control layer between camera lens and image sensor to under the circumstances that electronic equipment applyed the voltage to luminousness control layer, the luminousness of luminousness control layer can change, and then changes image sensor's luminousness, then electronic equipment can be according to the luminousness of the image sensor after changing, confirms image sensor's noise data. In the scheme, the electrochromic material and the image sensor are arranged oppositely, so that the light transmittance obtained by the image sensor can be changed, namely the light transmittance obtained by the photosensitive pixels in any area in the image sensor can be changed by the electronic equipment, the noise data of the photosensitive pixels in the image sensor can be determined according to the changed light transmittance, then the electronic equipment can compensate the shot image according to the noise data, and the image quality shot by the electronic equipment is improved.

In the image processing method provided in the embodiment of the present application, the execution subject may be an image processing apparatus. In the embodiments of the present application, an image processing apparatus that executes an image processing method is taken as an example to describe the image processing apparatus provided in the embodiments of the present application.

Fig. 8 is a schematic diagram showing a possible configuration of the image processing apparatus according to the embodiment of the present application. As shown in fig. 8, the image processing apparatus 70 may include: an acquisition module 71, a determination module 72 and a processing module 73.

The obtaining module 71 is configured to obtain a first original image and M second original images, where M is a positive integer. And a determining module 72, configured to determine target noise data of the image sensor according to the M second original images. A processing module 73, configured to perform noise reduction processing on the first original image based on the target noise data to obtain a target image; when the first original image is obtained, the light transmittance of the light transmittance control layer is larger than 0; when a second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

In a possible implementation manner, the obtaining module 71 is specifically configured to control the light transmittance of the light transmittance control layer to be a first light transmittance, and acquire a first original image through an image sensor; and controlling the light transmittance of the light transmittance control layer to be 0, and acquiring M second original images through the image sensor.

In a possible implementation manner, the determining module 72 is specifically configured to calculate K difference matrices according to M second original images; respectively calculating the standard deviations of the K difference value matrixes to obtain K standard deviations; and calculating the average value of the K standard deviations to obtain read noise data.

In a possible implementation manner, the processing module 73 is specifically configured to correct the initial noise calibration curve based on the read noise data and the photon noise data, so as to obtain an actual noise calibration curve; and performing noise reduction processing on the first original image based on the actual noise calibration curve.

In a possible implementation manner, the determining module 72 is specifically configured to calculate an average pixel matrix according to M second original images; and calculating the row standard deviation and the column standard deviation of the average pixel matrix to obtain the target fixed pattern noise data.

In a possible implementation manner, the processing module 73 is specifically configured to perform noise reduction processing on the first original image based on the target fixed pattern noise data.

In one possible implementation, the target noise data includes read noise data; the processing module 73 is further configured to, before the obtaining module 71 collects M second original images through the image sensor, adjust the first exposure parameter to a second exposure parameter, where the first exposure parameter is an exposure parameter when the image sensor collects the first original image, and the second exposure parameter is an exposure parameter when the image sensor collects M second original images; wherein the parameter value of the first exposure parameter is larger than the parameter value of the second exposure parameter.

In one possible implementation, in a case where the target noise data includes fixed pattern noise data; the acquiring module 71 is specifically configured to acquire M second original images by using a first exposure parameter, where the first exposure parameter is an exposure parameter when the image sensor acquires the first original image.

In a possible implementation manner, the material of the light transmittance control layer is an electrochromic material. The embodiment of the application provides an image processing device, because image processing device can obtain target noise data in real time according to M pieces of second original images, and carry out noise reduction processing on first original images according to the target noise data, obtain the target image, avoided when image processing device carries out noise reduction processing to the shot image, because image sensor can't confirm accurate end noise in real time, and lead to the fact image processing device can't accurate remove the noise, lead to the image quality that image processing device shot poor, so, promoted the image quality that image processing device shot.

The image processing apparatus in the embodiment of the present application may be an apparatus, and may also be a component, an integrated circuit, or a chip in an electronic device. The device can be mobile electronic equipment or non-mobile electronic equipment. By way of example, the Mobile electronic Device may be a Mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic Device, a Mobile Internet Device (MID), an Augmented Reality (AR)/Virtual Reality (VR) Device, a robot, a wearable Device, a super-Mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and may also be a server, a Network Attached Storage (Network Attached Storage, personal computer (NAS PC), a Television (TV), a teller machine, a self-service machine, and the like, and embodiments of the present application are not limited in particular.

The image processing apparatus in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an IOS operating system, or other possible operating systems, which is not specifically limited in the embodiments of the present application.

The image processing apparatus provided in the embodiment of the present application can implement each process implemented by the method embodiments of fig. 1 to fig. 8, and is not described herein again to avoid repetition.

Optionally, as shown in fig. 9, an electronic device 90 is further provided in this embodiment of the present application, and includes a processor 91 and a memory 92, where the memory 92 stores a program or an instruction that can be executed on the processor 91, and when the program or the instruction is executed by the processor 91, the steps of the embodiment of the image processing method are implemented, and the same technical effects can be achieved, and are not described again to avoid repetition.

It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 10 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes, but is not limited to: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

Those skilled in the art will appreciate that the electronic device 100 may further include a power source (e.g., a battery) for supplying power to the various components, and the power source may be logically connected to the processor 110 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in fig. 10 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.

The processor 110 is configured to obtain a first original image and M second original images, where M is a positive integer; determining target noise data of the image sensor according to the M second original images; performing noise reduction processing on the first original image based on the target noise data to obtain a target image; wherein, when the first original image is obtained, the light transmittance of the light transmittance control layer is greater than 0; when a second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

The embodiment of the application provides an electronic device, which can acquire target noise data in real time according to M pieces of second original images and perform noise reduction processing on a first original image according to the target noise data to obtain a target image, so that the problem that when the electronic device performs noise reduction processing on a shot image, noise cannot be accurately removed due to the fact that an image sensor cannot determine accurate background noise in real time, and the quality of the image shot by the electronic device is poor is avoided, and the quality of the image shot by the electronic device is improved.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to control the light transmittance of the electrochromic material to be a first light transmittance, and acquire a first original image through an image sensor; and controlling the light transmittance of the electrochromic material to be 0, and acquiring M second original images through the image sensor.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to calculate K difference matrices according to M second original images; respectively calculating the standard deviations of the K difference value matrixes to obtain K standard deviations; and calculating the average value of the K standard deviations to obtain read noise data.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to correct the initial noise calibration curve based on the read noise data and the photon noise data, so as to obtain an actual noise calibration curve; and performing noise reduction processing on the first original image based on the actual noise calibration curve.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to calculate an average pixel matrix according to M second original images; and calculating the row standard deviation and the column standard deviation of the average pixel matrix to obtain the noise data of the target fixed graph.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to perform noise reduction processing on the first original image based on the target fixed pattern noise data.

Alternatively, in the embodiment of the present application, in the case where the above-described target noise data includes read-out noise data; the processor 110 is specifically configured to adjust a first exposure parameter to a second exposure parameter, where the first exposure parameter is an exposure parameter when the image sensor acquires a first original image, and the second exposure parameter is an exposure parameter when the image sensor acquires M second original images; and the parameter value of the first exposure parameter is larger than the parameter value of the second exposure parameter.

Optionally, in this embodiment of the application, in a case where the target noise data includes fixed pattern noise data; the processor 110 is specifically configured to acquire M second original images by using a first exposure parameter, where the first exposure parameter is an exposure parameter when the image sensor acquires the first original image.

The electronic device provided by the embodiment of the application can realize each process realized by the method embodiment, can achieve the same technical effect, and is not repeated herein for avoiding repetition.

The beneficial effects of the various implementation manners in this embodiment may specifically refer to the beneficial effects of the corresponding implementation manners in the above method embodiments, and are not described herein again to avoid repetition.

It should be understood that, in the embodiment of the present application, the input Unit 104 may include a Graphics Processing Unit (GPU) 1041 and a microphone 1042, and the Graphics Processing Unit 1041 processes image data of a still picture or a video obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also referred to as a touch screen. The touch panel 1071 may include two parts of a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in further detail herein.

The memory 109 may be used to store software programs as well as various data. The memory 109 may mainly include a first storage area storing a program or an instruction and a second storage area storing data, wherein the first storage area may store an operating system, an application program or an instruction (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, memory 109 may comprise volatile memory or non-volatile memory, or memory 109 may comprise both volatile and non-volatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. The volatile Memory may be a Random Access Memory (RAM), a Static Random Access Memory (Static RAM, SRAM), a Dynamic Random Access Memory (Dynamic RAM, DRAM), a Synchronous Dynamic Random Access Memory (SDRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (ddr SDRAM ), an Enhanced SDRAM (ESDRAM), a SLDRAM (Synchronous link DRAM), and a Direct Rambus RAM (DRRAM). Memory 109 in the embodiments of the subject application includes, but is not limited to, these and any other suitable types of memory.

Processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor that primarily handles operations involving the operating system, user interface, and applications, and a modem processor that primarily handles wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 110.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements the processes of the foregoing method embodiments, and can achieve the same technical effects, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory ROM, a random access memory RAM, a magnetic or optical disk, and the like.

The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to execute a program or an instruction to implement each process of the foregoing method embodiment, and the same technical effect can be achieved.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as a system-on-chip, or a system-on-chip.

Embodiments of the present application provide a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the processes of the foregoing embodiments of the image processing method, and achieve the same technical effects, and in order to avoid repetition, details are not repeated here.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one of 8230, and" comprising 8230does not exclude the presence of additional like elements in a process, method, article, or apparatus comprising the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order, depending on the functionality involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the above embodiment method can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application may be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (which may be a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. An image processing method applied to an electronic device, the electronic device comprising: a light transmittance control layer disposed between a lens and an image sensor of the electronic device; the method comprises the following steps:

acquiring a first original image and M second original images, wherein M is a positive integer;

determining target noise data of the image sensor according to the M second original images;

based on the target noise data, carrying out noise reduction processing on the first original image to obtain a target image;

when the first original image is obtained, the light transmittance of the light transmittance control layer is greater than 0; when the second original image is obtained, the light transmittance of the light transmittance control layer is 0; the ISO sensitivity corresponding to the first original image is the same as the ISO sensitivity corresponding to the second original image.

2. The method of claim 1, wherein the acquiring the first original image and the M second original images comprises:

controlling the light transmittance of the light transmittance control layer to be a first light transmittance, and acquiring the first original image through the image sensor;

and controlling the light transmittance of the light transmittance control layer to be 0, and acquiring the M second original images through the image sensor.

3. The method of claim 1, wherein determining target noise data for the image sensor from the M second raw images comprises:

calculating to obtain K difference matrixes according to the M second original images;

respectively calculating the standard deviation of the K difference value matrixes to obtain K standard deviations;

and calculating the average value of the K standard deviations to obtain read noise data.

4. The method of claim 3, wherein the denoising the first original image based on the target noise data comprises:

correcting the initial noise calibration curve based on the read noise data and the photon noise data to obtain an actual noise calibration curve;

and performing noise reduction processing on the first original image based on the actual noise calibration curve.

5. The method of claim 1, wherein determining target noise data for the image sensor from the M second raw images comprises:

calculating to obtain an average pixel matrix according to the M second original images;

and calculating the row standard deviation and the column standard deviation of the average pixel matrix to obtain the noise data of the target fixed graph.

6. The method of claim 5, wherein the denoising the first original image based on the target noise data comprises:

and performing noise reduction processing on the first original image based on the target fixed pattern noise data.

7. The method of claim 1, wherein in the case where the target noise data includes read-out noise data;

before the acquiring, by the image sensor, the M second original images, the method further includes:

adjusting a first exposure parameter to a second exposure parameter, wherein the first exposure parameter is an exposure parameter when the image sensor acquires the first original image, and the second exposure parameter is an exposure parameter when the image sensor acquires the M second original images;

wherein the parameter value of the first exposure parameter is greater than the parameter value of the second exposure parameter.

8. The method of claim 1, wherein in the case where the target noise data comprises fixed pattern noise data;

the acquiring, by the image sensor, the M second original images includes:

and acquiring the M second original images by adopting a first exposure parameter, wherein the first exposure parameter is an exposure parameter when the image sensor acquires the first original image.

9. The method of claim 1, wherein the material of the transmittance control layer is an electrochromic material.

10. An image processing apparatus applied to an electronic device, the electronic device comprising: a light transmittance control layer disposed between a lens and an image sensor of the electronic device; the image processing apparatus includes: the device comprises an acquisition module, a determination module and a processing module;

the acquisition module is used for acquiring a first original image and M second original images, wherein M is a positive integer;

the determining module is used for determining target noise data of the image sensor according to the M second original images;

the processing module is used for carrying out noise reduction processing on the first original image based on the target noise data to obtain a target image;

11. The apparatus according to claim 10, wherein the obtaining module is specifically configured to control the light transmittance of the light transmittance control layer to be a first light transmittance, and acquire the first original image through the image sensor; and controlling the light transmittance of the light transmittance control layer to be 0, and acquiring the M second original images through the image sensor.

12. The apparatus according to claim 10, wherein the determining module is specifically configured to obtain K difference matrices by calculation according to the M second original images obtained by the obtaining module; respectively calculating the standard deviations of the K difference value matrixes to obtain K standard deviations; and calculating the average value of the K standard deviations to obtain read noise data.

13. The apparatus according to claim 12, wherein the processing module is specifically configured to correct an initial noise calibration curve based on the readout noise data and the photon noise data to obtain an actual noise calibration curve; and performing noise reduction processing on the first original image based on the actual noise calibration curve.

14. An electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor, which program or instructions, when executed by the processor, implement the steps of the image processing method according to any one of claims 1 to 9.

15. A readable storage medium, characterized in that it stores thereon a program or instructions which, when executed by a processor, implement the steps of the image processing method according to any one of claims 1 to 9.