WO2022127738A1

WO2022127738A1 - Image processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022127738A1
Application number: PCT/CN2021/137491
Authority: WO
Inventors: 张华�; 王津男
Original assignee: 华为技术有限公司
Priority date: 2020-12-14
Filing date: 2021-12-13
Publication date: 2022-06-23
Also published as: CN114626995A

Abstract

The present application relates to an image processing method and apparatus, an electronic device, and a storage medium, the method comprising: acquiring an image to be processed; the image to be processed is an image collected by an under-screen imaging system, the under-screen imaging system comprising a display screen and a camera arranged below the display screen; the image to be processed comprises a highlight object and glare produced thereby, the highlight object being a photographed object of a specified colour or brightness in the image to be processed; on the basis of the spectral composition of the highlight object, generating a first point spread function; the first point spread function is a point spread function of at least one channel in the under-screen imaging system; and performing deconvolution processing on the first point spread function and the image to be processed to obtain a de-glared image. In the present application, the glare produced by an image photographed by a camera under a display screen can be improved or eliminated, in particular rainbow glare, thus increasing the clarity of the image and enhancing the user experience.

Description

An image processing method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application filed on December 14, 2020 with the application number 202011474431.3 and the invention titled "An image processing method, device, electronic device and storage medium", the entire contents of which are approved by Reference is incorporated in this application.

technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method, apparatus, electronic device, and storage medium.

Background technique

The existing mobile phone adopts a full-screen mobile phone composed of a lifting front lens, and a sliding-type full-screen mobile phone. These designs make the visual experience of the mobile phone close to 100% full screen, but its complex mechanical structure will increase the thickness and weight of the mobile phone, occupying too much internal space, and the repeated use of the mechanical mechanism is also difficult to meet the needs of mobile phone users.

Based on the display design using organic light-emitting diodes (Organic Light Emitting Diode, OLED), the mobile phone can increase the screen ratio of the display. The camera can be set under the display screen, but due to the occlusion of the display screen, the images captured by the camera set under the display screen have glare, especially when shooting bright objects, the glare is serious, which greatly reduces the imaging effect of the camera and affects the user. experience.

SUMMARY OF THE INVENTION

In view of this, an image processing method, apparatus, electronic device and storage medium are proposed.

In a first aspect, an embodiment of the present application provides an image processing method, the method includes: acquiring an image to be processed; the to-be-processed image is an image collected by an under-screen imaging system, and the under-screen imaging system includes a display a screen and a camera arranged under the display screen; the to-be-processed image includes: a highlight object and the glare generated by it, wherein the highlight object is a specified color and/or brightness in the to-be-processed image photographing an object; generating a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point spread function of at least one channel in the under-screen imaging system; for the first point spread function The function and the image to be processed are subjected to deconvolution processing to obtain the image after the glare has been removed.

Based on the above technical solution, for the to-be-processed image collected by the under-screen imaging system, the to-be-processed image includes: a highlight object and the glare generated by it, and according to the spectral components of the highlight object in the to-be-processed image, a different highlight object is generated. The point spread function of at least one channel in the adaptive off-screen imaging system, and then the first point spread function and the to-be-processed image are deconvolved to obtain an image after removing the glare generated by the bright object; thus, it can effectively improve or Eliminates glare, especially rainbow glare, generated by images captured by cameras under the display screen, improving image clarity and user experience.

According to the first aspect, in a first possible implementation manner of the first aspect, the generating a first point spread function according to the spectral components of the highlighted object includes: according to the spectrum of the highlighted object components and the model of the display screen to generate a second point spread function; wherein, the second point spread function is the point spread function corresponding to different wavelengths of the highlighted object after passing through the under-screen imaging system; according to the The second point spread function and the photosensitive characteristic curve of the camera are used to generate the first point spread function.

Based on the above technical solution, according to the model of the display screen, combined with the spectral components of the highlighted object, the point spread function of the different wavelengths of the highlighted object passing through the under-screen imaging system can be generated, and then combined with the photosensitive characteristic curve of the camera to obtain the under-screen imaging system. The point spread function of at least one channel. In this way, the point spread function corresponding to the real glare can be obtained through physical modeling by using the spectral components of the high-brightness objects, so that the dispersion effect due to the periodic structure of the display screen can be effectively eliminated, and the glare caused by the periodic structure of the display screen can be effectively eliminated. , especially the rainbow glare has an excellent suppression effect, which improves the imaging quality of the under-screen imaging system.

According to the first aspect, in a second possible implementation manner of the first aspect, the generating a first point spread function according to the spectral components of the highlighted object includes: according to the spectrum of the highlighted object composition and a preset third point spread function to generate the first point spread function; wherein, the third point spread function is the point spread function corresponding to different wavelengths after passing through the under-screen imaging system.

Based on the above technical solution, the point spread function of at least one channel in the under-screen imaging system can be quickly generated through the preset point spread functions of different wavelengths after passing through the under-screen imaging system, combined with the spectral components of the above-mentioned high-brightness objects. The calculation efficiency is improved, and the processing speed of anti-glare is improved, and it can be applied to the anti-glare processing scene with a large amount of processing data and high processing speed requirements such as video shooting.

According to the first aspect, in a third possible implementation manner of the first aspect, the method further includes: acquiring the spectral components of the high-brightness object collected by the spectral measurement device.

Based on the above technical solutions, the spectral components of high-brightness objects can be acquired in real time during shooting through the spectral measurement equipment built into the electronic equipment, without relying on external equipment, which improves the functional integrity of the electronic equipment; The spectral components can be obtained to obtain the spectral components of bright objects, which saves the cost of electronic equipment.

According to the first aspect, in a fourth possible implementation manner of the first aspect, the method further includes: performing image recognition on the to-be-processed image to determine the light source type of the highlighted object; The spectral components of different light source types are determined to determine the spectral components of the highlighted object.

Based on the above technical solution, by pre-storing the spectral components corresponding to different light source types, the light source type of the highlighted object in the photographed object can be identified during shooting, so as to determine the spectral composition of the highlighted object; in this way, the electronic device does not need to be additionally configured with a spectral measurement device , the spectral composition of the bright object can be obtained, and the cost of electronic equipment can be saved.

According to the first aspect, in a fifth possible implementation manner of the first aspect, the method further includes: dividing the to-be-processed image into a first area and a second area; wherein the first area is the area where the bright object and its glare are located; the second area is the area other than the first area in the image to be processed; the pair of the first point spread function and the image to be processed Performing deconvolution processing to obtain the image after removing the glare, comprising: performing deconvolution processing on the first point spread function and the first area to obtain the first area after removing the glare; The glare-removed first area is fused with the second area to obtain the glare-removed image.

Based on the above technical solution, the image to be processed is divided into a first area where the highlighted object and the glare generated by it are located, and a second area that does not include the highlighted object and the glare generated by it. Further anti-glare processing is performed in the first area where it is located, which improves processing efficiency and saves processing resources; at the same time, it can reduce or eliminate the first area that does not contain bright objects and the glare generated by them in the image to be processed during the anti-glare processing. The effect of the second area.

According to a first possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the generation of the The two point spread function includes: taking the model of the display screen as the transmittance function, combining the light wave propagation factor, and taking the spectral component of the highlighted object as the weight to obtain the second point spread function; wherein, the The model of the display screen includes: an amplitude modulation function and a phase modulation characteristic function of the display screen; the light wave propagation factor is determined according to the focal length when the camera captures the to-be-processed image and the wavelength of the highlighted object.

Based on the above technical solutions, starting from the root cause of glare, that is, the diffraction effect caused by the periodic pixel arrangement of the display screen is sensitive to wavelength, the display screen model constructed by the amplitude modulation function and the phase modulation characteristic function is used as the transparent The over-rate function is used to determine the light wave propagation factor according to the focal length of the camera to capture the image to be processed and the actual wavelength of the highlighted object. At the same time, the spectral components of the highlighted object are used as weights to obtain the different wavelengths of the highlighted object after passing through the under-screen imaging system. Corresponding point spread function, the obtained point spread function can more realistically express the process of glare generated by bright objects through under-screen imaging.

In a second aspect, an embodiment of the present application provides an image processing apparatus, the apparatus includes: an acquisition module, configured to acquire an image to be processed; the image to be processed is an image collected by an under-screen imaging system, and the screen The lower imaging system includes a display screen and a camera arranged under the display screen; the to-be-processed image includes: a highlight object and the glare generated by it, wherein the highlight object is a specified color in the to-be-processed image and/or a photographic object of brightness; a generating module, configured to generate a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point of at least one channel in the under-screen imaging system a spread function; a processing module configured to perform deconvolution processing on the first point spread function and the to-be-processed image to obtain the image after the glare has been removed.

According to the second aspect, in a first possible implementation manner of the second aspect, the generating module is further configured to: generate a second point spread function; wherein, the second point spread function is the point spread function corresponding to the light of different wavelengths of the highlighted object after passing through the under-screen imaging system; according to the second point spread function and the camera The photosensitive characteristic curve is used to generate the first point spread function.

Based on the above technical solutions, according to the displayed screen model, combined with the spectral components of the highlighted objects, the point spread functions of the different wavelengths of the highlighted objects passing through the under-screen imaging system can be generated, and then combined with the photosensitive characteristic curve of the camera, the under-screen imaging system can be obtained. The point spread function of at least one channel. In this way, the point spread function corresponding to the real glare can be obtained through physical modeling by using the spectral components of the high-brightness objects, so that the dispersion effect due to the periodic structure of the display screen can be effectively eliminated, and the glare caused by the periodic structure of the display screen can be effectively eliminated. , especially the rainbow glare has an excellent suppression effect, which improves the imaging quality of the under-screen imaging system.

According to the second aspect, in a second possible implementation manner of the second aspect, the generation module is further configured to: generate the generation module according to the spectral components of the highlighted object and a preset third point spread function The first point spread function; wherein, the third point spread function is the point spread function corresponding to different wavelengths after passing through the under-screen imaging system.

Based on the above technical solution, the point spread function of at least one channel in the under-screen imaging system can be quickly generated by using the corresponding point spread functions of different preset wavelengths after passing through the under-screen imaging system, combined with the spectral components of the above-mentioned high-brightness objects, The calculation efficiency is significantly improved, and the processing speed of anti-glare is improved, which can be applied to the anti-glare processing scene with large amount of processing data and high processing speed requirements such as video shooting.

According to the second aspect, in a third possible implementation manner of the second aspect, the apparatus further includes: a spectral measurement device, configured to collect spectral components of the highlighted object.

According to the second aspect, in a fourth possible implementation manner of the second aspect, the device further includes: an acquisition module, configured to: perform image recognition on the to-be-processed image, and determine the size of the highlighted object. Light source type; according to preset spectral components of different light source types, determine the spectral components of the highlighted object.

According to the second aspect, in a fifth possible implementation manner of the second aspect, the apparatus further includes: a segmentation module, configured to: segment the to-be-processed image into a first area and a second area; wherein , the first area is the area where the bright object and the glare generated by it are located; the second area is the area other than the first area in the image to be processed; the processing module is also used for: by Perform deconvolution processing on the first point spread function and the first region to obtain the first region after removing the glare; deconvolute the first region and the second region after removing the glare Fusion to obtain the image after removing the glare.

According to a first possible implementation manner of the second aspect, in a sixth possible implementation manner of the second aspect, the generating module is further configured to: use the model of the display screen as a transmittance function , combined with the light wave propagation factor, and using the spectral components of the highlighted object as the weight, the second point spread function is obtained; wherein, the model of the display screen includes: the amplitude modulation function and phase modulation characteristics of the display screen function; the light wave propagation factor is determined according to the focal length when the camera captures the to-be-processed image and the wavelength of the highlighted object.

In a third aspect, embodiments of the present application provide an electronic device, including: a display screen, a camera below the display screen, a processor, and a memory; wherein, the camera is used to collect images to be processed through the display screen; The display screen is used to display the to-be-processed image and the image after glare removal; the memory is used to store processor-executable instructions; the processor is configured to implement the above-mentioned first aspect or One or more image processing methods in multiple possible implementation manners of the first aspect.

In a fourth aspect, embodiments of the present application provide a non-volatile computer-readable storage medium on which computer program instructions are stored, characterized in that, when the computer program instructions are executed by a processor, the above-mentioned first aspect is implemented Or one or more image processing methods in multiple possible implementation manners of the first aspect.

In a fifth aspect, embodiments of the present application provide a computer program product, comprising computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in an electronic When running in the device, the processor in the electronic device executes the first aspect or one or more of the image processing methods in multiple possible implementations of the first aspect.

These and other aspects of the present application will be more clearly understood in the following description of the embodiment(s).

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features and aspects of the application and together with the description, serve to explain the principles of the application.

1 shows a schematic diagram of a mobile phone configured with an under-screen camera according to an embodiment of the present application;

2A-2B show schematic diagrams of displaying images through the mobile phone 10 according to an embodiment of the present application;

FIG. 3 shows a schematic diagram of a shooting interface according to an embodiment of the present application;

FIG. 4 shows a flowchart of an image processing method according to an embodiment of the present application;

5 shows a schematic diagram of extracting the shape of the highlighted object and the glare starburst area in the glare area 302 in the above-mentioned FIG. 3 according to an embodiment of the present application;

FIG. 6 shows the spectral components of a strong light source collected by a color temperature sensor according to an embodiment of the present application;

FIG. 7 shows a schematic diagram of a broadening of the point spread function of the under-screen imaging system according to an embodiment of the present application;

FIG. 8 shows a partial schematic diagram of a display screen according to an embodiment of the present application;

9 shows a schematic diagram of a filter transmittance curve according to an embodiment of the present application;

10 shows a schematic diagram of the point spread function of the generated RBG three-channel according to an embodiment of the present application;

11 shows a schematic diagram of the point spread function corresponding to the pre-calibrated light of different wavelengths passing through the under-screen imaging system according to an embodiment of the present application;

FIG. 12 shows a schematic diagram of a partial image after deglare according to an embodiment of the present application;

FIG. 13 shows a flowchart of an image processing method according to an embodiment of the present application;

FIG. 14 shows a structural diagram of an image processing apparatus according to an embodiment of the present application;

FIG. 15 shows a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 16 shows a block diagram of a software structure of an electronic device according to an embodiment of the present application.

Detailed ways

Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

In addition, in order to better illustrate the present application, numerous specific details are given in the following detailed description. It should be understood by those skilled in the art that the present application may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present application.

For ease of understanding, some terms involved in the embodiments of the present application are explained.

Point spread function (PSF), for an optical system, the light field distribution of the output image when the input object is a point light source is called the point spread function, also known as the point spread function. Mathematically, a point light source can be represented by a delta function (point pulse), and the light field distribution of the output image is called an impulse response, so the point spread function is also the impulse response function of the optical system. The imaging performance of an optical system can be expressed by the point spread function of the optical system.

Non-blind deconvolution (Non-blind deconvolution), also known as non-blind deconvolution; the image formed by the optical system can be understood as the result of the convolution of the original image and the point spread function of the optical system. The point spread function of the optical system, the process of restoring the original image is called non-blind deconvolution.

The principle of OLED is the phenomenon that organic semiconductor materials and light-emitting materials are driven by an electric field and lead to light emission through carrier injection and recombination. OLED is a device that uses a multilayer organic thin film structure to generate electroluminescence. It is easy to fabricate and only requires low driving voltage. These main features make OLED very prominent in the application of flat panel displays. OLED displays are lighter and thinner than LCDs, with high brightness, low power consumption, fast response, high definition, good flexibility and high luminous efficiency.

The embodiment of the present application provides an image processing method, and the method can be applied to an electronic device, and the electronic device can include a display screen and a camera under the display screen. Among them, the display screen can be a touch screen or a non-touch screen. The touch screen can control the electronic device by clicking or sliding on the screen with a finger, a stylus, etc., and the non-touch screen device can be connected to Input devices such as mouse, keyboard, touch panel, etc., through which the electronic device is controlled; the camera may include a front camera and/or a rear camera.

Exemplarily, the electronic device of the present application may be a smart phone, a netbook, a tablet computer, a notebook computer, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a car device, a TV, a virtual reality device, an audio system, and an electronic ink. , etc., the embodiments of the present application do not limit the specific type of the electronic device.

Taking the electronic device as a mobile phone as an example, FIG. 1 shows a schematic diagram of a mobile phone configured with an under-screen camera according to an embodiment of the present application; as shown in FIG. 1 , the mobile phone 10 may include a display screen 101 and a camera 102 ; wherein, the display screen 101 For displaying images or videos, etc., the camera 102 is arranged below the display screen 101, and is used to shoot images or videos through the display screen 101; One), the display screen 101 and the camera 102 constitute the under-screen imaging system of the mobile phone 10 .

Since the camera is set under the display screen, when the camera shoots through the display screen, the captured image or video will have glare, especially when shooting bright objects, the glare is serious; A schematic diagram of displaying an image through the mobile phone 10 according to an embodiment of the present application. 2A is a schematic diagram of the original image displayed by the mobile phone 10 before being processed by the image processing method according to the embodiment of the present application. As shown in FIG. 2A , obvious glare 202 appears in the original image 201 ; FIG. 2B is an image displayed by the mobile phone 10 A schematic diagram of a high-definition image processed by the image processing method according to the embodiment of the present application, as shown in FIG. 2B , in the high-definition image 203 , a bright object - the sun 204 can be clearly seen; The photographed sun 204 appears in the original image 201 in the form of rainbow glare (not shown in the figure).

The main reasons for glare are: (1) The display screen used to display images and other content consists of many pixels representing red, green and blue colors, and these pixels are periodically arranged; when the camera located under the display screen passes through the display Diffraction effect occurs when external light passes through the monitor when shooting with a screen.

(2) When there is a bright object in the shot scene, such as a strong light source, under the illumination of the strong light source, the diffraction effect of the display screen will be further strengthened, and the light from the central area of the strong light source will be diffracted to the surrounding area, causing the camera At the same time, the limited dynamic range of the image sensor of the camera cannot capture the changing law of diffracted light.

(3) The diffraction effect of the display screen is sensitive to the wavelength, that is, the diffraction effect is different for light of different wavelengths, and the diffraction broadening of the light of different wavelengths after passing through the display screen and the lens of the camera is different, which is called dispersion. Phenomenon; this phenomenon of chromatic dispersion gives rise to rainbow glare in captured images.

In some embodiments, an anti-glare solution is: increasing the lens aperture of the camera to increase the amount of light entering; using a large-sized image sensor to improve the image sensor's ability to sense dark light; adding a new type of high transmittance light Materials, improve the light transmittance of the display screen; adopt the high dynamic range image (High Dynamic Range, HDR) data acquisition mode to improve the dynamic range of the image obtained by the camera and reduce the glare characteristics. In these embodiments, only the problem of low imaging throughput of the camera located under the display screen is considered, and the incoming light of the lens of the camera, the photosensitive ability of the image sensor or the light transmission characteristics of the display screen are increased; or only the pixel structure of the display screen is optimized. , which changes the energy distribution at the location of the glare. The effect of this solution in eliminating glare is limited, because it does not directly face the root cause of glare: the diffraction effect caused by the periodic pixel arrangement of the display screen is sensitive to wavelengths, and this solution cannot eliminate the rainbow generated by the camera imaging under the display screen. glare.

In other embodiments, there is another anti-glare solution: optimizing the pixel arrangement, pixel structure, and pixel driving circuit design of the OLED display screen. By improving the local characteristics of the periodic pixel structure of the display screen, the energy distribution of the point spread function of the under-screen imaging system and the distribution of glare can be optimized. However, due to the limitation of the size of the light-emitting area of the display screen, this solution can only improve the glare effect to a certain extent, and the improvement effect is limited; This solution cannot eliminate the rainbow glare caused by the imaging of the camera under the display.

In order to solve the above technical problems, some embodiments of the present application provide an image processing method, the image processing method can generate a point spread function suitable for different highlighted objects according to the spectral components of the highlighted objects in the image, and then solve the problem by solving The convolution process obtains a high-definition image after glare removal; thus, the glare generated by the image captured by the camera under the display screen can be effectively improved or eliminated, especially the rainbow glare, which improves the clarity of the image and enhances the user experience.

In the following, an image processing method provided by an embodiment of the present application is specifically introduced in conjunction with an application scenario.

In some scenarios, when a user shoots through an electronic device, real-time anti-glare processing is performed on the captured image.

Exemplarily, when there is a bright object in the captured image, the electronic device can turn on the anti-glare function, so as to improve or eliminate the glare generated by the image captured by the camera under the display screen, especially the rainbow glare. The highlighted objects may include strong light sources, such as the sun, the moon, lights, display screens, etc., and may also include objects whose surfaces reflect strong light, such as glass, metal, and the like.

Taking the mobile phone in Figure 1 above as an example, the front camera of the mobile phone is set under the display screen, the user takes a selfie through the front camera of the mobile phone outdoors, and the user turns on the front camera function of the mobile phone, so that the display screen enters the shooting mode. interface, the user can observe the captured picture in real time through the preview image displayed in the shooting interface; exemplarily, the preview image is the original image 201 shown in the above-mentioned FIG. 2A, and glare can be seen in the preview image, that is The sun appears as rainbow glare, and at the same time, the rainbow glare blocks some objects (such as mountain peaks), and the mobile phone turns on the anti-glare function, performs real-time anti-glare processing, and obtains a high-definition image with no glare removed. The high-definition image is shown in Figure 2B above. In the high-definition image 203, the glare is eliminated in the high-definition image displayed on the shooting interface, and the sun and some mountain peaks originally blocked by the rainbow glare of the sun can be seen; Preview the glare of the sun (rainbow glare) in the image so that you can get a high-definition image with no glare.

The following is an example of how the display screen of the mobile phone enters the shooting interface:

The user can trigger an instruction to enter the shooting, the mobile phone responds to the instruction, turns on the camera, and at the same time the display screen enters the shooting interface; for example, the user can click the main page in the display screen, or the boot page, or take a photo in other applications. icon to make the display screen enter the shooting interface; the user can also press the physical button for shooting set on the mobile phone to make the display screen enter the shooting interface; the user can also make the display screen enter the shooting interface through voice commands; the user can also The display screen is brought into the shooting interface by means of a shortcut gesture; in practical applications, the user can also make the display screen enter the shooting interface by other means, which is not limited in this embodiment of the present application.

3 shows a schematic diagram of a shooting interface according to an embodiment of the present application; as shown in FIG. 3, in addition to displaying a preview image in the shooting interface; the shooting interface may also include aperture, night scene, portrait, photo, video, professional, etc. There are options for multiple operation modes, and users can switch between different operation modes to enter specific operation modes such as shooting, video recording, night scene, etc.; the shooting interface can also include options for functions such as flash, HDR, AI, settings, and color tone. , the user can select different functions, turn on or off the flash, adjust the color tone, etc. The shooting interface can also include focal length adjustment options, so that the preview image can be enlarged or reduced accordingly by adjusting the focal length; in addition, the shooting interface It may also include a shooting button, an album button, a camera switching button, and the like.

The following is an example of how to enable the anti-glare function on a mobile phone:

Method 1. The mobile phone detects whether there is a bright object in the preview image. When it detects that there is a bright object in the preview image, the mobile phone automatically turns on the anti-glare function; when no bright object is detected in the preview image, the mobile phone does not. Turn on the anti-glare function. Wherein, the highlighted object includes a photographic object exhibiting a specified color and/or brightness in the preview image. Illustratively, the specified color may be white, the specified brightness may be the luminance value of white in YUV color coding, or the gray value of white in RGB color mode, and so on.

Exemplarily, the mobile phone can detect whether there are highlighted objects in the preview image on the condition that the gray values of most of the pixels in a certain area in the preview image are not less than the preset gray value threshold. When the grayscale values of most of the pixels are not less than the preset grayscale threshold, it is determined that there are bright objects in the preview image, and at this time, the mobile phone automatically turns on the anti-glare function. The grayscale value range of each pixel in the preview image is 0-255, white is 255, and black is 0. Considering that the area where the highlighted object is usually white, you can set the preset grayscale threshold equal to 255 or a value around 255. For example, the preset grayscale threshold can be 255, a window with a size of 100*100 can be selected, and the window can be sequentially slid in the preview image, so that the entire preview image can be spread; each time the window is slid, each pixel in the window is calculated. If the gray value of more than half of the pixels in the window is 255, it is determined that there is a bright object in the preview image, and at this time, the mobile phone automatically turns on the anti-glare function.

Exemplarily, when the mobile phone does not detect that there is a bright object in the preview image, the mobile phone may continue to execute the current shooting mode or other preset shooting modes (eg, a preset image enhancement mode).

In this method, the mobile phone can automatically detect whether there is a highlight object in the preview image, and automatically turn on the anti-glare function when a highlight object is detected; the anti-glare function can be turned on without the user's manual operation, which improves the performance of the mobile phone. How intelligent the operation is.

Manner 2: The mobile phone receives an instruction to enable the anti-glare function triggered by the user, and in response to the instruction, enables the anti-glare function.

Exemplarily, the user can observe and determine whether there is a highlight object by previewing the image or the photographed object, and when the user determines that there is a highlight object, the user can click the preset on the shooting interface to enable the anti-glare function or enter the anti-glare mode. pictures, trigger the command to enable the anti-glare function; the command to enable the anti-glare function may also be triggered by means of preset voice commands, shortcut gestures, physical buttons, etc. of the mobile phone, which is not limited in this embodiment of the present application. For example, as shown in FIG. 3 , the camera interface of the mobile phone 10 may include an icon 301 of the anti-glare function. The user can click the icon 301 of the anti-glare function to trigger an instruction to enable the anti-glare function. Anti-glare function.

In this method, the user independently judges whether there is a bright object when taking a photo, and selects an instruction to trigger the anti-glare function according to the judgment result, so that the mobile phone can turn on the anti-glare function and meet the user's demand for autonomous control of the mobile phone.

Method 3: The mobile phone detects whether there is a highlight object in the preview image. When it detects that there is a highlight object in the preview image, the mobile phone can send a prompt message to the user to prompt the user whether to enable the anti-glare function; if the user triggers the anti-glare function function command, the mobile phone will turn on the anti-glare function in response to the command.

Exemplarily, for the specific implementation of the mobile phone detecting whether there is a highlighted object in the preview image, reference may be made to the relevant description in the above-mentioned method 1 for enabling the anti-glare function on the mobile phone, which will not be repeated here. The prompt information that the mobile phone can send to the user can include: voice prompt information, vibration, light flashing, icon flashing, etc.; for example, as shown in FIG. 3 above, the mobile phone 10 can detect that there is a highlighted object in the preview image. At the time, the icon 301 of the anti-glare function in the control shooting interface flashes, thereby prompting the user to select and click the icon 301 of the anti-glare function. If the user clicks the icon 301 of the anti-glare function, the mobile phone will enable the anti-glare function. For another example, the mobile phone may display the prompt message "Enable the anti-glare function" in the preview image area of the shooting interface, thereby prompting the user to enable the anti-glare mode, wherein, the specific implementation of enabling the anti-glare mode by the user may refer to the above-mentioned method for enabling the anti-glare function on the mobile phone. The related descriptions in Section 2 will not be repeated here; if the user triggers an instruction to enable the anti-glare function, the mobile phone responds to the instruction to enable the anti-glare function.

In this method, the user may forget the anti-glare function of the mobile phone or be unfamiliar with the anti-glare function when taking pictures, and the user can be prompted to turn on the anti-glare function through the prompt information, thereby improving the user experience.

In addition, the mobile phone can also display a prompt box around the detected image area where there are highlighted objects to remind that there are highlighted objects in this area. The user can select the prompt box (for example, click the inner area of the prompt box) to choose to perform glare removal. Feature image area.

In practical applications, the mobile phone may also enable the anti-glare function in other ways, which is not limited in this embodiment of the present application.

The following is an example of how to perform real-time anti-glare processing after the mobile phone has the anti-glare function turned on to obtain a high-definition image with no glare:

FIG. 4 shows a flowchart of an image processing method according to an embodiment of the present application. As shown in FIG. 4 , the method may include the following steps:

Step 400: The mobile phone determines the glare area where the highlighted object in the preview image is located.

The number of highlighted objects may be one or more, and each glare area includes at least one highlighted object. For example, each glare area may include one highlighted object. In this case, the number of glare areas and the number of highlighted objects same. In the preview image, the highlighted object occupies a plurality of connected pixels whose gray value is not less than a preset threshold (eg, 255), and the glare area where the highlighted object is located at least includes these connected pixels occupied by the highlighted object. of pixels. Wherein, the glare area may be in the shape of a rectangle, a circle, or the like.

Exemplarily, the glare area may be a rectangular area, and a rectangular window with a size of m*n may be constructed, where m and n are both integers greater than 1. For example, the initial values of m and n may be 100. Sliding on the preview image, when a pixel with a gray value exceeding 255 appears in the rectangular window, adjust the values of m and n, so that multiple connected pixels, including the pixel, whose gray value exceeds 255 are connected to each other. When the pixels all fall on the rectangular window, the values of m and n are fixed at this time, and the rectangular window is the glare area. Multiple pixels connected to each other with a gray value of more than 255 in the rectangular window are a highlight object. , thereby determining the glare area where the highlighted object is located; covering the entire preview image, the glare area where each highlighted object in the preview image is located can be obtained. Correspondingly, the area other than the glare area in the preview image is non-glare area. Exemplarily, as shown in FIG. 3 above, the glare area 302 is a rectangular area, and the glare area includes the pixels occupied by the sun in the preview image.

It should be noted that this step 400 is an optional step, that is, after the anti-glare function is enabled on the mobile phone, this step 400 can be executed, so that the processing efficiency of the following steps can be improved and processing resources can be saved; at the same time, the processing process can be reduced or eliminated. In addition, after the mobile phone enables the anti-glare function, the mobile phone can also directly perform the following step 401.

Step 401 , the mobile phone identifies the shape of the highlighted object, and extracts the glare starburst area.

In this step, the mobile phone can identify the shape of the highlighted object in the glare area where the highlighted object is located in the preview image determined in the above step 400, and extract the glare starburst area in the glare area; Identify the shape of the highlighted object, and extract the glare starburst area in the preview image, where the glare starburst area is the area where glare appears and is in the shape of a starburst, and the area other than the glare starburst area in the preview image is non-glare Astral region.

Exemplarily, the mobile phone can automatically identify the shape of the highlighted object through the trained neural network. For example, when the highlighted object is a strong light source, the shape of the strong light source may include a ring light source, a strip light source, a point light source, and the like. Pre-select photos of light sources of different shapes to train the neural network, input the glare area or preview image into the trained neural network, and then output the glare area or the shape of the highlighted object in the preview image; the mobile phone can also use conventional overexposure Detection and other methods to identify the shape of the highlighted object.

Further, after recognizing the shape of the highlighted object, the mobile phone can detect the glare area or the glare starburst area in the preview image based on the position of the highlighted object. For example, the center of the highlighted object can be used as the cross star. At the center, continuously adjust the size of the cross star, so that the pixels occupied by the highlighted object all fall within the cross star. At this time, the pixels included in the cross star are the glare star area. After the glare starburst region is determined, the glare starburst region or the glare starburst region in the preview image is cropped to extract the glare starburst region; in this way, further anti-glare processing can be performed on the extracted glare starburst region to improve the This improves processing efficiency and saves processing resources; at the same time, it can reduce or eliminate the impact of processing on other areas in the image that do not contain bright objects and their glare.

It should be noted that this step 401 is an optional step, that is, after the anti-glare function is enabled on the mobile phone, this step 401 may be executed, and the following step 402 may also be directly executed. The following is an exemplary description of performing step 402 after performing step 401 .

FIG. 5 shows a schematic diagram of extracting the shape of the highlighted object and the glare starburst area in the glare area 302 in FIG. 3 according to an embodiment of the present application. As shown in FIG. 5 , in the glare area 302 , the shape of the sun is extracted 501, and at the same time, the glare starburst region 502 is extracted.

Step 402, the mobile phone acquires the spectral component _Ispe (λ) of the highlighted object, where λ represents the wavelength.

The manner of acquiring the spectral components of the highlighted object in this step will be illustrated below with an example.

Mode 1: The mobile phone is equipped with a spectral measurement device (for example, a spectral sensor, a color temperature sensor, a spectral camera, etc.), and the spectral components of a bright object (such as a strong light source) are collected by the spectral sensor.

For example, the mobile phone 10 in FIG. 1 may be configured with a color temperature sensor, and FIG. 6 shows the spectral components of the strong light source collected by the color temperature sensor according to an embodiment of the present application. As shown in FIG. The spectral component _Ispe (λ) of the sun 204 in the photographed object in FIG. 2B shows the relative intensities corresponding to different wavelengths λ.

Exemplarily, the mobile phone may also be configured with a hyperspectral detection device, and the hyperspectral detection device collects the spectral components of the bright object, thereby further improving the accuracy of the collected spectral components.

In this method, the spectral components of the highlighted objects can be obtained in real time during shooting through the color temperature sensor or hyperspectral detection equipment built in the mobile phone, without relying on external equipment, which improves the functional integrity of the mobile phone.

In a second way, the mobile phone can determine the spectral components of the identified highlighted objects by identifying the light source types of the highlighted objects in the photographed object, and according to preset spectral components corresponding to different light source types.

Exemplarily, for the strong light sources that often appear in practice, such as the sun, the moon, fluorescent lamps, incandescent lamps, fluorescent lamps, etc., the spectral components of different strong light sources are fixed. The spectral components are stored in the mobile phone; when shooting, the mobile phone can identify the light source type of the highlighted object in the subject through computer vision and other methods. From the pre-stored spectral components of a plurality of light source types, the spectral components of the sun are determined.

In this method, the spectral components corresponding to different light source types are stored in advance, and the light source type of the highlighted object in the subject is identified during shooting, so as to determine the spectral composition of the highlighted object; in this way, the mobile phone does not need to be additionally equipped with a spectral sensor or hyperspectral spectrum. Spectral measurement equipment such as detection equipment can obtain the spectral components of bright objects, saving the cost of mobile phones.

Method 3: The mobile phone can receive the spectral components of the bright object input from the outside.

Exemplarily, a spectral measurement device (such as a hyperspectral detection device, a color temperature sensor, a spectral camera, etc.) disposed in the external environment collects the spectral components of the highlighted objects, and inputs the collected spectral components of the highlighted objects into the mobile phone. , the mobile phone receives the inputted spectral components, so as to obtain the spectral components of the bright objects.

In this way, the mobile phone can obtain the spectral components of the bright object by using the spectral components collected by the external device, which saves the cost of the mobile phone.

Step 403: The mobile phone generates a point spread function of three RGB channels according to the spectral components of the highlighted object.

Considering that the influence of the periodic pixel arrangement of the display screen on the imaging of the camera under the display screen is manifested as the broadening of the point spread function, FIG. 7 shows a schematic diagram of the broadening of the point spread function of the under-screen imaging system according to an embodiment of the present application, as shown in FIG. As shown in Figure 7, the distribution of the point spread function of the under-screen imaging system will expand horizontally and vertically along the display screen, resulting in a serious smearing effect, which greatly reduces the imaging quality of the camera under the display screen. Therefore, in order to improve the imaging quality, in this step, the mobile phone uses the above-mentioned acquired spectral components of the bright object to generate the point spread function of the RGB three channels of the under-screen imaging system.

The following is an example of how to generate the point spread function of the RGB three-channel according to the spectral components of the highlighted object in this step:

Method 1. The mobile phone generates the point spread function of the RGB three channels of the under-screen imaging system through the display screen model, the spectral components of the highlighted objects obtained above, and the photosensitive characteristic curve (transmittance curve) of the filter of the image sensor of the camera. .

Among them, the display screen model can be pre-stored in the mobile phone, and the display screen model can include: amplitude modulation function A(m,n) and phase modulation function P(m,n), etc., m,n represent the pixels on the display screen The amplitude modulation function and phase modulation function can be determined according to factors such as the pixel point distribution on the display screen, the wiring method of the display screen and the material of the display screen. FIG. 8 shows a partial schematic diagram of a display screen according to an embodiment of the present application. As shown in FIG. 8 , the distribution of pixel points on the display screen can be obtained, wherein each pixel point includes a red photon pixel, a green photon pixel, and a blue photon pixel. subpixels. The photosensitive characteristic curve of the filter of the image sensor of the camera can be determined according to the relevant parameters of the camera configured on the mobile phone, and the filter of the image sensor can include: red filter, green filter, blue filter; A schematic diagram of the filter transmittance curve of an embodiment of the application, as shown in FIG. 9 , the transmittance curve _{Fr, g, b} (λ) of the filter includes: the transmittance curve _Fr of the red filter ( λ), the transmittance curve F _g (λ) of the green filter, and the transmittance curve F _b (λ) of the blue filter, where λ is the wavelength.

Exemplarily, the mobile phone can use the spectral composition I _spe (λ) of the sun in the photographed object collected by the color temperature sensor in the above-mentioned FIG. 6 and the preset display model (A(m,n) and P(m,n)) , generate the point spread function corresponding to the different wavelengths of the bright object (the sun) after passing through the under-screen imaging system, and then pass the transmittance curve F _r,g,b (λ) of the filter of the image sensor to generate the under-screen imaging The point spread function of the RGB three channels of the system.

Exemplarily, the mobile phone can also generate the point spread function corresponding to the different wavelengths of the highlighted object after passing through the under-screen imaging system through the spectral components of the highlighted object collected by the hyperspectral detection device and the preset display screen model, so that the The point spread function of different wavelengths of the generated highlight objects after passing through the under-screen imaging system is closer to the actual situation, thereby improving the effect of anti-glare.

Exemplarily, the display screen model can be used as the transmittance function, combined with the light wave propagation factor, and the spectral components of the highlighted object obtained above are used as weights to obtain the points corresponding to the different wavelengths of the highlighted object after passing through the under-screen imaging system. Spread function; wherein, the model of the display screen may include: the amplitude modulation function and the phase modulation characteristic function of the display screen; the light wave propagation factor may be determined according to the focal length of the camera to capture the image to be processed and the wavelength of the highlighted object.

In this way, starting from the root cause of glare, that is, the diffraction effect caused by the periodic pixel arrangement of the display screen is sensitive to wavelength, the display screen model constructed by the amplitude modulation function and the phase modulation characteristic function is used as the transmittance function. , and determine the light wave propagation factor according to the focal length of the camera to capture the image to be processed and the actual wavelength of the highlighted object, and at the same time use the spectral component of the highlighted object as the weight to obtain the different wavelengths of the highlighted object after passing through the under-screen imaging system. Spread function, the obtained point spread function can more realistically express the glare process of bright objects through under-screen imaging.

Exemplarily, the point spread function I(u, v; λ) corresponding to different wavelengths of the highlighted object after passing through the under-screen imaging system is shown in the following formula (1):

In formula (1), λ represents the wavelength, I _spe (λ) represents the spectral component of the highlighted object, f represents the actual focal length of the camera when shooting, m, n represent the coordinates of the pixels on the display screen, u, v Represents the coordinates of the pixel on the image sensor; A(m,n) represents the amplitude modulation function, and P(m,n) represents the phase modulation function.

The generated point spread function of the three-channel RGB of the under-screen imaging system is shown in the following formula (2):

In formula (2), PSF _r represents the point spread function of the red channel, PSF _g represents the point spread function of the green channel, PSF _b represents the point spread function of the blue channel, and I(u, v; λ) represents the highlight The point spread function corresponding to different wavelengths of the object after passing through the under-screen imaging system, λ represents the wavelength, u, v represent the coordinates of the pixel points on the image sensor, F _r (λ) represents the transmittance curve of the red filter of the image sensor , F _g (λ) represents the transmittance curve of the green filter of the image sensor, and F _b (λ) represents the transmittance curve of the blue filter of the image sensor.

FIG. 10 shows a schematic diagram of point spread functions of three RBG channels generated according to an embodiment of the present application. As shown in FIG. 10 , the point spread functions of the red light channel, the point spread function of the green light channel, and the point spread function, wherein the point spread function of the red light channel of the same pixel and the point spread function of the green light channel of the pixel point are separated by a sub-pixel point, and the point spread function of the red light channel of the same pixel point is the same as the pixel point. The point spread function of the blue light channel is separated by one sub-pixel point, and the point spread function of the green light channel of the same pixel point and the point spread function of the blue light channel of the pixel point are separated by one sub-pixel point.

In this method, when the computing power and storage capacity of the mobile phone meet the requirements, the display screen model can be stored in the mobile phone in advance, and combined with the acquired spectral components of the highlighted object, different wavelengths of the highlighted object can be generated and imaged under the screen. After the system, that is, the point spread functions corresponding to different wavelengths of different wavelengths of the highlighted object passing through the display screen and the lens of the camera under the display screen, and then the point spread function of the three RGB channels of the under-screen imaging system is obtained. In this way, the point spread function corresponding to the real glare can be obtained through physical modeling by using the spectral components of the high-brightness objects, so that the dispersion effect due to the periodic structure of the display screen can be effectively eliminated, and the glare caused by the periodic structure of the display screen can be effectively eliminated. , especially the rainbow glare has an excellent suppression effect, which improves the imaging quality of the under-screen imaging system.

Method 2: The mobile phone generates the point spread function of the RGB three channels of the under-screen imaging system according to the pre-calibrated point spread functions of different wavelengths after passing through the under-screen imaging system and the spectral components of the highlighted objects obtained above.

Exemplarily, the point spread functions corresponding to different wavelengths after passing through the under-screen imaging system can be pre-calibrated in the laboratory and stored in the mobile phone in advance; The schematic diagram of the point spread function corresponding to the wavelength after passing through the under-screen imaging system, as shown in Figure 11, is the point spread function corresponding to different wavelengths in the visible light band (400nm-780nm) after passing through the under-screen imaging system, Wherein, the interval between wavelengths may be 6 nm, 3 nm, or the like.

When the anti-glare function is enabled on the mobile phone, the point spread function corresponding to the different wavelengths stored in the mobile phone after passing through the under-screen imaging system can be directly called, and integrated with the spectral components of the highlighted objects obtained above to obtain the under-screen imaging system. RGB three-channel point spread function.

In this method, the point spread functions corresponding to different wavelengths after passing through the under-screen imaging system are pre-stored in the mobile phone, and the point spread functions of the RGB three channels of the under-screen imaging system can be quickly generated in combination with the above-mentioned acquisition of the spectral components of the highlighted objects. The computing efficiency is significantly improved, and the processing speed of the mobile phone's anti-glare is improved. It can be applied to the anti-glare processing scene with a large amount of processing data and high processing speed requirements such as video shooting.

It should be noted that the above step 402 and step 403 can be executed before the above step 400 or step 401, can also be executed after step 400 or step 401, and can also be executed simultaneously with step 400 or step 401. Not limited.

Step 404 , using the generated point spread function of the three RGB channels and the above-mentioned glare starburst area, and adopting a preset non-blind deconvolution algorithm to remove the glare in the glare starburst area.

Exemplarily, the preset non-blind deconvolution algorithm may include: neural network learning, convex optimization deconvolution, non-convex optimization deconvolution and other existing non-blind deconvolution algorithms, combined with the above-identified highlighted objects. Using the point spread function of the RGB three channels of the generated under-screen imaging system to perform non-blind deconvolution processing with the above-mentioned glare starburst area, the deconvolution results of the RGB three channels are obtained, so as to obtain the local area after the glare removal. image.

FIG. 12 shows a schematic diagram of a partial image after glare removal according to an embodiment of the present application. As shown in FIG. 12 , the partial image 1201 after glare removal is generated by removing the glare in the glare starburst area 502 in FIG. 5 above. The image, there is no glare in the deglared partial image 1202, and the sun shape 1202 can be clearly seen.

Step 405: Perform fusion processing on the de-glare partial image and the preview image to obtain a de-glare high-definition image.

Exemplarily, the glare area and the non-glare area where the partial image after glare removal is located may be spliced, so as to obtain a final high-definition image without glare.

Exemplarily, the high-definition image after de-glare can be displayed in real time on the shooting interface of the display screen. As shown in FIG. 2B above, in the high-definition image after de-glare, there is no glare or there is less glare, especially it can be eliminated. Rainbow glare, so that the brightness in the image can be smoothly transitioned to more realistically reflect the captured subject.

In the embodiment of the present application, for the glare generated in the captured image when the user shoots through the camera under the display screen, especially the rainbow glare, this glare is sensitive to the wavelength. This generates a point spread function suitable for different bright objects, and then obtains a high-definition image with no glare through non-blind deconvolution processing, thereby effectively improving or even eliminating the glare generated by the image captured by the camera under the display screen, especially the rainbow glare. , which can achieve the glare-free imaging effect of the camera under the display screen when the display screen is blocked, so that the user cannot perceive the effect of rainbow glare on the final image when using the camera shooting function under the display screen.

In some scenarios, the user performs anti-glare processing on the image or video that has been captured by the electronic device. The images or videos that have been captured are images or videos captured by the camera under the display screen.

Exemplarily, when there is a bright object in the image or video already captured by the electronic device, the electronic device can turn on the glare removal function, so as to improve or eliminate the glare generated by the image captured by the camera under the display screen, especially the rainbow glare.

Taking the mobile phone in FIG. 1 above as an example, the front camera of the mobile phone is set under the display screen, and the user takes a selfie through the front camera of the mobile phone outdoors. Exemplarily, the captured image is the one shown in FIG. 2A above. The original image 201, after the self-timer is completed, the captured image is stored in the mobile phone. When viewing the image, the user turns on the image editing function of the mobile phone, so that the display screen enters the image editing interface. Glare, that is, the sun appears as rainbow glare, and at the same time, the rainbow glare blocks some objects (such as mountain peaks), and the mobile phone turns on the anti-glare function and performs anti-glare processing, thereby obtaining a high-definition image with no glare. The high-definition image with no glare is shown in Figure 2B above. In the high-definition image 203 shown in the edited image, the glare has been eliminated, and the sun and some mountain peaks originally blocked by the rainbow glare of the sun can be seen; The glare of the sun (rainbow glare) in the captured image, so that a glare-removed high-definition image can be obtained.

Exemplarily, when viewing an image, the user can make the display screen enter the image editing interface by clicking an editing button, or by using a voice command, or the like.

Exemplarily, after the display screen enters the image editing interface, for the specific manner of enabling the anti-glare function of the mobile phone, reference may be made to the foregoing description about enabling the anti-glare function of the mobile phone, which will not be repeated here.

Exemplarily, after the anti-glare function is enabled on the mobile phone, the anti-glare processing is performed to obtain the high-definition image with the glare removed. Reference may be made to the relevant description in FIG. 4 above, which will not be repeated here.

In the embodiment of the present application, when the user edits the image captured by the camera under the display screen, the glare generated in the captured image, especially the rainbow glare, is sensitive to the wavelength. Therefore, by obtaining the spectrum of the highlighted object components, and thus generate a point spread function suitable for different bright objects, and then obtain a high-definition image with no glare through non-blind deconvolution processing, so as to effectively improve or even eliminate the glare generated by the image captured by the camera under the display screen. Especially the rainbow glare improves the clarity of the image.

FIG. 13 shows a flowchart of an image processing method according to an embodiment of the present application. As shown in FIG. 13 , the execution subject of the method may be an electronic device, for example, the mobile phone in FIG. 1 , and the method may be Include the following steps:

Step 1301: Acquire an image to be processed.

The image to be processed is an image collected by the under-screen imaging system. As shown in FIG. 2A , the under-screen imaging system may include a display screen and a camera disposed below the display screen; for example, the image to be processed may be the user in the above embodiment. The real-time preview image when an image or video is captured by an electronic device, as shown in FIG. 3 , can also be the image or video that has been captured in the above embodiment and stored in the electronic device, and can also be an image captured by other external devices, etc. , which is not limited in the embodiments of the present application.

The image to be processed may include: a highlight object and the glare generated by it, wherein the highlight object may be a photographic object showing a specified color and/or brightness in the image to be processed; glare is the light of the highlight object passing through the display screen and the The dazzling light from the camera under the display, which can be a rainbow-shaped glare. The specified color can be white, and the specified brightness can be the brightness value of white in YUV color coding, or the gray value of white in RGB color mode.

Exemplarily, it may be detected whether there is a bright object in the image to be processed under the condition that the grayscale values of most of the pixels in a certain area of the image to be processed are not less than a preset grayscale threshold. For a specific implementation process, reference may be made to the relevant expressions in the foregoing embodiments, and details are not repeated here.

Step 1302: Generate a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point spread function of at least one channel in the under-screen imaging system.

Among them, the spectral components of the highlighted objects can be collected by the spectral measurement device configured in the electronic device or the external spectral measurement device, or the spectral components corresponding to different light source types can be pre-stored in the electronic device, so as to identify the high brightness in the image to be processed. The light source type of the bright object determines the spectral composition of the bright object. For example, the spectral composition of the bright object is shown in Figure 6.

In a possible implementation manner, the step may further include: acquiring the spectral components of the bright object collected by the spectral measurement device. The spectral measurement device may be a configured spectral measurement device of the electronic device or an external spectral measurement device. For the specific description of this implementation, reference may be made to the relevant expressions about "the first and third modes of the method for obtaining the spectral components of the highlighted object" in step 402 in FIG.

In this way, the spectral components of the high-brightness objects can be acquired in real time when shooting through the spectral measurement equipment built into the electronic equipment, without relying on external equipment, which improves the functional integrity of the electronic equipment; or, with the help of spectral components collected by external equipment, Thus, the spectral components of the bright object are obtained, and the cost of electronic equipment is saved.

In a possible implementation manner, this step may further include: performing image recognition on the image to be processed to determine the light source type of the highlighted object; and determining the spectral composition of the highlighted object according to preset spectral components of different light source types. For the specific description of this implementation manner, reference may be made to the relevant expression about "the second manner in the manner of acquiring the spectral component of the highlighted object" in step 402 in FIG. 4 in the foregoing embodiment, which will not be repeated here.

In this way, considering that the spectral components of different light sources are fixed, by pre-storing the spectral components corresponding to different light source types, the light source type of the highlighted object in the subject can be identified during shooting, thereby determining the spectral composition of the highlighted object; , the electronic device can obtain the spectral composition of the bright object without additional configuration of the spectral measurement device, which saves the cost of the electronic device.

Wherein, when the under-screen imaging system includes three RGB channels, the first point spread function may be the point spread function of the red light channel, the point spread function of the green light channel, or the point spread function of the blue light channel, or the point spread function of the blue light channel. It may be the point spread function of any two channels, or may be the point spread function of three channels, which is not limited in this embodiment of the present application. For example, it may be the point spread function of three channels as shown in FIG. 10 .

In a possible implementation manner, step 1302 may include: generating a second point spread function according to the spectral components of the highlighted object and the model of the display screen; The corresponding point spread function behind the under-screen imaging system; the first point spread function is generated according to the second point spread function and the photosensitive characteristic curve of the camera.

The model of the display screen may be pre-stored in the electronic device, and the display screen model may be generated while performing step 1302; the display screen model may include: an amplitude modulation function A(m,n) and a phase modulation function P(m,n) ), etc., m, n represent the coordinates of the pixels on the display screen.

For the specific description of this implementation, reference may be made to the relevant statement in step 403 in FIG. 4 in the above-mentioned embodiment about “method 1 of generating the point spread function of three RGB channels according to the spectral components of the highlighted object”, which will not be repeated here.

Exemplarily, when the computing power and storage capacity of the electronic device meet the requirements, a display screen model can be stored in the electronic device in advance, and then a highlight can be generated according to the model of the display screen in combination with the spectral components of the highlighted object. Different wavelengths of the object pass through the point spread function of the under-screen imaging system, and then combine with the photosensitive characteristic curve of the camera to obtain the point spread function of at least one channel of the under-screen imaging system. In this way, the point spread function corresponding to the real glare can be obtained through physical modeling by using the spectral components of the high-brightness objects, so that the dispersion effect due to the periodic structure of the display screen can be effectively eliminated, and the glare caused by the periodic structure of the display screen can be effectively eliminated. , especially the rainbow glare has an excellent suppression effect, which improves the imaging quality of the under-screen imaging system.

In a possible implementation manner, generating the second point spread function according to the spectral components of the highlighted object and the model of the display screen may include: taking the model of the display screen as the transmittance function, combining the light wave propagation factor, and using The spectral components of the highlighted objects are used as weights to obtain the second point spread function; wherein, the model of the display screen includes: the amplitude modulation function and the phase modulation characteristic function of the display screen; The wavelength of bright objects is determined.

Exemplarily, the above formula (1) can be used to determine the actual focal length of the camera when shooting, the coordinates of the pixels on the display screen, the coordinates of the pixels on the image sensor of the camera, the spectral components of the highlighted objects, and the coordinates of the display screen. The phase modulation function and the amplitude modulation function determine the second point spread function.

Further, by the above formula (2), according to the second point spread function, the transmittance curve of the red filter of the image sensor, the transmittance curve of the green filter and the transmittance curve of the blue filter, the Integrate operation to obtain the first point spread function, wherein the transmittance curve of the red filter, the transmittance curve of the green filter and the transmittance curve of the blue filter of the image sensor can be shown in FIG. 9 .

In a possible implementation manner, step 1302 may include: generating the first point spread function according to the spectral components of the highlighted object and a preset third point spread function; wherein the third point spread function is different The point spread function corresponding to the wavelength after passing through the under-screen imaging system.

Exemplarily, the third point spread function can be obtained by pre-calibrating in the laboratory and stored in the electronic device, for example, the third point spread function can be shown in FIG. The point spread function corresponding to the under-screen imaging system, combined with the spectral components of the above-mentioned bright objects, can quickly generate the point spread function of at least one channel in the under-screen imaging system, which significantly improves the computational efficiency and improves the anti-glare effect. The processing speed can be applied to the anti-glare processing scene with a large amount of processing data and high processing speed requirements, such as video shooting.

For the specific description of this implementation, refer to the relevant expression in step 403 in FIG. 4 in the above-mentioned embodiment about “Method 2 of generating the point spread function of three RGB channels according to the spectral components of the highlighted object”, which will not be repeated here.

Step 1303: Perform deconvolution processing on the first point spread function and the to-be-processed image to obtain a glare-removed image.

Exemplarily, non-blind deconvolution algorithms such as neural network learning, convex optimization deconvolution, and non-convex optimization deconvolution can be used to perform deconvolution operations on the first point spread function and the image to be processed, thereby obtaining glare removal. The resulting image, for example, after removing the glare is shown in Figure 2B.

In a possible implementation, before step 1303, the image to be processed may also be divided into a first area and a second area; wherein the first area is the area where the highlighted object and the glare generated by it are located; the second area The area is the area other than the first area in the image to be processed. Correspondingly, step 1303 may include: performing deconvolution processing on the first point spread function and the first region to obtain the first region after removing glare; and merging the first region and the second region after removing the glare to obtain Image after removing glare.

For example, the first area may be the glare area where the highlighted object is located in the preview image in the foregoing embodiment, such as the glare area 302 shown in FIG. 3 . Correspondingly, the second area may be the glare area except for the preview image in the foregoing embodiment. A non-glare area other than the glare area; for another example, the first area may be the glare starburst area in the above-mentioned embodiment, such as the glare area 502 shown in FIG. 5 , and correspondingly, the second area may be the above-mentioned embodiment. A non-glare starburst area other than the glare starburst area. When the first area is the glare starburst area in the above embodiment, the first area after removing the glare may be as shown in FIG. 12 .

In this way, the image to be processed is divided into a first area where the highlighted object and the glare generated by it are located, and a second area that does not include the highlighted object and the glare generated by it, and then the first area where the highlighted object and the glare generated by it are located. Further anti-glare processing is performed on one area, which improves processing efficiency and saves processing resources; at the same time, it can reduce or eliminate the second area in the image to be processed that does not contain bright objects and the glare generated by them during the anti-glare processing. influences.

For the specific description of the implementation manner, reference may be made to the relevant expressions in step 400, step 401, step 404, and step 405 in FIG. 4 in the foregoing embodiment, which will not be repeated here.

In the embodiment of the present application, for the image to be processed collected by the under-screen imaging system, the image to be processed includes: a highlight object and the glare generated by it, and according to the spectral components of the highlight object in the image to be processed, different highlight objects are generated. The corresponding point spread function of at least one channel in the under-screen imaging system is used, and then the first point spread function and the image to be processed are deconvolved to obtain an image after removing the glare generated by the bright object; thus, it can effectively improve Or eliminate the glare generated by the image captured by the camera under the display screen, especially the rainbow glare, which improves the clarity of the image and enhances the user experience.

Various possible implementation manners or descriptions of the foregoing embodiments are referred to above, and details are not repeated here.

Based on the same inventive concept of the above method embodiments, the embodiments of the present application further provide an image processing apparatus, and the image processing is used to execute the technical solutions described in the above method embodiments.

FIG. 14 shows a structural diagram of an image processing apparatus according to an embodiment of the present application. As shown in FIG. 14 , the apparatus may include: an acquisition module 1401 for acquiring an image to be processed; the image to be processed is off-screen imaging For the images collected by the system, the under-screen imaging system includes a display screen and a camera set below the display screen; the images to be processed include: highlighted objects and the glare generated by them, wherein the highlighted objects are the specified colors and colors in the image to be processed. / or a photographic object of brightness; the generating module 1402 is used to generate a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point spread function of at least one channel in the under-screen imaging system; the processing module 1403 , which is used to perform deconvolution processing on the first point spread function and the image to be processed to obtain an image after glare removal.

In a possible implementation manner, the generation module is further configured to: generate a second point spread function according to the spectral components of the highlighted object and the model of the display screen; wherein the second point spread function is different wavelengths of the highlighted object The point spread function corresponding to the light passing through the under-screen imaging system; the first point spread function is generated according to the second point spread function and the photosensitive characteristic curve of the camera.

In a possible implementation manner, the generating module is further configured to: generate a first point spread function according to the spectral components of the highlighted object and a preset third point spread function; wherein the third point spread function is a different wavelength The corresponding point spread function after passing through the under-screen imaging system.

In a possible implementation manner, the apparatus may further include: a spectral measurement device, configured to collect spectral components of the highlighted object.

In a possible implementation manner, the device may further include: a collection module, configured to perform image recognition on the image to be processed, and determine the light source type of the highlighted object; and determine the highlighted object according to preset spectral components of different light source types spectral composition.

In a possible implementation manner, the apparatus may further include: a segmentation module, configured to: segment the to-be-processed image into a first area and a second area; wherein the first area is the highlighted object and its generated the area where the glare is located; the second area is an area other than the first area in the image to be processed; the processing module is also used for: deconvolution processing the first point spread function and the first area to obtain the glare-removed image The first area; the glare-removed first area is fused with the second area to obtain the glare-removed image.

In a possible implementation manner, the generation module is further used to: take the model of the display screen as the transmittance function, combine the light wave propagation factor, and take the spectral component of the highlighted object as the weight to obtain the second point spread function; Wherein, the model of the display screen includes: the amplitude modulation function and the phase modulation characteristic function of the display screen; the light wave propagation factor is determined according to the focal length when the camera captures the image to be processed and the wavelength of the highlighted object.

An embodiment of the present application provides an electronic device, which may include: a display screen, a camera below the display screen, a processor, and a memory; wherein, the camera is used to collect images to be processed through the display screen; the display screen is used for Display the image to be processed and the image after deglare; the memory is used to store the processor-executable instructions; the processor is configured to implement the image processing method of the above embodiment when the instruction is executed.

FIG. 15 shows a schematic structural diagram of an electronic device according to an embodiment of the present application. Taking the electronic device as a mobile phone as an example, FIG. 15 shows a schematic structural diagram of the mobile phone 200 .

The mobile phone 200 may include a processor 210, an external memory interface 220, an internal memory 221, a USB interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 251, a wireless communication module 252, Audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, sensor module 280, buttons 290, motor 291, indicator 292, camera 293, display screen 294, SIM card interface 295, etc. The sensor module 280 may include a gyroscope sensor 280A, an acceleration sensor 280B, a proximity light sensor 280G, a fingerprint sensor 280H, and a touch sensor 280K (of course, the mobile phone 200 may also include other sensors, such as a color temperature sensor, a temperature sensor, a pressure sensor, and a distance sensor. , magnetic sensor, ambient light sensor, air pressure sensor, bone conduction sensor, etc., not shown in the figure).

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the mobile phone 200 . In other embodiments of the present application, the mobile phone 200 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or Neural-network Processing Unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. The controller may be the nerve center and command center of the mobile phone 200 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in processor 210 is cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 210 . If the processor 210 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided, and the waiting time of the processor 210 is reduced, thereby improving the efficiency of the system.

The processor 210 can run the image processing method provided by the embodiments of the present application, so as to effectively improve or eliminate glare generated by the image captured by the camera under the display screen, especially rainbow glare, which improves the clarity of the image and improves the user experience. The processor 210 may include different devices. For example, when the CPU and the GPU/NPU are integrated, the CPU and the GPU/NPU may cooperate to execute the image processing method provided by the embodiments of the present application. For example, some algorithms in the image processing method are executed by the CPU, and another part of the algorithms Executed by GPU/NPU for faster processing efficiency.

Display screen 294 is used to display images, videos, and the like. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, cell phone 200 may include 1 or N display screens 294, where N is a positive integer greater than 1. The display screen 294 may be used to display information entered by or provided to the user as well as various graphical user interfaces (GUIs). For example, display 294 may display photos, videos, web pages, or documents, and the like. After the processor 210 detects a touch event of the user's finger (or stylus, etc.) on an application icon, in response to the touch event, the user interface of the application corresponding to the application icon is opened and displayed on the display 294 The user interface of the application.

In this embodiment of the present application, the display screen 294 may be an integrated flexible display screen, or a spliced display screen composed of two rigid screens and a flexible screen located between the two rigid screens.

The camera 293 (a front camera or a rear camera, or a camera can be used as both a front camera and a rear camera) is used to capture still images or videos, and the camera 293 can be arranged below the display screen 294 . Generally, the camera 293 may include a photosensitive element such as a lens group and an image sensor, wherein the lens group includes a plurality of lenses (convex or concave) for collecting the light signal reflected by the object to be photographed, and transmitting the collected light signal to the image sensor . The image sensor generates an original image of the object to be photographed according to the light signal.

Internal memory 221 may be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the mobile phone 200 by executing the instructions stored in the internal memory 221 . The internal memory 221 may include a storage program area and a storage data area. Wherein, the storage program area may store the operating system, the code of the application (such as a camera application, etc.), and the like. The storage data area may store data created during the use of the mobile phone 200 (such as images and videos collected by the camera application) and the like.

The internal memory 221 may also store one or more computer programs corresponding to the image processing methods provided in the embodiments of the present application. The one or more computer programs are stored in the above-mentioned memory 221 and configured to be executed by the one or more processors 210, and the one or more computer programs include instructions that can be used to perform the corresponding embodiments described above. The computer program may include an acquisition module 1401 for acquiring an image to be processed; the image to be processed is an image collected by an under-screen imaging system, and the under-screen imaging system includes a display screen and a camera set below the display screen; The processed image includes: a highlight object and the glare generated by it, wherein the highlight object is a photographic object exhibiting a specified color and/or brightness in the image to be processed; the generating module 1402 is configured to generate a first a point spread function; the first point spread function is the point spread function of at least one channel in the under-screen imaging system; the processing module 1403 is used to perform deconvolution processing on the first point spread function and the image to be processed to obtain a glare-removed image Image.

In addition, the internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

Certainly, the code of the image processing method provided by the embodiment of the present application may also be stored in an external memory. In this case, the processor 210 may execute the code of the image processing method stored in the external memory through the external memory interface 220 .

Exemplarily, the display screen 294 of the mobile phone 200 displays a main interface, and the main interface includes icons of multiple applications (such as a camera application, etc.). The user clicks the icon of the camera application in the main interface through the touch sensor 280K, which triggers the processor 210 to start the camera application and turn on the camera 293 . Display screen 294 displays an interface of a camera application, such as a capture interface.

The mobile communication module 251 can provide a wireless communication solution including 2G/3G/4G/5G, etc. applied on the mobile phone 200 . In this embodiment of the present application, the mobile communication module 251 may also be used for information interaction with other devices (eg, acquiring spectral components, images to be processed, etc.).

The mobile phone 200 can implement audio functions through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, and an application processor. Such as music playback, recording, etc. The cell phone 200 can receive key 290 input and generate key signal input related to user settings and function control of the cell phone 200 . The mobile phone 200 can use the motor 291 to generate a vibration prompt (for example, a prompt for turning on the anti-glare function). The indicator 292 in the mobile phone 200 can be an indicator light, which can be used to indicate a charging state, a change in power, or a message, a prompt message, a missed call, a notification, and the like.

It should be understood that, in practical applications, the mobile phone 200 may include more or less components than those shown in FIG. 13 , which are not limited in this embodiment of the present application. The illustrated handset 200 is merely an example, and the handset 200 may have more or fewer components than those shown, two or more components may be combined, or may have different component configurations. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily describe the software structure of an electronic device.

FIG. 16 is a block diagram of a software structure of an electronic device according to an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer.

The application layer can include a series of application packages.

As shown in Figure 16, the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 16, the application framework layer may include window managers, content providers, view systems, telephony managers, resource managers, notification managers, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

Content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The resource manager provides various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a brief pause without user interaction. For example, text information is prompted in the status bar, a prompt sound is issued, the electronic device vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

An embodiment of the present application provides an image processing apparatus, including: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions.

Embodiments of the present application provide a non-volatile computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the above method.

Embodiments of the present application provide a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (Electrically Programmable Read-Only-Memory, EPROM or flash memory), static random access memory (Static Random-Access Memory, SRAM), portable compact disk read-only memory (Compact Disc Read-Only Memory, CD - ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices, such as punch cards or raised structures in grooves on which instructions are stored, and any suitable combination of the foregoing .

Computer readable program instructions or code described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions used to perform the operations of the present application may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, may be connected to an external computer (eg, use an internet service provider to connect via the internet). In some embodiments, electronic circuits, such as programmable logic circuits, Field-Programmable Gate Arrays (FPGA), or Programmable Logic Arrays (Programmable Logic Arrays), are personalized by utilizing state information of computer-readable program instructions. Logic Array, PLA), the electronic circuit can execute computer readable program instructions to implement various aspects of the present application.

Aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in hardware (eg, circuits or ASICs (Application) that perform the corresponding functions or actions. Specific Integrated Circuit, application-specific integrated circuit)), or can be implemented by a combination of hardware and software, such as firmware.

While the invention has been described herein in connection with various embodiments, those skilled in the art will understand and understand from a review of the drawings, the disclosure, and the appended claims in practicing the claimed invention. Other variations of the disclosed embodiments are implemented. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that these measures cannot be combined to advantage.

Various embodiments of the present application have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

An image processing method, characterized in that the method comprises:

Obtain an image to be processed; the image to be processed is an image collected by an under-screen imaging system, and the under-screen imaging system includes a display screen and a camera disposed below the display screen; the image to be processed includes: a highlighted object and the resulting glare, wherein the highlighted object is a photographic object showing a specified color or brightness in the to-be-processed image;

generating a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point spread function of at least one channel in the under-screen imaging system;

Perform deconvolution processing on the first point spread function and the to-be-processed image to obtain an image after removing the glare.
The method according to claim 1, wherein the generating the first point spread function according to the spectral components of the highlighted object comprises:

According to the spectral components of the highlighted object and the model of the display screen, a second point spread function is generated; wherein, the second point spread function is obtained after the different wavelengths of the highlighted object pass through the under-screen imaging system The corresponding point spread function;

The first point spread function is generated according to the second point spread function and the photosensitive characteristic curve of the camera.
The method according to claim 1, wherein the generating the first point spread function according to the spectral components of the highlighted object comprises:

The first point spread function is generated according to the spectral components of the highlighted object and a preset third point spread function; wherein, the third point spread function is corresponding to different wavelengths after passing through the under-screen imaging system point spread function.
The method according to claim 1, wherein the method further comprises:

Obtain the spectral components of the bright object collected by the spectral measuring device.
The method according to claim 1, wherein the method further comprises:

Perform image recognition on the to-be-processed image to determine the light source type of the highlighted object;

According to preset spectral components of different light source types, the spectral components of the highlighted object are determined.
The method according to claim 1, wherein the method further comprises:

Divide the image to be processed into a first area and a second area; wherein, the first area is the area where the bright object and the glare generated by it are located; the second area is the area in the image to be processed areas other than the first area;

The performing deconvolution processing on the first point spread function and the to-be-processed image to obtain the image after removing the glare, including:

By performing deconvolution processing on the first point spread function and the first region, the first region after removing the glare is obtained;

The glare-removed first area is fused with the second area to obtain the glare-removed image.
The method according to claim 2, wherein the generating a second point spread function according to the spectral components of the highlighted object and the model of the display screen, comprising:

Taking the model of the display screen as a transmittance function, combining the light wave propagation factor, and taking the spectral components of the highlighted object as a weight, the second point spread function is obtained;

Wherein, the model of the display screen includes: the amplitude modulation function and the phase modulation characteristic function of the display screen; the light wave propagation factor is based on the focal length when the camera captures the to-be-processed image and the wavelength of the highlighted object Sure.
An image processing device, characterized in that the device comprises:

an acquisition module, configured to acquire an image to be processed; the image to be processed is an image collected by an under-screen imaging system, and the under-screen imaging system includes a display screen and a camera arranged under the display screen; the image to be processed Including: a highlight object and the glare generated by it, wherein the highlight object is a shooting object showing a specified color or brightness in the image to be processed;

a generating module, configured to generate a first point spread function according to the spectral components of the highlighted object; the first point spread function is the point spread function of at least one channel in the under-screen imaging system;

A processing module, configured to perform deconvolution processing on the first point spread function and the to-be-processed image to obtain an image after removing the glare.
The device according to claim 8, wherein the generating module is further configured to: generate a second point spread function according to the spectral components of the highlighted object and the model of the display screen; wherein, the The second point spread function is the point spread function corresponding to the light of different wavelengths of the highlighted object after passing through the under-screen imaging system; according to the second point spread function and the photosensitive characteristic curve of the camera, the first point spread function is generated. A little spread function.
The device according to claim 8, wherein the generating module is further configured to: generate the first point spread function according to the spectral components of the highlighted object and a preset third point spread function; Wherein, the third point spread function is a point spread function corresponding to different wavelengths after passing through the under-screen imaging system.
The apparatus according to claim 8, characterized in that, the apparatus further comprises: a spectrum measuring device, configured to collect the spectral components of the highlighted object.
The device according to claim 8, wherein the device further comprises: an acquisition module, configured to: perform image recognition on the to-be-processed image, and determine the light source type of the highlighted object; The spectral composition of the light source type, to determine the spectral composition of the highlighted object.
The device according to claim 8, wherein the device further comprises: a segmentation module, configured to: segment the to-be-processed image into a first area and a second area; wherein the first area is the The area where the highlighted object and the glare generated by it are located; the second area is the area other than the first area in the image to be processed;

The processing module is further configured to: perform deconvolution processing on the first point spread function and the first region to obtain the first region after removing the glare; The first area of is fused with the second area to obtain the image after removing the glare.
The device according to claim 9, wherein the generating module is further configured to: take the model of the display screen as a transmittance function, combine with a light wave propagation factor, and use the spectral component of the highlighted object As the weight, the second point spread function is obtained; wherein, the model of the display screen includes: the amplitude modulation function and the phase modulation characteristic function of the display screen; The focal length of the image and the wavelength of the highlighted object are determined.
An electronic device, characterized by comprising: a display screen, a camera below the display screen, a processor and a memory; wherein,

The camera is used to collect images to be processed through the display screen;

the display screen is used for displaying the to-be-processed image and the image after glare removal;

the memory for storing processor executable instructions;

The processor is configured to implement the method of any of claims 1-7 when executing the instructions.
A non-volatile computer-readable storage medium on which computer program instructions are stored, characterized in that, when the computer program instructions are executed by a processor, the method described in any one of claims 1-7 is implemented.