WO2024103746A1

WO2024103746A1 - Image processing method, electronic device, computer program product, and storage medium

Info

Publication number: WO2024103746A1
Application number: PCT/CN2023/103443
Authority: WO
Inventors: 英豪; 杨杰
Original assignee: 华为技术有限公司
Priority date: 2022-11-18
Filing date: 2023-06-28
Publication date: 2024-05-23
Also published as: CN118057830A

Abstract

Embodiments of the present application relate to the field of terminals, and provide an image processing method, an electronic device, a computer program product, and a computer readable storage medium. The image processing method comprises: acquiring a multispectral image and an original formed image of a photography scene; performing spectral estimation on the basis of the multispectral image to obtain spectral power of a light source in the photography scene; and determining color reduction parameters according to the spectral power, and performing color reduction on the original formed image on the basis of the color reduction parameters to obtain a color-reduced image. According to the present application, the color deviation between an image captured by an electronic device and a real scene observed by human eyes can be corrected, so that the overall color perception of the captured image matches human vision, and the use experience of users is improved.

Description

Image processing method, electronic device, computer program product and storage medium

This application claims priority to the Chinese patent application filed with the China Patent Office on November 18, 2022, with application number 202211449305.1 and application name “Image processing method, electronic device, computer program product and storage medium”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of terminals, and in particular to an image processing method, an electronic device, a computer program product, and a computer-readable storage medium.

Background technique

There is a difference between the photosensitivity of the camera's imaging sensor and that of the human eye, which may cause the original color information of the image obtained by the camera to be inconsistent with the color information actually seen by the human eye.

In order to make the color information of the scene captured by the camera consistent with the color information perceived by the human eye, the image captured by the camera can be processed for color restoration. The existing color restoration processing method is generally based on the color correction parameters of several typical light sources that have been calibrated in advance. However, the light source composition in the actual shooting scene is relatively complex, resulting in various light source illumination situations in the shooting area. Based on the above processing method, it is difficult for the camera to accurately restore the color, resulting in a large color deviation between the captured photos and the real scene observed by the human eye.

Summary of the invention

In view of this, it is necessary to provide an image processing method to solve the problem in the prior art that it is difficult to accurately restore the color of the captured image, resulting in a large color deviation between the captured image and the real scene observed by the human eye.

A first aspect of an embodiment of the present application discloses a color restoration method, which is applied to an electronic device. The method includes: acquiring a multispectral image and an original imaging image of a shooting scene; performing spectral estimation based on the multispectral image to obtain the spectral power of the light source in the shooting scene; determining color restoration parameters based on the spectral power, and performing color restoration on the original imaging image based on the color restoration parameters to obtain a color restored image.

By adopting the above technical scheme, a multispectral image of the shooting scene is obtained. Compared with the original imaging image obtained by camera imaging (generally, the original imaging image is obtained based on imaging of the three primary colors (red, green, blue, RGB) image sensor), the multispectral image has a narrower channel bandwidth. Therefore, the imaging spectral resolution is higher. Based on this, the light source spectrum of the shooting scene can be accurately estimated to obtain the light source spectrum of the shooting scene, and then the color restoration parameters corresponding to the shooting scene can be accurately obtained to correct the color deviation between the photos taken by the camera and the real scenes observed by the human eye, so that the overall color perception of the captured image matches human vision, thereby improving the user experience.

In some embodiments, spectral estimation is performed based on the multispectral image to obtain the spectral power of the light source in the shooting scene, including: performing spectral estimation based on the multispectral image of the shooting scene to obtain the local spectral power of each area in the shooting scene; determining color restoration parameters according to the spectral power, including: determining the local color restoration parameters of each area according to the local spectral power of each area in the shooting scene.

By adopting the above technical solution, when the spectral powers of different areas in the same shooting scene are different, color restoration can be performed for each area, so that the image color of each area in the same shooting scene can be accurately restored, thereby improving the accuracy of color restoration of the original imaging image.

In some embodiments, spectral estimation is performed based on a multispectral image of a captured scene to obtain local spectral power of each area in the captured scene, including: performing highlight detection on the multispectral image to obtain a highlight area of the multispectral image; and obtaining the local spectral power of each area based on the highlight area.

With this technical solution, as the light source illuminates the photographed object, the photographed object may form a highlight area after reflection. It is easier to determine the spectral power of the light source based on the highlight area, and then the local spectral power of each area can be accurately obtained, saving calculation amount.

In some embodiments, the multispectral image includes an auxiliary accessory for highlight detection, and highlight detection is performed on the multispectral image to obtain a highlight area of the multispectral image, including: performing highlight detection on the auxiliary accessory in the multispectral image, and obtaining the highlight area of the multispectral image based on the position of the auxiliary accessory determined to be a highlight.

By adopting this technical solution, the detection success rate and accuracy of high-light areas can be improved through auxiliary accessories.

In some embodiments, highlight detection is performed on a multispectral image to obtain a highlight area of the multispectral image, including: counting the brightness of each pixel in the multispectral image, and taking the area where pixels with pixel brightness ranking before a preset position are located as the highlight area of the multispectral image; or detecting the brightness of each pixel in the multispectral image, and taking the area where pixels with pixel brightness greater than a preset brightness threshold are located as the highlight area of the multispectral image.

By adopting this technical solution, the highlight area of the multispectral image can be accurately extracted by taking the area where pixels with pixel brightness ranking before the preset position, or the area where pixels with pixel brightness greater than the preset brightness threshold, as the highlight area of the multispectral image, thereby improving the success rate and accuracy of highlight area detection.

In some embodiments, based on the highlight area, the local spectral power of each area is obtained, including: performing principal component analysis on the highlight area to obtain a first principal component vector of the highlight area and a second principal component vector of the highlight area; projecting the image data of the highlight area to a plane formed by the first principal component vector of the highlight area and the second principal component vector of the highlight area; determining a linear cluster with a linear distribution based on the distribution of the projected image data in the plane; performing principal component analysis on the linear cluster to obtain a first principal component vector of the linear cluster; and obtaining the local spectral power based on the first principal component vector of the highlight area, the second principal component vector of the highlight area, and the first principal component vector of the linear cluster.

By adopting this technical solution, by performing principal component analysis on the highlight area, the noise in the image data can be eliminated, the calculation overhead of subsequent spectral power analysis can be reduced, and the local spectral power corresponding to the area of different shooting scenes can be generated to improve the accuracy of the local spectral power.

In some embodiments, local spectral power is obtained based on the first principal component vector of the highlight area, the second principal component vector of the highlight area and the first principal component vector of the linear cluster, including: performing pseudo-inverse operation on the first principal component vector of the highlight area and the second principal component vector of the highlight area to obtain a pseudo-inverse matrix; and obtaining local spectral power based on the pseudo-inverse matrix and the first principal component vector of the linear cluster.

By adopting this technical solution and introducing a pseudo-inverse operation, the local spectral power corresponding to each local area can be accurately calculated, thereby improving the accuracy of the local spectral power.

In some embodiments, the electronic device includes a camera module, the local color restoration parameters include a local color correction matrix, and the local color restoration parameters of each area are determined according to the local spectral power of each area in the shooting scene, including: obtaining the color card reflectance function, spectral sensitivity function and standard observer color matching function of the camera module; obtaining the photosensitive data of the camera module based on the color card reflectance function, spectral sensitivity function and local spectral power; obtaining the standard observer tristimulus values based on the color card reflectance, the standard observer color matching function and the local spectral power; and obtaining the local color correction matrix according to the photosensitive data of the camera module and the standard observer tristimulus values.

By adopting this technical solution, the local color correction matrix, the color card reflectance function, the camera sensitivity function and the standard observer color matching function are fitted in the process of calculating the color correction matrix, thereby improving the accuracy of the color correction matrix and further improving the accuracy of color reproduction.

In some embodiments, the local color restoration parameters include a local chromatic adaptation conversion matrix, and the local color restoration parameters of each area in the shooting scene are determined according to the local spectral power of each area in the shooting scene, including: obtaining a standard observer color matching function and a spectral power of a target light source; obtaining white point tristimulus values of the light source in the shooting scene based on the local spectral power and the standard observer color matching function; obtaining white point tristimulus values of the target light source based on the spectral power of the target light source and the standard observer color matching function; obtaining a local chromatic adaptation conversion matrix according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source.

With this technical solution, a local chromatic adaptation conversion matrix is fitted based on the standard observer color matching function, the spectral power of the target light source, and the local spectral power, so that when color restoration is performed, the three stimulus values of the light source in the shooting scene can be converted to the three stimulus values under the target light source, thereby improving the accuracy of color restoration. For example, if the target light source is D65 light source.

In some embodiments, a local chromatic adaptation conversion matrix is obtained according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source, including: obtaining a first response value based on the white point tristimulus values of the light source in the shooting scene and a preset chromatic adaptation model, the first response value being the response value of the human eye to the long wave, medium wave and short wave of the light source in the shooting scene; obtaining a second response value based on the white point tristimulus values of the target light source and the chromatic adaptation model, the second response value being the response value of the human eye to the long wave, medium wave and short wave of the target light source; and obtaining the local chromatic adaptation conversion matrix according to the first response value and the second response value.

By adopting this technical solution, the three stimulus values are converted into the response values of the human eye to the long-wave, medium-wave and short-wave of the light source and the target light source in the shooting scene, and then the local chromatic adaptation conversion matrix is calculated, so that the calculated chromatic adaptation conversion matrix is more compatible with human vision.

In some embodiments, the color restoration parameters include: a color correction matrix and a color adaptation conversion matrix, and color restoration is performed on the original image based on the color restoration parameters to obtain a color restored image, including: correcting each pixel in the original image based on the color correction matrix to obtain a color corrected image; converting the color corrected image based on the color adaptation conversion matrix to obtain a color restored image.

Using this technical solution, the color correction matrix can convert the camera's photosensitivity data into the three stimulus values of a standard observer, and the chromatic adaptation correction matrix can convert the three stimulus values of the light source in the shooting scene into the three stimulus values under the target light source, thereby achieving color restoration.

In some embodiments, spectral estimation is performed based on a multispectral image to obtain the spectral power of the light source in the shooting scene, including: performing spectral estimation based on the multispectral image to obtain the spectral power and light source distribution information of the light source in the shooting scene; performing color restoration on the original imaging image based on color restoration parameters to obtain a color restored image, and further including: determining critical pixels in the color restored image based on the light source distribution information, the critical pixels being located in the light source boundary area; and smoothing the critical pixels to obtain a smoothed image.

By adopting this technical solution, critical pixels in the color restored image are smoothed based on the light source distribution information, avoiding large color differences in the image in the light source boundary area, achieving smooth color transition, and thus improving the color restoration effect.

In some embodiments, spectral estimation is performed based on a multispectral image to obtain the spectral power of the light source in the shooting scene, including: in response to a request to enter a color fidelity mode, generating a human-computer interaction interface for setting the light source information of the shooting scene; obtaining the light source information of the shooting scene set by the user from the human-computer interaction interface; and performing light source spectrum estimation based on the shooting scene light source information and the multispectral image to obtain the spectral power of the light source in the shooting scene.

By adopting this technical solution and combining the acquired light source information of the shooting scene with the multispectral image for spectral estimation, the spectral power of the light source in the shooting scene can be estimated more accurately. Spectral estimation is only performed in color fidelity mode, which can reduce the energy consumption of electronic equipment and improve user experience when the user has low demand for color restoration of the image.

In some embodiments, obtaining a multispectral image of a captured scene includes: obtaining a multispectral initial image of the captured scene through multiple multispectral image sensors; and merging the multispectral initial images to obtain a multispectral image of the captured scene.

By adopting this technical solution, a plurality of multispectral initial images are merged, so that the obtained multispectral image of the shooting scene contains more information and the spectral power estimation is more accurate.

In a second aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method as described in the first aspect.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the image processing method described in the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device (such as a computer), the electronic device executes the image processing method described in the first aspect.

In a fifth aspect, a device is provided, wherein the device has the function of implementing the electronic device behavior in the method provided in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The software includes one or more modules corresponding to the above functions.

It can be understood that the computer-readable storage medium described in the second aspect, the electronic device described in the third aspect, the computer program product described in the fourth aspect, and the device described in the fifth aspect all correspond to the method of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of color restoration of a captured image based on a color correction matrix provided by an embodiment of the present application;

FIG2 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

FIG3 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

FIG4 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application;

FIG5 is a schematic diagram of the architecture of an electronic device performing color restoration according to an embodiment of the present application;

FIG6 is a flow chart of color restoration performed by an electronic device according to an embodiment of the present application;

FIG7 is a schematic diagram of the architecture of an electronic device performing color restoration according to another embodiment of the present application;

FIG8 is a schematic diagram of an interface for setting light source information of a shooting scene in a high-fidelity mode provided by an electronic device according to an embodiment of the present application;

FIG9 is a schematic diagram of an interface for setting a highlight area of an electronic device provided by an embodiment of the present application;

FIG10 is a schematic diagram of a multispectral filter array of a multispectral image sensor provided in an embodiment of the present application;

FIG11 is a schematic diagram of a spectral sensitivity curve of a multi-spectral image sensor with two staggered peaks provided by an embodiment of the present application;

FIG12 is a schematic diagram of a scenario in which an electronic device performs spectrum estimation according to an embodiment of the present application;

FIG13 is a schematic diagram of a process of performing spectrum estimation by an electronic device provided in an embodiment of the present application;

FIG14 is a schematic diagram of setting auxiliary accessories in a shooting scene according to an embodiment of the present application;

FIG15 is a schematic diagram of the structure of a multi-spectral color temperature sensor provided in an embodiment of the present application;

FIG. 16 is a flow chart of an image processing method provided in an embodiment of the present application.

Detailed ways

It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And/or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

Spectrum: After the complex light is split by the dispersion system, the dispersed monochromatic light is arranged in sequence according to wavelength or frequency. The full name is optical spectrum.

Multispectral image: A multispectral image is an image that contains many bands, each band is a grayscale image that represents the brightness of the scene according to the sensitivity of the sensor used to produce that band.

Spectral reflectivity: When light source shines on the surface of an object, the object will selectively reflect electromagnetic waves of different wavelengths. Spectral reflectivity refers to the ratio of the luminous flux reflected by the object in a certain band to the luminous flux incident on the object, which characterizes the essential properties of the object's surface.

Spectral Sensitivity Function (SSF): A function that measures the sensitivity to light of different wavelengths.

Color Matching Function (CMF): The color matching function is the quantity of red, green and blue primary colors required to match each monochromatic light in the equal-energy spectrum. It is the basic data for color measurement and calculation.

Tristimulus value: The amount of stimulation of the three primary colors that causes the human retina to perceive a certain color, expressed as X (red primary color stimulation), Y (green primary color stimulation) and Z (blue primary color stimulation).

Chromatic Adaptation (CA): When the observation conditions change, the human visual system automatically adjusts the relative sensitivity of the three types of cone cells in the retina to try to keep the color perception of a certain physical target surface unchanged, that is, the color appearance remains unchanged.

Spectral Power Distribution (SPD): The distribution of light source radiation power according to wavelength, also known as spectral power.

The CIE 1931 XYZ color space (also called the CIE 1931 color space) is a mathematically defined color space created by the International Commission on Illumination (CIE) in 1931.

The photosensitivity of the imaging sensor in the camera is different from that of the human eye. Therefore, when the camera shoots a scene, there is a certain difference between the original color information of the scene obtained by the camera and the color information of the scene directly observed by the human eye. For example, after shooting mahogany-colored solid wood furniture, the color displayed in the image after imaging may turn blood red; after shooting yellow-green curtains, the color displayed in the image after imaging may be reddish. In order to make the color of the final image generated by the camera consistent with the real color observed by the human eye, the original imaging image obtained by the camera can be color restored. The role of color restoration is to convert the color information of the scene captured by the camera into the color information perceived by the human eye.

Lighting conditions will affect the color information obtained by the human eye and the camera, but the lighting conditions of the actual shooting scene are uncontrollable. For example, there may be multiple types of light sources such as natural light and fluorescent lights. It is difficult for the camera to accurately restore the color based on the light source of the actual shooting scene, resulting in color deviations between the captured photos and the real scene observed by the human eye.

As shown in reference figure 1, the electronic device can obtain in advance the standard color card photographed by the camera under various typical light sources to obtain known data such as the spectral reflectance of the standard color card, the camera's spectral sensitivity function, and the standard observer color matching function; the color correction matrix (Color Correction Matrix, CCM) of several typical light sources can be calibrated based on these known data; for example, when the camera shoots an actual scene, the two typical light sources close to each light source in the actual shooting scene and the weights of the two typical light sources can be calculated; a weighted sum is performed based on the color correction matrices of the two typical light sources and the weights of the two typical light sources, and the weighted sum value is used as the color correction matrix corresponding to the light source in the shooting scene.

For example, in FIG1 , there are multiple mixed light sources such as fluorescent light sources and natural light sources. The color correction matrices of the two closest typical light sources correspond to CCM_A and CCM_D, respectively. CCM_A accounts for 40% and CCM_D accounts for 60%. 40%*CCM_A and CCM_D*60% can be used as the color correction matrices corresponding to the light sources in the shooting scene.

The above method directly uses the linear combination of the color correction matrices of two typical light sources as the color correction matrix corresponding to the light source in the shooting scene, which will result in low accuracy of the color correction matrix and difficulty in accurate color restoration. In addition, the above method relies on the calibration of typical light sources. If the number of calibrated typical light sources is too small or the calibration is incorrect, it will be difficult to perform color restoration based on the calibrated light sources.

In some image processing methods, the electronic device may also perform the following steps: S1: taking the sum of the p parameters of each channel as the objective function, calculating the optimal spectral transformation matrix from the camera spectral sensitivity function to the CIE1931 XYZ color matching function, and satisfying the chromatic aberration constraints of the ideal reflective surface under several typical light sources; wherein the p parameter is defined as the degree of approximation between a pair of sensitivity functions s1(λ) and s2(λ) with respect to the wavelength λ; S2: for the original imaging image to be color corrected, directly applying the spectral transformation matrix obtained in S1 to the original RGB response value of each pixel to convert it to the CIE1931 XYZ color space; at the same time, directly applying the spectral transformation matrix obtained in S1 to the light source color response value estimated by the automatic white balance module to convert it to the CIE1931 XYZ color space; S3: using the CAT02 color adaptation transformation model in the CIECAM02 color appearance model to calculate the corresponding color of the object color under the standard light source after color adaptation; S4: converting the CIE1931 color response value of each pixel after color adaptation to the CIE1931 The XYZ tristimulus values are converted to the target color space for the camera's final output or file storage, completing the color correction process.

The above method converts the original RGB response signal of the camera into the device-independent CIE1931 XYZ space by calculating the spectral transformation relationship between the camera spectral sensitivity function and the CIE1931 color matching function, and uses the CAT02 color adaptation transformation model to calculate the corresponding color response value after color adaptation under the reference light source, thereby realizing a color correction process that does not rely on pre-calibrated parameters. However, this method only considers the light source spectrum function and the object spectral reflectance function when performing color correction. The parameters considered are relatively simple, and it is difficult to accurately determine the spectrum of the shooting scene, resulting in low color restoration effect.

To solve the above technical problems, the embodiment of the present application also provides an image processing method, which can accurately obtain the color restoration parameters corresponding to the shooting scene to correct the color deviation between the photos taken by the camera and the real scene observed by the human eye, so that the overall color perception of the captured image matches human vision.

The image processing method provided in the embodiment of the present application can be applied to electronic devices, and the electronic devices can communicate with other electronic devices or servers through a communication network. The electronic devices of the present application may include mobile phones, foldable electronic devices, tablet computers, personal computers (personal computers, PCs), laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPCs), netbooks, cellular phones, personal digital assistants (personal digital assistants, PDAs), augmented reality (augmented reality, AR) devices, virtual reality (virtual reality, VR) devices, artificial intelligence (artificial intelligence, AI) devices, wearable devices, smart home devices, and at least one of smart city devices. The embodiment of the present application does not impose special restrictions on the specific type of electronic devices. The communication network can be a wired network or a wireless network. For example, the communication network can be a local area network (LAN) or a wide area network (WAN), such as the Internet. When the communication network is a local area network, illustratively, the communication network may be a short-distance communication network such as a wireless fidelity (Wi-Fi) hotspot network, a Wi-Fi P2P network, a Bluetooth network, a Zigbee network, or a near field communication (NFC) network. When the communication network is a wide area network, illustratively, the communication network may be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future-evolved public land mobile network (PLMN) or the Internet.

In some embodiments, the electronic device may install one or more APPs (Applications). APP can be referred to as an application, which is a software program that can implement one or more specific functions. For example, communication applications, video applications, audio applications, image capture applications, cloud desktop applications, and the like. Among them, communication applications may include text messaging applications, for example. Image capture applications may include shooting applications (system cameras or third-party shooting applications). Video applications may include Huawei Video, for example. Audio applications may include Huawei Music. The applications mentioned in the following embodiments may be system applications installed on the electronic device when it leaves the factory, or they may be third-party applications downloaded from the Internet or obtained from other electronic devices by the user during the use of the electronic device.

Electronic equipment includes but is not limited to Windows or other operating systems.

FIG. 2 shows a schematic structural diagram of an electronic device 10 .

The electronic device 10 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, an audio module 150, a sensor module 160, a camera module 170, a display screen 180, etc.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural network processor. Network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. The memory may store instructions or data that have been used or are frequently used by the processor 110. If the processor 110 needs to use the instructions or data, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface. The processor 110 may be connected to an audio module, a wireless communication module, a display, a camera, and other modules through at least one of the above interfaces.

It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The wireless communication function of the electronic device 10 can be implemented through the antenna 1, the antenna 2, the mobile communication module 130, the wireless communication module 140, the modem processor and the baseband processor.

The mobile communication module 130 may provide solutions for wireless communications including 2G/3G/4G/5G, etc., applied to the electronic device 10. The mobile communication module 130 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. In some embodiments, at least some functional modules of the mobile communication module 130 may be arranged in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 may be arranged in the same device as at least some modules of the processor 110.

The wireless communication module 140 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 10. The wireless communication module 140 can be one or more devices integrating at least one communication processing module.

In some embodiments, the electronic device 10 can communicate with a network and other electronic devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).

The electronic device 10 can realize the display function through a GPU, a display screen 180, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 180 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The camera module 170 includes a camera. The display screen 180 is used to display images, videos, etc. The display screen 180 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 10 may include one or more display screens 180.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 10, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 10 by running instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 can be arranged in the processor 110.

The software system of the electronic device 10 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 10.

FIG. 3 is a software structure block diagram of the electronic device 10 according to an embodiment of the present application.

The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime (Android runtime, ART) and native C/C++ library, hardware abstract layer (HAL) and kernel layer.

The application layer can include a series of application packages.

As shown in FIG. 3 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

As shown in FIG. 3 , the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

The window manager provides window management services (Window Manager Service, WMS). WMS can be used for window management, window animation management, surface management and as a transit station for the input system.

Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

The Activity Manager can provide Activity Manager Service (AMS). AMS can be used to start, switch, and schedule system components (such as activities, services, content providers, broadcast receivers) as well as manage and schedule application processes.

The input manager can provide input management service (IMS), which can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. IMS takes events from input device nodes and distributes them to the appropriate window through interaction with WMS.

The Android runtime includes the core library and the Android runtime. The Android runtime is responsible for converting source code into machine code. The Android runtime mainly uses the ahead-of-time (AOT) compilation technology and the just-in-time (JIT) compilation technology.

The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides an API for users to develop Android applications.

The native C/C++ library can include multiple functional modules, such as surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of multiple commonly used audio and video formats, as well as static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides drawing and operation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for applications of electronic devices 10.

The hardware abstraction layer runs in user space, encapsulates kernel layer drivers, and provides a calling interface to the upper layer.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

The following is an illustrative introduction to an application scenario diagram of the image processing method provided in an embodiment of the present application in conjunction with FIG. 4 .

In an embodiment of the present application, the electronic device 10 is taken as a mobile phone with a camera function, and the mobile phone may have a front camera and/or a rear camera, which is not limited in the present embodiment. As shown in FIG4 , the electronic device 10 includes a camera module 170 and a display screen 180, and the camera module 170 includes a multispectral image sensor and an RGB image sensor.

In other embodiments of the present application, the electronic device 10 may also be a vehicle-mounted camera device, etc., but the embodiments are not limited thereto. The above-mentioned camera module 170 and display screen 180 may also be arranged on another electronic device that is communicatively connected to the electronic device 10. For example, the other electronic device may be communicatively connected to the electronic device 10 wirelessly. The electronic device 10 may be used to control the RGB image sensor in the camera module to obtain a first image based on the imaging of the shooting scene, and control the multispectral image sensor to obtain a second image based on the imaging of the shooting scene. The electronic device 10 is also used to perform color restoration on the first image based on the second image, and to control the display screen 180 to display the color-restored image, that is, the color-restored image is used as the final displayed captured image.

In application scenarios where there is a high-fidelity requirement for the color of objects in the shooting scene, for example, in mobile phone video live streaming, the color of many sales items serves as an important sales attribute, where the sales items may include jade, furniture, lipstick, clothes, and colorful pictures. The deviation in jade color may have a great impact on the price. Therefore, when shooting jade with a mobile phone camera, the accuracy of color restoration is very high; the color difference caused by lipstick is sufficient to cause deviation in color number. If the above-mentioned items cannot be accurately restored in color after being imaged by a mobile phone camera, disputes may arise due to color difference. The mobile phone can use the image processing method provided in the embodiment of the present application to accurately restore the color of the imaged image of the camera module 170, correct the color deviation between the imaged image of the camera module 170 and the real scene observed by the human eye, so that the overall color perception of the image finally displayed on the display screen 180 is consistent with human vision. Match and improve user experience.

As shown in FIG5 , it is a core architecture diagram of color restoration of an electronic device 10 according to an embodiment of the present application. The electronic device 10 may include a multi-spectral image sensor, an RGB image sensor, a processor and a display screen. The color restoration of the electronic device 10 may include the following implementation process:

51. The multispectral image sensor images the shooting scene to obtain a multispectral image of the shooting scene.

52. The RGB image sensor images the shooting scene to obtain an RGB image of the shooting scene (i.e., the original imaging image).

53. The processor performs spectrum estimation based on the multi-spectral image and the light source information of the shooting scene to obtain the spectral power and light source distribution information of the light source in the shooting scene.

In some embodiments, the light source information of the shooting scene can be set by the user. For example, the electronic device 10 is installed with a shooting application. After the electronic device 10 starts the shooting application, the multispectral image sensor can image the shooting scene to obtain a multispectral image of the shooting scene, and the RGB image sensor can image the shooting scene to obtain an RGB image of the shooting scene. The shooting application also has a human-computer interaction interface for setting the light source information of the shooting scene. For example, the light source information of the shooting scene that can be set includes: the number of light sources, the highlight area, the light source position, the light source boundary, the light source type, etc.

54. The processor obtains color restoration parameters according to the spectral power of the light source and the light source distribution information. The color restoration parameters include a color correction matrix and a color adaptation conversion matrix.

In some embodiments, the color correction matrix can be used to correct the RGB image, converting the RGB image from the RGB color space to the XYZ color space, and the chromatic adaptation conversion matrix can be used to perform chromatic adaptation transformation on the image in the XYZ color space, chromatic adaptation transformation of the XYZ image data under the current shooting scene light source to the XYZ image data under the target light source, for example, the target light source is D65 light source (also known as international standard artificial daylight (Artificial Daylight), whose color temperature is 6500K).

55. The processor performs color restoration on the RGB image according to the color restoration parameters to obtain a color restored RGB image.

In some embodiments, the color restored RGB image may be output to a display screen for display.

6 and 7, FIG6 is a flowchart of color restoration of the electronic device 10 according to an embodiment of the present application, and FIG7 is an overall architecture diagram of color restoration of the electronic device 10 according to an embodiment of the present application. The color restoration process is further described below in conjunction with FIG6 and FIG7.

Step 601: If a color fidelity mode information configuration request is received, a human-computer interaction interface for setting the shooting scene light source information is generated, and the scene light source information set by the user is obtained from the human-computer interaction interface.

In one embodiment of the present application, the electronic device 10 may be installed with a shooting application, and the shooting application may provide a color fidelity mode. In the color fidelity mode, the electronic device 10 may accurately restore the color of the imaging of the shooting scene.

Refer to Figure 8, which is a schematic diagram of the interface for the electronic device 10 to set the shooting scene light source information in the color fidelity mode. If the user has a high-fidelity color shooting requirement, the electronic device 10 can respond to the user's operation instructions to start the shooting application and enter the color fidelity mode. The electronic device 10 can also respond to the user's operation instructions to enter the information configuration interface of the color fidelity mode. The information configuration interface is a human-computer interaction interface for setting the shooting scene light source information; the electronic device 10 can obtain the shooting scene light source information set by the user from the human-computer interaction interface.

In some embodiments, the electronic device 10 may be configured to perform color restoration processing only in the color high-fidelity mode, which can improve the camera performance of the electronic device 10 and save power consumption of the electronic device 10. The electronic device 10 may also display an information configuration interface when entering the color fidelity mode. The information configuration interface may include an "OK" icon. After the user completes the information configuration and clicks the "OK" icon, the information configuration interface may be closed, and the shooting application displays a preview screen shot in the color fidelity mode.

For example, in Figure 8, the light source information that can be set in the human-computer interaction interface includes: the number of light sources, highlight area, light source position, light source boundary, and light source type. For example, the light source types include daylight, halogen light, fluorescence, and light emitting diode (LED).

Refer to Figure 9, which is a schematic diagram of setting the highlight area. The light source shines on the object and then reflects to the human body. When the eyes see the object, each part of the object has its corresponding brightness in the human eye. The user can set the highlight area in the displayed shooting scene, and the brightest point on the object can be set as the highlight area. For example, the human eye observes that there is a highlight area 802 on the jade 801, and a mark 803 can be placed on the highlight area 802.

It can be understood that in this embodiment, the mode for color restoration is called the color fidelity mode. In actual application, it can also be named "color mode", "color authenticity mode", "color-specific mode" and other similar names. The embodiment of the present application does not limit this.

Step 602: imaging the shooting scene by using a multispectral image sensor to obtain a multispectral image of the shooting scene.

A multispectral image has multiple frequency bands. For example, referring to FIG10 , FIG10 is a schematic diagram of a multispectral filter array (MSFA), in which a number represents a frequency band, and the MSFA has 16 frequency bands. The higher the spectral resolution of a multispectral image sensor, the greater the spatial resolution, and the higher the accuracy of spectral estimation.

In some embodiments, a plurality of multispectral image sensors may also be installed on the electronic device, and narrow-band distribution intervals are staggered between different multispectral image sensors. As shown in FIG11 , FIG11 is a schematic diagram of a spectral sensitivity curve of a multispectral image sensor with two staggered peaks. The electronic device 10 can obtain a multispectral initial image of the shooting scene through a plurality of multispectral image sensors; and merge the multispectral initial images to obtain a multispectral image of the shooting scene. This embodiment merges the multispectral images generated by a plurality of multispectral image sensors, and uses the merged multispectral image as the input of spectral estimation, which can improve the spectral resolution of the multispectral image, and thus improve the accuracy of subsequent spectral estimation.

For example, referring to Fig. 12, Fig. 12 is a schematic diagram of a spectrum estimation scene. After acquiring the multi-spectral image, spectrum estimation can be performed based on the multi-spectral image. The spectrum estimation step can refer to step 603.

Step 603 , performing spectrum estimation based on the multi-spectral image and the light source information of the shooting scene to obtain the spectral power and light source distribution information of the light source in the shooting scene.

The spectral power of the light source may also be referred to as the light source spectrum. The light source distribution information may include: position information of the light source distribution in the shooting scene.

Referring to FIG. 13 , in some embodiments, performing spectrum estimation according to the multispectral image in step 603 to obtain the spectral power and light source distribution information of the light source in the shooting scene may include:

Step 6031, perform highlight detection on the multispectral image to obtain the highlight area on the multispectral image.

Highlights are bright spots on objects observed by the human eye when light source shines on the object and then reflects into the human eye.

For example, when the electronic device 10 actually performs highlight detection, the brightness of each pixel in the multispectral image can be counted, and the pixels with the top 5% brightness among the pixels can be used as highlight areas. A brightness threshold can also be set. If a pixel exceeds the brightness threshold, the pixel can be used as a highlight area. The embodiments of the present application are not limited to this.

For another example, the highlight area can be preliminarily detected through a two-color reflection model, a center surround filter or a dark channel method, and then, the edges of the preliminarily detected highlight area that are misjudged as the highlight area are removed through a low-pass filter to obtain the highlight area on the multispectral image.

In some embodiments, as shown in FIG. 14 , an auxiliary accessory 1301 may be further provided in the shooting scene. The electronic device 10 may perform highlight detection on the auxiliary accessories in the multispectral image, obtain the auxiliary accessories in the multispectral image that are in a highlight state and the position information of these auxiliary accessories in the highlight state, and then determine the highlight area on the multispectral image based on the position information of these auxiliary accessories in the highlight state. In this embodiment, the auxiliary accessories can improve the success rate of highlight area detection and improve the accuracy of spectral estimation.

The auxiliary accessory may be an object of neutral color that can reflect the light emitted by the light source, and the size and position of the auxiliary accessory may be adjusted according to the actual scene. For example, the auxiliary accessory is a gray sphere with a glossy surface, and the gray spheres are evenly distributed in the shooting scene. The gray spheres can reflect the highlights formed by the light from the light source, and the electronic device 10 can provide more accurate highlight area data for spectral estimation based on the highlights detected by the gray spheres.

In some embodiments, the highlight area may be determined in combination with the scene light source information obtained in step 601. For example, the highlight area marked by the user may be obtained from the human-computer interaction interface in step 601, and the position corresponding to the marked highlight area may be determined in the multispectral image, and the position may also be used as the highlight area in the multispectral image.

Step 6032: Perform principal component analysis on the highlight area to obtain the first principal component vector and the second principal component vector. quantity.

Principal Component Analysis (PCA) is a statistical method. Through orthogonal transformation, a set of variables that may be correlated is converted into a set of linearly unrelated variables. The converted set of variables is called principal components. PCA is a dimensionality reduction method, which is often used to reduce the dimensionality of high-dimensional data sets. Its main idea is to map high-dimensional features to k dimensions. This k dimension is the principal component and can retain most of the information of the original variable. The information here refers to the variance of the original variable. The one with the largest variance among the principal components is called the first principal component vector, and the one with the second largest variance is called the second principal component vector.

Among them, principal component analysis can be implemented through methods such as singular value decomposition (SVD), which is not limited in the embodiments of the present application.

Step 6033, projecting the image data of the highlight area onto the plane formed by the first principal component vector and the second principal component vector.

Step 6034, determining the light source direction information according to the distribution of the image data of the highlight area on the plane.

For example, a linear cluster of linear distribution can be determined in the distribution of the projected image data in the plane, and the linear cluster represents the specular reflection of the light source on the object; then, principal component analysis is performed on the linear cluster to obtain the first principal component vector of the linear cluster, and the first principal component vector of the linear cluster represents the light source direction information of the corresponding area in the shooting scene.

Step 6035, obtaining the local spectral power according to the light source direction information, the first principal component vector of the highlight area, and the second principal component vector of the highlight area.

For example, a pseudo-inverse operation may be performed on the first principal component vector of the highlight area and the second principal component vector of the highlight area to obtain a pseudo-inverse matrix; the pseudo-inverse matrix is multiplied by the first principal component vector of the linear cluster to obtain the local spectral power.

That is, the pseudo-inverse matrix includes [the pseudo-inverse of the first principal component vector of the highlight area, the pseudo-inverse of the second principal component vector of the highlight area], and the local spectral power = pseudo-inverse matrix * the first principal component vector of the linear cluster.

For each highlight region on the multispectral image obtained in step 6031, the above steps 6032 to 6035 may be performed to obtain the local spectral power of each region in the multispectral image.

Step 6036, obtaining light source distribution information according to the local spectral power of each area in the multi-spectral image.

Based on the local spectral power of each region in the multispectral image, the regional distribution information of various spectral powers of the multispectral image can be obtained. The regional distribution information of spectral power represents the light source distribution information.

The above steps 602 and 603 are based on obtaining a multispectral image by a multispectral image sensor, and performing spectral estimation based on the multispectral image. In other embodiments, referring to FIG. 15 , which is a schematic diagram of a multispectral color temperature sensor, the electronic device can obtain the original RGB image of the camera and the intensity values of each channel of the multispectral color temperature sensor; the above two types of data are input into a pre-trained neural network model to obtain the probability that the light source of the current shooting scene belongs to each type of typical light source; based on the probability and the known spectrum of the typical light source, the light source spectrum of the current scene is determined.

However, this embodiment has the following problems:

1. Multispectral color temperature sensors cannot form images, have no spatial resolution, and have low accuracy.

2. The light received by the multi-spectral color temperature sensor includes the light from the light source as well as the light reflected by the object, which is equivalent to the object color mixed into the light source. The color temperature sensor cannot distinguish between the object color and the light source color, resulting in the inability to accurately calculate the true light source spectrum in the shooting scene.

3. The light source spectrum of this embodiment is obtained based on known typical light source spectrum data after judging and classifying the multi-spectral color temperature sensor data. The light source spectrum obtained based on the typical light source is not accurate.

4. This embodiment calculates the global light source spectrum in the shooting scene. However, in actual application, the local light source spectra of different areas of the shooting scene are not the same. Therefore, the color restoration of the global light source spectrum is not accurate.

Therefore, compared with the above-mentioned scheme of performing spectrum estimation by using a multi-spectral color temperature sensor, the spectrum estimation scheme of step 602 and step 603 in the embodiment of the present application has the following effects:

1. Step 602 and step 603 use a multispectral image sensor. The multispectral image obtained by the multispectral image sensor has a higher spectral resolution. Based on this, the spectrum of the light source at the shooting scene can be accurately estimated. Get the light source spectrum of the shooting scene.

2. The multispectral image sensor can distinguish the color of the object and the light source, so as to accurately obtain the real light source spectrum of the shooting scene.

3. Step 602 and step 603 do not rely on a typical light source spectrum that is pre-calibrated, but can improve the accuracy of spectrum estimation by estimating the light source spectrum of the multi-spectral image of the captured scene in real time.

4. The embodiment of the present application can estimate the local light source spectrum of each area in the shooting scene based on the local multi-spectral image, so that the subsequent color restoration is more accurate.

After the spectral power is acquired, step 604 may be performed.

Step 604: based on the spectral power, fit the color restoration parameters corresponding to the spectral power.

Color restoration parameters may include a color correction matrix (CCM) and a color adaptation conversion matrix. For the local spectral power of each area in the shooting scene, local color restoration parameters corresponding to the local spectral power, namely, the local color correction matrix and the local color adaptation conversion matrix, may be fitted.

In some embodiments, the electronic device 10 fits a local color correction matrix, which may include: obtaining a color card reflectance function of the camera module 170, a spectral sensitivity function of the camera module 170, and a standard observer color matching function; integrating the color card reflectance function, the camera spectral sensitivity function, and the local spectral power to obtain photosensitivity data of the camera module 170; integrating the color card reflectance, the standard observer color matching function, and the local spectral power to obtain standard observer tristimulus values; and obtaining a local color correction matrix based on the photosensitivity data of the camera module 170 and the standard observer tristimulus values.

For example, the local spectral power is recorded as E, the color card reflectance function is recorded as R, the camera spectral sensitivity function is recorded as S _cam , and the standard observer color matching function is recorded as S _cmf . Based on the above calculation of the photosensitive data ∫E*R*S _cam of the camera module 170 and the standard observer tristimulus value ∫E*R*S _cmf, the local color correction matrix CCM can be obtained by fitting. For example, through a conversion model such as a linear model, a polynomial model, or a root polynomial model, the local color correction matrix used in the process of converting the human eye photosensitive data into the standard tristimulus value is fitted and calculated.

Specifically, refer to the following formula:

∫E(λ)*R(λ)* _Scam (λ)=CCM*∫E(λ)*R(λ)* _Scmf (λ).

The above-mentioned camera spectral sensitivity function can be measured by selecting a certain model of camera module, and the color card reflectance function and the standard observation color matching function can also be measured in advance. The embodiment of the present application does not limit the measurement method of the above-mentioned embodiment.

In this embodiment, the CCM is calculated by taking into account the spectral power, the standard observer color matching function, the camera spectral sensitivity function and the color card reflectance function, and the parameters considered are more accurate, so that the calculated CCM is more comprehensive.

The step of fitting the chromatic adaptation conversion matrix by the electronic device 10 may include:

(1) Multiply the local spectral power E and the standard observer color matching function S _cmf to obtain the white point tristimulus value XYZ _WS of the light source in the shooting scene, where the red primary color stimulus amount in XYZ _WS is recorded as X _WS , the green primary color stimulus amount is recorded as Y _WS , and the blue primary color stimulus amount is recorded as Z _WS , that is, XYZ _WS =E*S _cmf .

(2) Multiply the spectral power E _tgt of the target light source and the standard observer color matching function S _cmf to obtain the white point tristimulus value XYZ _tgt of the target light source. The red primary color stimulus amount in XYZ _tgt is recorded as X _tgt , the green primary color stimulus amount is recorded as Y _tgt , and the blue primary color stimulus amount is recorded as Z _tgt , that is, XYZ _tgt = E _tgt * S _cmf .

For example, the target light source may be a D65 light source, the spectral power E _tgt of the D65 light source may be recorded as _ED65 , the tristimulus values of the D65 light source may be recorded as XYZ _D65 , the red primary color stimulation amount of the D65 light source may be recorded as X _WD , the green primary color stimulation amount may be recorded as Y _WD , and the blue primary color stimulation amount may be recorded as Z _WD .

(3) The local chromatic adaptation conversion matrix is obtained according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source.

For example, the white point tristimulus values of the light source in the shooting scene can be multiplied by a preset color adaptation model to obtain a first response value, which is the response value of the human eye to the long wave, medium wave and short wave of the light source in the shooting scene (also known as the degree of color adaptation).

The preset color adaptation model can be the CAT02 color adaptation model, that is, M _cat02 , and the first response value can be recorded as [ρ _S γ _S β _S ], the formula is as follows:

Then, the white point tristimulus values of the target light source are multiplied by the color adaptation model to obtain a second response value, which is the response value of the human eye to the long wave, medium wave and short wave of the target light source. The second response value can be recorded as [ρ _D γ _D β _D ], and the formula is as follows:

Finally, the local chromatic adaptation conversion matrix M _ca is obtained according to the first response value and the second response value.

Specifically, the formula can be referred to as follows, where is the inverse matrix of M _cat02 :

Step 605: Perform color restoration on the original image captured by the camera module based on the color restoration parameters.

In some embodiments, each pixel of the original imaging image may be processed through a color correction matrix to convert the original imaging image from an RGB color space to an XYZ color space to obtain a color-corrected image XYZsrc, where XYZsrc is the tristimulus value corresponding to the light source in the shooting scene. Therefore, XYZsrc may also be processed through a chromatic adaptation conversion matrix to convert the tristimulus values corresponding to the light source in the shooting scene to the tristimulus values under the target light source (e.g., a D65 light source) to obtain a color-restored image XYZ _tgt .

The above-mentioned original imaging image may be an RGB image obtained by imaging the shooting scene by the RGB image sensor in the camera module.

In some embodiments, when the RGB image sensor images the captured scene, the initially obtained raw image may be a Bayer format image, namely a Bayerraw image. Therefore, the Bayerraw image may be subjected to de-mosaicing and other processing to obtain the RGB image raw image.

When the color correction matrix is the local color correction matrix of each area in the shooting scene, and the color adaptation conversion matrix is the local color adaptation conversion matrix of each area in the shooting scene, before processing the pixel of the original imaging image, the local color correction matrix and the local color adaptation conversion matrix corresponding to the pixel can be determined first, and then the pixel can be processed based on the corresponding local color correction matrix and the local color adaptation conversion matrix.

However, each area in the above-mentioned shooting scene has a corresponding local color correction matrix and a local color adaptation conversion matrix. When color correction is performed based on the local color correction matrix, it may cause a sudden color change in the light source boundary area. Therefore, in the embodiment of the present application, when processing each pixel of the original imaging image through the color correction matrix, the critical pixel in the image after color restoration can be determined according to the light source distribution information. The critical pixel is located in the light source boundary area; after the color adaptation conversion matrix is processed, the critical pixel is smoothed to obtain a smoothed image. In this embodiment, the light source boundary area is smoothed to avoid sudden color changes in the light source boundary area and improve the smoothness of the image color.

For example, if the light source boundary area is a boundary area between a first area and a second area in the shooting scene, the first area The local color correction matrix of the first region is CCM1, and the local color correction matrix of the second region is CCM2. Each pixel located in the light source boundary region can be processed by interleaving CCM1 and CCM2, such as the first pixel located in the light source boundary region is processed by CCM1, the second pixel is processed by CCM2, the third pixel is processed by CCM1, and so on. For another example, CCM1 and CCM2 can be averaged, and each pixel in the light source boundary region can be processed by the average.

In actual application, other processing methods may also be used, and the embodiments of the present application do not limit the specific method of smoothing processing.

For example, after obtaining the tristimulus values under the target light source, a smoothing process may be performed based on the light source distribution information, and the tristimulus values of the target light source after the smoothing process may be converted to a standard RGB color space such as standard Red Green Blue (sRGB) for display on the display screen 180. That is, the image XYZ _tgt after color restoration may refer to an image in the sRGB color space, and the image in the sRGB color space is used as the captured image to be finally displayed.

16 , an embodiment of the present application provides an image processing method, which is applied to an electronic device 10. In this embodiment, the image processing method may include:

Step 1601, obtaining a multispectral image and an original imaging image of a shooting scene.

In some embodiments, the electronic device 10 may acquire a multispectral image and an original imaging image of the shooting scene in response to a shooting instruction.

In some embodiments, the electronic device 10 may also respond to a shooting instruction in a color fidelity mode to obtain a multispectral image and an original imaging image of the shooting scene to save power consumption, while in other shooting modes, the original imaging image of the shooting scene is obtained without performing color restoration processing on the original imaging image.

In some embodiments, at least one multispectral image sensor and an RGB image sensor may be installed on the electronic device 10. The electronic device 10 may obtain a multispectral image of the shooting scene through the multispectral image sensor and obtain an original imaging image of the shooting scene through the RGB image sensor.

In the case where multiple multispectral image sensors can be installed on the electronic device 10, the narrow-band distribution intervals between the multispectral image sensors can be staggered so as to obtain a multispectral image including more light source information. After the electronic device 10 obtains the multispectral initial image of the shooting scene through each multispectral image sensor, the multispectral initial images can be merged, and the merged multispectral image is used as the multispectral image of the shooting scene. The spectrum estimation is performed through the merged multispectral image, which can improve the accuracy of the spectrum estimation.

Step 1602: perform spectrum estimation based on the multispectral image to obtain the spectral power of the light source in the shooting scene.

In some embodiments, the local spectral power of each area in the shooting scene can be obtained based on the multispectral image. With this technical solution, when the spectral power of each area in the same shooting scene is different, color restoration can be performed for each area, so that the color of each area in the same shooting scene can be accurately restored.

In some embodiments, a highlight detection may be performed on a multispectral image to obtain a highlight region of the multispectral image; and each local spectral power may be obtained based on each highlight region. In order to accurately detect the highlight region of the shooting scene, an auxiliary accessory may be placed in the shooting scene, and the highlight region of the shooting scene may be determined by detecting the auxiliary accessory.

For example, the brightness of each pixel in the multispectral image can be counted, and the area where pixels with a preset brightness ranking in the front (such as the top 5%) are located is taken as the highlight area of the multispectral image; or the brightness of each pixel in the multispectral image can be detected, and the area where pixels with a brightness greater than a preset brightness threshold (which can be set according to actual needs) are located is taken as the highlight area of the multispectral image.

In some embodiments, highlight detection is performed on a multispectral image, and the highlight area of the multispectral image can be preliminarily detected through a two-color reflection model, a center surround filter, or a dark channel method, and then a low-pass filter is used to remove the edges of the preliminarily detected highlight area that are misjudged as the highlight area, thereby obtaining the highlight area on the multispectral image.

In other embodiments, the electronic device 10 generates a human-computer interaction interface for setting the light source information of the shooting scene in response to a request to enter the color fidelity mode. In the interface, the light source information that can be set may include: the number of light sources, the highlight area, the light source position, the light source boundary, and the light source type, such as sunlight, halogen light, fluorescence, and light-emitting diode. The electronic device 10 can obtain the light source information of the shooting scene from the human-computer interaction interface, and based on the shooting scene The light source information of the scene is used to obtain the highlight area.

In some embodiments, obtaining each local spectral power according to each highlight area includes: performing principal component analysis on the highlight area to obtain the principal component vector of the highlight area; obtaining the local spectral power based on the principal component vector of the highlight area. Specifically, the electronic device 10 can determine the first principal component vector and the second principal component vector in the principal component vector of the highlight area; project the image data of the highlight area to the plane formed by the first principal component vector of the highlight area and the second principal component vector of the highlight area; determine a linear cluster with a linear distribution in the distribution of the projected image data in the plane; perform principal component analysis on the linear cluster to obtain the first principal component vector of the linear cluster; obtain the local spectral power based on the first principal component vector of the highlight area, the second principal component vector of the highlight area, and the first principal component vector of the linear cluster.

The first principal component vector of the linear cluster represents the light source direction of the light source, and the distribution of the light source can be determined by the above method.

Furthermore, obtaining the local spectral power based on the first principal component vector of the highlight area, the second principal component vector of the highlight area and the first principal component vector of the linear cluster can include: performing a pseudo-inverse operation on the first principal component vector of the highlight area and the second principal component vector of the highlight area to obtain a pseudo-inverse matrix; multiplying the pseudo-inverse matrix with the principal component vector of the linear cluster to obtain the local spectral power.

Step 1603, determining a color restoration parameter according to the spectral power, and performing color restoration on the original image based on the color restoration parameter to obtain a color restored image.

In some embodiments, the color restoration parameters may include: a color correction matrix and a color adaptation conversion matrix. For the local spectral power of each area in the shooting scene, local color restoration parameters corresponding to the local spectral power, namely, the local color correction matrix and the local color adaptation conversion matrix, can be fitted.

For example, each pixel in the original imaging image is processed by a color correction matrix to obtain a color-corrected image; and the color-corrected image is processed by a chromatic adaptation conversion matrix to obtain a color-restored image.

In some embodiments, processing each pixel in the original imaging image through a color correction matrix to obtain a color-corrected image can include: determining critical pixels in the color-restored image based on light source distribution information, where the critical pixels are located in the light source boundary area; smoothing the critical area based on the local color correction matrix corresponding to each pixel in the light source boundary area to obtain a smoothed image, which can avoid large color differences in the image in the light source boundary area and achieve smooth color transition, thereby improving the color restoration effect.

In some embodiments, the electronic device 10 fits a local color correction matrix, which may include: obtaining a color card reflectance function, a spectral sensitivity function, and a standard observer color matching function of a camera module; integrating the color card reflectance function, the camera spectral sensitivity function, and the local spectral power to obtain photosensitivity data of the camera module; integrating the color card reflectance, the standard observer color matching function, and the local spectral power to obtain standard observer tristimulus values; and obtaining a local color correction matrix based on the photosensitivity data of the camera module and the standard observer tristimulus values.

In some embodiments, the step of fitting the local chromatic adaptation conversion matrix by the electronic device 10 may include: obtaining the standard observer color matching function and the spectral power of the target light source; multiplying the local spectral power by the standard observer color matching function to obtain the white point tristimulus values of the light source in the shooting scene; multiplying the spectral power of the target light source by the standard observer color matching function to obtain the white point tristimulus values of the target light source; and obtaining the local chromatic adaptation conversion matrix according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source.

Specifically, obtaining a local chromatic adaptation conversion matrix based on the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source can include: multiplying the white point tristimulus values of the light source in the shooting scene with a preset chromatic adaptation model to obtain a first response value, where the first response value is the response value of the human eye to the long wave, medium wave and short wave of the light source in the shooting scene; multiplying the white point tristimulus values of the target light source with the chromatic adaptation model to obtain a second response value, where the second response value is the response value of the human eye to the long wave, medium wave and short wave of the target light source; and obtaining a local chromatic adaptation conversion matrix based on the first response value and the second response value.

Wherein, obtaining the local chromatic adaptation conversion matrix according to the first response value and the second response value may include: generating a diagonal matrix based on the first response value and the second response value, wherein the values of the diagonal matrix respectively include: the ratio of the long wave of the target light source to the long wave of the light source in the shooting scene, the ratio of the medium wave of the target light source to the medium wave of the light source in the shooting scene, and the ratio of the short wave of the target light source to the short wave of the light source in the shooting scene; converting the inverse matrix of the chromatic adaptation model into The matrix is multiplied by the diagonal matrix and the chromatic adaptation model to obtain the local chromatic adaptation transformation matrix.

This embodiment performs spectral estimation of the real light source of the shooting scene based on the multispectral image by region to obtain the light source spectrum, and then obtains the local color conversion parameters corresponding to each region by region, such as the local color correction matrix and the color adaptation conversion matrix, so that the calculated color conversion parameters are more accurate and reasonable, and the color restoration of each object in the shooting scene is more accurate, which can effectively improve the overall color cast of the image and achieve the effect of matching the overall color perception of the scene with human vision.

In addition, the embodiment of the present application performs spectral estimation through multispectral images. The multispectral images have high spectral resolution and can accurately estimate the light source spectrum of each area in the scene. In the spectral estimation process, the light source information of the shooting scene set in the color fidelity mode is also combined, thereby further improving the accuracy of the light source spectrum estimation.

Moreover, the color restoration parameters are calculated based on the actual spectrum of the mixed light source scene, and the colors of each area in the shooting scene can be accurately restored through the local color correction matrix and the local color adaptation conversion matrix of the divided regions. The embodiment of the present application gets rid of the dependence on the typical light source in the calibration, covers a wider range of scenes, and ensures the accuracy of color restoration in different scenes.

In the electronic device 10 provided in the embodiment of the present application, the internal memory 121 can be used to store instructions, and the processor 110 can be used to call the instructions in the internal memory 121, so that the electronic device 10 executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on the electronic device 10, the electronic device 10 executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

Among them, the computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be in the form of a software product. It is embodied that the software product is stored in a storage medium, including several instructions for enabling a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

An image processing method, applied to electronic equipment, is characterized in that the method comprises:

Acquire multispectral images and original imaging images of the shooting scene;

Performing spectral estimation based on the multispectral image to obtain spectral power of the light source in the shooting scene;

Color restoration parameters are determined according to the spectral power, and color restoration is performed on the original image based on the color restoration parameters to obtain a color restored image.
The image processing method according to claim 1, characterized in that the performing spectral estimation based on the multispectral image to obtain the spectral power of the light source in the shooting scene comprises:

Performing spectral estimation based on the multispectral image of the shooting scene to obtain local spectral power of each area in the shooting scene;

The determining of the color restoration parameter according to the spectral power comprises:

According to the local spectral power of each area in the shooting scene, the local color restoration parameter of each area is determined.
The image processing method according to claim 2, characterized in that the performing spectral estimation based on the multispectral image of the shooting scene to obtain the local spectral power of each area in the shooting scene comprises:

Performing highlight detection on the multispectral image to obtain a highlight area of the multispectral image;

According to the highlight areas, the local spectral power of each area is obtained.
The image processing method according to claim 3, characterized in that the multispectral image includes an auxiliary accessory for highlight detection, and the highlight detection of the multispectral image to obtain the highlight area of the multispectral image includes:

Highlight detection is performed on the auxiliary accessories in the multispectral image, and based on the position of the auxiliary accessories determined as highlights, a highlight area of the multispectral image is obtained.
The image processing method according to claim 3, characterized in that the step of performing highlight detection on the multispectral image to obtain a highlight area of the multispectral image comprises:

Counting the brightness of each pixel in the multispectral image, and taking the area where the pixels with the highest brightness ranking are located as the highlight area of the multispectral image; or

The brightness of each pixel in the multispectral image is detected, and the area where the pixels whose brightness is greater than a preset brightness threshold are located is taken as the highlight area of the multispectral image.
The image processing method according to claim 3, characterized in that obtaining the local spectral power of each region according to the highlight region comprises:

Performing principal component analysis on the highlight area to obtain a first principal component vector of the highlight area and a second principal component vector of the highlight area;

Projecting the image data of the highlight area onto a plane formed by a first principal component vector of the highlight area and a second principal component vector of the highlight area;

Determining a linear cluster of linear distribution based on the distribution of the projected image data in the plane;

Performing principal component analysis on the linear cluster to obtain a first principal component vector of the linear cluster;

The local spectral power is obtained based on the first principal component vector of the highlight area, the second principal component vector of the highlight area, and the first principal component vector of the linear cluster.
The image processing method according to claim 6, characterized in that the obtaining of the local spectral power based on the first principal component vector of the highlight area, the second principal component vector of the highlight area and the first principal component vector of the linear cluster comprises:

Performing a pseudo-inverse operation on the first principal component vector of the highlight area and the second principal component vector of the highlight area to obtain a pseudo-inverse matrix;

The local spectral power is obtained based on the pseudo-inverse matrix and the first principal component vector of the linear cluster.
The image processing method according to claim 2, characterized in that the electronic device includes a camera module, the local color restoration parameters include a local color correction matrix, and the local color restoration parameters include a local color correction matrix according to the local color of each area in the shooting scene. The local spectral power of each region is determined to determine the local color restoration parameters of each region, including:

Obtaining a color card reflectance function, a spectral sensitivity function, and a standard observer color matching function of the camera module;

Based on the color card reflectance function, the spectral sensitivity function and the local spectral power, obtaining the photosensitivity data of the camera module;

Obtaining standard observer tristimulus values based on the color card reflectance, the standard observer color matching function and the local spectral power;

The local color correction matrix is obtained according to the photosensitive data of the camera module and the standard observer tristimulus values.
The image processing method according to claim 2, characterized in that the local color restoration parameters include a local chromatic adaptation conversion matrix, and determining the local color restoration parameters of each area according to the local spectral power of each area in the shooting scene includes:

Obtain the standard observer color matching function and the spectral power of the target light source;

Based on the local spectral power and the standard observer color matching function, obtaining white point tristimulus values of the light source in the shooting scene;

Obtaining white point tristimulus values of the target light source based on the spectral power of the target light source and the standard observer color matching function;

The local chromatic adaptation conversion matrix is obtained according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source.
The image processing method according to claim 9, characterized in that the step of obtaining the local chromatic adaptation conversion matrix according to the white point tristimulus values of the light source in the shooting scene and the white point tristimulus values of the target light source comprises:

Based on the white point tristimulus values of the light source in the shooting scene and a preset chromatic adaptation model, a first response value is obtained, where the first response value is a response value of the human eye to the long wave, medium wave and short wave of the light source in the shooting scene;

Based on the white point tristimulus values of the target light source and the chromatic adaptation model, a second response value is obtained, where the second response value is a response value of the human eye to the long wave, the medium wave and the short wave of the target light source;

The local chromatic adaptation conversion matrix is obtained according to the first response value and the second response value.
The image processing method according to claim 1, characterized in that the color restoration parameters include: a color correction matrix and a color adaptation conversion matrix, and the color restoration of the original image based on the color restoration parameters to obtain the color restored image includes:

Performing correction processing on each pixel in the original image based on the color correction matrix to obtain a color-corrected image;

The color-corrected image is converted based on the chromatic adaptation conversion matrix to obtain the color-restored image.
The image processing method according to claim 1, characterized in that the performing spectral estimation based on the multispectral image to obtain the spectral power of the light source in the shooting scene comprises:

Performing spectral estimation based on the multispectral image to obtain spectral power and light source distribution information of the light source in the shooting scene;

After performing color restoration on the original image based on the color restoration parameters to obtain the color restored image, the image processing method further comprises:

Determining critical pixels in the color restored image according to the light source distribution information, wherein the critical pixels are located in a light source boundary area;

The critical pixels are smoothed to obtain a smoothed image.
The image processing method according to any one of claims 1 to 12, characterized in that the performing spectral estimation based on the multispectral image to obtain the spectral power of the light source in the shooting scene comprises:

In response to a request to enter a color fidelity mode, generating a human-computer interaction interface for setting light source information of a shooting scene;

Acquiring shooting scene light source information set by the user from the human-computer interaction interface;

The light source spectrum is estimated based on the light source information of the shooting scene and the multi-spectral image to obtain the spectral power of the light source in the shooting scene.
The image processing method according to any one of claims 1 to 12, characterized in that acquiring a multispectral image of a shooting scene comprises:

Acquire a multispectral initial image of the shooting scene by using a plurality of multispectral image sensors;

The multispectral initial images are combined to obtain a multispectral image of the shooting scene.
A computer-readable storage medium, characterized in that it includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the image processing method according to any one of claims 1 to claim 14.
An electronic device, characterized in that the electronic device comprises a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the image processing method described in any one of claims 1 to claim 14.
A computer program product, characterized in that it comprises computer instructions, and when the computer instructions are executed on a processor, an electronic device executes the image processing method according to any one of claims 1 to 14.