CN116385566A

CN116385566A - Light source estimation method, device, electronic equipment, chip and storage medium

Info

Publication number: CN116385566A
Application number: CN202210593898.2A
Authority: CN
Inventors: 张雪岩
Original assignee: Shanghai Xuanjie Technology Co ltd
Current assignee: Shanghai Xuanjie Technology Co ltd
Priority date: 2022-05-27
Filing date: 2022-05-27
Publication date: 2023-07-04
Anticipated expiration: 2042-05-27

Abstract

The disclosure relates to the technical field of computer vision, and particularly provides a light source estimation method, a light source estimation device, electronic equipment, a chip and a storage medium. A light source estimation method, comprising: acquiring multispectral data of a current scene light source; determining a first weight set corresponding to the standard light source data set according to the preset standard light source data set and multispectral data; the standard light source data set comprises spectrum data of at least two standard light sources, and the first weight set comprises a first weight corresponding to each standard light source; and determining an estimation result of the current scene light source according to the spectrum data of each standard light source in the standard light source data set and the first weight. In the embodiment of the disclosure, the accuracy of light source estimation is improved, the color of the image is restored to be closer to the color actually perceived by human eyes, and the imaging effect is improved.

Description

Light source estimation method, device, electronic equipment, chip and storage medium

Technical Field

The disclosure relates to the technical field of image processing, and in particular relates to a light source estimation method, a light source estimation device, electronic equipment, a chip and a storage medium.

Background

For the human visual system, in the case of a change in field Jing Guangyuan, the human eye is able to perceive the inherent color of the object itself without being affected by the scene light source change, this capability is referred to as color constancy. For computer vision systems, when the scene light source changes, the imaging effect of the image sensor may be color cast to different degrees, such as blue or yellow of the object.

The light source estimation algorithm can estimate the correlated color temperature or white point coordinates of the current scene light source, so that the scene image is restored to the standard light source, and color restoration of the image is realized. However, in the related art, the light source estimation effect is poor, so that the color of the finally restored image deviates from the actual perception of human eyes, and the color effect of the image is poor.

Disclosure of Invention

In order to improve light source estimation accuracy, embodiments of the present disclosure provide a light source estimation method, apparatus, electronic device, chip, and storage medium.

In a first aspect, embodiments of the present disclosure provide a light source estimation method, including:

acquiring multispectral data of a current scene light source;

determining a first weight set corresponding to a standard light source data set according to the preset standard light source data set and the multispectral data; the standard light source data set comprises spectrum data of at least two standard light sources, and the first weight set comprises a first weight corresponding to each standard light source;

And determining an estimation result of the current scene light source according to the spectrum data and the first weight of each standard light source in the standard light source data set.

In some embodiments, the determining, according to a preset standard light source data set and the multispectral data, a first weight set corresponding to the standard light source data set includes:

carrying out regression processing on the initial weight of each standard light source according to the standard light source data set and the multispectral data to obtain a first weight corresponding to each standard light source;

and determining the first weight set according to the first weight value corresponding to each standard light source.

In some embodiments, the standard light source dataset comprises correlated color temperature data for each standard light source; the determining, according to the spectral data and the first weight of each standard light source in the standard light source data set, an estimation result of the current scene light source includes:

determining at least two candidate standard light sources from the standard light source dataset according to the first weight of each of the first weight sets;

and determining an estimation result of the current scene light source according to the spectrum data and the correlated color temperature data of each candidate standard light source and the multispectral data.

In some embodiments, the determining at least two candidate standard light sources from the standard light source dataset according to the first weights of the respective standard light sources comprises:

sequencing the first weights in the first weight set from large to small to obtain a first sequencing set;

and determining standard light sources corresponding to a preset number of first weights before sequencing as the candidate standard light sources according to the relation between the maximum value in the first sequencing set and a preset weight threshold.

In some embodiments, the determining, according to the relationship between the maximum value in the first sorted set and the preset weight threshold, the standard light source corresponding to the first weight of the preset number before sorting as the candidate standard light source includes:

determining standard light sources corresponding to a first weight of a first quantity before sorting as the candidate standard light sources in response to the maximum value in the first sorting set being not smaller than the preset weight threshold;

and/or the number of the groups of groups,

determining standard light sources corresponding to a first weight of a second number before sorting as the candidate standard light sources in response to the maximum value in the first sorting set being smaller than the preset weight threshold; wherein the second number is greater than the first number.

In some embodiments, the determining the estimation result of the current scene light source according to the spectrum data and the correlated color temperature data of each candidate standard light source and the multispectral data comprises:

carrying out regression processing on the first weight of each candidate standard light source according to the spectrum data, the correlated color temperature data and the multispectral data of each candidate standard light source to obtain the second weight of each candidate standard light source;

determining a target number of target standard light sources from the candidate standard light sources according to the second weight;

and determining an estimation result of the current scene light source according to the spectrum data of each target standard light source and the second weight.

In some embodiments, after obtaining the estimation result of the current scene illuminant, the method further comprises:

and determining a confidence score of the estimation result according to the estimation result of the current scene light source and the multispectral data.

In some embodiments, the process of pre-setting the standard light source dataset includes:

acquiring initial spectrum data of the at least two standard light sources through a multispectral sensor, and obtaining a spectrum curve of each standard light source according to the initial spectrum data;

For each standard light source, extracting spectral data of a preset wave band close to a response peak point of the multispectral sensor according to the spectral curve, and taking the spectral data as the spectral data of the standard light source;

and obtaining the standard light source data set according to the spectrum data of all the standard light sources.

In a second aspect, embodiments of the present disclosure provide a light source estimating apparatus, including:

the acquisition module is configured to acquire multispectral data of the current scene light source;

the weight determining module is configured to determine a first weight set corresponding to a standard light source data set according to the preset standard light source data set and the multispectral data; the standard light source data set comprises spectrum data of at least two standard light sources, and the first weight set comprises a first weight corresponding to each standard light source;

and the result determining module is configured to determine an estimation result of the current scene light source according to the spectrum data of each standard light source in the standard light source data set and the first weight.

In some embodiments, the weight determination module is configured to:

In some embodiments, the standard light source dataset comprises correlated color temperature data for each standard light source; the result determination module is configured to:

In some embodiments, the result determination module is configured to:

And/or the number of the groups of groups,

In some embodiments, the result determination module is configured to:

In some embodiments, the light source estimation device of the present disclosure further comprises a data set setting module configured to:

In a third aspect, embodiments of the present disclosure provide an electronic device, including:

a processor; and

a memory storing computer instructions for causing the processor to perform the method according to any implementation of the first aspect.

In a fourth aspect, an embodiment of the present disclosure provides a storage medium storing computer instructions for causing a computer to perform the method according to any embodiment of the first aspect.

In a fifth aspect, embodiments of the present disclosure provide a chip comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and to send the signal to the processor, the signal including computer instructions stored in the memory; the computer instructions, when executed by the processor, cause the electronic device to perform the method of any implementation of the first aspect.

The light source estimation method of the embodiment of the disclosure comprises the steps of obtaining multispectral data of a current scene light source, determining a first weight set corresponding to a standard light source data set according to a preset standard light source data set and multispectral data, and determining an estimation result of the current scene light source according to the spectrum data and the first weight value of each standard light source in the standard light source data set. In the embodiment of the disclosure, the estimation result of the current scene light source is determined by utilizing the spectrum data of the standard light source, so that the spectrum type of the current scene light source can be accurately reflected, the spectrum difference of the light source can be accurately identified, the accuracy of color restoration is improved, the obtained image is more similar to the color actually perceived by human eyes, and the imaging effect is improved.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the prior art, the drawings that are required in the detailed description or the prior art will be briefly described, it will be apparent that the drawings in the following description are some embodiments of the present disclosure, and other drawings may be obtained according to the drawings without inventive effort for a person of ordinary skill in the art.

Fig. 1 is a block diagram of an electronic device in accordance with some embodiments of the present disclosure.

Fig. 2 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 3 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 4 is a graph of spectra in accordance with some embodiments of the present disclosure.

Fig. 5 is a response graph of a spectral sensor in accordance with some embodiments of the present disclosure.

Fig. 6 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 7 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 8 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 9 is a flow chart of a light source estimation method in accordance with some embodiments of the present disclosure.

Fig. 10 is a block diagram of a light source estimating apparatus according to some embodiments of the present disclosure.

Fig. 11 is a block diagram of an electronic device in accordance with some embodiments of the present disclosure.

Detailed Description

The following description of the embodiments of the present disclosure will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure are intended to be within the scope of this disclosure. In addition, technical features related to different embodiments of the present disclosure described below may be combined with each other as long as they do not make a conflict with each other.

For the human visual system, in the case of a change in field Jing Guangyuan, the human eye is able to perceive the inherent color of the object itself without being affected by the scene light source change, this capability is referred to as color constancy. However, for the computer vision system, when the scene light source changes, the imaging effect of the image sensor may appear in different degrees of color cast, such as blue cast, yellow cast, etc. of the finally imaged object.

In order to realize the color constancy of a computer vision system, a light source estimation algorithm is generally used to estimate a real light source of a current scene to obtain a correlated color temperature value or white point coordinate of the current real light source, so as to correct the color of an image and restore the color of the image to the standard light source.

However, the inventor of the present application found that the light source estimation algorithm in the related art is difficult to be applied in a part of scenes, and is easy to cause erroneous color reproduction. Further research has found that this is because conventional light source technology algorithms often give the correlated color temperature or white point coordinates of the current scene light source, and the spectrum type of the light source cannot be determined.

Thus, for two light sources of close color but different spectra, if a conventional light source estimation algorithm is used, the system determines that the two light sources are of the same or similar light source type because the correlated color temperature and white point coordinates of the two light sources are substantially identical. That is, the conventional light source estimation algorithm cannot effectively identify light sources with colors close to each other but different spectrums, so that the color reproduction effect deviates from the color actually perceived by human eyes, and the color cannot be accurately reproduced.

Based on the defects in the related art, the embodiment of the disclosure provides a light source estimation method, a device, an electronic device, a chip and a storage medium, which aim to accurately identify a current scene light source by estimating the spectrum type of the light source and improve the accuracy of light source estimation and color restoration.

In a first aspect, embodiments of the present disclosure provide a light source estimation method, which may be applied to an electronic device. It will be appreciated that the electronic device of the embodiments of the present disclosure may be any suitable type of device for implementation, such as a smart phone, a tablet, a wearable device, a palm terminal, a vehicle-mounted device, a server, a cloud platform, etc., to which the present disclosure is not limited.

Fig. 1 shows a block diagram of an electronic device according to some embodiments of the present disclosure, and an application environment of the method of the present disclosure will be described below with reference to the structure of the electronic device shown in fig. 1.

As shown in fig. 1, an electronic device 600 in some examples of the present disclosure includes a processor 601, a memory 602, an image sensor 604, a multispectral sensor 605, and a display device 606.

A communicable connection between any two is established between the processor 601, the memory 602, the image sensor 604, the multispectral sensor 605, and the display device 606 via the bus 603.

Processor 601 may be any type of processor having one or more processing cores. It may perform single-threaded or multi-threaded operations for parsing instructions to perform operations such as fetching data, performing logical operation functions, and delivering operational processing results.

The memory 602 may include a non-volatile computer-readable storage medium such as at least one magnetic disk storage device, a flash memory device, a distributed storage device remotely located relative to the processor 601, or other non-volatile solid state storage device. The memory may have program storage areas for storing non-volatile software programs, non-volatile computer-executable programs, and modules which are called by the processor 601 to cause the processor 601 to perform one or more method steps. The memory 602 may also include a volatile random access memory medium, or a storage portion such as a hard disk, as a data storage area for storing the result of the arithmetic processing and data issued and output by the processor 601.

The image sensor 604 refers to a functional module for receiving external light to generate image data, such as a camera module commonly found in smartphones, an image sensor in a camera, and the like. In some embodiments, the image sensor 604 may comprise a CMOS (Complementary Metal Oxide Semiconductor ) sensor or a CCD (Charged Coupled Device, electrically coupled device) sensor or the like, as this disclosure is not limited.

The display device 606 refers to a functional module of the electronic apparatus for displaying images, such as a screen module of the electronic apparatus. In some embodiments, the display device 606 may include, for example, an LCD (Liquid Crystal Display) display module or an OLED (Organic Light-Emitting Diode) display module, which is not limited by the present disclosure.

The image sensor 604 receives external light to generate image data, the image data is processed by the processor 601 and then sent to the display device 606, and finally the image data is imaged on the display device 606, so that a user can see the image effect on the electronic equipment.

In the disclosed embodiment, the electronic device 600 further includes a multispectral sensor 605, the multispectral sensor 605 having a plurality of optical path channels, e.g., 8 channels, 10 channels, etc. Each channel of the multispectral sensor 605 can collect light in a wavelength range, so as to output the collected value in the wavelength range, and by combining the collected values of all the light path channels, multispectral data of the current scene light source can be obtained. For example, in one example, the multispectral sensor 605 of embodiments of the present disclosure may collect spectral data of visible light from 405nm to 690 nm.

Based on the electronic device structure shown in fig. 1, a light source estimating method of an example of the present disclosure will be described below with reference to fig. 2.

As shown in fig. 2, in some embodiments, the light source estimation method of the examples of the present disclosure includes:

s210, multispectral data of the current scene light source are acquired.

In the embodiment of the disclosure, as shown in fig. 1, the multispectral sensor 605 of the electronic device 600 may be utilized to acquire multispectral data of the current scene light source in real time.

Taking a photographing scene as an example, a user may press a shutter key of the electronic device 600 when photographing with the electronic device 600, and then the image sensor 604 may collect image data of an image of the current scene, while the multispectral sensor 605 may collect multispectral data of a light source of the current scene.

Taking the video recording scene as an example, when the user uses the electronic device 600 to record video, the user can press a recording start key of the electronic device 600, then the image sensor 604 collects image data of an image of the current scene frame by frame according to a preset collection frequency, and meanwhile, the multispectral sensor 605 can synchronously collect multispectral data of a light source of the current scene. For example, in one example, where the video recording has a frame rate of 30FPS, the image sensor 604 may collect 30 frames of image data per second, while the multispectral sensor 605 may also collect 30 frames of multispectral data per second.

In the embodiment of the present disclosure, a light source estimation method of an example of the present disclosure will be described taking a single frame image as an example. For video stream data, only each frame of image needs to be implemented according to the method disclosed herein, which is not described in detail in the disclosure.

S220, determining a first weight set corresponding to the standard light source data set according to the preset standard light source data set and the multispectral data.

In the embodiment of the present disclosure, a standard light source data set, which refers to a set of spectrum data including a plurality of standard light sources, needs to be established in advance.

In some embodiments, n standard light sources with different colors and different spectrums can be preselected, and spectrum data of each standard light source is collected by using a multispectral sensor, so that a data set including spectrum data of a plurality of standard light sources, namely a standard light source data set disclosed in the disclosure, is obtained.

In some embodiments, the standard light source data set disclosed in the present disclosure may further include correlated color temperature (CCT, correlated color temperature) data of each standard light source in addition to the spectrum data of each standard light source, so as to perform light source estimation in combination with the correlated color temperature CCT data, thereby improving accuracy.

The above-described process of pre-establishing a standard light source data set is specifically described in the following embodiments of the present disclosure, and will not be described in detail herein.

It should be noted that the standard light source data set is equivalent to a database including a plurality of standard light sources, and a light source in a real scene can be theoretically obtained by fusion of spectral data of one or more standard light sources in the database.

Therefore, the light source estimation method in the embodiment of the disclosure mainly finds a corresponding relationship between the spectrum data of each standard light source included in the standard light source dataset and the collected multispectral data of the current scene light source, the corresponding relationship can be understood as a weight value of each standard light source, and the spectrum data and the weight value of each standard light source are used for carrying out the multi-light source fusion, so that the obtained multi-light source fusion result is used for representing the light source estimation result of the current scene light source.

Specifically, the standard light source data set is denoted as a, the collected multispectral data of the current scene light source is denoted as B, and the two have the following relationship:

A*x＝B (1)

in the formula (1), x represents a correspondence between a standard light source data set and a current scene light source, that is, a first weight set according to the embodiment of the disclosure, which includes a first weight value corresponding to spectrum data of each standard light source.

In the embodiment of the disclosure, according to the standard light source data set, the acquired multispectral data and other constraint conditions, an optimization algorithm is utilized to estimate each standard light source to obtain a first weight corresponding to each standard light source, and a set of the first weights of all the standard light sources, namely, a first weight set.

S230, determining an estimation result of the current scene light source according to the spectrum data and the first weight of each standard light source in the standard light source data set.

In the embodiment of the disclosure, after the first weight set is obtained, the spectrum data of each standard light source can be fused according to the spectrum data of each standard light source and the corresponding first weight, and the obtained fusion result is the estimation result of the current scene light source.

It should be noted that, in the estimation result obtained in the embodiment of the present disclosure, the correspondence between the current scene light source and the standard light source is accurately represented, so that the spectrum type of the current scene light source can be accurately reflected, and in the subsequent color restoration algorithm, color restoration can be accurately performed on the image according to the estimation result.

Especially, for two light sources with close colors but different spectrums, the spectrum difference of the two light sources can be accurately identified in the light source estimation results obtained by the method, so that different color reduction treatments are respectively carried out, the obtained image is consistent or close to the color actually perceived by human eyes, the color cast influence is reduced, the color reduction of 'what you see is what you get' is realized, and the imaging effect is improved.

In some embodiments, since the number of standard light sources in the standard light source data set is large, if all standard light sources are used to represent the estimation result of the current scene light source, the data size is large, so that only part of standard light sources with high weights can be selected to represent the estimation result of the current scene light source, that is, the most representative part of standard light sources are selected from all standard light sources, thereby simplifying the data size. The following embodiments of the present disclosure are described in detail herein.

As can be seen from the foregoing, in the embodiment of the present disclosure, the estimation result of the current scene light source is determined by using the spectrum data of the standard light source, so that the spectrum type of the current scene light source can be accurately reflected, the spectrum difference of the light source can be accurately identified, the accuracy of color reduction is improved, the obtained image is closer to the color actually perceived by human eyes, and the imaging effect is improved.

In some embodiments, the spectral data of a plurality of standard light sources may be acquired in advance using a multispectral sensor, thereby yielding a standard light source dataset.

Taking 1 standard light source as an example, spectrum data of the standard light source is collected by utilizing a multispectral sensor, and the spectrum data can be represented by the following light source matrix:

Each element in the light source matrix represents: and (3) sampling the corresponding spectral characteristic value at one wavelength. For example, in one example, the multispectral sensor collects light with a wavelength range of 300nm to 1000nm, so that m=701 in the corresponding light source matrix, that is, each element represents a spectral feature value corresponding to a 1nm wavelength sampling point.

In some embodiments, n standard light sources may be co-calibrated in the above manner, i.e. n light source matrices may be obtained, and then all spectral data included in the final standard light source data set may utilize the following standard light source matrix a _m*n Representation of：

Standard light source matrix A _m*n Each column of (2) represents the spectral data of 1 standard light source, i.e. the spectral data of n standard light sources in total. Wherein each element

I.e. the characteristic value of the ith standard light source at the jth wavelength sampling point.

In some embodiments, consider a standard light source matrix A constructed in the manner described above _m*n The data volume is larger, the requirement on the system hardware performance is higher, and the deployment of lightweight equipment such as mobile terminals or wearable equipment is not facilitated. For example, in the above example, the multispectral sensor collects light having a wavelength in the range of 300nm to 1000nm altogether, so that the spectral data of each standard light source includes m=701 eigenvalues. Assuming that the standard light source data set includes n=20 standard light sources in total, the standard light source matrix a obtained as described above _m*n Together, the data volume is large, including m×n=701×20=14020 eigenvalues.

Accordingly, in some embodiments of the present disclosure, the amount of data of the standard light source matrix is reduced by extracting eigenvalues of the standard light source matrix, which is described below in connection with the embodiment of fig. 3.

As shown in fig. 3, in some embodiments, the process of pre-setting the standard light source data set includes:

s310, acquiring initial spectrum data of at least two standard light sources through a multispectral sensor, and obtaining a spectrum curve of each standard light source according to the initial spectrum data.

In the embodiment of the disclosure, taking a standard light source as an example, similar to the foregoing process, a multispectral sensor may be used to collect spectrum data of the standard light source, that is, initial spectrum data, and the spectrum matrix is expressed as:

it will be appreciated that when each element in the spectrum matrix represents a characteristic value of one wavelength sampling point, for example, m=701, the light source matrix of the standard light source includes 701 characteristic values in total.

After initial spectrum data of the standard light source, namely a light source matrix is obtained, a corresponding spectrum curve can be obtained by fitting according to the light source matrix. For example, in one example, the fitted spectral curve may be as shown in fig. 4. In fig. 4, the horizontal axis represents the wavelength range collected by the multispectral sensor, i.e., 300nm to 1000nm; the vertical axis represents the feature value. Spectral matrix

I.e., one point on the graph of fig. 4.

The above description describes the process of determining the spectrum curves of the standard light sources by taking one standard light source as an example, and the initial spectrum data and the spectrum curves of each standard light source can be obtained by sequentially and respectively executing the above processes for n standard light sources. This disclosure is not repeated here.

S320, for each standard light source, extracting spectral data of a preset wave band close to a response peak point of the multispectral sensor according to a spectral curve, and taking the spectral data as spectral data of the standard light source.

The response peak point of the multispectral sensor refers to the peak point of the response curve corresponding to each light path channel in the multispectral sensor. For example, a schematic diagram of a response curve of the multispectral sensor is shown in fig. 5, each curve in fig. 5 represents a response curve corresponding to one optical path channel, and as can be seen by observing fig. 5, each curve corresponds to a peak, and the peak represents the best response at the wavelength sampling point.

Thus, in the embodiment of the present disclosure, spectral data in a range near the response peak point can be extracted as spectral data of the standard light source.

For example, in the examples of fig. 4 and 5, spectral data of a preset band range near each response peak point can be selected and then combined as the spectral number of the standard light sourceAccording to the above. For example, taking the first peak at the left side of fig. 5 as an example, the wavelength sampling point corresponding to the response peak point is 410nm, and the preset wave band may be the wavelength sampling point ±5nm, so that the spectral data with the wavelength range of 410nm±5nm can be extracted from the spectral curve of fig. 4, and the process is performed on the peak in each response curve, so that the final spectral data can be obtained. Using a spectral matrix can be expressed as

I.e. the spectral matrix comprises k elements in total, k < m.

S330, obtaining a standard light source data set according to the spectrum data of all the standard light sources.

For each standard light source, the foregoing process is executed, so that the spectrum data corresponding to each standard light source can be obtained, and then the spectrum data of all n standard light sources are combined to obtain a final standard light source data set, and the standard light source matrix can be used for representing:

it can be understood that since the light source matrix of each standard light source extracts k eigenvalues close to the response peak point from m eigenvalues, k is far smaller than m, and the standard light source matrix A is finally obtained _k*n The matrix size is much smaller than the standard light source matrix a _m*n Thereby greatly reducing the amount of data.

As can be seen from the foregoing, in the embodiment of the present disclosure, the spectral data near the response peak point is extracted from the spectral data of each standard light source to obtain the standard light source data set, so that the data volume of the standard light source data set is reduced, the complexity of the algorithm is reduced, and the method is beneficial to being deployed in lightweight electronic devices such as smart phones and wearable devices.

In some embodiments, the pre-established standard light source data set includes not only spectral data of each standard light source, but also correlated color temperature data of each standard light source. Correlated color temperature (CCT, correlated Color Temperature) is a parameter used to characterize the color of a light source, which refers to the temperature of a blackbody radiator that is most similar in color to the light source having the same intensity stimulus.

In some embodiments of the present disclosure, after the spectrum data of each standard light source is acquired by the multispectral sensor, a correlated color temperature CCT value of each standard light source may be calculated according to the spectrum data.

For example, taking a standard light source as an example, after spectrum data of the standard light source is obtained by a multispectral sensor, channel values can be converted into XYZ values according to spectrum data based on a CIE XYZ-Yxy color model, then the xy values are calculated, and then CCT values are calculated by the following formula:

CCT(x，y)＝-449n ³ +3525n ² -6823.3n+5520.33 (3)

In the formula (3), n= (x-x) _e )/(y-y _e )，x _e ＝0.3320，y _e = 0.1858. The CCT value of each standard light source obtained through final calculation, namely correlated color temperature data of each standard light source disclosed in the disclosure.

It will be appreciated by those skilled in the art that, for the above process of calculating the standard illuminant CCT value, it is certainly understood and fully implemented with reference to the CIE XYZ-Yxy color model in the related art, and this will not be repeated in this disclosure.

In some embodiments, after obtaining the spectral data and correlated color temperature data of each standard light source, the spectral data and correlated color temperature data may also be normalized, so that the data values may be constrained to a smaller range of values such that the data values are on the same order of magnitude. For example, in one example, the spectral data for each standard light source may be preprocessed using mean normalization and correlated color temperature data may be preprocessed using inverse normalization. Those skilled in the art will understand and fully realize this, and this disclosure is not repeated.

After the above establishment of the standard light source data set is completed, the light source estimation method of the present disclosure may be implemented using the pre-established standard light source data set, which is further described below.

As shown in fig. 6, in some embodiments, the light source estimation method of the examples of the present disclosure, the process of determining the first set of weights from the standard light source data set and the multispectral data includes:

s610, carrying out regression processing on the initial weight of each standard light source according to the standard light source data set and the multispectral data to obtain a first weight corresponding to each standard light source.

S620, determining a first weight set according to the first weight value corresponding to each standard light source.

In the embodiment of the present disclosure, the initial weight refers to an initial weight given to each standard light source, for example, in one example, the standard light source data set includes n standard light sources in total, so that the initial weight of each standard light source may be set to 1/n.

From the foregoing, it can be seen that the standard light source dataset has a relationship as shown in equation (1) with the multispectral data of the current scene light source. In the embodiment of the disclosure, regression processing may be performed on the initial weight according to the standard light source dataset, the multispectral data and the preset constraint condition based on the least square method, so as to obtain a first weight corresponding to each standard light source, which is expressed as:

in the formula (4), A represents a standard light source data set, b represents multispectral data, H represents a constraint condition, and x is a first weight.

The regression processing can be performed on the initial weight of each standard light source through the above formula (4), so as to obtain a first weight corresponding to each standard light source, wherein the set of all n first weights is a first weight set, which can be expressed as: [ w ] ₁ ，w ₂ ，w ₃ ，……，w _n ]。

In some embodiments, after the first weight set is obtained, weighted fusion may be performed according to each standard light source spectrum data in the standard light source data set, so as to obtain an estimation result that is finally used to represent the current scene light source spectrum type. And the spectrum data of the Top-k standard light sources can be ranked according to the first weight to carry out weighted fusion, so that an estimation result which is finally used for representing the spectrum type of the current scene light source is obtained. The present disclosure is not limited in this regard.

According to the method, in the embodiment of the disclosure, the estimation result of the current scene light source is determined by utilizing the spectrum data of the standard light source, so that the spectrum type of the current scene light source can be accurately reflected, the spectrum difference of the light source can be accurately identified, the accuracy of color restoration is improved, the obtained image is more similar to the color actually perceived by human eyes, and the imaging effect is improved.

In some embodiments, after the first weight set is obtained, the estimation result of the current scene light source is not determined directly based on the first weight set and the spectrum data of the standard light source, but the correlated color temperature data (CCT) of the standard light source is further fused, and the correlated color temperature data can describe the color characteristics of the light source, so that the accuracy of light source estimation can be further improved, and the problem of confusion in light source estimation is avoided. The following describes the embodiment of fig. 7.

As shown in fig. 7, in some embodiments, the light source estimation method of the present disclosure includes a process of obtaining an estimation result of a current scene light source, including:

s710, determining at least two candidate standard light sources from the standard light source data set according to each first weight value in the first weight set.

S720, determining an estimation result of the current scene according to the spectrum data and the correlated color temperature data of each candidate standard light source and the multispectral data.

In this embodiment of the disclosure, each first weight in the first weight set indicates a degree of correlation between a standard light source corresponding to the first weight and a current scene light source, and a larger first weight indicates a higher degree of correlation between a corresponding standard light source and a current scene light source, whereas a smaller first weight indicates a lower degree of correlation between a corresponding standard light source and a current scene light source.

Based on this, in some embodiments of the present disclosure, a plurality of candidate standard light sources with a higher degree of correlation may be selected from the standard light source dataset based on the magnitude of the first weight, so as to reduce the calculation amount. The following describes the embodiment of fig. 8.

As shown in fig. 8, in some embodiments, the light source estimation method of the present disclosure examples, a process of determining candidate standard light sources, includes:

S711, sorting all the first weights in the first weight set from big to small to obtain a first sorting set.

S712, determining standard light sources corresponding to a preset number of first weights before sequencing as candidate standard light sources according to the relation between the maximum value in the first sequencing set and a preset weight threshold.

In the embodiment of the disclosure, the first weight set includes a first weight of each standard light source, so that the first weights are ranked from large to small, and a first ranking set can be obtained.

In some embodiments, after the first sorted set is obtained, the standard light sources corresponding to Top-k first weights in the Top order may be directly determined as candidate standard light sources. For example, the standard light sources corresponding to the first weights of the top 4 of the ranks may be determined as candidate standard light sources.

In other embodiments, it is contemplated that the sum of all first weights in the first sorted set is equal to 1. Therefore, if the first weight of a certain standard light source is far greater than that of other standard light sources, the correlation degree between the standard light source and the current scene light source is high, and a large number of standard light sources are not needed to be selected for fusion to obtain an estimation result, so that the calculation amount can be reduced. Otherwise, if the first weight distribution of all the standard light sources is more balanced, the fact that a larger number of standard light sources are needed to be fused to obtain an accurate estimation result is indicated.

Based on this, in some embodiments of the present disclosure, the candidate standard light sources may be determined according to a relationship between a maximum value in the first sorted set and a preset weight threshold. Specifically, the following two cases may be included:

1) And determining standard light sources corresponding to the first weight values of the first quantity before sequencing as candidate standard light sources in response to the fact that the maximum value in the first sequencing set is not smaller than a preset weight threshold.

For example, the preset weight threshold may be set to 0.5. If the maximum value in the first sorting set is greater than 0.5, the sum of all the first weights in the first sorting set is 1, so that the first weight of the standard light source corresponding to the maximum value is far greater than that of other standard light sources, and in this case, a smaller number of candidate standard light sources can be selected. For example, the standard light source corresponding to the first weight of the top 2 ranked items may be determined as the candidate standard light source, that is, the number of candidate standard light sources is 2.

2) And determining standard light sources corresponding to the first weight values of the second quantity before sequencing as candidate standard light sources in response to the maximum value in the first sequencing set being smaller than a preset weight threshold.

Still further to the above example, the preset weight threshold may be set to 0.5. If the maximum value in the first sorting set is smaller than 0.5, the sum of all the first weights in the first sorting set is 1, so that the first weight distribution of each standard light source is relatively uniform, and in this case, a larger number of candidate standard light sources can be selected. For example, the standard light sources corresponding to the first weight of the top 4 of the ranks may be determined as candidate standard light sources, that is, the number of candidate standard light sources is 4.

Of course, those skilled in the art will appreciate that the foregoing is merely an example of the disclosure, and that those skilled in the art may implement the disclosure according to specific scenarios and are not limited to the foregoing examples.

Through the above process, a preset number of candidate standard light sources can be obtained, and the estimation result of the current scene light source can be calculated according to the spectrum data, correlated color temperature data and multispectral data of the candidate standard light sources, which is described below in connection with the embodiment of fig. 9.

As shown in fig. 9, in some embodiments, the light source estimation method of the present disclosure includes a process of determining an estimation result of a current scene light source, including:

s721, carrying out regression processing on the first weight of each candidate standard light source according to the spectrum data and correlated color temperature data of each candidate standard light source and multispectral data to obtain the second weight of each candidate standard light source.

In the embodiment of the disclosure, after determining the candidate standard light sources, the first weights of the candidate standard light sources may be normalized, where the purpose of normalization is to make the sum of the first weights of the candidate standard light sources equal to 1.

For example, in one example, candidate standard light sources include: a standard light source 1, the first weight of which is 0.6; the first weight of the standard light source 2 is 0.3. The first weight of the standard light source 1 is normalized: 0.6/(0.6+0.3) =0.67. The first weight of the standard light source 2 is normalized: 0.3/(0.6+0.3) =0.33, so that the sum of the normalized first weights of the two candidate standard light sources is 1.

In this embodiment of the present disclosure, the normalized first weight is used as a regression initial value of each candidate target light source, and then the first weight may be subjected to regression processing according to the spectral data, the correlated color temperature data and the preset constraint condition of each candidate standard light source based on the least square method, so as to obtain the optimized second weight of each candidate standard light source. For example, in the above example, after the least squares regression, the second weights of the two candidate standard light sources are 0.7 and 0.3, respectively.

Specific procedures will be understood and fully implemented by those skilled in the art with reference to the foregoing embodiment of fig. 5, and this disclosure will not be repeated here.

S722, determining target standard light sources of target quantity from the candidate standard light sources according to the second weight.

In the embodiment of the disclosure, the target number is a preset value, which may be determined by a subsequent color restoration algorithm, that is, the color restoration algorithm requires that the estimation result of the current scene light source is represented by t standard light sources, and the target data may be set to t. Of course, the target number may also be selected according to specific scenario requirements, which is not limited by the present disclosure.

It is worth noting that the target number of target standard light sources may be the same as the number of candidate standard light sources. For example, in the above example, the number of candidate standard light sources is 2, and the data of the target standard light source may be 2 as well, that is, all of the 2 candidate standard light sources are determined as the target standard light source.

Of course, the target number of the target standard light sources may be smaller than the number of the candidate standard light sources, for example, the number of the candidate standard light sources is 4, and the target number required by the color reduction algorithm is 2, and 2 target standard light sources may be determined from the 4 candidate standard light sources according to the second weight of the candidate standard light sources.

For example, in the above example, the second weights of the 4 candidate standard light sources may be ranked from large to small, and then the first 2 candidate standard light sources of the ranking may be selected as the target standard light source. Of course, the second weights of the 2 target standard light sources may be normalized, so that the sum of the second weights of the two may be equal to 1, which is not described in detail in this disclosure.

S723, determining an estimation result of the current scene light source according to the spectrum data of each target standard light source and the second weight.

After the determined target standard light sources are obtained, fusion processing is carried out according to the spectrum data of each target standard light source and the second weight, and an estimation result used for representing the spectrum type of the current scene light source is obtained.

As can be seen from the foregoing, in the embodiment of the present disclosure, after the first weight set is obtained, the correlated color temperature data is further fused to obtain the second weight of the target standard light source, so that the problem of confusion of the light source type in the light source estimation is effectively avoided by using the correlated color temperature data, and the accuracy of the light source estimation is improved.

In some embodiments, after obtaining the estimation result of the current scene illuminant, a corresponding confidence score may also be calculated according to the estimation result, where the confidence score is used to represent the reliability of the estimation result.

It will be appreciated that the estimation result is an estimation value of the spectrum type of the current scene light source represented by the spectrum data of the target standard light source and the second weight, and the aforementioned multispectral data collected by the multispectral sensor represents the actual value of the current scene light source.

The smaller the difference between the two, the more reliable the estimation result is, namely, the better the image effect obtained by the subsequent color reduction is; otherwise, if the difference between the two is larger, the estimation result is lower in reliability, and the image effect obtained by the subsequent social color reduction is relatively poor. Thus, in the disclosed embodiments, the confidence score may be utilized to represent the difference between the current scene illuminant estimation result and the multispectral data.

For example, in one example, the confidence score may be calculated from the adjusted symmetric mean absolute percentage error, specifically expressed as:

confidene＝1-sMAPE_adjusted (6)

in equations (5) and (6), confidence represents confidence score, sMAPE_adjusted represents the symmetric mean absolute percentage error, y _i The multi-spectral data is represented by a multi-spectral data,

and represents the estimation result.

As can be seen from the foregoing, in the embodiments of the present disclosure, by determining the corresponding confidence score according to the estimation result, the reliability of the current scene light source estimation result can be visually represented, which is conducive to algorithm optimization and improves the light source estimation effect.

The present disclosure has been described above by taking a light source estimation process of a single frame image as an example, and for a video recording scene, in some embodiments, the above method process may be directly performed for each frame image in sequence.

In other embodiments, the second weight obtained from the previous frame of image may be used as the initial weight of the standard light source corresponding to the next frame of image when the image has not changed greatly. For example, the frame rate of video recording is 30FPS, that is, 30 frames of image data are processed in 1 second, in a short time (for example, 1 second), the current scene light source is basically unchanged, so that the second weight obtained by the previous frame of image can be used as the initial weight of the corresponding standard light source for the next frame of image processing, and therefore, the initial weight can be returned, the initial weight can be quickly converged on the basis of the previous result, a better processing result can be obtained, and meanwhile, the processing efficiency is improved.

Based on the light source estimation method, in the embodiment of the disclosure, the estimation result of the current scene light source is determined by using the spectrum data of the standard light source, so that the spectrum type of the current scene light source can be accurately reflected, the spectrum difference of the light source can be accurately identified, the accuracy of color restoration is improved, the obtained image is closer to the color actually perceived by human eyes, and the imaging effect is improved. After the first weight set is obtained, correlated color temperature data are further fused to obtain a second weight of the target standard light source, the problem of light source type confusion in light source estimation is effectively avoided by utilizing the correlated color temperature data, and accuracy of the light source estimation is improved. The reliability of the light source estimation result of the current scene can be visually represented by determining the corresponding confidence score according to the estimation result, so that the optimization of an algorithm is facilitated, and the light source estimation effect is improved.

In a second aspect, embodiments of the present disclosure provide a light source estimating apparatus that is applicable to an electronic device. It will be appreciated that the electronic device of the embodiments of the present disclosure may be any suitable type of device for implementation, such as a smart phone, a tablet, a wearable device, a palm terminal, a vehicle-mounted device, a server, a cloud platform, etc., to which the present disclosure is not limited.

As shown in fig. 10, in some embodiments, a light source estimating apparatus of an example of the present disclosure includes:

an acquisition module 10 configured to acquire multispectral data of a current scene light source;

the weight determining module 20 is configured to determine a first weight set corresponding to the standard light source data set according to the preset standard light source data set and multispectral data; the standard light source data set comprises spectrum data of at least two standard light sources, and the first weight set comprises a first weight corresponding to each standard light source;

the result determining module 30 is configured to determine an estimation result of the current scene light source according to the spectrum data and the first weight of each standard light source in the standard light source data set.

In some implementations, the weight determination module 20 is configured to:

and determining a first weight set according to the first weight value corresponding to each standard light source.

In some embodiments, the standard light source dataset comprises correlated color temperature data for each standard light source; the result determination module 30 is configured to:

determining at least two candidate standard light sources from the standard light source data set according to the first weight value of each of the first weight sets;

In some implementations, the result determination module 30 is configured to:

sequencing all the first weights in the first weight set from big to small to obtain a first sequencing set;

And determining standard light sources corresponding to a preset number of first weights before sequencing as candidate standard light sources according to the relation between the maximum value in the first sequencing set and a preset weight threshold.

In some implementations, the result determination module 30 is configured to:

determining standard light sources corresponding to a first weight of a first quantity before sequencing as candidate standard light sources in response to the maximum value in the first sequencing set not being smaller than a preset weight threshold;

and/or the number of the groups of groups,

determining standard light sources corresponding to a first weight of a second quantity before sequencing as candidate standard light sources in response to the maximum value in the first sequencing set being smaller than a preset weight threshold; wherein the second number is greater than the first number.

In some implementations, the result determination module 30 is configured to:

carrying out regression processing on the first weight of each candidate standard light source according to the spectrum data and correlated color temperature data of each candidate standard light source and the multispectral data to obtain the second weight of each candidate standard light source;

In some implementations, the result determination module 30 is configured to:

acquiring initial spectrum data of at least two standard light sources through a multispectral sensor, and obtaining a spectrum curve of each standard light source according to the initial spectrum data;

for each standard light source, extracting spectral data of a preset wave band close to a response peak point of the multispectral sensor according to a spectral curve, and taking the spectral data as spectral data of the standard light source;

and obtaining a standard light source data set according to the spectrum data of all the standard light sources.

a processor; and

a memory storing computer instructions for causing a processor to perform a method according to any of the embodiments of the first aspect.

A block diagram of an electronic device according to some embodiments of the present disclosure is shown in fig. 11, and the principles related to the electronic device and the storage medium according to some embodiments of the present disclosure are described below with reference to fig. 11.

Referring to fig. 11, the electronic device 1800 may include one or more of the following components: a processing component 1802, a memory 1804, a power component 1806, a multimedia component 1808, an audio component 1810, an input/output (I/O) interface 1812, a sensor component 1816, and a communication component 1818.

The processing component 1802 generally controls overall operation of the electronic device 1800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1802 may include one or more processors 1820 to execute instructions. Further, the processing component 1802 may include one or more modules that facilitate interactions between the processing component 1802 and other components. For example, the processing component 1802 may include a multimedia module to facilitate interaction between the multimedia component 1808 and the processing component 1802. As another example, the processing component 1802 may read executable instructions from a memory to implement electronic device-related functions.

The memory 1804 is configured to store various types of data to support operations at the electronic device 1800. Examples of such data include instructions for any application or method operating on the electronic device 1800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 1804 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The power supply assembly 1806 provides power to the various components of the electronic device 1800. The power components 1806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1800.

The multimedia component 1808 includes a display screen between the electronic device 1800 and the user that provides an output interface. In some embodiments, the multimedia component 1808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 1800 is in an operational mode, such as a shooting mode or a video mode, the front-facing camera and/or the rear-facing camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 1810 is configured to output and/or input audio signals. For example, the audio component 1810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 1800 is in operating modes, such as a call mode, a recording mode, and a speech recognition mode. The received audio signals may be further stored in the memory 1804 or transmitted via the communication component 1818. In some embodiments, audio component 1810 includes a speaker for outputting audio signals.

The I/O interface 1812 provides an interface between the processing component 1802 and a peripheral interface module, which may be a keyboard, click wheel, button, or the like. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 1816 includes one or more sensors for providing status assessment of various aspects of the electronic device 1800. For example, the sensor assembly 1816 may detect the on/off state of the electronic device 1800, the relative positioning of the components, such as the display and keypad of the electronic device 1800, the sensor assembly 1816 may detect the change in position of the electronic device 1800 or a component of the electronic device 1800, the presence or absence of a user's contact with the electronic device 1800, the orientation or acceleration/deceleration of the electronic device 1800, and the change in temperature of the electronic device 1800. The sensor assembly 1816 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1816 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1816 may include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1818 is configured to facilitate communication between the electronic device 1800 and other devices, either wired or wireless. The electronic device 1800 may access a wireless network based on a communication standard, such as Wi-Fi,2G,3G,4G,5G, or 6G, or a combination thereof. In one exemplary embodiment, the communication component 1818 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 1818 includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 1800 can be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements.

In some embodiments, the Chip provided in the present disclosure may be any Chip type suitable for implementation, such as a CPU (central processing unit ) Chip, GPU (graphics processing unit, graphics processor) Chip, soC (System on Chip), etc., and may also be an acceleration Chip dedicated to the artificial intelligence technology, such as an AI (Artificial Intelligence ) accelerator, etc., which is not limited in this disclosure.

It should be apparent that the above embodiments are merely examples for clarity of illustration and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the present disclosure.

Claims

1. A method of estimating a light source, comprising:

acquiring multispectral data of a current scene light source;

2. The method according to claim 1, wherein determining a first set of weights corresponding to the standard light source dataset according to the preset standard light source dataset and the multispectral data comprises:

3. The method according to claim 1 or 2, wherein the standard light source dataset comprises correlated color temperature data for each standard light source; the determining, according to the spectral data and the first weight of each standard light source in the standard light source data set, an estimation result of the current scene light source includes:

4. A method according to claim 3, wherein said determining at least two candidate standard light sources from said standard light source dataset based on said first weights for each of said standard light sources comprises:

5. The method of claim 4, wherein determining, as the candidate standard light sources, standard light sources corresponding to a first weight of a preset number before sorting according to a relation between a maximum value in the first sorted set and a preset weight threshold, comprises:

and/or the number of the groups of groups,

6. A method according to claim 3, wherein said determining an estimation of the current scene illuminant from said spectral data and said correlated color temperature data for each candidate standard illuminant, and said multispectral data, comprises:

7. The method of claim 1, further comprising, after obtaining the estimation of the current scene illuminant:

8. The method of claim 1, wherein the step of pre-setting the standard light source dataset comprises:

9. A light source estimating apparatus, comprising:

10. An electronic device, comprising:

a processor; and

memory storing computer instructions for causing the processor to perform the method according to any one of claims 1 to 8.

11. A storage medium having stored thereon computer instructions for causing a computer to perform the method according to any one of claims 1 to 8.

12. A chip is characterized in that,

comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and to send the signal to the processor, the signal including computer instructions stored in the memory; the computer instructions, when executed by the processor, cause the electronic device to perform the method of any of claims 1 to 8.