CN117061881A - Image processing and image processing model training method and device - Google Patents
Image processing and image processing model training method and device Download PDFInfo
- Publication number
- CN117061881A CN117061881A CN202310872067.3A CN202310872067A CN117061881A CN 117061881 A CN117061881 A CN 117061881A CN 202310872067 A CN202310872067 A CN 202310872067A CN 117061881 A CN117061881 A CN 117061881A
- Authority
- CN
- China
- Prior art keywords
- image
- color
- image processing
- processing
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 420
- 238000012549 training Methods 0.000 title claims abstract description 87
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000003672 processing method Methods 0.000 claims abstract description 74
- 238000010801 machine learning Methods 0.000 claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims description 83
- 230000004044 response Effects 0.000 claims description 69
- 238000001228 spectrum Methods 0.000 claims description 52
- 238000012937 correction Methods 0.000 claims description 46
- 230000009467 reduction Effects 0.000 claims description 44
- 230000003595 spectral effect Effects 0.000 claims description 35
- 238000007781 pre-processing Methods 0.000 claims description 29
- 230000015654 memory Effects 0.000 claims description 28
- 238000013528 artificial neural network Methods 0.000 claims description 21
- 230000006978 adaptation Effects 0.000 claims description 17
- 238000013508 migration Methods 0.000 claims description 14
- 230000005012 migration Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000011946 reduction process Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000003860 storage Methods 0.000 claims description 5
- 238000007499 fusion processing Methods 0.000 claims description 3
- 230000000873 masking effect Effects 0.000 claims 3
- 230000000875 corresponding effect Effects 0.000 description 47
- 230000008569 process Effects 0.000 description 29
- 238000010586 diagram Methods 0.000 description 25
- 230000006870 function Effects 0.000 description 23
- 238000011176 pooling Methods 0.000 description 17
- 206010047289 Ventricular extrasystoles Diseases 0.000 description 16
- 238000005129 volume perturbation calorimetry Methods 0.000 description 16
- 230000000694 effects Effects 0.000 description 13
- 230000004913 activation Effects 0.000 description 11
- 230000003044 adaptive effect Effects 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 9
- 208000036971 interstitial lung disease 2 Diseases 0.000 description 7
- 239000003086 colorant Substances 0.000 description 6
- 125000004122 cyclic group Chemical group 0.000 description 6
- 238000010606 normalization Methods 0.000 description 6
- 238000012805 post-processing Methods 0.000 description 6
- 230000007935 neutral effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 230000003190 augmentative effect Effects 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 239000011248 coating agent Substances 0.000 description 2
- 238000000576 coating method Methods 0.000 description 2
- 230000000052 comparative effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 239000012633 leachable Substances 0.000 description 2
- 229910044991 metal oxide Inorganic materials 0.000 description 2
- 150000004706 metal oxides Chemical class 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000010408 sweeping Methods 0.000 description 2
- 241001428800 Cell fusing agent virus Species 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000003705 background correction Methods 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 238000000701 chemical imaging Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000004737 colorimetric analysis Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/843—Demosaicing, e.g. interpolating colour pixel values
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/88—Camera processing pipelines; Components thereof for processing colour signals for colour balance, e.g. white-balance circuits or colour temperature control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
Abstract
The embodiment of the application discloses an image processing method, which comprises the following steps: inputting a first image into an image processing module, wherein the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by a first image sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning; and performing first color restoration processing on the first image by using the image processing model to obtain a second image, wherein the first color restoration processing comprises the operation of adjusting the color of the first image based on first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image. Therefore, color reproduction accuracy can be improved by performing color reproduction processing on the first image based on the machine learning pre-trained image processing model and the multispectral original image, and an image with better color reproduction degree can be obtained. The embodiment of the application also discloses an image processing device, an image processing model training method and an image processing model training device.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, and an image processing model training method and apparatus.
Background
Electronic devices such as cameras and mobile phones, which are widely used at present, are equipped with color image sensors, for example, RGB (Red, R), green (G), blue (B) image sensors. However, the response of the RGB image sensor to the spectrum is sampling or integrating the complete spectrum signal, and the RGB image sensor cannot completely capture the spectrum signal, which easily causes problems such as Metamerism (Metamerism), so that in some scenes, the image color reproduction performance is limited, the image color cannot be accurately reproduced, and the image quality and the user experience are affected. A Multispectral (Multispectral) image sensor can capture raw spectral signals more completely than an RGB image sensor. Therefore, how to obtain an image with better color reproducibility based on the original spectrum signal captured by the multispectral image sensor, and improve the image quality and the user experience is a current problem to be solved.
Disclosure of Invention
The embodiment of the application provides an image processing method, an image processing model training method and an image processing model training device, which can solve the technical problems, namely, an image with better color reproducibility can be obtained based on an original spectrum signal captured by a multispectral image sensor, and the image quality and the user experience are effectively improved.
To solve the above technical problem, in a first aspect, an embodiment of the present application provides an image processing method, including: inputting a first image into an image processing module, wherein the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by a first sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning; and performing first color restoration processing on the first image by using the image processing model to obtain a second image, wherein the first color restoration processing comprises the operation of adjusting the color of the first image based on first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.
The first image sensor is a multispectral image sensor, and an image corresponding to a RAW spectral signal captured by the multispectral image sensor may be referred to as a multispectral RAW (RAW) image. The first image may include, for example, only a multispectral RAW image, may include a multispectral RAW image and a high-resolution color RAW image, or may include a multispectral RAW image and a high-resolution grayscale RAW image. Wherein the multispectral RAW image is one example of a first sub-image included in the first image, the high-resolution color RAW image is one example of a second sub-image included in the first image, and the high-resolution gray-scale RAW image is another example of the second sub-image included in the first image. That is, the first image may include only the first sub-image, or may include the first sub-image and the second sub-image. Of course, the first image may also include other images, which may be selectively set as desired. In addition, the high resolution color (or grayscale) RAW image may be acquired by a second image sensor, which may be, for example, a normal image sensor, i.e., may be, for example, a normal 3-channel color image sensor or a 1-channel grayscale image sensor. The common 3-channel color image sensor is used for acquiring a high-resolution color RAW image with spatial resolution larger than that of the multispectral RAW image, and the 1-channel gray scale image sensor is used for acquiring a high-resolution gray scale RAW image with spatial resolution larger than that of the multispectral RAW image.
If the first image only includes a multispectral RAW image, the second image is a color image (i.e., a color image with lower resolution, simply referred to as a color image), and if the first image includes a multispectral RAW image and a high-resolution color (or grayscale) RAW image, the second image is a high-resolution color image.
In one possible implementation of the first aspect, the first color reproduction process may include, for example, a color subspace projection process, a white balance process, a subspace color correction process, and so on. Of course, other processes may be included, which may be set as desired.
In a possible implementation of the first aspect, the first image adjustment information includes, for example, a color subspace projection matrix, a light source power spectrum raw domain response, a subspace color correction matrix, an image for color migration, and the like, and of course, other information may be included, which may be set as needed.
The image processing model is a parameter-learning image processing model which is trained in advance based on machine learning or deep learning, so in the embodiment, color restoration processing is performed on the first image through the image processing model, color restoration accuracy can be improved, a color image with better color restoration degree is obtained, image quality is effectively improved, and user experience is further improved.
And, through the image processing model, the color reduction processing is performed on the first image, which may be to obtain corresponding image adjustment information based on the multispectral RAW image, perform the color reduction processing on the multispectral RAW image according to the image adjustment information corresponding to the multispectral RAW image, or perform the color reduction processing on the high-resolution color (or gray scale) RAW image, and also effectively improve the color reduction accuracy, so as to obtain a color image with better color reduction degree, effectively improve the image quality, and further improve the user experience.
In a possible implementation manner of the first aspect, performing a first color reduction process on the first image using the image processing model to obtain a second image includes: processing with an image processing model based on the first sub-image to obtain first image adjustment information; and carrying out first color restoration processing on the first sub-image according to the first image adjustment information to obtain a second image.
In this embodiment, color reproduction processing is performed on the multispectral RAW image by using image adjustment information corresponding to the multispectral RAW image obtained by the image processing model, so that color reproduction accuracy can be improved, a color image with better color reproduction degree can be obtained, image quality can be effectively improved, and user experience can be further improved.
In a possible implementation of the first aspect, processing using an image processing model to obtain the first image adjustment information based on the first sub-image includes: processing the first sub-image by using an image processing model to acquire spectral features of the first sub-image; based on the spectral features, first image adjustment information is obtained using an image processing model.
Therefore, the image adjustment information can be conveniently and accurately obtained based on the spectral characteristics of the multispectral RAW image so as to carry out color restoration processing on the multispectral RAW image, the color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
In a possible implementation manner of the first aspect, the first image adjustment information includes a color subspace projection matrix, a light source power spectrum original domain response, and a subspace color correction matrix, and the performing, according to the first image adjustment information, a first color restoration process on the first sub-image to obtain a second image includes: performing color subspace projection processing on the first sub-image according to the color subspace projection matrix to obtain a subspace response image; obtaining a white balance diagonal matrix according to the color subspace projection matrix and the original domain response of the light source power spectrum, and performing white balance processing on the subspace response image according to the white balance diagonal matrix to obtain a white balance image; and performing color correction processing on the white balance image according to the subspace color correction matrix to obtain a color-adaptive image as a second image, or performing color gamut conversion processing on the color-adaptive image to obtain the second image.
In this embodiment, the image processing model may be a model capable of realizing the functions of light source estimation and color adaptation, that is, may be a model of light source estimation and color adaptation. In addition, in this embodiment, the first sub-image may be the aforementioned multispectral RAW image, and the second image is a color image. Additionally, the illuminant may be referred to as an ambient illuminant, and thus, the illuminant estimation and color adaptation model may also be referred to as an ambient illuminant estimation and color adaptation model.
In a possible implementation manner of the first aspect, the first image further includes a second sub-image, where the second sub-image is a color image or a gray scale image acquired by a second image sensor, a spatial resolution of the second sub-image is greater than a spatial resolution of the first sub-image, and the performing, with an image processing model, a first color reduction process on the first image to obtain the second image includes: processing with an image processing model based on the first sub-image to obtain first image adjustment information; and carrying out first color restoration processing on the second sub-image according to the first image adjustment information to obtain a second image. The second image sensor may be a color image sensor or a gray image sensor, where the color sensor captures a color image and the gray sensor captures a gray image.
In this embodiment, the first sub-image may be the multispectral RAW image, the second sub-image may be the high-resolution color RAW image, and the second image may be the high-resolution color image. Therefore, a high-resolution color image is generated based on the multispectral RAW image and the high-resolution color RAW image, namely, based on image adjustment information obtained by the multispectral RAW image, color restoration processing is carried out on the high-resolution color RAW image, color restoration accuracy can be improved, a color image with better color restoration degree is obtained, image quality is effectively improved, and user experience is further improved.
In a possible implementation manner of the first aspect, the first adjustment information includes a white balance adjustment parameter and a subspace color correction matrix, the white balance adjustment parameter includes a white balance diagonal matrix, the second sub-image is a color image, and the performing, according to the first image adjustment information, a first color restoration process on the second sub-image to obtain the second image includes: performing white balance processing on the second sub-image according to the white balance adjustment parameters to obtain a white balance image; and performing color correction processing on the white balance image according to the subspace color correction matrix to obtain a color adaptation image as a second image.
In a possible implementation of the first aspect, the white balance diagonal matrix is obtained from a color subspace projection matrix and an original domain response of the light source power spectrum.
In a possible implementation manner of the first aspect, the first image adjustment information includes a third image, and the performing, according to the first image adjustment information, a first color reduction process on the second sub-image to obtain a second image includes: and performing color migration processing on the second sub-image according to the third image to obtain a second image.
Therefore, the color restoration effect can be better improved through color migration processing, a color image with better color restoration degree is obtained, the image quality is effectively improved, and the user experience is further improved. In addition, the second sub-image may be a color image or a gray-scale image having a spatial resolution greater than that of the first sub-image.
In addition, in this embodiment, the image processing model may include at least one neural network, for example, after the first image is input, the at least one neural network performs color reduction and other processes on the first image to obtain a third image, and performs end-to-end color reduction processing, for example, color migration processing, on the multispectral RAW image through the third image, so that color reduction accuracy can be improved, a color image with better color reduction degree can be obtained, image quality is effectively improved, and user experience is further improved.
In a possible implementation manner of the first aspect, the image processing model includes at least one neural network, and the performing, with the image processing model, a first color reduction process on the first image to obtain a second image includes: and performing first color restoration processing on the first sub-image by using at least one neural network to obtain a second image.
In this embodiment, through the image processing model including at least one neural network, the end-to-end color reproduction processing can be conveniently performed on the multispectral RAW image, so that the color reproduction accuracy can be improved, a color image with better color reproduction degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
In a possible implementation of the first aspect, inputting the first image to the image processing module includes: preprocessing the first image to adjust the image quality of the first image and obtain a preprocessed first image; and inputting the preprocessed first image to an image processing module.
The preprocessing can be, for example, image demosaicing and the like, and through the preprocessing, noise in the multispectral RAW image can be removed, and/or useful information in the image can be enhanced, so that the image quality is improved, further, a color image with better color reproducibility can be obtained, the image quality is effectively improved, and further, the user experience is improved.
In a possible implementation of the first aspect, the method further includes: and performing stylization processing on the second image to obtain a fourth image with the image style being a preset image style.
Therefore, the corresponding stylized color image can be obtained, the image quality is further effectively improved, different requirements of users can be met, and the user experience is improved.
In a possible implementation of the first aspect, the method further includes: obtaining second image adjustment information according to the first sub-image; and carrying out second color reduction processing on the first target image according to the second image adjustment information to obtain a fifth image, wherein the first target image comprises a second image or a fourth image.
Therefore, the color image obtained can be further subjected to color reproduction processing, so that a color image with better color reproduction degree is obtained, the image quality is effectively improved, and the user experience is further improved.
In a possible implementation manner of the first aspect, the second image adjustment information includes image overexposure region covering information and a sixth image, the image overexposure region covering information is image overexposure region covering information corresponding to a sub-channel response overexposure region in the first sub-image, the sixth image is an image obtained by performing third color reduction processing according to a non-overexposed sub-channel response in the first sub-image, and performing second adjustment processing on the second target image according to the second image adjustment information, to obtain a fifth image, where the method includes: and carrying out image fusion processing according to the first target image, the image overexposure region covering information and the sixth image to obtain a fifth image.
In this embodiment, by processing the overexposed region, a color image with better color reproducibility can be obtained, so that the image quality is effectively improved, and further the user experience is improved.
In a possible implementation manner of the first aspect, the second image adjustment information includes image brightness information, the image brightness information is determined according to a response of a target sub-channel in the first sub-image, and the second adjustment processing is performed on the first target image according to the second image adjustment information to obtain a fifth image, where the second image adjustment processing includes: and adjusting the brightness of the second image according to the image brightness information to obtain a fifth image.
In the embodiment, the color image with better color reproducibility can be obtained by processing the brightness of the image, so that the image quality is effectively improved, and the user experience is further improved.
In a second aspect, an embodiment of the present application provides an image processing model training method, including: inputting a training sample image into an initial image processing model, wherein the training sample image comprises a multispectral original image acquired by a first image sensor; the initial image processing model obtains an output image according to the training sample image, wherein the output image is a color image; training an initial image processing model according to the output image to obtain an image processing model, wherein the image processing model is applied to the image processing method.
Therefore, based on an image processing model obtained by model training, the multispectral RAW image is subjected to color restoration processing, so that the color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, the image quality is effectively improved, and the user experience is further improved.
In a third aspect, an embodiment of the present application provides an image processing apparatus including: the first input module is used for inputting a first image into the image processing module, the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by the first image sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning; the image processing module is used for carrying out first color restoration processing on the first image by utilizing the image processing model to obtain a second image, the first color restoration processing comprises an operation of adjusting the color of the first image based on first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.
In a fourth aspect, an embodiment of the present application provides an image processing model training apparatus, including: the second input module is used for inputting a training sample image into the initial image processing model, the initial image processing model module comprises an initial image processing model, and the training sample image comprises a multispectral original image acquired by the first image sensor; the initial image processing model module is used for obtaining an output image according to the training sample image, wherein the output image is a color image; the training module is used for training the initial image processing model according to the output image to obtain an image processing model, and the image processing model is applied to the image processing method.
In a fifth aspect, embodiments of the present application provide an electronic device, including: a memory for storing a computer program, the computer program comprising program instructions; and a processor for executing program instructions to cause the electronic device to perform the aforementioned image processing method or to cause the electronic device to perform the aforementioned image processing model training method.
In a sixth aspect, embodiments of the present application provide a cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the aforementioned image processing method or to cause the cluster of computing devices to perform the aforementioned image processing model training method.
In a seventh aspect, embodiments of the present application provide a computer program product comprising instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the aforementioned image processing method, or cause the computing device cluster to perform the aforementioned image processing model training method.
In an eighth aspect, embodiments of the present application provide a computer readable storage medium comprising computer program instructions which, when executed by a computing device cluster, perform the aforementioned image processing method, or the computing device cluster performs the aforementioned image processing model training method.
The relevant advantageous effects of the third aspect to the eighth aspect may be referred to the relevant description of the first aspect or the second aspect, and are not described herein.
Drawings
In order to more clearly illustrate the technical solution of the present application, the following description will briefly explain the drawings used in the description of the embodiments.
FIG. 1 shows a schematic diagram of the corresponding CFA arrangement of RGB image sensors and the response of the RGB image sensors to the spectrum;
FIG. 2 shows a schematic representation of human eye cones and the response of human eye cones to spectra;
FIG. 3 shows a schematic diagram of the corresponding CFA arrangement of the multispectral image sensor and the response of the multispectral image sensor to the spectrum;
FIG. 4 is a flow diagram illustrating an image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 5 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 6 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 7 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 8 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 9 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 10 is a flow diagram illustrating the structure of a global illuminant estimation and color adaptation model and the corresponding image tuning parameter generation, according to some embodiments of the present application;
FIG. 11 is a flow diagram illustrating a structure of a local illuminant estimation and color adaptation model and corresponding image adjustment parameter generation, according to some embodiments of the present application;
FIG. 12 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 13 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 14 is a comparative schematic diagram illustrating an image processing effect, according to some embodiments of the application;
FIG. 15 is a schematic diagram illustrating a number of different images corresponding to global and local estimation processes, according to some embodiments of the present application;
FIG. 16 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 17 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 18 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 19 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 20 is a diagram illustrating a comparison of image effects of disabling and enabling response cutoff compensation, according to some embodiments of the application;
FIG. 21 is a flow diagram illustrating another image processing system and corresponding image processing method, according to some embodiments of the application;
FIG. 22 is a diagram showing a comparison of image effects to disable and enable image brightness transitions, according to some embodiments of the application;
FIG. 23 is a schematic diagram illustrating a configuration of an image processing apparatus according to some embodiments of the present application;
FIG. 24 is a schematic diagram illustrating one configuration of an image processing model training apparatus, according to some embodiments of the application;
25A and 25B are diagrams illustrating some configurations of computing devices, according to some embodiments of the application;
26A and 26B are schematic diagrams illustrating some configurations of clusters of computing devices, according to some embodiments of the application.
Detailed Description
The technical scheme provided by the embodiment of the application is further described below with reference to the accompanying drawings.
As described above, electronic devices having photographing or image processing capabilities, such as cameras and mobile phones, which are widely used at present, are each equipped with a color image sensor. Common color imaging systems including color image sensors typically use a color filter array (Color Filter Array, CFA) arranged over a photosensitive area. CFA is also known as Bayer Filter (Bayer Filter), and the photosensitive region may be implemented, for example, based on complementary metal oxide semiconductor (Complementary Metal-Oxide Semiconductor, CMOS) technology or the like.
Fig. 1 (a) shows a common arrangement of CFAs, which is composed of 2×2 four pixel points, and a cyclic unit, that is, a cyclic unit can be understood as a 2×2 pixel arrangement. Each cyclic unit contains three primary colors: r, G, B. Accordingly, color image sensors are also commonly referred to as RGB image sensors. In addition, the response of the RGB image sensor to the spectrum is shown in fig. 1 (b). The response of an image sensor to a spectrum can be represented by a correlated (QE) curve, which describes the photoelectric conversion efficiency of the image sensor. Since the absorption efficiency of a semiconductor material for an optical signal is wavelength dependent, the abscissa of the QE curve is wavelength, for example in nanometers (nm), and the ordinate is the percentage corresponding to the photoelectric conversion efficiency, for example, where 0.2 corresponds to 20%.
The human eye typically has 3 types of cone cells (S, M, L, i.e., L/M/S) within it that are each responsive to light of a different range of wavelengths. The RGB image sensor is intended to directly mimic the response of 3 cones of the human eye (which may be abbreviated as LMS cones) to the spectrum. The structure of human eye cones is shown in fig. 2 (a). The response of 3 cone cells of the human eye to the spectrum is shown in fig. 2 (b). Wherein, the S cone cells respond to light with the wavelength of 400-500nm, and the peak appears at 420-440 nm; m cone cells, for example, will respond to light having wavelengths in the range of 450-630nm, with peaks at 534-555 nm; l cone cells, for example, respond to light having wavelengths in the range of 500-700nm, with peaks at 564-580 nm. And, the responses of 3 cone cells to the spectrum correspond to three primary colors: r, G, B.
As can be seen from fig. 1 (b) and fig. 2 (b), the response QE curve of the RGB image sensor to the spectrum is deviated from the response QE curve of the human eye cone cell to the spectrum. Therefore, the RAW (RAW) response of the RGB image sensor generally needs to undergo color correction processing to restore the color of the image, so that the color condition of the image seen by the human eye in the shooting scene can be reproduced more accurately. The RAW response of the RGB image sensor, referred to as the camera sensor RAW imaging signal (Camera Raw Imaging Signal), may simply be referred to as a RAW image response or RAW image signal. In addition, due to Color Constancy (Color constant) characteristics of human eyes, it is also generally necessary to correct RAW image response by white balance or the like. For example, paper "Hakki Can Karaimer, et al A Software Platform for Manipulating the Camera Imaging Pipeline" describes a typical image color processing operational flow for RGB image sensors. Color constancy refers to the perceptual property that a person's perception of the color of an object surface remains unchanged when the color light illuminating the object surface changes. In the field of image processing science, based on the cognitive characteristics of human eyes on scenes, the separation of background elements and illumination elements in an image is a key point for solving the problem.
In addition, since the RGB image sensor responds to the spectrum by sampling or integrating the complete spectrum signal (may also be referred to as spectrum information), the spectrum signal cannot be completely captured, and problems such as metamerism are easily caused, so that the RGB camera including the RGB image sensor has limited image color restoration performance in some difficult-to-see scenes such as containing large-area pure-color areas, and cannot accurately restore the image color, thereby affecting the image quality and the user experience.
Multispectral image sensors based on multispectral technology aim to capture raw spectral signals more completely than RGB image sensors. Multispectral refers to a spectrum detection technology capable of acquiring a plurality of optical spectrum bands (usually more than 3) simultaneously and expanding towards infrared light and ultraviolet light on the basis of visible light.
As shown in fig. 3 (a), in one implementation, the multispectral image sensor includes a CFA, and a cyclic unit is formed by 4×4 sixteen pixels above the photosensitive area, where each pixel corresponds to a color filter or a color filter coating. With this design, the resolution of the spectrum by the multispectral image sensor is increased from 3 dimensions to 16 dimensions. The response of the multispectral image sensor to the spectrum, as shown in fig. 3 (b), includes more response curves. Therefore, the multispectral image sensor can capture the spectrum signals more completely, so that the image color reproduction performance in difficult scenes is improved. In addition, the cyclic unit included in the CFA included in the multispectral image sensor can be understood as an n×n pixel arrangement, N being 3 or more.
Based on the above, the embodiment of the application provides an image processing method, which can be applied to electronic equipment such as cameras, mobile phones and the like. As shown in fig. 4, in one embodiment of the application, an electronic device may include an image acquisition system and an image processing system. The image acquisition system may comprise a multispectral image sensor (i.e. a first sensor) which may acquire an image resulting in a multispectral RAW image (i.e. a first sub-image), RAW representing the RAW image signal read out from the image sensor, so that the multispectral RAW image represents the RAW image signal read out from the multispectral image sensor. And, the multispectral image sensor may send the acquired multispectral RAW image to an image processing system, which obtains a corresponding color image (as an example of the second image) from the multispectral RAW image.
In addition, in one embodiment of the present application, the image acquisition system may further include a general image sensor (i.e., a second image sensor), such as a general 3-channel color image sensor or a 1-channel grayscale image sensor acquisition. The common 3-channel color image sensor is used for acquiring a high-resolution color RAW image with spatial resolution larger than that of the multispectral RAW image, and the 1-channel gray scale image sensor is used for acquiring a high-resolution gray scale RAW image with spatial resolution larger than that of the multispectral RAW image. The high resolution color (or grayscale) RAW image represents the original image signal read out from the normal image sensor. And, the general image sensor may transmit the acquired high-resolution color (or grayscale) RAW image to an image processing system, which obtains a corresponding high-resolution color image from the multispectral RAW image and the high-resolution color (or grayscale) RAW image.
Further, as shown in fig. 4, in the embodiment of the present application, the image processing system includes a parameter-learnable image processing module (as an example of the image processing module), and the parameter-learnable image processing module includes a parameter-learnable image processing model. The parameter-learnable image processing model is a pre-trained image processing model based on machine learning or deep learning. That is, the parameter-learnable image processing model is an image processing model obtained based on model training. The corresponding color image can be obtained by performing color reproduction processing (which may also be referred to as color adjustment processing) on the multispectral RAW image transmitted from the multispectral image sensor and the multispectral RAW image by the image processing model with the parameter learning. Or, the high-resolution color (or gray) RAW image sent by the common image sensor is subjected to color reduction processing through the image processing model with the parameter capable of learning and the multispectral RAW image sent by the multispectral image sensor, so that a corresponding high-resolution color image can be obtained.
That is, the embodiment of the application provides a multi-spectrum image color restoration method based on learning, or a new path based on multi-spectrum image color restoration based on learning, and the electronic device can perform color restoration processing on the multi-spectrum RAW image or the high-resolution color (or gray scale) RAW image based on the image processing model with the parameter capable of being learned, so that the color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
Further, the parameter-learnable image processing model may be constituted by at least one of a neural network, a differentiable image processing operator (for example, it may be understood that the image processing is based on a color space projection matrix, a color correction matrix, or the like), and a non-differentiable image processing operator (for example, it may be understood that the image processing is based on a color space projection matrix, a color correction matrix, or the like that invokes other modules to process). Wherein the neural network and/or the differentiable image processing operator comprises a learnable parameter.
Further, as shown in fig. 5, in one embodiment of the present application, the image acquisition system includes a multispectral image sensor for acquiring a multispectral RAW image 101 (i.e. a multispectral RAW image with a lower resolution), the multispectral RAW image 101 is input to a parameter-learning image processing module 301, and the parameter-learning image processing module 301 can obtain a color image 401 according to the multispectral RAW image 101.
The resolution of an image is typically represented by Spatial/spectral resolution (Spatial/Spectral Resolution), for example, using H 'x W' x C 'to represent an image, where H' x W 'represents the Spatial resolution of the image and C' represents the spectral resolution of the image. For example, a common 3-channel color image has a spectral resolution C 'of 3, whereas for multispectral images, it is common for the spectral resolution C' to be >3.
In this embodiment, the resolution of the multispectral RAW image 101 may be, for example, h×w×n, where h×w represents the width and height of the image (i.e., spatial resolution), and N represents the number of channels of the image (i.e., spectral resolution), where N is typically greater than 3. The resolution of the color image 401 may be, for example, h×w×3.
Therefore, the electronic device performs color reproduction processing on the multispectral RAW image 101 based on the image processing model with the parameter capable of being learned, so that color reproduction accuracy can be improved, a color image 401 with better color reproduction degree is obtained, image quality is effectively improved, and user experience is further improved.
As shown in fig. 6, in one embodiment of the present application, a multispectral image sensor included in the image acquisition system acquires a multispectral RAW image 101 (i.e., a multispectral RAW image with a lower resolution), a common image sensor acquires a high-resolution color (or grayscale) RAW image 102, and the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102 are input to an image processing module 301 with a parameter learning function, so that a high-resolution color image 402 can be acquired.
In the present embodiment, the resolution of the high-resolution color (or grayscale) RAW image 102 may be, for example, h×w×c, where h×w represents the width and height (i.e., spatial resolution) of the image, respectively, and H > H, W > W. For the high resolution gray scale RAW image 102, the resolution thereof may be, for example, h×w×1, i.e., c=1. Similarly, for the high resolution color RAW image 102, the resolution thereof may be, for example, h×w×3, i.e., c=3. The resolution of the high-resolution color image 402 may be, for example, h×w×3.
In this way, the electronic device performs color reproduction processing on the high-resolution color (or grayscale) RAW image 102 based on the parameter-learnable image processing model and the multispectral RAW image 101, and can obtain a high-resolution color image 402. Therefore, color reproduction accuracy can be improved, a high-resolution color image 402 with better color reproduction degree can be obtained, image quality is effectively improved, and user experience is further improved.
Further, as shown in fig. 7, in some embodiments of the present application, the image processing system may further include at least one of a preprocessing module and a stylized post-processing module. Wherein:
the preprocessing module comprises, for example, a preprocessing module 201 and/or a preprocessing module 202, the preprocessing module 201 being configured to preprocess the multispectral RAW image 101, and the preprocessing module 202 being configured to preprocess the high resolution color (or grayscale) RAW image 102. In addition, the preprocessing includes at least one image processing operation of image demosaicing, image denoising, image super-resolution, sensor black level correction, lens shading correction, lens distortion correction, sensor spectral response correction, sensor response normalization and the like. Of course, the preprocessing may also include other image processing operations, which may be selected and set as desired. The multispectral RAW image is preprocessed through the preprocessing module, so that the image quality of the multispectral RAW image can be effectively improved, and the quality of the obtained color image can be further improved.
The post-stylization processing module is configured to perform a stylization process on the color image obtained by the parameter-learning image processing module 301, to obtain a final color image 401 or a high-resolution color image 402, for example. The stylizing process may be, for example, a process of adjusting the saturation, contrast, or the like of an image, or may be another process, and an image processing operation included in the stylizing process and a manner of performing the stylizing process may be selected and set as necessary. The stylized post-processing module is used for performing stylized processing on the color image, so that the image which meets the requirements of users can be obtained, the image quality is improved, and the user experience is effectively improved.
In summary, in some embodiments of the present application, it may be understood that the electronic device mainly includes the following 5 parts:
a module 101 for inputting a multispectral RAW image 101.
A module 102 for inputting a high resolution color (or grayscale) RAW image 102.
The modules 201 and 202 correspond to the preprocessing module 201 and the preprocessing module 202, respectively, and are used for preprocessing the image.
The module 301, corresponding to the parameter-learnable image processing module 301, comprises the aforementioned parameter-learnable image processing model. If only the multispectral RAW image 101 is input or only the image processed by the preprocessing module 201 corresponding to the multispectral RAW image 101 is input, the image processing module 301 with the parameter learning function outputs the color image 401. If the multispectral RAW image 101 (or the image processed by the preprocessing module 201 corresponding to the multispectral RAW image 101) and the high-resolution color (or gray scale) RAW image 102 (or the image processed by the preprocessing module 202 corresponding to the high-resolution color (or gray scale) RAW image 102) are input at the same time, the parameter-learning image processing model module 301 outputs the high-resolution color image 402.
Modules 401, 402 correspond to color image 401 output and high resolution color image 402 output, respectively.
Therefore, the electronic equipment can carry out color restoration processing on the multispectral RAW image based on the image processing model with the parameter capable of being learned, the color restoration accuracy can be improved, the color image with better color restoration degree is obtained, the image quality is effectively improved, and the user experience is further improved.
In other embodiments of the present application, the multispectral RAW image 101 and the parameter-learnable image processing module 301 described above are necessary, and other, e.g., high-resolution color (or grayscale) RAW image 102, preprocessing module, stylized post-processing module, high-resolution color image 402, etc., are optional, which may be selected and set as desired.
The structure of the image processing system and the procedure of performing image processing related to obtaining the color image 401 from the multispectral RAW image 101 will be further described.
As shown in fig. 8, in one embodiment of the present application, an image processing system includes a preprocessing module 201 and a parameter-learnable image processing module 301, the parameter-learnable image processing module 301 including a parameter-learnable image processing model. The parameter-learnable image processing model comprises at least one neural network, i.e. the parameter-learnable image processing model may be implemented by one neural network or a set of multiple neural networks. The parameter-learnable image processing model may comprise one or more of the general neural network components, such as convolutional layer, pooling layer, activation layer, upsampling layer, downsampling layer, self-attention layer, residual connection, dense connection, etc., which may be selected and set as desired. In addition, the image processing model with the parameter capable of being learned can be obtained through model training, and the structure and the training mode of the image processing model can be set according to requirements.
In this embodiment, the multispectral RAW image 101 is input to the preprocessing module 201, and the preprocessing module 201 performs the aforementioned preprocessing such as image demosaicing on the multispectral RAW image 101 to obtain a preprocessed multispectral RAW image 101'. Then, the preprocessing module 201 inputs the preprocessed multispectral RAW image 101' into the parameter-learnable image processing module 301, i.e., into a parameter-learnable image processing model (which may also be referred to as a neural network or a set of neural networks). The parameter-learnable image processing model performs image color restoration processing (as an example of the first color restoration processing) on the basis of the pre-processed multispectral RAW image 101', resulting in a color image 401. The image color reproduction process may be, for example, to adjust the color of the image by means of filtering, matrix aggregation, or the like.
Of course, in other embodiments of the present application, the electronic device may include only the parameter-learning image processing module 301 in fig. 8, and not include the preprocessing module 201. The multispectral RAW image 101 is directly input into the parameter-learnable image processing module 301, and a corresponding color image 401 can be obtained.
The embodiment is an end-to-end image color restoration scheme based on the multispectral RAW image 101, and based on the embodiment, the electronic device can perform color restoration processing on the multispectral RAW image 101 based on the image processing model with the parameter capable of being learned and the multispectral RAW image 101, so that color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, image quality is effectively improved, and user experience is further improved.
As shown in fig. 9, in one embodiment of the present application, the image processing system includes a parameter-learnable image processing module 301 and a stylized post-processing module, wherein the parameter-learnable image processing module 301 includes a light source estimation and color adaptation model (as an example of a parameter-learnable image processing model) module, a matrix module, and a color Gamut (Gamut) transformation module, and the matrix module may include a single matrix or may include a matrix set of a plurality of matrices.
The light source estimation and color adaptive model module includes a light source estimation and color adaptive model for extracting spectral features from the multispectral RAW image 101 (as an example of the first image) to obtain image adjustment parameters (as an example of the first image adjustment information), and inputting the image adjustment parameters to the matrix module. In this way, estimation and adjustment of image adjustment parameters can be achieved. The image adjustment parameter may be, for example, at least one of a color subspace projection matrix, a RAW response of the light source power spectrum (i.e., light source power spectrum RAW response), and a subspace color correction matrix (Color Calibration Matrix, CCM). Of course, the image adjustment parameter may be other parameters, which may be selected and set as needed. The extraction method of the spectral features of the light source estimation corresponding to the color adaptive model may be performed by a model with a learnable parameter such as a neural network, or may be performed by a non-parametric feature extraction method such as histogram statistics.
The matrix module is configured to perform color reproduction processing (as an example of the first color reproduction processing) on the multispectral RAW image 101 according to the image adjustment parameters, obtain a color-adapted image, and input the color-adapted image to the Gamut transform module.
The Gamut transform module is configured to perform a color Gamut transform process on the color-adapted image to obtain a color image 401 '(as an example of a second image), and then input the color image 401' to the post-stylization processing module for further stylization processing to obtain the color image 401 (as an example of a fourth image).
Further, in one embodiment of the present application, the image adjustment parameters include a color subspace projection matrix, a RAW response of the light source power spectrum, and a subspace color correction matrix. For example, the color subspace projection matrix is T N×3 RAW response of light source power spectrum is L, subspace color correction matrix T M×3 . Wherein N and M can be based on multispectralThe channel dimension of the RAW image is determined, that is, N is the number of channels of the multispectral image, and represents the spectral resolution, where M is typically a multiple of 3, which can be determined according to practical situations. The light source estimation and color self-adaptive model can be realized by a neural network, the input of the light source estimation and color self-adaptive model is a multispectral RAW image, the spectral characteristics are obtained through spectral characteristic extraction, the spectral characteristics are output to a plurality of branches, and the multipath branches are provided with different characteristic processing flows, such as at least one of characteristic dimension reduction processing, global average pooling processing, activation processing, scaling and normalization processing. Of course, the feature process flow may also include other processes, which may be selected and set as desired.
Illustratively, in one mode of the present application, the illuminant estimation and color adaptation model may be a global illuminant estimation and color adaptation model, and the global illuminant estimation and color adaptation model is configured as shown in fig. 10, and includes a trunk model module and three branch modules. Wherein the trunk model module comprises a trunk model, which can be any neural network based on convolutional neural network (Convolutional Neural Networks, CNN) or transducer, and which is applied to multispectral RAW image 101, such as X ε R h×w×N Spectral feature extraction is performed to obtain spectral features, such as F.epsilon.R h ′×w′×N′ . The spectral features are then input to the 3 tributary modules, tributary 1 module (as an example of a first tributary module), tributary 2 module (as an example of a second tributary module), and tributary 3 module (as an example of a third tributary module), respectively. Each branch module comprises a characteristic dimension reduction module, a global average pooling module and an Exp activation module respectively, wherein the characteristic dimension reduction modules of each branch module can be the same or different, and the global average pooling module and the Exp activation module of each branch module can be the same or different. The feature dimension reduction module performs dimension reduction processing on the spectrum features, and then inputs the spectrum features to the global average pooling and Exp activation module to perform global pooling processing and activation processing.
Further, in this embodiment, global average pooling and Exp excitation in the tributary 1 moduleThe output of the active module is the RAW response of the light source power spectrum, such as L E R N 。
In this embodiment, the tributary 2 module may further include a scaling module, and the output of the global average pooling and Exp activation module in the tributary 2 module is input to the scaling module to perform scaling processing to obtain a color subspace projection matrix, for example, T 1 ∈R N×3 。
In this embodiment, the tributary 3 module may further include a scaling and normalization module, and the output of the global average pooling and Exp activation module in the tributary 3 module is input to the scaling and normalization module for scaling and normalization to obtain a subspace color correction matrix, for example, T 2 ∈R 3×3 Or T 2 ∈R 6×3 。
In this embodiment, each of the tributary modules may not include the feature dimension reduction module, that is, the feature dimension reduction module may be optional. In addition, each of the branching modules may also include other modules, which may be selected and arranged as desired.
In one embodiment of the present application, the number of channels corresponding to the spectral feature may be greater than the number of channels of the multispectral RAW image 101, for example, the number of channels of the multispectral RAW image 101 may be 9, 16, 25, etc., and the number of channels corresponding to the spectral feature may be 32, 64, 128, etc. Of course, in other embodiments of the present application, the number of channels corresponding to the spectral features and the number of channels of the multispectral RAW image 101 may be other values, which may be set as required.
For example, in one embodiment of the present application, the illuminant estimation and color adaptive model may be a local illuminant estimation and color adaptive model, and the structure of the local illuminant estimation and color adaptive model is shown in fig. 11, and compared to the global illuminant estimation and color adaptive model shown in fig. 10, the local illuminant estimation and color adaptive model includes an Exp activation module, that is, only Exp activation processing is performed on the feature after the dimension reduction, and no global pooling processing is performed. In addition, each tributary module also includes a reverse pooling (or replication) module.
In the present embodimentThe output of the Exp activation module in the branch 1 module is the RAW response of the light source power spectrum, for example, L E R h′×w′×N . Then, RAW response L epsilon R of the light source power spectrum h′×w′×N Inputting to a reverse pooling (or copying) module for reverse pooling to obtain RAW response L E R of final light source power spectrum h×w×N 。
In this embodiment, the color subspace projection matrix obtained by the branch 2 module scaling module is, for example, T 1 ∈R h ′×w′×N×3 Then, the color subspace projection matrix T is used for 1 ∈R h′×w′×N×3 Inputting to a reverse pooling (or copying) module, and performing reverse pooling processing to obtain final color subspace projection matrix T 1 ∈R h×w×N×3 。
In this embodiment, the sub-3 module scaling and normalization module obtains a subspace color correction matrix, e.g., T 2 ∈R h′×w′×M×3 Then, the subspace color correction matrix T is used 2 ∈R h′×w′×M×3 Inputting to a reverse pooling (or copying) module, and performing reverse pooling processing to obtain final subspace color correction matrix T 2 ∈R h×w×M×3 。
In this embodiment, each of the tributary modules may not include the feature dimension reduction module, that is, the feature dimension reduction module may be optional. In addition, each of the branching modules may also include other modules, which may be selected and arranged as desired.
According to the realization mode, through the source estimation and color self-adaptive model structure of the main network and the multi-path branch structure, as the multi-path branches have different characteristic processing flows, a plurality of different image adjustment parameters can be obtained, so that color restoration processing can be conveniently and accurately carried out on an image, color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, image quality is effectively improved, and user experience is further improved.
Further, as shown in fig. 12 and 13, corresponding to fig. 9, in one embodiment of the present application, the matrix module adjusts the parameter pair multispectral RAW graph according to the image The image 101 is subjected to color reproduction processing to obtain a color-adapted image, which comprises the following steps: according to the color subspace projection matrix T N×3 I.e. a three-dimensional color subspace projection matrix, the h×w×n-dimensional multispectral RAW image 101 is projected to the h×w×3-dimensional color subspace, so as to adapt to the data dimension required by the three-primary-color (R/G/B) display device (i.e. electronic device) or realize dimension reduction processing, and obtain a subspace response image. Then, according to the color subspace projection matrix T N×3 And the RAW response of the light source power spectrum is used for obtaining a White Balance diagonal matrix, and White Balance (WB) processing or automatic White Balance (Auto White Balance, AWB) processing is carried out on the subspace response image according to the White Balance diagonal matrix, so that a White Balance image is obtained. That is, the projected light source RAW domain response is aligned to the neutral color (r=g=b) response of the display device, the colors of the different light sources are decoupled, and color uniformity is achieved. Then, in the projected 3-channel color subspace, the matrix T is corrected according to the normalized subspace color M×3 That is, the three-dimensional color subspace correction matrix performs color correction (Color Calibration) on the white balance image to realize the color correction related to the light source, thereby obtaining a color-adapted image.
The basic concept of white balance is to restore a white object to white regardless of any light source, and to compensate for the color shift phenomenon occurring when photographing under a specific light source by enhancing the corresponding complementary color.
The purpose of color correction is to ensure that the colors of the image are reproduced more accurately as seen by the human eye at the scene of the shot. Color correction techniques are critical to image color reproduction.
Still further, as shown in fig. 13, in one embodiment of the present application, the multispectral RAW image 101 may be input to the parameter-learnable image processing module 301 after passing through the preprocessing module 201.
And, as shown in fig. 13, in one embodiment of the present application, the multispectral RAW response is obtained by a multispectral image sensor based on a filter array (or a filter coating), and a cyclic unit is formed by 3×3 nine pixel points, that is, the spectral resolution is 9. And, color subSpace projection matrix T N×3 For example, a matrix of dimensions 9 x 3, e.gThe RAW response L of the light source power spectrum may be, for example, a 9-dimensional vector, such as [0.564,0.362, … …, … …,0.498,0.548]Subspace color correction matrix T M×3 For example, a matrix of 3 x 3 dimensions, e.g The resulting white balance diagonal matrix may be, for example, a 3 x 3 dimensional matrix, e.g
Of course, in other embodiments of the application, the color subspace projection matrix, the RAW response of the light source power spectrum, and the subspace color correction matrix may be other formats or values, which may be selected and set as desired. And, the RAW response of the light source power spectrum can be understood as the output vector of the light source estimation.
The matrix module described above may be understood as an abstract representation without regard to the specific physical meaning of the color subspace projection matrix, the white balance diagonal matrix, and the subspace color correction matrix described above.
The present embodiment is a modularized image color restoration scheme based on the multispectral RAW image 101, and the image processing method provided by the present embodiment has better effect of the obtained color image compared with some image processing methods in the prior art. For example, the image effect may be represented by a Color Difference (Color Difference) of the image obtained under different light sources. Color difference, also known as color distance, is a point of interest in colorimetry. It quantifies a concept. The color difference may be obtained by simple calculation of Euclidean distance (e.g. Angle Error (AE)) in the color space, or by more complex and uniform human perception formula calculation (e.g. Delta E (E) by the International Commission of illumination, and dE is a measure describing the difference between two colors.
Illustratively, AE and dE (e.g., dE 2000) of the color image, and the corresponding average value, are used as evaluation measures. The Light source may include, for example, a fluorescent lamp (fluorescent tube lamp, CFL), an incandescent lamp (incandescent lamp, INC), a Light-Emitting Diode (LED), and Sunlight (SUN).
Illustratively, as shown in fig. 14, a comparative way of image processing effect is shown. As can be seen from fig. 14, the Average value (Average) of AE of the color image obtained by the inventive scheme is reduced compared with the Average value of AE in the related art method. In some scenarios, the reduction may be more than 50%. In addition, the average value of the dE of the color image is reduced compared with that of the prior art. In some scenarios, it may also be reduced by more than 50%. That is, the electronic device can perform color restoration processing on the multispectral RAW image based on the image processing model with the parameter capable of being learned, so that color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, image quality is effectively improved, and user experience is further improved.
In some scenarios, the effect of local light source estimation is substantially the same as the effect of global light source estimation. However, in some typical scenarios, the image adjustment parameters obtained by the local illuminant estimation compared with the global illuminant estimation are parameters with resolution, i.e. the local illuminant estimation can be processed differently for different areas compared with the global illuminant estimation, so that the effect of the local illuminant estimation is better than that of the global illuminant estimation. Especially, under difficult scenes such as low color temperature, mixed color temperature and the like, the effect of local light source estimation is better than that of global light source estimation.
For example, referring to fig. 15, fig. 15 shows some color images based on global illuminant estimation and local illuminant estimation. In the outdoor sunlight scene, the color saturation of the color image estimated based on the local light source is better than the color saturation of the color image estimated based on the global light source. In a low color temperature difficult scene, the color image obtained based on the global light source estimation is yellow, and the color of the color image obtained based on the local light source estimation is closer to the color seen by human eyes, so that the white balance of the local light source estimation is more accurate compared with the white balance of the global light source estimation. In a mixed color temperature difficult scene, the color image obtained based on the global light source estimation is overall bluish, and the color of the color image obtained based on the local light source estimation is closer to the color seen by human eyes, so that the color reproduction of the local light source estimation is more accurate compared with the color reproduction of the global light source estimation.
Further, in a large-area monochromatic background scene, for example, a white ball is placed on a green background, if the estimation accuracy of a light source is insufficient, the white ball is greenish or yellowish, and the image processing method provided by the embodiment of the application remarkably improves the light source estimation accuracy of the difficult scene, so that the color reduction of neutral color (R=G=B) is more accurate.
In low color temperature (yellow-warm) lighting scenes, if the estimation accuracy of the light source is not enough, the overall color of the image is yellowish (e.g. white objects are yellowish). The image processing method provided by the embodiment of the application obviously improves the light source estimation precision of the low-color-temperature scene, and ensures that the color reproduction of neutral color (R=G=B) is more accurate.
In a mixed color temperature scene with shadows, if the estimation accuracy of the light source is insufficient, the overall color of the image is bluish. The image processing method provided by the embodiment of the application obviously improves the light source estimation precision of the scene, and ensures that the color reproduction of neutral color (R=G=B) is more accurate.
In a mixed color temperature scene with shadows, if the estimation accuracy of the light source is insufficient, the overall color will be blue. The image processing method provided by the embodiment of the application obviously improves the light source estimation precision of the scene, and ensures that the color reproduction of neutral color (R=G=B) is more accurate.
In a pure-color background and a face scene with darker skin, the problem that a face with darker skin cannot be shot into a real black color stably exists, and the color consistency is poor. The image processing method provided by the embodiment of the application can accurately recover the color of the face with darker skin under various solid-color backgrounds, and has better color consistency.
In addition, the image processing method provided by the embodiment of the application can accurately recover the image color under different light sources, and has better color consistency.
In summary, in the embodiment of the present application, the light source estimation is implemented in the multispectral RAW domain space, so that on one hand, the metamerism problem can be alleviated, the light source estimation accuracy is improved, and on the other hand, the light source estimation is implemented through the matrix T 1 And T is 2 The self-adaptive color correction related to the light source is realized, the color reproduction accuracy is improved, a color image with better color reproduction degree is obtained, the consistency of the colors of the images under different light sources is effectively improved, the image quality is effectively improved, and the user experience is further improved.
In addition, the image processing method provided by the embodiment of the application can realize the end-to-end processing from the multispectral RAW image to the color image (such as RGB image), reduce the complexity of the channel, realize the end-to-end joint optimization among all modules, relieve the metamerism phenomenon by utilizing the spectrum information and improve the color distinguishing performance. And based on the light source estimation and the color self-adaptive model, the color reproduction accuracy is remarkably improved, the light source estimation accuracy and the color reproduction accuracy in difficult scenes can also be improved, and the color reproduction accuracy can be improved by realizing the self-adaptive color correction related to the light source.
The structure of the image processing system and the procedure for performing image processing associated with obtaining the high resolution color image 402 from the multispectral RAW image 101 and the high resolution color (or grayscale) RAW image 102 are further described below.
In another embodiment of the present application, as shown in fig. 16, the image processing system includes a preprocessing module 202, a parameter-learnable image processing module 301, and a stylized post-processing module, wherein the parameter-learnable image processing module 301 includes a light source estimation and color adaptation model (as an example of a parameter-learnable image processing model) module and a matrix module.
Wherein the illuminant estimation and color adaptation model module is configured to estimate the color of the image from the multispectral RAW image 101 (e.g., X 2 ∈R h ×w×N As an example of the first sub-image), an image adjustment parameter (as another example of the first image adjustment information) is obtained by extracting a spectral feature, and the image adjustment parameter is input to the matrix module, where the image adjustment parameter may be at least one of a white balance parameter and a subspace color correction matrix, and the white balance parameter may be a white balance diagonal matrix obtained from the foregoing RAW response of the color subspace projection moment and the light source power spectrum, or may be another parameter, which may be set as required. Of course, the image adjustment parameter may be other parameters, which may be selected and set as needed.
The matrix module is used for adjusting parameters of the high-resolution color RAW image 102 (such as X 1 ∈R H×W×3 As an example of the second sub-image) is subjected to color reduction processing to obtain a high-resolution color image 402 '(as another example of the second image) as an image intermediate processing result, and then the high-resolution color image 402' is input to a post-stylization processing module to be subjected to further stylization processing to obtain the high-resolution color image 402 (as another example of the fourth image). The high resolution color image 402 may be, for example, Y ε R H×W×3 。
Further, as shown in fig. 17, in one embodiment of the present application, the image adjustment parameter may be, for example, a white balance parameter T 1 ∈R 3 And subspace color correction matrix T 2 ∈R 3×3 . The matrix module performs color reproduction processing on the high-resolution color RAW image 102 according to the image adjustment parameters to obtain a high-resolution color image 402', which includes the following steps: according to the white balance parameter T 1 ∈R 3 The white balance processing is performed on the high-resolution color RAW image 102 to obtain a white balance image. Then, the matrix T is corrected according to the subspace color 2 ∈R 3×3 And performing color correction processing on the white balance image to realize the color correction related to the light source, thereby obtaining a color-adaptive image. The white balance parameters may comprise, for example, a white balance diagonal matrix derived from the color subspace projection matrix and the source power spectrum raw domain response.
In this embodiment, the white balance and the color correction function in the same manner as described above, but may be selected and set as needed.
The embodiment is a modularized image color restoration scheme based on a multispectral RAW image 101 and a high-resolution color RAW image 102, and based on the embodiment, an electronic device can perform color restoration processing on the high-resolution color RAW image 102 based on a parameter-learning image processing model and the multispectral RAW image 101, so that color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, image quality can be effectively improved, and user experience can be further improved.
And white balance parameter T is obtained through light source estimation and color self-adaptive model 1 And a color correction matrix T 2 The method is respectively applied to the 3-channel high-resolution color RAW (or the pre-processed RAW) image, and can improve the color reproduction quality of the image on the basis of keeping the high resolution of H multiplied by W.
In another embodiment of the present application, as shown in fig. 18, the parameter-learnable image processing module 301 includes a light source estimation and color adaptation model module and a color migration module. The light source estimation and color adaptive model module is configured to extract spectral features from the multispectral RAW image 101, obtain an image adjustment parameter (as another example of the first image adjustment information), and input the image adjustment parameter to the color migration module, where the image adjustment parameter may be, for example, a color image a (as an example of the third image), where the resolution of the color image a is smaller than the resolution of the high-resolution color (or grayscale) RAW image 102. Of course, the image adjustment parameter may be other parameters, which may be selected and set as needed.
The color migration module is used to migrate the color image A to the high resolution color (or grayscale) RAW image 102 (e.g., X 1 ∈R H×W×G ) Performing color reproduction processing, i.e., color migration processing, to obtain a high-resolution color image 402 '(as another example of the second image), and then inputting the high-resolution color image 402' to a stylized post-processing moduleThe rows are further stylized resulting in a high resolution color image 402 (as another example of a fourth image).
The color migration module may be implemented by a classical image filtering algorithm (such as a bilateral filtering algorithm) or by one (or more) neural networks.
The present embodiment is a modularized image color restoration scheme based on a multispectral RAW image 101 and a high-resolution color (or gray scale) RAW image 102, and based on the color image a obtained from the multispectral RAW image 101 by a light source estimation and color adaptive model module in the present embodiment, color restoration processing is performed on the high-resolution color (or gray scale) RAW image 102, so that color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, image quality is effectively improved, and user experience is further improved.
In some scenes, there is an overexposure problem in the process of taking pictures, and in the case of the overexposure problem, when the response of a part of channels exceeds the dynamic range of a sensor, the response is truncated, which breaks the subspace projection relationship established by the linear matrix, and causes color distortion at the overexposure position.
Based on this, in another embodiment of the present application, as shown in fig. 19, the image processing system further includes an overexposure processing module, configured to process the obtained color image 1 (for example, the color image 401 or the high-resolution color image 402 described above) and perform compensation of the multi-spectral response truncation, so as to obtain the color image 3 after the overexposure processing.
As shown in fig. 19, in one embodiment of the present application, the overexposure processing module includes an overexposure region determination module, an overexposure region adjustment module, a non-overexposure channel determination module, a parameter-leachable image processing module 301, and an image combining module. The overexposure region determining module is configured to determine a region in the multispectral RAW image 101 that responds to overexposure, for example, if the response of a certain channel is greater than a preset response threshold, and then consider the region as an overexposed region. The overexposure region adjustment module is configured to process the overexposure region by means of overexposure region coverage (Mask) and the like, to obtain an image corresponding to the overexposure region adjustment (as an example of image overexposure region coverage information), where the overexposure region coverage may be implemented by, for example, binarization processing. The non-overexposed channel determining module is configured to determine that the non-overexposed sub-channel response image is input to the parameter-leachable image processing module 301, and perform, for example, the aforementioned color reduction process (as an example of the third color reduction process) to obtain the color image 2 (as an example of the sixth image). The image combining module then performs an image fusion process (as an example of a second color reproduction process) based on the color image 1, the overexposed region adjustment image, and the color image 2 (as an example of second image adjustment information), and obtains and outputs a combined color image 3 (as an example of a fifth image).
As shown in fig. 20, color artifacts (artifacts) at overexposure are eliminated after the response post-truncation compensation scheme is applied. Therefore, a color image with better color reproducibility can be obtained, the image quality is effectively improved, and the user experience is further improved.
In other implementations of the application, color image 2 may also be derived from the unexposed sub-channel response by other means such as 3-channel color reduction (e.g., AWB and color correction, etc.).
In some scenes, there is a problem of unrealistic color in taking pictures. Based on this, in one embodiment of the present application, as shown in fig. 21, the image processing system further includes a brightness adjustment module for performing brightness adjustment processing on the obtained color image (for example, the color image 401 or the high-resolution color image 402) to obtain a color image after the brightness adjustment processing.
As shown in fig. 21, in one embodiment of the present application, the luminance adjustment module includes a sub-channel determination module, a luminance estimation module, and a luminance migration module. The sub-channel determining module is configured to select a single (or multiple) channel response (i.e. a target sub-channel) according to a preset channel selection rule to perform image brightness estimation according to characteristics of the response of the multispectral RAW image 101, and input the determined sub-channel to the brightness estimating module to obtain estimated brightness (as another example of the second image adjustment information). The brightness estimation method may be various, and one common method is to use the average value of each pixel point in the image as the brightness estimation. And the brightness migration module adjusts the brightness of the color image according to the brightness migration module to obtain a final color image. For example, by applying the result of the estimation in the form of multiplication (or division) to the color image (i.e., by multiplying or dividing the estimated brightness by the brightness of the color image to obtain the adjusted image brightness), an output color image (as another example of the fifth image) is obtained.
As shown in fig. 22, in the color image after the brightness migration process, the response at the image overexposure can be close to the maximum value of the 8-bit color image, so that the brightness distribution of the output image is more reasonable, and the visual effect is better. Namely, a color image with better color reproducibility can be obtained, the image quality is effectively improved, and the user experience is further improved.
In addition, in another implementation manner of the present application, the color restoration processing may be performed on the selected sub-channel response by an independent parameter-learning image processing module to obtain a color restored image, and then the image brightness estimation may be performed according to the color restored image, or another mode such as 3-channel color restoration (for example, AWB, CC) may be used to obtain a color restored image, and then the image brightness estimation may be performed according to the color restored image.
In summary, the Image processing method provided by the implementation manner of the present application may be applied to the multispectral Imaging/Image (MSI) field, and the electronic device may perform color reduction processing on the multispectral RAW Image or the high-resolution color (or gray level) Image based on the Image processing model and the multispectral RAW Image that may be learned by parameters, so as to improve color reduction accuracy, obtain a color Image with better color reduction degree, effectively improve Image quality, and further improve user experience.
In addition, in some implementations, the multi-spectrum RAW image may be subjected to, for example, channel-by-channel image preprocessing, spectrum image interpolation, piecewise gaussian approximation function processing, and the like to obtain a color image, and in a manner of correcting the color image, a final color image may be obtained. In the method, the multispectral RAW image is processed based on the traditional interpolation and function fitting algorithm, the interpolation and function fitting algorithm is fixed, the multispectral RAW image cannot be suitable for more scenes, the steps are related and independent, and the end-to-end image optimization of the whole path can be realized. According to the image processing method provided by the embodiment of the application, based on the image processing model with the parameter capable of being learned and the multispectral RAW image, the multispectral RAW image or the high-resolution color (or gray level) image is subjected to color restoration processing, so that the image processing method can be suitable for more scenes, a color image with better color restoration degree can be obtained, the image quality is effectively improved, and further the user experience is improved.
In some implementations, the image color may also be corrected by locally averaging the spectral response, but processing the spectral response in this manner involves only a simple averaging operation, making it difficult to adequately extract the spectral information. According to the image processing method provided by the embodiment of the application, based on the image processing model and the multispectral RAW image which can be learned based on the parameters, the spectral characteristics can be fully extracted, the multispectral RAW image or the high-resolution color (or gray level) image is subjected to color reduction processing, the color reduction accuracy can be improved, the color image with better color reduction degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
In some implementations, the corresponding parameters may also be derived based on classical statistical assumptions, and a classical white balance algorithm for the color image is extended to the multispectral image, which is white balanced according to the classical white balance algorithm. However, in this way, the classical white balance algorithm is obtained based on classical statistical assumptions, and is effective only for part of the scene, and the overall applicability is weak. According to the image processing method provided by the embodiment of the application, based on the image processing model and the multispectral RAW image which can be learned based on the parameters, the spectral characteristics can be fully extracted, the multispectral RAW image or the high-resolution color (or gray level) image is subjected to color reduction processing, the color reduction accuracy can be improved, the color image with better color reduction degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
In summary, the embodiment of the application provides a new image color processing flow, which can improve color reproduction accuracy, obtain a color image with better color reproduction degree, effectively improve image quality and further improve user experience.
In particular, according to the light source estimation and color self-adaptive model based on the multispectral RAW image provided by the embodiment of the application, more accurate image adjustment information can be obtained based on the multispectral RAW image, and color restoration processing is performed on the multispectral RAW image or the high-resolution color (or gray scale) RAW image according to the image adjustment information, so that the color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, the image quality can be effectively improved, and the user experience can be further improved.
In one embodiment of the application, the image processing system may be implemented, for example, by an image signal processor (Image Signal Processor, ISP) for processing the image signal output by the image signal sensor. It takes the core dominant role in camera systems and is an important device constituting a camera. Its main functional characteristics include: demosaicing, automatic exposure, automatic white balance, lens shading elimination, gamma correction, color space conversion, dynamic range correction, image cropping, and the like.
The embodiment of the application also provides an image processing model training method, which comprises the following steps: inputting a training sample image into an initial image processing model, wherein the training sample image comprises a multispectral original image acquired by a multispectral image sensor; the initial image processing model obtains an output image according to the training sample image, wherein the output image is a color image; training the initial image processing model according to the output image to obtain an image processing model.
In the present embodiment, the obtained image processing model may be, for example, the image processing model including one or more neural networks shown in fig. 8 described above, or may be the light source estimation and color adaptation model shown in fig. 9 and the like described above.
In addition, the training sample image may also include the aforementioned high resolution color (or grayscale) image, which may be selected and set as desired.
In this implementation manner, the training sample image is a multispectral RAW image, the output image is a color image after performing color restoration processing, and the structure of the initial image processing model is the same as that of the image processing module. And training the initial image processing model according to the obtained output image, and adjusting model parameters in the initial image processing model until reaching training termination conditions, so that the image processing model after model training can be obtained. The training termination condition may be, for example, that the corresponding objective loss function converges or reaches a preset iteration number, the convergence condition of the objective loss function and the objective loss function, or the iteration number may be specifically set according to needs, which is not specifically limited in the present application. The multispectral RAW image is subjected to color restoration processing based on the image processing model after model training, so that the color restoration accuracy can be improved, a color image with better color restoration degree can be obtained, the image quality is effectively improved, and the user experience is further improved.
The embodiment of the present application also provides an image processing apparatus, as shown in fig. 23, including: a first input module and an image processing module. Wherein: the first input module is used for inputting a first image into the image processing module, the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by the multispectral image sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning; an image processing module for performing a first color restoration process on the first image by using the image processing model to obtain a second image, wherein the first color restoration process comprises an operation of adjusting the color of the first image based on the first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image
The first input module and the image processing module can be realized by software or hardware. By way of example, an embodiment of the first input module will be described below using the first input module as an example. Similarly, embodiments of the image processing module may refer to embodiments of the first input module.
Module as an example of a software functional unit, the first input module may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the first input module may include code running on multiple hosts/virtual machines/containers. Multiple hosts/virtual machines/containers for running the code may be distributed in the same region (region) or may be distributed in different regions. Further, multiple hosts/virtual machines/containers for running the code may be distributed in the same availability zone (availability zone, AZ) or may be distributed in different AZs, each AZ comprising a data center or multiple geographically close data centers. Wherein typically a region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same virtual private cloud (virtual private cloud, VPC) or in multiple VPCs. In general, one VPC is disposed in one region, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same region and between VPCs in different regions.
Module as an example of a hardware functional unit, the first input module may comprise at least one computing device, such as a server or the like. Alternatively, the first input module may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (programmable logic device, PLD), etc. The PLD may be implemented as a complex program logic device (complex programmable logical device, CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (generic array logic, GAL), or any combination thereof.
The plurality of computing devices included in the first input module may be distributed in the same region or may be distributed in different regions. The plurality of computing devices included in the first input module may be distributed in the same AZ or may be distributed in different AZ. Also, the plurality of computing devices included in the first input module may be distributed in the same VPC or may be distributed in a plurality of VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL.
In other embodiments, the first input module may be used to perform any step in the image processing method, the image processing module may be used to perform any step in the image processing method, the steps that the first input module and the image processing module are responsible for implementing may be specified as needed, and the first input module and the image processing module implement different steps in the image processing method to implement all functions of the image processing apparatus.
In this embodiment, the process of implementing the corresponding function by each module may refer to the foregoing related content about the image processing method, which is not described herein again.
In this embodiment, the image processing apparatus may be applied to a computing device such as a computer or a server, or may be applied to a computing device cluster including at least one computing device, so as to implement an image processing function.
The present application also provides an image processing model training apparatus, as shown in fig. 24, comprising: the system comprises a second input module, an initial image processing model module and a training module. Wherein: the second input module is used for inputting a training sample image into the initial image processing model module, the initial image processing model module comprises an initial image processing model, and the training sample image comprises a multispectral original image acquired by the multispectral image sensor; the initial image processing model module is used for obtaining an output image according to the training sample image, wherein the output image is a color image; the training module is used for training the initial image processing model according to the output image to obtain an image processing model, and the image processing model is applied to the image processing method.
The second input module, the initial image processing model module and the training module can be realized by software or hardware. By way of example, an embodiment of the second input module will be described below using the second input module as an example. Similarly, embodiments of the initial image processing model module and the training module may refer to embodiments of the second input module.
Module as an example of a software functional unit, the second input module may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the second input module may include code running on multiple hosts/virtual machines/containers. Multiple hosts/virtual machines/containers for running the code may be distributed in the same region (region) or may be distributed in different regions. Further, multiple hosts/virtual machines/containers for running the code may be distributed in the same availability zone (availability zone, AZ) or may be distributed in different AZs, each AZ comprising a data center or multiple geographically close data centers. Wherein typically a region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same virtual private cloud (virtual private cloud, VPC) or in multiple VPCs. In general, one VPC is disposed in one region, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same region and between VPCs in different regions.
Module as an example of a hardware functional unit, the second input module may comprise at least one computing device, such as a server or the like. Alternatively, the second input module may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (programmable logic device, PLD), etc. The PLD may be implemented as a complex program logic device (complex programmable logical device, CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (generic array logic, GAL), or any combination thereof.
The plurality of computing devices included in the second input module may be distributed in the same region or may be distributed in different regions. The plurality of computing devices included in the second input module may be distributed in the same AZ or may be distributed in different AZ. Likewise, the plurality of computing devices included in the second input module may be distributed in the same VPC or may be distributed in a plurality of VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL.
In other embodiments, the second input module may be used to perform any step in the image processing model training method, the initial image processing model module may be used to perform any step in the image processing model training method, the training module may be used to perform any step in the image processing model training method, the steps that the second input module, the initial image processing model module, and the training module are responsible for implementing may be specified as needed, and different steps in the image processing model training method are implemented by the second input module, the initial image processing model module, and the training module, respectively, to implement all functions of the image processing model training device.
In this embodiment, the process of implementing the corresponding function by each module may refer to the content related to the image processing model training method, which is not described herein.
In this embodiment, the image processing model training apparatus may be applied to a computing device such as a computer or a server, or may be applied to a computing device cluster including at least one computing device, so as to implement an image processing model training function.
The image processing method provided by the embodiment of the application has wide application scenes, can be applied to various application scenes related to image processing, such as camera or mobile phone photographing, automatic driving, terminal equipment (such as a sweeping robot, augmented Reality (Augmented Reality, AR)/Virtual Reality (VR) glasses) and the like, and can effectively improve the image processing effect. That is, the electronic device may be an electronic device having an image processing function such as a camera, a mobile phone, an automated driving car, a terminal device (e.g., a robot for sweeping floor, AR/VR glasses), or the like.
Furthermore, the image processing method provided by the embodiment of the application can be deployed on the computing node of the related equipment, and the image color processing performance is improved through software modification and hardware (such as an image sensor and the like) adaptation. The computing node may be a central processing unit (Central Processing Unit/Processor, CPU), a graphics Processor (Graphics Processing Unit, GPU), a neural network Processor (Neural Processor Unit, NPU), a tensor Processor (Tensor Processing Unit, TPU), or an application specific integrated chip (Application Specific Integrated Circuit, ASIC), etc.
That is, the foregoing image processing system may be realized by CPU, GPU, NPU, TPU or ASIC or the like.
The present application also provides a computing device 10. As shown in fig. 25A and 25B, computing device 10 includes: bus 102, processor 104, memory 106, and communication interface 108. Communication between the processor 104, the memory 106, and the communication interface 108 is via the bus 102. Computing device 10 may be a server or a terminal device. It should be understood that the present application is not limited to the number of processors, memories in computing device 10.
Bus 102 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one line is shown in fig. 25A and 25B, but not only one bus or one type of bus. Bus 102 may include a path to transfer information between various components of computing device 10 (e.g., memory 106, processor 104, communication interface 108).
The processor 104 may include any one or more of a central processing unit (central processing unit, CPU), a graphics processor (graphics processing unit, GPU), a Microprocessor (MP), or a digital signal processor (digital signal processor, DSP).
The memory 106 may include volatile memory (RAM), such as random access memory (random access memory). The processor 104 may also include non-volatile memory (ROM), such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state disk (solid state drive, SSD).
As shown in fig. 25A, executable program codes are stored in the memory 106, and the processor 104 executes the executable program codes to realize the functions of the aforementioned first input module and image processing module, respectively, that is, the functions of the aforementioned image processing apparatus, thereby realizing the image processing method. That is, the memory 106 has stored thereon instructions for performing the image processing method.
Alternatively, as shown in fig. 25B, executable codes are stored in the memory 106, and the processor 104 executes the executable codes to implement the functions of the aforementioned second input module, initial image processing model module, and training module, that is, the aforementioned image processing model training device, respectively, thereby implementing the image processing model training method. That is, the memory 106 has instructions stored thereon for performing the image processing model training method.
Communication interface 108 enables communication between computing device 10 and other devices or communication networks using a transceiver module such as, but not limited to, a network interface card, transceiver, or the like.
The embodiment of the application also provides a computing device cluster. The cluster of computing devices includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop, notebook, or smart phone.
As shown in fig. 26A, a cluster of computing devices includes at least one computing device 10. The same instructions for performing the image processing method may be stored in the memory 106 in one or more computing devices 10 in the cluster of computing devices.
In some possible implementations, part of the instructions for performing the image processing method may also be stored in the memory 106 of one or more computing devices 10 in the computing device cluster, respectively. In other words, a combination of one or more computing devices 10 may collectively execute instructions for performing the image processing method.
The memories 106 in different computing devices 10 in the cluster of computing devices may store different instructions for performing part of the functions of the image processing apparatus, respectively. That is, the instructions stored by the memory 106 in the different computing devices 10 may implement the functionality of one or more of the first input module and the first image processing module.
As shown in fig. 26B, the computing device cluster includes at least one computing device 10. The same instructions for performing the image processing model training method may be stored in memory 106 in one or more computing devices 10 in the computing device cluster.
In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also each have stored therein a portion of instructions for performing the image processing model training method. In other words, a combination of one or more computing devices 10 may collectively execute instructions for performing an image processing model training method.
The memory 106 in different computing devices 10 in the cluster of computing devices may store different instructions for performing part of the functions of the image processing model training apparatus, respectively. That is, the instructions stored by the memory 106 in the different computing devices 10 may implement the functionality of one or more of the second input module, the initial image processing model module, and the training module.
In some possible implementations, one or more computing devices in a cluster of computing devices may be connected through a network. Wherein the network may be a wide area network or a local area network, etc.
Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored in any useful medium. The computer program product, when run on at least one computing device, causes the at least one computing device to perform an image processing method, or an image processing model training method.
The embodiment of the application also provides a computer readable storage medium. Computer readable storage media can be any available media that can be stored by a computing device or data storage device such as a data center containing one or more available media. Usable media may be magnetic media (e.g., floppy disks, hard disks, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk), among others. The computer-readable storage medium includes instructions that instruct a computing device to perform an image processing method or instruct a computing device to perform an image processing model training method.
In addition, the image processing method and/or the image processing model training method can be applied to the computing device or the computing device cluster comprising at least one computing device. The computing device or the computing device cluster is used for deploying algorithms related to the image processing method and/or algorithms related to the image processing model training method according to the embodiment of the application so as to realize corresponding functions. The computing device may be the electronic device such as the vehicle, the intelligent driving server, the mobile phone, the tablet computer, the wearable device, or other types of devices, which may be selected and set as required.
An embodiment of the present application provides an electronic apparatus including: a memory for storing a computer program, the computer program comprising program instructions; and a processor for executing program instructions to cause the electronic device to perform the aforementioned image processing method or to cause the electronic device to perform the aforementioned image processing model training method.
The color quality of the image is a key factor influencing the photographing performance of electronic equipment such as cameras, mobile phones and the like, so that the image is subjected to color reduction processing to obtain an image with better color reduction performance, the image quality is improved, and the use experience of users on the electronic equipment can be effectively improved.
The electronic device may be, for example, a vehicle-mounted device, a cell phone, a tablet computer, a notebook computer, a palm top computer, a mobile internet device (mobile internet device, MID), a wearable device (including, for example, a smart watch, a smart bracelet, a pedometer, etc.), a personal digital assistant, a portable media player, a navigation device, a video game device, a set top box, a virtual reality and/or augmented reality device, an internet of things device, an industrial control device, a streaming client device, an electronic book, a reading device, a POS device, and other devices.
In embodiments of the present application, the terms "first," "second," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.
In the embodiments of the application, some structural or methodological features may be shown in a particular arrangement and/or order in the drawings. However, it should be understood that such a particular arrangement and/or ordering may not be required. Rather, in some embodiments, these features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of structural or methodological features in a particular figure is not meant to imply that such features are required in all embodiments, and in some embodiments, may not be included or may be combined with other features.
While the application has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that the foregoing is a further detailed description of the application with reference to specific embodiments, and it is not intended to limit the application to the specific embodiments described. Various changes in form and detail may be made therein by those skilled in the art, including a few simple inferences or alternatives, without departing from the spirit and scope of the present application.
Claims (20)
1. An image processing method, the method comprising:
inputting a first image to an image processing module, wherein the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by a first image sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning;
and performing first color restoration processing on the first image by using the image processing model to obtain a second image, wherein the first color restoration processing comprises an operation of adjusting the color of the first image based on first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.
2. The image processing method according to claim 1, wherein performing a first color reduction process on the first image using the image processing model to obtain a second image, comprises:
processing with the image processing model based on the first sub-image to obtain the first image adjustment information;
and carrying out the first color reduction processing on the first sub-image according to the first image adjustment information to obtain the second image.
3. The image processing method according to claim 2, wherein processing with the image processing model based on the first sub-image to acquire the first image adjustment information includes:
processing with the image processing model based on the first sub-image to obtain spectral features of the first image;
and obtaining the first image adjustment information by using the image processing model based on the spectral characteristics.
4. The image processing method according to claim 2 or 3, wherein the first image adjustment information includes a color subspace projection matrix, a light source power spectrum original domain response, and a subspace color correction matrix, and the performing the first color reduction processing on the first sub-image according to the first image adjustment information to obtain the second image includes:
performing color subspace projection processing on the first sub-image according to the color subspace projection matrix to obtain a subspace response image;
obtaining a white balance diagonal matrix according to the color subspace projection matrix and the original domain response of the light source power spectrum, and performing white balance processing on the subspace response image according to the white balance diagonal matrix to obtain a white balance image;
And performing color correction processing on the white balance image according to the subspace color correction matrix to obtain a color-adaptive image as the second image, or performing color gamut conversion processing on the color-adaptive image to obtain the second image.
5. The image processing method according to claim 1, wherein the first image further includes a second sub-image, the second sub-image being a color image or a grayscale image acquired by a second image sensor, the spatial resolution of the second sub-image being greater than the spatial resolution of the first sub-image, and performing a first color reduction process on the first image using the image processing model to obtain a second image, including:
processing with the image processing model based on the first sub-image to obtain the first image adjustment information;
and carrying out the first color reduction processing on the second sub-image according to the first image adjustment information to obtain the second image.
6. The image processing method according to claim 5, wherein the first image adjustment information includes a white balance adjustment parameter and a subspace color correction matrix, the white balance adjustment parameter includes a white balance diagonal matrix, the second sub-image is a color image, the first color reduction processing is performed on the second sub-image according to the first image adjustment information, and the second image is obtained, including:
Performing white balance processing on the second sub-image according to the white balance adjustment parameters to obtain a white balance image;
and performing color correction processing on the white balance image according to the subspace color correction matrix to obtain a color adaptation image serving as the second image.
7. The image processing method according to claim 5, wherein the first image adjustment information includes a third image, the first color reduction processing is performed on the second sub-image according to the first image adjustment information, and the second image is obtained, comprising:
and performing color migration processing on the second sub-image according to the third image to obtain the second image.
8. The image processing method according to claim 1, wherein the image processing model includes at least one neural network, and performing a first color reduction process on the first image using the image processing model to obtain a second image includes:
and carrying out the first color restoration processing on the first sub-image by using the at least one neural network to obtain the second image.
9. The image processing method according to any one of claims 1 to 8, wherein inputting the first image to the image processing module includes:
Preprocessing the first image to adjust the image quality of the first image, so as to obtain the preprocessed first image;
and inputting the preprocessed first image to the image processing module.
10. The image processing method according to any one of claims 1 to 9, characterized in that the method further comprises:
and performing stylization processing on the second image to obtain a fourth image with the image style being a preset image style.
11. The image processing method according to claim 10, characterized in that the method further comprises:
obtaining second image adjustment information according to the first sub-image;
and carrying out second color reduction processing on the first target image according to the second image adjustment information to obtain a fifth image, wherein the first target image comprises the second image or the fourth image.
12. The image processing method according to claim 11, wherein the second image adjustment information includes image overexposure region masking information and a sixth image, the image overexposure region masking information being image overexposure region masking information corresponding to an overexposed region in response to a sub-channel in the first sub-image, the sixth image being an image obtained by performing a third color reduction process according to a non-overexposed sub-channel response in the first sub-image,
And performing a second color reduction process on the first target image according to the second image adjustment information to obtain a fifth image, including:
and performing image fusion processing according to the first target image, the image overexposure region covering information and the sixth image to obtain the fifth image.
13. The image processing method of claim 11, wherein the second image adjustment information includes image brightness information, the image brightness information being determined based on a response of a target subchannel in the first sub-image,
and performing a second color reduction process on the first target image according to the second image adjustment information to obtain a fifth image, including:
and adjusting the brightness of the first target image according to the image brightness information to obtain the fifth image.
14. A method of training an image processing model, the method comprising:
inputting a training sample image into an initial image processing model, wherein the training sample image comprises a multispectral original image acquired by a first image sensor;
the initial image processing model obtains an output image according to the training sample image, wherein the output image is a color image;
Training the initial image processing model according to the output image to obtain an image processing model, wherein the image processing model is applied to the image processing method as claimed in any one of claims 1 to 13.
15. An image processing apparatus, comprising:
the first input module is used for inputting a first image into the image processing module, the first image comprises a first sub-image, the first sub-image is a multispectral original image acquired by the first image sensor, and the image processing module comprises an image processing model which is trained in advance based on machine learning;
the image processing module is used for carrying out first color restoration processing on the first image by utilizing the image processing model to obtain a second image, the first color restoration processing comprises an operation of adjusting the color of the first image based on first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.
16. An image processing model training apparatus, comprising:
the second input module is used for inputting a training sample image into the initial image processing model module, the initial image processing model module comprises an initial image processing model, and the training sample image comprises a multispectral original image acquired by the first image sensor;
The initial image processing model module is used for obtaining an output image according to the training sample image, wherein the output image is a color image;
a training module, configured to train the initial image processing model according to the output image, so as to obtain an image processing model, where the image processing model is applied to the image processing method according to any one of claims 1-13.
17. An electronic device, comprising:
a memory for storing a computer program, the computer program comprising program instructions;
a processor for executing the program instructions to cause the electronic device to perform the image processing method of any one of claims 1-13 or to cause the electronic device to perform the image processing model training method of claim 14.
18. A cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the image processing method of any one of claims 1-13 or to cause the cluster of computing devices to perform the image processing model training method of claim 14.
19. A computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the image processing method of any of claims 1-13 or cause the cluster of computing devices to perform the image processing model training method of claim 14.
20. A computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the image processing method according to any of claims 1-13 or the image processing model training method according to claim 14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310872067.3A CN117061881A (en) | 2023-07-14 | 2023-07-14 | Image processing and image processing model training method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310872067.3A CN117061881A (en) | 2023-07-14 | 2023-07-14 | Image processing and image processing model training method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117061881A true CN117061881A (en) | 2023-11-14 |
Family
ID=88654360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310872067.3A Pending CN117061881A (en) | 2023-07-14 | 2023-07-14 | Image processing and image processing model training method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117061881A (en) |
-
2023
- 2023-07-14 CN CN202310872067.3A patent/CN117061881A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220207680A1 (en) | Image Processing Method and Apparatus | |
| US11037278B2 (en) | Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures | |
| CN109712102B (en) | Image fusion method and device and image acquisition equipment | |
| KR101698876B1 (en) | Generation of ghost-free high dynamic range images | |
| CN115442515A (en) | Image processing method and apparatus | |
| TWI737979B (en) | Image demosaicer and method | |
| CN112348747A (en) | Image enhancement method, device and storage medium | |
| US20220189029A1 (en) | Semantic refinement of image regions | |
| US20160104270A1 (en) | Method for obtaining at least one high dynamic range image, and corresponding computer program product, and electronic device | |
| CN115550570B (en) | Image processing method and electronic equipment | |
| CN113676628A (en) | Multispectral sensor, imaging device and image processing method | |
| CN116438804A (en) | Frame processing and/or capturing instruction systems and techniques | |
| CN110717864B (en) | Image enhancement method, device, terminal equipment and computer readable medium | |
| CN114862698A (en) | Method and device for correcting real overexposure image based on channel guidance | |
| CN116416122B (en) | Image processing method and related device | |
| CN110868548A (en) | Image processing method and electronic equipment | |
| CN113379609A (en) | Image processing method, storage medium and terminal equipment | |
| US11457189B2 (en) | Device for and method of correcting white balance of image | |
| CN113781326A (en) | Demosaicing method and device, electronic equipment and storage medium | |
| CN107295261A (en) | Image defogging processing method, device, storage medium and mobile terminal | |
| CN116723413A (en) | RAW domain image denoising method and shooting device | |
| KR20200145670A (en) | Device and method for correcting white balance of image | |
| CN115802183B (en) | Image processing method and related device | |
| Pan et al. | Joint demosaicking and denoising for CFA and MSFA images using a mosaic-adaptive dense residual network | |
| CN115550575B (en) | Image processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |