WO2023082811A1 - 一种图像颜色的处理方法及装置 - Google Patents

一种图像颜色的处理方法及装置 Download PDF

Info

Publication number
WO2023082811A1
WO2023082811A1 PCT/CN2022/117564 CN2022117564W WO2023082811A1 WO 2023082811 A1 WO2023082811 A1 WO 2023082811A1 CN 2022117564 W CN2022117564 W CN 2022117564W WO 2023082811 A1 WO2023082811 A1 WO 2023082811A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
light source
depth
point cloud
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/117564
Other languages
English (en)
French (fr)
Inventor
钱彦霖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honor Device Co Ltd
Original Assignee
Honor Device Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202111387638.1A external-priority patent/CN116152360B/zh
Application filed by Honor Device Co Ltd filed Critical Honor Device Co Ltd
Publication of WO2023082811A1 publication Critical patent/WO2023082811A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the present application relates to the field of electronic information technology, and in particular to an image color processing method and device.
  • Image processing is a common function of electronic devices and can be used in scenes such as taking photos and videos.
  • Image processing includes image color processing.
  • the image color processing effect is directly related to the user's experience, how to improve the image color processing effect is one of the current research directions.
  • the present application provides an image color processing method and device, aiming at solving the problem of how to improve the image color processing effect.
  • the first aspect of the present application provides an image color processing method, which is applied to an electronic device, and the electronic device includes a first camera and a second camera, and the method includes: acquiring a first image and a depth image, and the first The image is an image collected by the first camera, the depth image is an image collected by the second camera, and the depth image is used to indicate part or all of the depth information of the first image. Based on the first image and the depth image, determine first light source information of the first image, where the first light source information includes a white point value of the first image, and based on the first light source information, The first image is processed.
  • the depth information indicated by the depth image is used as the basis for determining the light source information, when determining the light source information, it can be considered that the "white spots" at different positions have different influences on the pixels, which is conducive to improving the accuracy of the light source information, thereby Improve the effect of image processing.
  • the first light source information of the first image is determined based on the first image and the depth image, and the first light source information is determined based on the first light source information.
  • the implementation of processing the first image includes: based on the first image and the depth image, determining the second light source information of the second image and the third light source information of the third image, the first image includes the For the second image and the third image, the second light source information includes the white point value of the second image, and the third light source information includes the white point value of the third image.
  • the second image is processed based on the second light source information
  • the third image is processed based on the third light source information.
  • the implementation of determining the first light source information of the first image based on the first image and the depth image includes: based on the first image and the depth image The depth image generates a point cloud, and any point in the point cloud has position information and color information. Based on the point cloud, the first light source information of the first image is determined. Point cloud can well integrate color information and depth information, laying the foundation for determining light source information based on color information and depth information.
  • the method of generating a point cloud based on the first image and the depth image includes: based on a target point as any point in the point cloud in the The coordinates of the corresponding pixel in the first image, the depth value of the corresponding pixel in the depth image, and the parameters of the second camera generate the two-dimensional position information of the target point.
  • the two-dimensional position information and the depth value are used as position information of the target point in the point cloud.
  • the color information of the corresponding pixel in the first image of the target point is used as the color information of the target point in the point cloud.
  • the determining the first light source information of the first image based on the point cloud includes: inputting the point cloud into a light source estimation model to obtain the The light source estimation vector output by the light source input model, wherein the light source input model is trained using sample point clouds and amplified sample point clouds.
  • the sample point cloud is obtained by amplifying.
  • the amplified sample point cloud enriches the training data, which is beneficial to improve the accuracy of the light source estimation model.
  • the point cloud includes: a local point cloud of points corresponding one-to-one to some pixels in the first image.
  • the determining the second light source information of the second image and the third light source information of the third image based on the first image and the depth image includes: determining the second light source information based on the local point cloud corresponding to the second image.
  • the second light source information of the second image is determined based on the local point cloud corresponding to the third image to determine the third light source information of the third image.
  • the local point cloud lays the foundation for dividing the first image into blocks to determine the light source information and processing them in blocks.
  • the first camera and the second camera are the same camera.
  • the first image includes: an RGB image.
  • the processing the first image includes: performing white balance processing on the first image, or performing relighting on the first image. Because the determination of the light source information takes into account the relative position information between the light source pixel and other pixels, the light source information is more accurate, and thus the white balance effect is better. Also, experiments have shown that the processing speed is faster, and the required images are smaller. Relighting preserves the relative relationship of pixel colors.
  • the acquiring the first image and the depth image includes: acquiring the first image and the Depth image for easy user manipulation.
  • the user interface includes: a real-time service interface, which is convenient for improving the image quality of the real-time service.
  • the second aspect of the present application provides an image color processing method applied to an electronic device, the electronic device includes a first camera and a second camera, the method includes: acquiring a first image and depth information, the first The image is an image collected by the first camera, the depth information is obtained by the second camera, and the depth information is part or all of the depth information of the first image. Based on the first image and the depth information, determine first light source information of the first image, where the first light source information includes a white point value of the first image, and based on the first light source information, The first image is processed.
  • the depth information is used as the basis for determining the light source information, when determining the light source information, it can be considered that the "white point" at different positions has different influences on the pixels, which is conducive to improving the accuracy of the light source information, thereby improving image processing. Effect.
  • the first light source information of the first image is determined based on the first image and the depth information, and the first light source information is determined based on the first light source information.
  • the implementation manner of processing the first image includes: determining the second light source information of the second image and the third light source information of the third image based on the first image and the depth information, the first image including the For the second image and the third image, the second light source information includes the white point value of the second image, and the third light source information includes the white point value of the third image.
  • the second image is processed based on the second light source information
  • the third image is processed based on the third light source information.
  • the implementation of determining the first light source information of the first image based on the first image and the depth information includes: based on the first image and the depth information The depth information generates a point cloud, and any point in the point cloud has position information and color information. Based on the point cloud, the first light source information of the first image is determined. Point cloud can well integrate color information and depth information, laying the foundation for determining light source information based on color information and depth information.
  • the method of generating a point cloud based on the first image and the depth information includes: based on a target point as any point in the point cloud in the The coordinates of the corresponding pixels in the first image, the depth value, and the parameters of the second camera generate two-dimensional position information of the target point.
  • the two-dimensional position information and the depth value are used as position information of the target point in the point cloud.
  • the color information of the corresponding pixel in the first image of the target point is used as the color information of the target point in the point cloud.
  • the determining the first light source information of the first image based on the point cloud includes: inputting the point cloud into a light source estimation model to obtain the The light source estimation vector output by the light source input model, wherein the light source input model is trained using sample point clouds and amplified sample point clouds.
  • the sample point cloud is obtained by amplifying.
  • the amplified sample point cloud enriches the training data, which is beneficial to improve the accuracy of the light source estimation model.
  • the point cloud includes: a local point cloud of points one-to-one corresponding to some pixels in the first image.
  • the determining the second light source information of the second image and the third light source information of the third image based on the first image and the depth information includes: determining the second light source information based on the local point cloud corresponding to the second image.
  • the second light source information of the second image is determined based on the local point cloud corresponding to the third image to determine the third light source information of the third image.
  • the local point cloud lays the foundation for dividing the first image into blocks to determine the light source information and processing them in blocks.
  • the first camera and the second camera are the same camera.
  • the first image includes: an RGB image.
  • the processing the first image includes: performing white balance processing on the first image, or performing relighting on the first image. Because the determination of the light source information takes into account the relative position information between the light source pixel and other pixels, the light source information is more accurate, and thus the white balance effect is better. Also, experiments have shown that the processing speed is faster, and the required images are smaller. Relighting preserves the relative relationship of pixel colors.
  • the acquiring the first image and depth information includes: acquiring the first image and the In-depth information, easy for users to operate.
  • the user interface includes: a real-time service interface, which is convenient for improving the image quality of the real-time service.
  • a third aspect of the present application provides an electronic device, including: a memory for storing application programs.
  • One or more processors are configured to run the application program to implement the image color processing method described in the first aspect or the second aspect of the present application.
  • the fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored, and when the computer device runs the application program, the image color processing method described in the first aspect or the second aspect of the present application is realized .
  • a fifth aspect of the present application provides a computer program product.
  • the computer program product runs on a computer, the computer executes the image color processing method described in the first aspect or the second aspect of the present application.
  • Figure 1 is an example diagram of a camera interface including AWB
  • Fig. 2 is an example diagram of the effect of the existing AWB
  • Fig. 3 is the hardware structural diagram of the electronic device provided by the present application.
  • FIG. 4 is a software architecture diagram of the electronic device provided by the present application.
  • FIG. 5 is a flow chart of training the light source estimation model provided by the embodiment of the present application.
  • FIG. 6 is a flow chart of an image color processing method provided in an embodiment of the present application.
  • FIG. 7 is an effect diagram of the global AWB method provided by the embodiment of the present application.
  • Fig. 8a is an example diagram of an application scenario of the image color processing method provided by the embodiment of the present application.
  • FIG. 8b is an example diagram of the image color processing method provided by the embodiment of the present application applied to a real-time scene
  • FIG. 9 is a flowchart of another image color processing method provided by the embodiment of the present application.
  • Figure 10a and Figure 10b are both example diagrams of application scenarios of the image color processing method provided by the embodiment of the present application.
  • Figure 11a is an effect diagram of the local AWB method provided by the embodiment of the present application.
  • Fig. 11b is another rendering of the partial AWB method provided by the embodiment of the present application.
  • Fig. 12 is a flowchart of another image color processing method provided by the embodiment of the present application.
  • Figure 13a is an example diagram before relighting provided by the embodiment of the present application.
  • Fig. 13b is an example diagram of the effect of relighting provided by the embodiment of the present application.
  • Image color processing includes, but is not limited to, automatic white balance (AWB) and Relighting.
  • ALB automatic white balance
  • Relighting Relighting
  • AWB The purpose of AWB is to reduce the influence of the light source in the solid imaging environment on the solid imaging color.
  • the idea of AWB is: determine the light source information from the image, commonly known as "white point", and then use the color of the "white point” to adjust the color of other pixels in the image to remove the color of the "white point” from affecting the color of other pixels. Influence.
  • AWB is a common image processing function of electronic equipment. Take the shooting interface of the camera shown in Figure 1 as an example. Assuming that AWB has been turned on, the captured image displayed on the shooting interface of the camera is the image after AWB, as shown in Figure 2 shown.
  • the correlated color temperature (Correlated Color Temperature, CCT) of area 1 in Figure 2 is 7000k, and the CCT of area 2 is 4000k. It can be seen that the same white has a large color temperature difference in FIG. 2 , indicating that the color difference is large, so at least one area has a large color difference from white. So the effect of AWB still has room for improvement.
  • CCT Correlated Color Temperature
  • the near-light The light in the distance actually affects area 1 more than area 2
  • the light in the distance actually affects area 2 more than area 1.
  • the nearby lights have a greater impact on the entire image than the far lights because they are closer to the camera.
  • the existing AWB regards the influence of nearby and distant lights on other pixels as the same, and determines the color of the "white point” based on the color average of the two light sources, and then uses the color of the "white point” to adjust other pixels s color.
  • the inventors further found that if the positional relationship of each pixel in the image in the actual space is used as a constraint, it is beneficial to improve the effect of AWB.
  • the depth image can express the positional relationship of each pixel in the image in the actual 3D space, and the configuration of time-of-flight (TOF) cameras on electronic devices is becoming more and more common. Therefore, it is possible to use TOF
  • TOF time-of-flight
  • the embodiments of the present application provide a method for processing image colors, using RGB images and depth images to obtain light source information (that is, "white point") in the RGB image, and use the obtained light source information as a basis , to adjust the color of the image to obtain more accurate light source information than the existing AWB, so that the processed image is closer to the real color than the existing image.
  • light source information that is, "white point”
  • Relighting refers to changing the color of the main light source without changing the direction of light irradiation and maintaining the relative color relationship between pixels in the image.
  • the light source information in the image is also the basis for relighting.
  • the image color processing method disclosed in the embodiment of the present application can use more accurate light source information to realize relighting.
  • the method provided in the embodiment of the present application is applicable but not limited to images imaged under multiple light sources, and image color processing can be performed based on at least one item of global light source information and local light source information.
  • the lighting mode of an RGB image is shown in formula (1):
  • ⁇ i is the incident angle of the light, the light comes from the surface ⁇ + of the upper hemisphere
  • ⁇ 0 is the observation angle of the light, which is reflected in the image as the reflection angle of the light entering the RGB camera when the RGB camera shoots
  • N is the normal vector of the surface
  • E represents the spectral energy distribution
  • S is the surface reflectance
  • R rgb is the response coefficient of the RGB sensor.
  • the global light source information of an RGB image is:
  • the global light source information can be understood as the information of light sources affecting all pixels in the image.
  • a RGB image is divided into multiple image blocks, and E of each image block is called the local light source information of an RGB image.
  • black body An object that neither reflects nor completely projects under the action of radiation, but can absorb all the radiation falling on it is called a black body or a complete radiator.
  • the black body When the black body is continuously heated, the maximum value of its relative spectral power distribution will move to the short-wave direction, and the corresponding light color will change in the order of red, yellow, white, and blue. At different temperatures, the light color corresponding to the black body will change.
  • the arc locus formed on the chromaticity diagram is called the black body locus or Planck locus.
  • CCT refers to the temperature of the black-body radiator closest to the color with the same luminance stimulus, expressed in K degrees, and is used to describe the measure of the color of light located near the Planckian locus.
  • Light sources other than thermal radiation light sources have linear spectra, and their radiation characteristics are quite different from black body radiation characteristics. Therefore, the light color of these light sources may not exactly fall on the black body locus on the chromaticity diagram.
  • CCT is usually used to describe the color characteristics of the light source.
  • AWB AWB Parameters involved in AWB include CCT and chromaticity distance D uv , but for the image provided in the embodiment of the present application, D uv can be ignored, so details are not repeated here.
  • CCT is still used to represent color.
  • 6500k one of the white CCTs recognized in the industry, is used as the white CCT.
  • the image color processing method provided in the embodiment of the present application is applied to an electronic device.
  • the electronic device may be a cell phone, tablet computer, desktop, laptop, notebook computer, Ultra-mobile Personal Computer (UMPC), handheld computer, netbook, personal digital assistant (Personal Digital Assistant) Assistant, PDA), wearable electronic devices, smart watches and other devices.
  • UMPC Ultra-mobile Personal Computer
  • PDA Personal Digital Assistant
  • FIG. 3 is an example of the structure of an electronic device, including: an RGB camera 1 , a TOF camera 2 , a processor 3 , a memory 4 , and an I/O subsystem 5 .
  • RGB camera 1 is used to collect RGB data.
  • RGB camera 1 can be set as a front camera or a rear camera.
  • the RGB camera 1 includes but not limited to an RGB sensor 11 (which may be called a first camera) and an RGB sensor controller 12 (which may be called a second camera).
  • the TOF camera 2 is used to collect TOF data for generating depth images.
  • TOF Camera 2 can be set as a front camera or a rear camera. It can be understood that, in the electronic device described in this embodiment, because the RGB image and the depth image need to be registered, in order to reduce the computational complexity, the positions of the RGB camera 1 and the TOF camera 2 on the electronic device are concentrated.
  • the TOF camera 2 includes but not limited to a TOF sensor 21 , a TOF sensor controller 22 , a TOF light source 23 and a TOF light source controller 24 .
  • the TOF light source controller 24 is controlled by the TOF sensor controller 22 to realize the control of the TOF light source 23 .
  • the TOF light source 23 emits infrared light or laser light under the control of the TOF light source controller 24 .
  • the TOF sensor 21 is used to sense the emitted light reflected by the object to collect TOF data.
  • the RGB sensor controller 12, the TOF sensor controller 22, and the TOF light source controller 24 are arranged in the I/O subsystem 5, and communicate with the processor 3 through the I/O subsystem 5.
  • the memory 4 is used for storing computer-executable program codes.
  • the memory 4 may include a program storage area and a data storage area.
  • the program storage area may store program codes required for implementing an operating system, a software system, at least one function, and the like.
  • the data storage area can store data acquired, generated, and used during the use of the electronic device.
  • the processor 3 may include one or more processing units, for example: the processor 3 may include an application processor (application processor, AP), a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP) )wait.
  • application processor application processor, AP
  • graphics processing unit graphics processing unit
  • ISP image signal processor
  • the structure shown in this embodiment does not constitute a specific limitation on the electronic device.
  • the electronic device may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components may be realized in hardware, software, or a combination of software and hardware.
  • the first camera and the second camera are the same camera, that is, the camera not only collects RGB data but also collects depth information.
  • the operating system implemented by the processor 3 by running the code stored in the memory 4 may be an iOS operating system, an Android open source operating system, a Windows operating system, and the like.
  • the Android open source operating system will be used as an example for illustration.
  • the Android system is divided into four layers, which are respectively an application program layer, an application program framework layer, a hardware abstraction layer, and a kernel layer from top to bottom.
  • the application layer can include a series of applications. As shown in FIG. 4 , in the embodiment of the present application, examples of applications related to AWB and relighting include cameras, video calls, and the like.
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • HAL Hardware Abstraction Layer
  • Android Runtime Android Runtime
  • the kernel layer is the layer between hardware and software.
  • the kernel layer includes at least RGB camera driver, TOF camera driver and so on. Each driver processes the acquired hardware data, and reports the processing result to the corresponding module of the hardware abstraction layer.
  • the embodiment of the present application uses the RGB image and the depth image to obtain the light source information in the RGB image, but it should be emphasized that the depth image represents the depth information of the pixels in the image, only the depth image is introduced, and It cannot directly improve the accuracy of light source information.
  • mapping relationship between the information obtained by combining color information and depth information and light source information, and this mapping relationship can be well expressed by a neural network model.
  • the model is called a light source estimation model.
  • the light source estimation model realizes the function of outputting a light source estimation vector based on the point cloud by learning the mapping relationship between the sample point cloud set and the label of the sample point cloud set.
  • the point cloud described in this embodiment not only has position information, but also has color information. That is, each point in the point cloud has position information and color information.
  • Fig. 5 is the training process of the light source estimation model, including the following steps:
  • sample RGB image can be obtained by using the RGB camera in the electronic device
  • sample depth image can be obtained by using the TOF camera in the electronic device
  • sample RGB image and the sample depth image can also be from the training image set.
  • the sample RGB images in this embodiment all include the imaging area of the color card.
  • the sample point cloud set is a collection of N point clouds, denoted as
  • the point cloud generated in this embodiment has not only position information, but also color information.
  • p i is any point in the point cloud, which can be expressed as a vector with six-dimensional information (x, y, z, r, g, b).
  • the generation rule of any point vector in the point cloud is:
  • u and v are the coordinates of the pixel corresponding to p i in the RGB image
  • d is the depth value of the pixel corresponding to p i in the depth image
  • fx, fy are the focal length of the TOF camera converted to the pixel coordinate system
  • c x and cy are the components under the transformation of the optical center of the TOF camera to the pixel coordinate system
  • r is the R component of p i in the RGB image
  • g is the G component of p i in the RGB image
  • b is p i B component in RGB image.
  • the sample RGB image corresponds to the pixel in the sample depth image, which not only refers to the coordinate correspondence in the pixel coordinate system, but also corresponds to the same imaging entity, that is, the corresponding pixel points represent the same in the imaging entity location point.
  • the points in the point cloud have a one-to-one correspondence with the pixel points in the sample RGB image and the sample depth image respectively. Therefore, any point in the point cloud has a relationship with the sample RGB image or the sample depth image The corresponding pixel points in represent the same position points in the imaging entity.
  • the sample point cloud has increased position information, and compared with the sample depth image, the sample point cloud has added color information, so the sample point cloud obtained in this embodiment combines the sample RGB
  • the information of the image and the sample depth image has both position information and color information.
  • sample point clouds can be augmented.
  • the rotation matrix R ⁇ SO(3) belonging to the orthogonal group is set to represent the different shooting angles of the camera for the same object.
  • the position data in the sample point cloud is multiplied by each rotation matrix R (the color data remains unchanged) to obtain the sample point cloud from different viewing angles, and realize the amplification of the sample point cloud.
  • the illumination change of an object can be changed by the change of the relative position between the object and the light source, and under the same light source, the change of illumination does not affect the change of the color of the light source, set the light intensity
  • the coefficient L multiplies the color data in the sample point cloud by L (the position data remains unchanged) to realize the amplification of the sample point cloud.
  • P' represents the sample point cloud after amplification
  • P pos represents the position data in the sample point cloud before amplification
  • P rgb represents the color data in the sample point cloud before amplification
  • L is a one-dimensional Gaussian distribution constant, Represents a link at the dimension level.
  • pointnet can be used to extract features point by point from the sample point cloud, and obtain a light source estimation vector based on the weighted information of the feature of each point.
  • the obtained in this step is the global light source estimation vector in the RGB sample image, that is, for an RGB image, a light source estimation vector is output as the light source information of this RGB image.
  • the light source estimation vector can be understood as the color of the light source, and can also be understood as the "white point" value of the RGB image.
  • the format of the sample point cloud can also be converted, for example, the dimensions of the sample point cloud can be reconstructed.
  • the method of obtaining the label of any sample point cloud can be as follows: identify the color card area in the sample RGB image, and determine the color value of the sample point cloud generated by the sample RGB image according to the color value of the pre-specified reference color block in the color card area.
  • a light source estimation vector such as the color value of the reference patch, is used as the label of the sample point cloud.
  • the training method shown in Figure 5 enables the light source estimation model to learn the mapping relationship between the point cloud with position information and color information and the light source information, laying the foundation for improving the effect of the image processing method based on light source information. Because the position information is added on the basis of the color information, it is beneficial for the model to learn to determine the light source distribution in the space from the mutual position relationship and color information between each point, and further determine the ability of the global light source, thereby improving the light source. Accuracy of Information. And, further, the amplification method of the sample point cloud can obtain enough sample point clouds, so as to obtain a better training effect.
  • FIG. 6 Based on the light source estimation model obtained through training, the flow of the image color processing method provided by the embodiment of the present application is shown in FIG. 6 , and the flow focuses on the description of AWB.
  • Figure 6 includes the following steps:
  • S61 may be triggered by the user's operation, for example, the user clicks the shooting button on the interface shown in FIG. 1 .
  • the RGB image is registered by pre-configuring the parameters of the RGB camera and the TOF camera in the electronic device, as well as the transformation matrix between the world coordinate system, the camera coordinate system, and the pixel coordinate system and depth images, which will not be repeated here.
  • the RGB image and depth image described in the following steps are the registered RGB image and depth image.
  • the global point cloud refers to a point cloud corresponding to all pixels in the RGB image.
  • the generation rules of the point cloud can be referred to formula (3), and will not be repeated here.
  • the points in the point cloud are in one-to-one correspondence with the pixels in the RGB image, and are in one-to-one correspondence with the pixels in the depth image. There is a one-to-one correspondence between pixels in the RGB image and pixels in the depth image. The corresponding specific meanings are as described above.
  • the obtained is the global light source estimation vector of the RGB image acquired in S61.
  • the light source vector includes the white point value of the RGB image.
  • the white point value can be understood as the color value of the light source.
  • the depth information can represent the positional relationship between the light source and other pixels in the image, or the depth information can represent the positional relationship of each object in the image, or the depth The information can represent the positional relationship between the light source in the image and each object in the image. Therefore, the process shown in Figure 6 and the introduction of depth information are beneficial to determine the "white point" in the image according to the degree of influence of the light source (that is, light source estimation). vector) to obtain a more accurate "white point".
  • Table 1 lists the comparison data of the process shown in Figure 6 and other existing white balance adjustment algorithms in the same data set:
  • existing algorithms 1-5 represent existing AWB algorithms.
  • the present invention represents the AWB algorithm shown in FIG. 6 .
  • Points refers to the number of pixels in the RGB image.
  • Angular error refers to the cosine value of the angle between the light source vector estimated by the algorithm and the sample light source vector, and is used to measure the difference between the light source vector estimated by the algorithm and the sample light source vector.
  • Angular error refers to the cosine value of the angle between the light source vector estimated by the existing algorithm 1 and the sample light source vector.
  • the sample light source vector refers to the label of the light source vector in the RGB image, that is, the label of the aforementioned sample point cloud.
  • the method of obtaining the sample light source vector please refer to the method of obtaining the label of the aforementioned sample point cloud.
  • the mean value Mean represents the mean value of the angle errors obtained by executing the algorithm on multiple (for example, 500) RGB images. Taking the existing algorithm 1 as an example, multiple angle errors are obtained by executing the existing algorithm 1 on multiple RGB images, and Mean represents multiple angle errors. The mean value of angle errors.
  • the median value Median represents the median value of the angle error obtained by executing the algorithm on multiple RGB images, the three-mean value Tri.
  • the mean value of the 25% of the angle errors with the smallest (best) value, the worst W25% means that among the angle errors obtained by executing the algorithm on multiple RGB images, the 25% of the largest (worst) values The mean value of the angular error.
  • the training length refers to the time required for training the light source estimation model in the method described in the present invention.
  • the present invention (256Points, w/o depth) refers to the parameter that does not consider the algorithm that depth information obtains, and it can be seen that, under the situation that does not consider depth information, effect obviously becomes worse.
  • the process described in this embodiment can be triggered and executed through the AWB control on the user interaction interface, as shown in FIG.
  • the test duration is the duration of processing an image after the actual deployment of the algorithm, which is more practical.
  • the training time is the training time of the model offline on the server before the algorithm is deployed.
  • the process shown in FIG. 6 can also be applied to real-time scenarios.
  • the electronic device executes S61 after acquiring the AWB instruction. It is understandable that before the electronic device obtains the AWB command, the RGB image may have been obtained through the RGB camera. As shown in Figure 8b, the user has already displayed the RGB images of both parties during the video call (the RGB image of the other party is The peer end transmits to the local end), but the TOF camera may not be used. After the electronic device obtains the AWB command, take Figure 8b as an example, the user clicks the AWB button in the interface to trigger the AWB command, the electronic device uses the TOF camera to obtain the depth image, and at the same time still uses the RGB camera to obtain the RGB image. Subsequent steps of S61 can be referred to as shown in FIG. 6 , and will not be repeated here.
  • the smaller size of the processed image in addition to being able to be applied to real-time scenes, it can also reduce the limitation on the resolution of the TOF camera, such as the TOF image obtained by a TOF camera with a resolution of 8*8, can meet the demand.
  • the processing flow based on global light source information shown in FIG. 6 is called global AWB.
  • the local light source information of the RGB image that is, the light source information of each image block
  • the information is AWB, and this method of AWB in blocks is called local AWB.
  • FIG. 9 another image color processing method disclosed in the embodiment of the present application is executed after acquiring the local AWB instruction.
  • the commands of local AWB can be obtained through the user interface, as shown in Figure 10 and Figure 10b, which respectively display the global AWB (abbreviated as GAWB) and local AWB (abbreviated as LAWB) options, and through the operation on the interface, determine the execution of Figure 6 Global AWB as shown or local AWB as shown in Figure 9.
  • GAWB global AWB
  • LAWB local AWB
  • the local point cloud refers to the point cloud blocks divided by the global point cloud. It can be understood that the points in the global point cloud correspond to all pixels in the RGB image, and the points in the local point cloud correspond to some pixels in the RGB image.
  • the local light source estimation vector must be obtained.
  • the point cloud block can be input into the light source estimation model to obtain the light source estimation vector of the RGB image block corresponding to the point cloud block. .
  • the RGB image and the depth image are divided into blocks first to obtain registered RGB image blocks and depth image blocks, and then the registered RGB image blocks and depth image blocks are used to generate local point cloud blocks. It can be understood that the parameters of each registered RGB image block and depth image block are sequentially input according to formula (3), and each point in each point cloud block is obtained.
  • the registered RGB image and depth image are used to generate a global point cloud, and then the global point cloud is divided into local point cloud.
  • the division of the RGB image, the depth image, or the global point cloud can be realized according to the depth value of each pixel in the depth image.
  • the output of the light source estimation model is the global light source estimation vector of the local point cloud, but for the entire RGB image or global point cloud, the output of the light source estimation model is the local light source estimation vector.
  • the RGB components of the pixels of each RGB image block are adjusted respectively, and for any RGB image block, the RGB components of the pixels in the image block are adjusted using the light source estimation vector obtained from the image block.
  • the local AWB method determines the "white point" in blocks and adjusts the color of pixels in blocks. It is a method of regional adjustment, which is closer to the function of the human eye.
  • Figure 11a and Figure 11b are examples of the effect of local AWB:
  • Figure 11a is the local AWB result of dividing the RGB image and the depth image into 5*5 image blocks
  • Figure 11b is the division of the RGB image and the depth image into 100*100 Local AWB results for image patches.
  • each image block estimates the "white point” and adjusts the color of other pixels according to its own “white point”, so in Fig. The ones are closer, and the area 2 is closer to white. It can be seen that the local AWB processing based on the local light source information is better than the global AWB processing.
  • the nearby wall in Figure 11b shows more color levels than the nearby wall in Figure 11a according to the distance from the light source, which is closer to the actual scene and closer to the perception of the human eye. to the effect.
  • the inventors further found that, in addition to performing local AWB using local light source information, the local light source information can also be used to provide other image processing functions, so as to improve user experience.
  • local light source estimation vectors can be used for relighting.
  • relighting refers to changing the color of the main light source without changing the direction of light and maintaining the relative color relationship between pixels in the image.
  • FIG. 12 shows another image color processing method disclosed in the embodiment of the present application, which is executed after the relighting instruction is acquired.
  • the relighting function can be packaged as an application program, and the relighting command is triggered by the user's operation.
  • the relighting function is packaged as a filter and displayed in an interface related to image processing.
  • buttons for various filters are displayed on the display interface of the camera application program, including buttons for resetting the light filter. After the user clicks the button of the relighting filter, the relighting function is started, and after the user clicks the shooting button, a relighting instruction is issued.
  • Figure 12 includes the following steps:
  • the color data refers to the data representing the color of the light source selected by the user, such as CCT. It can be understood that the color data can be obtained through the user interface.
  • FIG. 13a an image taken in the morning on a sunny day is displayed.
  • Various filter controls such as studio light are displayed in the interface.
  • the interface shown in Figure 13b is displayed.
  • the relighting effect includes sunrise and cloudy sky.
  • the image shown in Figure 13a is processed to the effect shown in Figure 13b.
  • a color card (not shown in Figure 13a and Figure 13b) can also be displayed, and the user can select the color value of the light source by clicking the color in the color card, or display the color Value input interface (not shown in Fig. 13a and Fig. 13b), etc.
  • the relighting method described in this embodiment can preserve the relative relationship of pixel colors, which is a pixel-level transformation.
  • the subsequent processing needs to use the relative relationship between pixels in the original image, after relighting
  • the images can be directly used for subsequent processing.
  • the AWB function is enabled in the interface as an example.
  • the global AWB or The local AWB that is, the method of turning on or off the global AWB or the local AWB is not limited in the embodiment of the present application.
  • the above embodiments all take RGB images as an example for illustration, but as long as the image with color information, that is, the color image can provide color information for the point cloud, so the embodiments of the present application are not limited to RGB images, and can also be YUV images, etc.
  • RGB images can be replaced by grayscale images (including black and white images).
  • the depth data collected by the TOF camera can also be used directly.
  • the depth data represents the distance from the camera to the pixel in the RGB image or grayscale image.
  • the embodiments of the present application also provide an electronic device, including: a memory and one or more processors.
  • the memory is used to store an application program, and one or more processors are used to run the application program to implement the image color processing method described in the above-mentioned embodiments.
  • the embodiment of the present application also discloses a computer-readable storage medium, on which a program is stored, and when the computer device runs the application program, the image color processing method described in the above-mentioned embodiments is realized.
  • the embodiment of the present application also discloses a computer program product.
  • the computer program product When the computer program product is run on a computer, the computer is made to execute the image color processing method described in the above embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)

Abstract

本申请提供一种图像颜色的处理方法，应用于包括第一摄像头和第二摄像头的电子设备，所述方法包括：获取第一图像和深度图像，所述第一图像为所述第一摄像头所采集的图像，所述深度图像为所述第二摄像头所采集的图像，所述深度图像用于指示部分或者全部的所述第一图像的深度信息。基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，所述第一光源信息包括所述第一图像的白点值，基于所述第一光源信息，对所述第一图像的进行处理。因为将深度图像指示的深度信息作为光源信息的确定依据，所以，能够在确定光源信息时，考虑到不同位置的"白点"对于像素的影响程度不同，有利于提高光源信息的准确性，从而改善图像处理的效果。

Description

一种图像颜色的处理方法及装置
本申请要求于2021年11月15日提交中国专利局、申请号为202111350182.1、发明名称为“图像颜色的处理方法及装置”以及于2021年11月22日提交中国专利局、申请号为202111387638.1、发明名称为“一种图像颜色的处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及电子信息技术领域，尤其涉及一种图像颜色的处理方法及装置。
背景技术
图像处理是电子设备常见的功能，可以用在拍照、视频等场景中。图像处理包括图像颜色的处理。
图像颜色的处理效果直接关系到用户的感受，如何改善图像颜色的处理效果，是目前的研究方向之一。
发明内容
本申请提供了一种图像颜色的处理方法及装置，目的在于解决如何改善图像颜色的处理效果的问题。
为了实现上述目的，本申请提供了以下技术方案：
本申请的第一方面提供一种图像颜色的处理方法，应用于电子设备，所述电子设备包括第一摄像头和第二摄像头，所述方法包括：获取第一图像和深度图像，所述第一图像为所述第一摄像头所采集的图像，所述深度图像为所述第二摄像头所采集的图像，所述深度图像用于指示部分或者全部的所述第一图像的深度信息。基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，所述第一光源信息包括所述第一图像的白点值，基于所述第一光源信息，对所述第一图像的进行处理。因为将深度图像指示的深度信息作为光源信息的确定依据，所以，能够在确定光源信息时，考虑到不同位置的“白点”对于像素的影响程度不同，有利于提高光源信息的准确性，从而改善图像处理的效果。
可选的，在本申请的第一方面中，所述基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，基于所述第一光源信息，对所述第一图像的进行处理的实现方式包括：基于所述第一图像和所述深度图像，确定第二图像的第二光源信息和第三图像的第三光源信息，所述第一图像包括所述第二图像和所述第三图像，所述第二光源信息包括所述第二图像的白点值，所述第三光源信息包括所述第三图像的白点值。基于所述第二光源信息对所述第二图像进行处理，基于所述第三光源信息对所述第三图像进行处理。将第一图像至少划分为第二图像和第三图像，分别确定第二图像和第三图像的光源向量，再分别依据各自的光源向量，处理第二图像和第三图像，这种将图像分块处理的方式，更接近人眼的感受，从而进一步改善图像处理的效果。
可选的，在本申请的第一方面中，所述基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息的实现方式包括：基于所述第一图像和所述深度图像，生成点云， 所述点云中的任意一个点具有位置信息和颜色信息。基于所述点云，确定所述第一图像的所述第一光源信息。点云能够很好地将颜色信息与深度信息融合，为基于颜色信息和深度信息确定光源信息奠定基础。
可选的，在本申请的第一方面中，所述基于所述第一图像和所述深度图像，生成点云的方式包括：基于作为所述点云中的任意一点的目标点在所述第一图像中对应像素的坐标、在所述深度图像中对应像素的深度值、以及所述第二摄像头的参数，生成所述目标点的二维位置信息。将所述二维位置信息以及所述深度值，作为所述目标点在所述点云中的位置信息。将所述目标点所述第一图像中对应像素的颜色信息，作为所述目标点在所述点云中的颜色信息。
可选的，在本申请的第一方面中，所述基于所述点云，确定所述第一图像的所述第一光源信息，包括：将所述点云输入光源估计模型，得到所述光源输入模型输出的光源估计向量，其中，所述光源输入模型使用样本点云以及扩增的样本点云训练得到，所述扩增的样本点云从相机视角以及光照强度的至少一个维度，对所述样本点云扩增得到。扩增的样本点云丰富了训练数据，有利于提高光源估计模型的准确性。
可选的，在本申请的第一方面中，所述点云包括：与所述第一图像中的部分像素一一对应的点的局部点云。所述基于所述第一图像和所述深度图像，确定第二图像的第二光源信息和第三图像的第三光源信息，包括：基于与所述第二图像对应的局部点云，确定所述第二图像的所述第二光源信息，基于与所述第三图像对应的局部点云，确定所述第三图像的所述第三光源信息。局部点云为将第一图像分块确定光源信息，并分块进行处理奠定了基础。
可选的，在本申请的第一方面中，所述第一摄像头与所述第二摄像头为同一个摄像头。
可选的，在本申请的第一方面中，所述第一图像包括：RGB图像。
可选的，在本申请的第一方面中，所述对所述第一图像进行处理，包括：对所述第一图像进行白平衡处理，或者，对所述第一图像进行重打光。因为光源信息的确定考虑了光源像素与其它像素的相对位置信息，所以，光源信息更为准确，从而白平衡的效果更优。并且，实验证明处理速度更快，且，所需的图像更小。重打光处理能够保留像素的颜色的相对关系。
可选的，在本申请的第一方面中，所述获取第一图像和深度图像，包括：响应于用户界面的自动白平衡或重打光的触发操作，获取所述第一图像和所述深度图像，便于用户操作。
可选的，在本申请的第一方面中，所述用户界面包括：实时业务界面，便于提高实时业务的图像质量。
本申请的第二方面提供一种图像颜色的处理方法，应用于电子设备，所述电子设备包括第一摄像头和第二摄像头，所述方法包括：获取第一图像和深度信息，所述第一图像为所述第一摄像头所采集的图像，所述深度信息通过所述第二摄像头获取，所述深度信息为部分或全部的所述第一图像的深度信息。基于所述第一图像和所述深度信息，确定所述第一图像的第一光源信息，所述第一光源信息包括所述第一图像的白点值，基于所述第一光源信息，对所述第一图像进行处理。因为将深度信息作为光源信息的确定依据，所以，能 够在确定光源信息时，考虑到不同位置的“白点”对于像素的影响程度不同，有利于提高光源信息的准确性，从而改善图像处理的效果。
可选的，在本申请的第二方面中，所述基于所述第一图像和所述深度信息，确定所述第一图像的第一光源信息，基于所述第一光源信息，对所述第一图像的进行处理的实现方式包括：基于所述第一图像和所述深度信息，确定第二图像的第二光源信息和第三图像的第三光源信息，所述第一图像包括所述第二图像和所述第三图像，所述第二光源信息包括所述第二图像的白点值，所述第三光源信息包括所述第三图像的白点值。基于所述第二光源信息对所述第二图像进行处理，基于所述第三光源信息对所述第三图像进行处理。将第一图像至少划分为第二图像和第三图像，分别确定第二图像和第三图像的光源向量，再分别依据各自的光源向量，处理第二图像和第三图像，这种将图像分块处理的方式，更接近人眼的感受，从而进一步改善图像处理的效果。
可选的，在本申请的第二方面中，所述基于所述第一图像和所述深度信息，确定所述第一图像的第一光源信息的实现方式包括：基于所述第一图像和所述深度信息，生成点云，所述点云中的任意一个点具有位置信息和颜色信息。基于所述点云，确定所述第一图像的所述第一光源信息。点云能够很好地将颜色信息与深度信息融合，为基于颜色信息和深度信息确定光源信息奠定基础。
可选的，在本申请的第二方面中，所述基于所述第一图像和所述深度信息，生成点云的方式包括：基于作为所述点云中的任意一点的目标点在所述第一图像中对应像素的坐标、深度值、以及所述第二摄像头的参数，生成所述目标点的二维位置信息。将所述二维位置信息以及所述深度值，作为所述目标点在所述点云中的位置信息。将所述目标点所述第一图像中对应像素的颜色信息，作为所述目标点在所述点云中的颜色信息。
可选的，在本申请的第二方面中，所述基于所述点云，确定所述第一图像的所述第一光源信息，包括：将所述点云输入光源估计模型，得到所述光源输入模型输出的光源估计向量，其中，所述光源输入模型使用样本点云以及扩增的样本点云训练得到，所述扩增的样本点云从相机视角以及光照强度的至少一个维度，对所述样本点云扩增得到。扩增的样本点云丰富了训练数据，有利于提高光源估计模型的准确性。
可选的，在本申请的第而方面中，所述点云包括：与所述第一图像中的部分像素一一对应的点的局部点云。所述基于所述第一图像和所述深度信息，确定第二图像的第二光源信息和第三图像的第三光源信息，包括：基于与所述第二图像对应的局部点云，确定所述第二图像的所述第二光源信息，基于与所述第三图像对应的局部点云，确定所述第三图像的所述第三光源信息。局部点云为将第一图像分块确定光源信息，并分块进行处理奠定了基础。
可选的，在本申请的第二方面中，所述第一摄像头与所述第二摄像头为同一个摄像头。
可选的，在本申请的第二方面中，所述第一图像包括：RGB图像。
可选的，在本申请的第二方面中，所述对所述第一图像进行处理，包括：对所述第一图像进行白平衡处理，或者，对所述第一图像进行重打光。因为光源信息的确定考虑了光源像素与其它像素的相对位置信息，所以，光源信息更为准确，从而白平衡的效果更优。并且，实验证明处理速度更快，且，所需的图像更小。重打光处理能够保留像素的颜色的 相对关系。
可选的，在本申请的第二面中，所述获取第一图像和深度信息，包括：响应于用户界面的自动白平衡或重打光的触发操作，获取所述第一图像和所述深度信息，便于用户操作。
可选的，在本申请的第二方面中，所述用户界面包括：实时业务界面，便于提高实时业务的图像质量。
本申请的第三方面提供一种电子设备，包括：存储器，用于存储应用程序。一个或多个处理器，用于运行所述应用程序，以实现本申请的第一方面或第二方面所述的图像颜色的处理方法。
本申请的第四方面提供一种计算机可读存储介质，其上存储有程序，在计算机设备运行所述应用程序时，实现本申请的第一方面或第二方面所述的图像颜色的处理方法。
本申请的第五方面提供一种计算机程序产品，当计算机程序产品在计算机上运行时，使得所述计算机执行本申请的第一方面或第二方面所述的图像颜色的处理方法。
附图说明
图1为包括AWB的相机界面的示例图；
图2为现有的AWB的效果示例图；
图3为本申请提供的电子设备的硬件结构图；
图4为本申请提供的电子设备的软件架构图；
图5为本申请实施例提供的对光源估计模型训练的流程图；
图6为本申请实施例提供的一种图像颜色的处理方法的流程图；
图7为本申请实施例提供的全局AWB方法的效果图；
图8a为本申请实施例提供的图像颜色的处理方法的应用场景示例图；
图8b为本申请实施例提供的图像颜色的处理方法应用在实时场景的示例图；
图9为本申请实施例提供的又一种图像颜色的处理方法的流程图；
图10a和图10b均为本申请实施例提供的图像颜色的处理方法的应用场景示例图；
图11a为本申请实施例提供的局部AWB方法的效果图；
图11b为本申请实施例提供的局部AWB方法的又一效果图；
图12为本申请实施例提供的又一种图像颜色的处理方法的流程图；
图13a为本申请实施例提供的重打光之前的示例图；
图13b为本申请实施例提供的重打光的效果示例图。
具体实施方式
图像的颜色处理包括但不限于自动白平衡(automatic white balance，AWB)以及重打光(Relighting)。
AWB的目的在于，减小实体成像环境中的光源对实体成像颜色的影响。AWB的思想为：从图像中确定光源信息，俗称为“白点”，再使用“白点”的颜色调整图像中的其它像素的颜色，以去除“白点”的颜色对其它像素的颜色的影响。
AWB是电子设备具备的常见的图像处理功能，以图1所示的相机的拍摄界面为例，假 设AWB已被开启，则相机的拍摄界面显示的拍摄图像，为AWB后的图像，如图2所示。
但目前的AWB的效果还有待改善，还以图2为例：
图2中主要的光源有两处，近处的墙壁上的区域1的实际颜色为白色，远处的墙壁上的区域2的实际颜色也为白色，但在光源的影响下，成像后并不是白色，AWB的目的之一为将墙壁的颜色还原为白色。
经AWB后，图2中区域1的相关色温(Correlated Color Temperature，CCT)为7000k，区域2的CCT为4000k。可以看出，同样的白色在图2中呈现出的色温差距较大，说明颜色相差较大，因此至少有一个区域的颜色与白色相差较大。所以AWB的效果还有改善的空间。
发明人在研究的过程中发现，现有的AWB的效果不佳的原因在于，没有考虑到不同位置的“白点”对其它像素的影响程度不同这一因素，例如图2中，近处发光的灯实际对区域1的影响大于对区域2的影响，远处的灯光实际对区域2的影响大于对区域1的影响。并且，近处的灯光因为距离相机较近，所以对于整个图像的影响大于远处的灯光。但现有的AWB，将近处和远处的灯光对其它像素的影响程度认作相同，并且依据两处光源的颜色均值确定“白点”的颜色，再使用“白点”的颜色调整其它像素的颜色。
基于上述原因，发明人进一步发现：如将图像中各个像素在实际空间中的位置关系作为约束，有利于改善AWB的效果。
并且，深度图像能够表达出图像中各个像素在实际的3D空间中的位置关系，而飞行时间(time-of-flight，TOF)相机在电子设备上的配置越来越普遍，因此，能够利用TOF相机采集的深度图像，改善AWB的效果。
结合上述问题以及发现，本申请的实施例提供一种图像颜色的处理方法，使用RGB图像以深度图像，获取RGB图像中的光源信息(即“白点”)，并将获取的光源信息作为依据，对图像的颜色进行调整，以获得比现有的AWB更准确的光源信息，从而处理后的图像相对于现有图像更接近真实的颜色。
重打光是指不改变光线的照射方向，并维持图像中像素间的相对颜色关系，而改变主体光源的颜色。图像中的光源信息，也是重打光的依据，本申请实施例公开的图像颜色的处理方法，能够使用更准确的光源信息实现重打光。
下面先对一些参数的含义进行澄清，再分别对图像的颜色处理的两种具体实现方式：AWB和重打光，分别进行说明。
需要说明的是，本申请实施例提供的方法，适用但并不限于多个光源下成像的图像，且可以基于全局光源信息和局部光源信息的至少一项，进行图像颜色的处理。
一张RGB图像的光照模式如式(1)所示：
Figure PCTCN2022117564-appb-000001
其中，ω i为光的入射角度，光来自于上半球曲面Ω+，ω 0是光照的观察角度，在图像中体现为RGB相机拍摄时光线进入RGB相机的反射角度，N是表面的法向量，E表示光谱能量分布，S是表面反射系数，R rgb是RGB传感器的响应系数。
基于式(1)，在本申请的实施例中，一张RGB图像的全局光源信息为：
Figure PCTCN2022117564-appb-000002
基于式(2)，全局光源信息可以理解为，影响图像中的全部像素的光源的信息。
将一张RGB图像划分为多个图像块，每一个图像块的E，称为一张RGB图像的局部光源信息。
需要说明的是，图2中以CCT来表示图像的颜色的原因如下：
在辐射作用下既不反射也不完全投射，而能把落在它上面的辐射全部吸收的物体称为黑体或完全辐射体。当黑体连续加热时，它的相对光谱功率分布的最大值将向短波方向移动，相应的光色将按照红、黄、白、蓝的顺序进行变化，在不同温度下，黑体对应的光色变化在色度图上形成的弧形轨迹，叫做黑体轨迹或普朗克轨迹。
CCT是指与具有相同亮度刺激的颜色最相近的黑体辐射体的温度，用k氏温度表示，用于描述位于普朗克轨迹附近的光的颜色的度量。
除热辐射光源以外的其它光源具有线状光谱，其辐射特性与黑体辐射特性差别较大，所以这些光源的光色在色度图上不一定准确地落在黑体轨迹上，对这样一类光源，通常用CCT来描述光源的颜色特性。
AWB涉及的参数包括CCT和色度距离D uv，但对于本申请实施例提供的图像而言，D uv可以忽略，所以不再赘述。
因此，在下文中的图像示例中，仍使用CCT表示颜色。并且，以业内公认的白色的CCT之一的6500k作为白色的CCT。
本申请实施例提供的图像颜色的处理方法，应用在电子设备。在一些实施例中，电子设备可以是手机、平板电脑、桌面型、膝上型、笔记本电脑、超级移动个人计算机(Ultra-mobile Personal Computer，UMPC)、手持计算机、上网本、个人数字助理(Personal Digital Assistant，PDA)、可穿戴电子设备、智能手表等设备。
图3为电子设备的结构的示例，包括：RGB相机1、TOF相机2，处理器3、存储器4、以及I/O子系统5。
RGB相机1用于采集RGB数据。RGB相机1可以被设置为前置相机或后置相机。RGB相机1包括但不限于RGB传感器11(可称为第一摄像头)以及RGB传感器控制器12(可称为第二摄像头)。
TOF相机2用于采集用于生成深度图像的TOF数据。TOF相机2可以被设置为前置相机或后置相机。可以理解的是，本实施例所述的电子设备，因为RGB图像和深度图像需要配准，所以为了降低计算复杂度，RGB相机1和TOF相机2在电子设备上的位置较为集中。
TOF相机2包括但不限于TOF传感器21、TOF传感器控制器22、TOF光源23以及TOF光源控制器24。
在某些实施方式中，TOF光源控制器24受TOF传感器控制器22的控制，实现对TOF光源23的控制。TOF光源23在TOF光源控制器24的控制下，发射红外光或激光。TOF传感器21用于感应发射的光在物体反射的光线，以采集TOF数据。
RGB传感器控制器12、TOF传感器控制器22、以及TOF光源控制器24设置在I/O子系 统5中，通过I/O子系统5与处理器3通信。
存储器4用于存储计算机可执行的程序代码。具体的，存储器4可以包括程序存储区和数据存储区。其中，程序存储区可存储用于实现操作系统、软件系统、至少一个功能所需的程序代码等。数据存储区可存储电子设备使用过程中所获取、生成以及使用的数据等。
处理器3可以包括一个或多个处理单元，例如：处理器3可以包括应用处理器(application processor，AP)，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)等。
可以理解的是，本实施例示意的结构并不构成对电子设备的具体限定。在另一些实施例中，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件、软件或软件和硬件的组合实现。例如，还可以，第一摄像头和第二摄像头为同一个摄像头，即摄像头即采集RGB数据又采集深度信息。
可以理解的是，处理器3通过运行存储器4中存储的代码，实现的操作系统可以为iOS操作系统、Android开源操作系统、Windows操作系统等。在以下实施例中，将以Android开源操作系统为例进行说明。
在一些实施例中，如图4所示，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，硬件抽象层，以及内核层。
应用程序层可以包括一系列应用程序。如图4所示，在本申请实施例中，与AWB以及重打光相关的应用程序的示例包括相机和视频通话等。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
硬件抽象层(HAL)，或称为安卓运行时(Android Runtime)，负责安卓系统的调度和管理，HAL可以配置功能函数或核心库，用于实现本实施例所述的基于光源信息的图像处理方法。
内核层是硬件和软件之间的层。在本申请实施例中，内核层至少包含RGB相机驱动，以及TOF相机驱动等。各个驱动用于经获取的硬件的数据进行处理，并将处理结果上报至硬件抽象层的相应模块。
下面将对应用在上述软件以及硬件框架的图像颜色的处理方法进行详细说明。
如前所述，本申请的实施例使用RGB图像和深度图像，获取RGB图像中的光源信息，但需要强调的是，深度图像表示的是图像中的像素的深度信息，仅仅引入深度图像，并不能直接提升光源信息的准确性。
发明人在研究的过程中发现，颜色信息与深度信息相结合得到的信息，与光源信息之间存在映射关系，且这一映射关系能够通过神经网络模型被很好地表达。
所述模型称为光源估计模型，光源估计模型通过学习样本点云集与样本点云集的标签之间的映射关系，实现依据点云输出光源估计向量的功能。
需要说明的是，本实施例所述的点云，不仅具有位置信息，还具有颜色信息。即点云中的每个点具有位置信息和颜色信息。
图5为对光源估计模型的训练过程，包括以下步骤：
S51、获取配准的样本RGB图像和样本深度图像。
可以理解的是，样本RGB图像可以使用电子设备中的RGB相机获得，样本深度图像可以使用电子设备中的TOF相机获得，样本RGB图像和样本深度图像还可以来自训练图像集。
为了便于获取标签，本实施例中的样本RGB图像中均包括色卡的成像区域。
S52、依据配准的样本RGB图像和样本深度图像，生成样本点云集。
样本点云集为具有N个点云的集合，记为
Figure PCTCN2022117564-appb-000003
本实施例中生成的点云，除了具有位置信息之外，还具有颜色信息，这里将样本点云集中的任意一个点云表示为P＝{p i|i∈1,......n}，p i为点云中的任意一个点，可被表示为具有六维信息(x,y,z,r,g,b)的向量。
在某些实现方式中，点云中的任意一个点向量的生成规则为：
Figure PCTCN2022117564-appb-000004
式(3)中，u和v为p i在RGB图像中对应的像素的坐标，d为p i在深度图像中对应的像素的深度值，fx，fy为TOF相机的焦距转换至像素坐标系下的分量，c x和c y为TOF相机的光心转换至像素坐标系下的分量，r为p i在RGB图像中R分量，g为p i在RGB图像中G分量，b为p i在RGB图像中B分量。可以理解的是，因为本实施例为训练过程，所以，本实施例中，式(3)中的参数涉及的RGB图像为样本RGB图像，深度图像为样本深度图像，生成的点为样本点云集中的点。
可以理解的是，因为样本RGB图像与样本深度图像配准，所以，样本RGB图像与样本深度图像中的像素点具有一一对应关系，因为后续要使用样本RGB图像和样本深度图像估计样本RGB图像中的光源向量，所以，样本RGB图像与样本深度图像中的像素点对应，不仅是指像素坐标系中的坐标对应，还包括对应相同的成像实体，即对应的像素点表示成像实体中相同的位置点。而基于式(3)可知，点云中的点，分别与样本RGB图像以及样本深度图像中的像素点具有一一对应关系，因此，点云中的任意一点，与样本RGB图像或样本深度图像中对应的像素点，表示成像实体中相同的位置点。
综上所述，与样本RGB图像相比，样本点云增加了位置信息，与样本深度图像相比，样本点云增加了颜色信息，所以，本实施例中获得的样本点云结合了样本RGB图像和样本深度图像的信息，既具有位置信息又具有颜色信息。
可以理解的是，为了增加样本点云的数量，以使得光源估计模型具有足够的训练样本，可以对样本点云进行扩增。
在一些实现方式中，因为在连续区域旋转相机坐标，并不改变主体光源的色度，所以设置属于正交群的旋转矩阵R∈SO(3)，表示相机对于同一对象的不同拍摄视角，将样本点 云中的位置数据与各个旋转矩阵R相乘(颜色数据不变)，得到不同视角的样本点云，实现对于样本点云的扩增。
在另一些实现方式中，依据一个对象的光照变化可由该对象与光源的相对位置的改变而发生改变，并且，在同一光源下，照度的改变并不影响光源颜色的改变的原理，设置光照强度系数L，将样本点云中的颜色数据与L相乘(位置数据不变)，实现对于样本点云的扩增。
可以理解的是，可以既进行拍摄角度的扩增，又进行光照强度的扩增，即如式(4)所示：
Figure PCTCN2022117564-appb-000005
其中，P'表示扩增后的样本点云，P pos表示扩增前的样本点云中的位置数据，P rgb表示扩增前的样本点云中的颜色数据。L为属于一维高斯分布
Figure PCTCN2022117564-appb-000006
的常数，
Figure PCTCN2022117564-appb-000007
表示维度层面的链接。
S53、将样本点云输入光源估计模型，得到光源估计模型输出的光源估计向量。
在某些实现方式中，可以使用pointnet对样本点云逐点提取特征，并依据对每一个点的特征的加权信息，获得光源估计向量。
可以理解的是，本步骤中获得是RGB样本图像中的全局光源估计向量，即对于一张RGB图像，输出一个光源估计向量，作为这一张RGB图像的光源信息。其中，光源估计向量可以理解为光源的颜色，也可以理解为该RGB图像的“白点”值。
可以理解的是，为了适应于光源估计模型对于输入数据的格式的要求，在执行S53之前，还可以，转换样本点云的格式，例如，重构样本点云的维度。
S54、使用样本点云的标签、光源估计模型输出的光源估计向量以及损失函数，调整光源估计模型的参数。
这里将样本点云集的标签记为ε＝{E 1,E 2，......E M}。任意一个样本点云的标签的获取方式可以为：在样本RGB图像中识别色卡区域，并依据色卡区域中的预先指定的参考色块的颜色值，确定样本RGB图像生成的样本点云的光源估计向量，例如参考色块的颜色值，作为样本点云的标签。
图5所示的训练方法，使得光源估计模型学习到具有位置信息和颜色信息的点云与光源信息之间的映射关系，为改善基于光源信息的图像处理方法的效果，奠定基础。因为在颜色信息的基础上增加了位置信息，所以有利于模型学习到，从每个点之间的相互位置关系和颜色信息，确定空间中的光源分布，进一步确定全局光源的能力，从而提高光源信息的准确性。并且，进一步的，样本点云的扩增方式，能够获得足够的样本点云，从而获得更好的训练效果。
基于训练得到的光源估计模型，本申请实施例提供的图像颜色的处理方法的流程如图6所示，该流程侧重对AWB的说明。图6中包括以下步骤：
S61、通过RGB相机获取RGB图像，并通过TOF相机获取深度图像。
可以理解的是，S61可以由用户的操作等方式触发执行，例如，用户在图1所示的界面点击拍摄按键。
可以理解的是，在某些实现方式中，通过预先配置在电子设备中的RGB相机和TOF相机的参数、以及世界坐标系、相机坐标系、像素坐标系之间的转换矩阵，配准RGB图像和深度图像，这里不再赘述。以下步骤所述的RGB图像和深度图像为配准的RGB图像和深度图像。
S62、使用RGB图像和深度图像，生成全局点云。
全局点云是指，对应RGB图像中的全部像素点的点云。
点云的生成规则可以参见式(3)，这里不再赘述。
可以理解的是，本实施例中，点云中的点与RGB图像中的像素一一对应，且与深度图像中的像素一一对应。RGB图像中的像素与深度图像中的像素一一对应。对应的具体含义如前所述。
S63、将全局点云输入光源估计模型，得到光源估计模型输出的全局光源估计向量。
可以理解的是，因为输入的是全局点云，所以得到的是S61中获取的RGB图像的全局光源估计向量。
可以理解的是，光源向量包括RGB图像的白点值。如前所述，白点值可以理解为光源的颜色值。
S64、依据全局光源估计向量，调整RGB图像中其它像素的RGB值。
可以理解的是，不同位置的光源对于像素的影响程度不同，而深度信息能够表示图像中的光源与其它像素点的位置关系，或者，深度信息能够表示图像中的各个物体的位置关系，或者深度信息能够表示图像中的光源与图像中的各个物体的位置关系，所以，图6所示的流程，深度信息的引入，有利于依据光源的影响程度确定图像中的“白点”(即光源估计向量)，从而获得更为准确的“白点”。
图6所示的流程得到的处理后的图像的示例如图7所示：
图7中，区域1的CCT为6500k，区域2的CCT接近3500k，对比图2和图7可以看出，图7中的区域1和区域2均更接近白色，选择对全局影响大的光源。
从图2以及图7中的两处光源的位置可以看出，因为近处的灯光距离镜头更近，所以对于图像中的像素的颜色的影响更大，远处的灯光对于图像中的像素的颜色的影响较小，图6所示的流程，能够将这种影响程度的差别引入“白点”的计算，即近处的灯光在“白点”计算中的权重更大，所以，使用“白点”调整后的图像，与图2相比，墙壁的颜色都更接近白色。
表1中列举了图6所示的流程与现有的其它白平衡调整算法，在相同的数据集执行的效果对比数据：
表1
Figure PCTCN2022117564-appb-000008
Figure PCTCN2022117564-appb-000009
表1中，现有算法1-5表示现有的AWB算法。本发明表示图6所示的AWB算法。其中，Points是指RGB图像具有的像素点的数量。
角度误差Angular error是指以算法估计的光源向量与样本光源向量之间的夹角的余弦值，用于度量算法估计的光源向量与样本光源向量之间的差异。以现有算法1为例，Angular error是指现有算法1估计的光源向量与样本光源向量之间的夹角的余弦值。对于任意一张RGB图像，样本光源向量是指RGB图像中光源向量的标签，即前述样本点云的标签，样本光源向量的获得方法可以参见前述样本点云的标签的获得方式。
均值Mean表示在多张(例如500张)RGB图像执行算法得到的角度误差的均值，以现有算法1为例，在多张RGB图像上执行现有算法1得到多个角度误差，Mean表示多个角度误差的均值。
中间值Median表示在多张RGB图像执行算法得到的角度误差的中间值，三均值Tri.表示在多张RGB图像执行算法得到的角度误差的三均值，最优B25％表示在多张RGB图像执行算法得到的角度误差中，数值最小(最好)的25％的角度误差的均值，最差W25％表示在多张RGB图像执行算法得到的角度误差中，数值最大(最差)的25％的角度误差的均值。
训练长度是指本发明所述的方法，训练光源估计模型所需的时长。
从角度误差的各个参数可以看出，本实施例所述的流程的效果优于现有算法。并且，本发明(256Points,w/o depth)是指不考虑深度信息的算法得到的参数，可以看出，不考虑深度信息的情况下，效果明显变差。
并且，从本发明的几行数据还可以看出，像素数量越多，效果越优，但16points的效果已经优于现有算法，因此，本实施例所述的方法适用在小尺寸的图像上，效果也优于现有算法。
本实施例所述的流程，可以通过用户交互界面上的AWB控件触发执行，如图8a所示，在用户触发AWB控件使得AWB处于开启状态后，电子设备对相机拍摄的图像进行图6所示的处理流程，得到图7所示的图像。
测试时长是实际部署算法后处理一张图像的时长，更有实际意义。训练时长是算法部署前在服务器离线的模型训练时长。
从表1中还可以看出，图6所示的流程除了具有更优的AWB效果之外，还具有更快的处理速度。
发明人在研究的过程中还发现，现有的AWB算法，因为需要更大的图像，所以，受限于硬件平台，很难实现对白平衡的实时调整，而图6所示的流程，因为适用于更小的图像以及具有更优的处理速度，因此能够实现对白平衡的实时调整。
因此，可以理解的是，图6所示的流程，除了应用于图1所示的场景外，还可以应用在实时场景，以图8b所示的视频通话场景为例，在视频通话过程中，用户可以在通话界面开启AWB功能，实现调整通话视频的白平衡的功能，从而能够为用户提供更优的通话画质。
在实时场景下，在某些实现方式中，电子设备在获取到AWB指令后，执行S61。可以理解的是，电子设备在获取到AWB指令之前，有可能已经通过RGB相机获取RGB图像，如图8b所示，用户在视频通话过程中，已经显示通话双方的RGB图像(对方的RGB图像为对端传输至本端)，但TOF相机可以不被使用。在电子设备获取到AWB指令后，还以图8b为例，用户点击界面中的AWB按键，以触发AWB指令，电子设备使用TOF相机获取深度图像，并同时仍使用RGB相机获取RGB图像。S61的后续步骤，可参见图6所示，这里不再赘述。
进一步的，得益于处理的图像的尺寸更小，除了能够应用于实时场景之外，也能够降低对于TOF相机的分辨率的限制，例如分辨率为8*8的TOF相机获得的TOF图像，即可满足需求。
上述实施例重点对于基于全局光源信息的图像颜色的处理进行说明，本申请的实施例中，将图6所示的基于全局光源信息的处理流程称为全局AWB。可以理解的是，在多光源的情况下，还可以基于训练得到的模型，获得RGB图像的局部光源信息，即每个图像块的光源信息，并对于每一个图像块，使用该图像块的光源信息进行AWB，这种分块进行AWB的方式，称为局部AWB。
如图9所示，本申请实施例公开的又一种图像颜色的处理方法，在获取局部AWB的指令后执行。
局部AWB的指令可以通过用户交互界面获得，如图10和图10b所示，分别显示全局AWB(简称为GAWB)和局部AWB(简称为LAWB)选项，并通过界面上的操作，确定执行图6所示的全局AWB或图9所示的局部AWB。
图9中包括以下步骤：
S91、获取RGB图像以及深度图像。
具体实现方式可参见S61。
S92、使用RGB图像和深度图像，获取局部点云。
局部点云是指全局点云划分的点云块。可以理解的是，全局点云中的点对应了RGB图像中的全部像素，局部点云中的点对应的是RGB图像中的部分像素。
因为要对RGB图像进行局部AWB，所以，要获得局部的光源估计向量，而基于光源估计模型的功能，可以将点云块输入光源估计模型，得到点云块对应的RGB图像块的光源估计向量。
在一些实现方式中，先对RGB图像和深度图像分块，得到配准的RGB图像块和深度图像块，再使用配准的RGB图像块和深度图像块，生成局部点云块。可以理解的是，依次将配准的各个RGB图像块和深度图像块的参数输入按照式(3)，得到各个点云块中的各点。
在另一些实现方式中，使用配准的RGB图像和深度图像，生成全局点云，再按照RGB图像块的划分方式，并依据全局点云的各个点的位置信息，将全局点云划分为局部点云。
可以理解的是，可以依据深度图像中各像素的深度值，实现对于RGB图像、深度图像， 或者全局点云的划分。
S93、依次将局部点云输入光源估计模型，得到光源估计模型输出的各个局部点云的光源估计向量。
可以理解的是，对于任意一个局部点云，光源估计模型输出的为该局部点云的全局光源估计向量，但对于整个RGB图像或全局点云，光源估计模型输出的为局部光源估计向量。
S94、依据各个局部点云的光源估计向量，对局部点云对应的RGB图像块调整像素的RGB分量。
也就是说，分别调整各个RGB图像块的像素的RGB分量，对于任意一个RGB图像块，使用该图像块获得的光源估计向量，调整该图像块中的像素的RGB分量。
与全局AWB方式不同的是，局部AWB方式分块确定“白点”并且分块调整像素的颜色，是一种分区域调整的方式，与人眼的功能更为接近。
图11a和图11b均为局部AWB的效果示例：图11a为将RGB图像和深度图像划分为5*5的图像块的局部AWB结果，图11b为将RGB图像和深度图像划分为100*100的图像块的局部AWB结果。
因为每个图像块均估计“白点”，并依据各自的“白点”调整其它像素的颜色，所以图11a与图7相比，区域1的CCT为6500k，区域2的CCT为6300k，两者更加接近，并且区域2更加接近白色，可见，依据局部光源信息进行的局部AWB处理，比全局AWB处理的效果更优。
图11b与图11a相比，区域1的CCT为6500k，区域2的CCT为6500k。可见，对图像或点云的分块的数量越多，局部AWB的效果越优。
进一步的，从视觉感应到的颜色的角度，对于近处的墙壁，因为区域3更接近近处的灯光，区域4比区域3更远离灯光，所以，区域3比区域4的颜色更接近灯光的颜色，区域3的颜色更接近白色。也就是说，图11b中的近处的墙壁，比图11a中近处的墙壁，按照与光源的距离，体现出更多的颜色层次，从而与实际场景更为贴近，也更加接近人眼感受到的效果。
由此可见，对图像或点云的分块的数量越多，局部AWB的效果越优，但占用的计算资源越过，并结合表1所示，耗时也越多，因此，实际中，能够依据效果与耗时以及所占资源的需求，选择分块的数量。
发明人进一步发现，除了可以利用局部光源信息进行局部AWB之外，还可以利用局部光源信息提供其它图像处理功能，以提升用户的使用体验。
在某些实现方式中，可以利用局部光源估计向量，进行重打光(Relighting)。如前所述，重打光是指不改变光线的照射方向，并维持图像中像素间的相对颜色关系，而改变主体光源的颜色。
图12所示为本申请实施例公开的又一种图像颜色的处理方法，在获取重打光指令后执行。重打光功能可以被封装为应用程序，由用户的操作触发重打光指令。
以图13为例，重打光功能被封装为滤镜，显示在与图像处理相关的界面中。图13中，拍照应用程序的显示界面中显示各种滤镜的按键，其中包括重打光滤镜的按键。在用户点 击重打光滤镜的按键后，启动重打光功能，并在用户点击拍摄按键后，发出重打光指令。
图12中包括以下步骤：
S121、获取RGB图像以及深度图像。
具体实现方式可以参见S61。
S122、使用RGB图像和深度图像，获取局部点云。
具体实现方式可参见S92。
S123、依次将局部点云输入光源估计模型，得到光源估计模型输出的各个局部点云的光源估计向量。
具体实现方式可参见S93。
S124、依据各个局部点云的光源估计向量以及获得的颜色数据，对RGB图像进行重打光。
颜色数据是指表示用户选定的光源的颜色的数据，例如CCT，可以理解的是，颜色数据可以通过用户界面获得。
例如，在图13a所示的界面中，显示的是晴天早上拍摄的图像。界面中显示有影棚光等各种滤镜控件。在用户选择重打光控件后，显示图13b所示的界面，这里假设重打光效果包括日出和阴天，在用户选择重打光滤镜并进一步选择阴天后，图13a所示的图像被处理为图13b所示的效果。
可以理解的是，除了通过控件选择重打光之外，还可以显示色卡(图13a和图13b中未画出)，用户可以通过点击色卡中的颜色选择光源的颜色值，或者显示颜色值的输入界面(图13a和图13b中未画出)等。
需要说明的是，虽然图13a和图13b中，重打光作为滤镜的一项展示给用户，但从图12所示的流程可以看出，重打光调整的是图像中的像素点的RGB数值，而非像其它现有的滤镜，例如剧场光等，在图像上增加具有剧场光效果的“掩膜”，实现滤镜功能，这种增加“掩膜”的方式，会改变像素之间的颜色相对关系。
所以，本实施例所述的重打光的方式，可以保留像素的颜色的相对关系，是一种像素级别的变换，在后续处理需要使用原始图像中像素间的相对关系时，重打光后的图像可以直接用于后续的处理。
可以理解的是，以上图例中，均以在界面中开启AWB功能为例，除此之外，还可以，在电子设备的配置界面中，开启或关闭本申请实施例中所述的全局AWB或局部AWB，即本申请实施例中不对全局AWB或局部AWB的开启或关闭的方式进行限定。
并且，以上实施例均以RGB图像为例进行说明，但只要具有颜色信息的图像，即彩色图像即可为点云提供颜色信息，所以，本申请的实施例并不限于RGB图像，还可以是YUV图像等。
可以理解的是，本申请的实施例也并不限于彩色图像，RGB图像可以被替换为灰度图像(包括黑白图像)。
基于TOF相机的原理，本申请的实施例中，并不限于获取到深度图像，还可以直接使用TOF相机采集的深度数据，深度数据表示RGB图像或灰度图像中的像素，距离摄像头的 距离。
可以理解的是，除了上述实施例所述的电子设备的结构之外，本申请的实施例还提供了一种电子设备，包括：存储器以及一个或多个处理器。存储器用于存储应用程序，一个或多个处理器，用于运行所述应用程序，以实现上述实施例所述的图像颜色的处理方法。
本申请的实施例还公开了一种计算机可读存储介质，其上存储有程序，在计算机设备运行所述应用程序时，实现上述实施例所述的图像颜色的处理方法。
本申请的实施例还公开了一种计算机程序产品，当计算机程序产品在计算机上运行时，使得所述计算机执行上述实施例所述的图像颜色的处理方法。

Claims (15)

  1. 一种图像颜色的处理方法，应用于电子设备，所述电子设备包括第一摄像头和第二摄像头，其特征在于，包括：
    获取第一图像和深度图像，所述第一图像为所述第一摄像头所采集的图像，所述深度图像为所述第二摄像头所采集的图像，所述深度图像用于指示部分或者全部的所述第一图像的深度信息；
    基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，所述第一光源信息包括所述第一图像的白点值；
    基于所述第一光源信息，对所述第一图像进行处理。
  2. 根据权利要求1所述的方法，其特征在于，所述基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，基于所述第一光源信息，对所述第一图像进行处理，包括：
    基于所述第一图像和所述深度图像，确定第二图像的第二光源信息和第三图像的第三光源信息，所述第一图像包括所述第二图像和所述第三图像，所述第二光源信息包括所述第二图像的白点值，所述第三光源信息包括所述第三图像的白点值；
    基于所述第二光源信息对所述第二图像进行处理，基于所述第三光源信息对所述第三图像进行处理。
  3. 根据权利要求1或2所述的方法，其特征在于，所述基于所述第一图像和所述深度图像，确定所述第一图像的第一光源信息，包括：
    基于所述第一图像和所述深度图像，生成点云，所述点云中的任意一个点具有位置信息和颜色信息；
    基于所述点云，确定所述第一图像的所述第一光源信息。
  4. 根据权利要求3所述的方法，其特征在于，所述基于所述第一图像和所述深度图像，生成点云，包括：
    基于目标点在所述第一图像中对应像素的坐标、在所述深度图像中对应像素的深度值、以及所述第二摄像头的参数，生成所述目标点的二维位置信息；所述目标点为所述点云中的任意一点；
    将所述二维位置信息以及所述深度值，作为所述目标点在所述点云中的位置信息；
    将所述目标点所述第一图像中对应像素的颜色信息，作为所述目标点在所述点云中的颜色信息。
  5. 根据权利要求3或4所述的方法，其特征在于，所述基于所述点云，确定所述第一图像的所述第一光源信息，包括：
    将所述点云输入光源估计模型，得到所述光源输入模型输出的光源估计向量；
    所述光源输入模型使用样本点云以及扩增的样本点云训练得到，所述扩增的样本点云从相机视角以及光照强度的至少一个维度，对所述样本点云扩增得到。
  6. 根据权利要求2-5任一项所述的方法，其特征在于，所述点云包括：
    局部点云；所述局部点云为包括与所述第一图像中的部分像素一一对应的点的点云；
    所述基于所述第一图像和所述深度图像，确定第二图像的第二光源信息和第三图像的第三光源信息，包括：
    基于与所述第二图像对应的局部点云，确定所述第二图像的所述第二光源信息；
    基于与所述第三图像对应的局部点云，确定所述第三图像的所述第三光源信息。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，所述第一摄像头与所述第二摄像头为同一个摄像头。
  8. 根据权利要求1-7任一项所述的方法，其特征在于，所述第一图像包括：
    RGB图像。
  9. 根据权利要求1-8任一项所述的方法，其特征在于，所述对所述第一图像进行处理，包括：
    对所述第一图像进行白平衡处理，或者，对所述第一图像进行重打光。
  10. 根据权利要求1-9任一项所述的方法，其特征在于，所述获取第一图像和深度图像，包括：
    响应于用户界面的自动白平衡或重打光的触发操作，获取所述第一图像和所述深度图像。
  11. 根据权利要求10所述的方法，其特征在于，所述用户界面包括：
    实时业务界面。
  12. 一种图像颜色的处理方法，应用于电子设备，所述电子设备包括第一摄像头和第二摄像头，其特征在于，包括：
    获取第一图像和深度信息，所述第一图像为所述第一摄像头所采集的图像，所述深度信息通过所述第二摄像头获取，所述深度信息为部分或全部的所述第一图像的深度信息；
    基于所述第一图像和所述深度信息，确定所述第一图像的第一光源信息，所述第一光源信息包括所述第一图像的白点值；
    基于所述第一光源信息，对所述第一图像进行处理。
  13. 一种电子设备，其特征在于，包括：
    存储器，用于存储应用程序；
    一个或多个处理器，用于运行所述应用程序，以实现权利要求1-12任一项所述的图像颜色的处理方法。
  14. 一种计算机可读存储介质，其上存储有程序，其特征在于，在计算机设备运行所述应用程序时，实现权利要求1-12任一项所述的图像颜色的处理方法。
  15. 一种计算机程序产品，其特征在于，当计算机程序产品在计算机上运行时，使得所述计算机执行权利要求1-12任一项所述的图像颜色的处理方法。
PCT/CN2022/117564 2021-11-15 2022-09-07 一种图像颜色的处理方法及装置 Ceased WO2023082811A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111350182 2021-11-15
CN202111350182.1 2021-11-15
CN202111387638.1 2021-11-22
CN202111387638.1A CN116152360B (zh) 2021-11-15 2021-11-22 一种图像颜色的处理方法及装置

Publications (1)

Publication Number Publication Date
WO2023082811A1 true WO2023082811A1 (zh) 2023-05-19

Family

ID=86335063

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/117564 Ceased WO2023082811A1 (zh) 2021-11-15 2022-09-07 一种图像颜色的处理方法及装置

Country Status (1)

Country Link
WO (1) WO2023082811A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120807592A (zh) * 2025-09-12 2025-10-17 先临三维科技股份有限公司 图像处理方法、扫描设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170116777A1 (en) * 2015-10-21 2017-04-27 Samsung Electronics Co., Ltd. Image processing method and apparatus
US20170171523A1 (en) * 2015-12-10 2017-06-15 Motorola Mobility Llc Assisted Auto White Balance
EP3358846A1 (en) * 2017-02-06 2018-08-08 Robo-Team Home Ltd. Method and device for stereoscopic vision
CN108876833A (zh) * 2018-03-29 2018-11-23 北京旷视科技有限公司 图像处理方法、图像处理装置和计算机可读存储介质
CN113518210A (zh) * 2020-04-10 2021-10-19 华为技术有限公司 图像自动白平衡的方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170116777A1 (en) * 2015-10-21 2017-04-27 Samsung Electronics Co., Ltd. Image processing method and apparatus
US20170171523A1 (en) * 2015-12-10 2017-06-15 Motorola Mobility Llc Assisted Auto White Balance
EP3358846A1 (en) * 2017-02-06 2018-08-08 Robo-Team Home Ltd. Method and device for stereoscopic vision
CN108876833A (zh) * 2018-03-29 2018-11-23 北京旷视科技有限公司 图像处理方法、图像处理装置和计算机可读存储介质
CN113518210A (zh) * 2020-04-10 2021-10-19 华为技术有限公司 图像自动白平衡的方法及装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120807592A (zh) * 2025-09-12 2025-10-17 先临三维科技股份有限公司 图像处理方法、扫描设备及存储介质

Similar Documents

Publication Publication Date Title
US20230050695A1 (en) Systems and methods for capturing digital images
CN109565551B (zh) 对齐于参考帧合成图像
US10554943B2 (en) Systems and methods for digital photography
JP6864449B2 (ja) イメージの明るさを調整する方法及び装置
US9406147B2 (en) Color balance in digital photography
CN111526351B (zh) 白平衡同步方法、系统、电子设备、介质及数字成像设备
CN108616700B (zh) 图像处理方法和装置、电子设备、计算机可读存储介质
US11699218B2 (en) Method controlling image sensor parameters
CN112866667A (zh) 图像的白平衡处理方法、装置、电子设备和存储介质
JP2017138927A (ja) 画像処理装置、撮像装置およびそれらの制御方法、それらのプログラム
JP2015090562A (ja) 画像処理装置、方法、及びプログラム
CN116152360B (zh) 一种图像颜色的处理方法及装置
US10621769B2 (en) Simplified lighting compositing
CN112995633B (zh) 图像的白平衡处理方法、装置、电子设备和存储介质
JP2020177619A (ja) 赤外線カメラによる対話型画像処理システム
CN118975264A (zh) 对原始传感器图像执行颜色变换的设备和方法
CN113766206A (zh) 一种白平衡调整方法、装置及存储介质
US9996969B2 (en) Dynamically creating and presenting a three-dimensional (3D) view of a scene by combining color, brightness, and intensity from multiple scan data sources
CN112995634B (zh) 图像的白平衡处理方法、装置、电子设备和存储介质
US12356083B2 (en) Techniques for correcting images in flash photography
JP2016218663A (ja) 照明光色推定装置、照明光色推定方法及び照明光色推定プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22891616

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22891616

Country of ref document: EP

Kind code of ref document: A1