WO2022227996A1 - 图像处理方法、装置、电子设备以及可读存储介质 - Google Patents

图像处理方法、装置、电子设备以及可读存储介质 Download PDF

Info

Publication number
WO2022227996A1
WO2022227996A1 PCT/CN2022/083822 CN2022083822W WO2022227996A1 WO 2022227996 A1 WO2022227996 A1 WO 2022227996A1 CN 2022083822 W CN2022083822 W CN 2022083822W WO 2022227996 A1 WO2022227996 A1 WO 2022227996A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature information
image
feature
target
normal
Prior art date
Application number
PCT/CN2022/083822
Other languages
English (en)
French (fr)
Inventor
王光伟
Original Assignee
北京字跳网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字跳网络技术有限公司 filed Critical 北京字跳网络技术有限公司
Priority to US18/551,392 priority Critical patent/US20240177409A1/en
Publication of WO2022227996A1 publication Critical patent/WO2022227996A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects
    • G06T15/506—Illumination models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the embodiments of the present disclosure relate to the technical field of image processing, and in particular, to an image processing method, an apparatus, an electronic device, and a readable storage medium.
  • Relighting technology is a technology that changes the lighting when an image is taken to obtain a new image.
  • the face relighting technique targets the face in the image, that is, the face is relighted.
  • Relighting technology is widely used in post-processing of images or post-production of movies.
  • the user can process the face in the image by using the re-lighting function in the image retouching software to change the light and shadow effect of the face.
  • the image processing apparatus can use illumination from a certain angle to render the object based on the normal information of the object, and the rendering effect is single.
  • Embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, and a readable storage medium, which can enrich the lighting rendering effect of an image.
  • an embodiment of the present disclosure provides an image processing method, including: acquiring an image to be processed, the image to be processed includes a target object; using a feature model to acquire normal feature information and target feature information of the target object , the target feature information includes: depth feature information and/or tangent feature information; based on the normal feature information and the target feature information, perform lighting rendering on the target object in the to-be-processed image, and obtain after lighting rendering image; output the image rendered by the lighting.
  • an image processing apparatus including:
  • the processing module is used to obtain an image to be processed, and the image to be processed includes a target object; adopt a feature model to obtain normal feature information and target feature information of the target object, and the target feature information includes: depth feature information and /or tangent feature information; based on the normal feature information and the target feature information, perform illumination rendering on the target object in the to-be-processed image to obtain an image after illumination rendering.
  • An output module configured to output the image rendered by the lighting.
  • embodiments of the present disclosure provide an electronic device, including: a processor and a memory;
  • the memory stores computer-executable instructions
  • the processor executes the computer-implemented instructions stored in the memory to cause the processor to perform the image processing method as described in the first aspect and various possible designs of the first aspect above.
  • embodiments of the present disclosure provide a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, the first aspect and the first Aspects of various possible designs of the image processing method described.
  • an embodiment of the present disclosure provides a computer program product, including a computer program or an instruction.
  • the computer program or the instruction is executed by a processor, the above first aspect and various possible designs of the first aspect are implemented.
  • the described image processing method is described in detail below.
  • an embodiment of the present disclosure provides a computer program that, when executed by a processor, executes the image processing method described in the first aspect and various possible designs of the first aspect.
  • the present embodiment provides an image processing method, apparatus, electronic device, and readable storage medium.
  • the method includes: acquiring an image to be processed, and the to-be-processed image includes a target object; using a feature model to acquire normal feature information of the target object and Target feature information, the target feature information includes: depth feature information and/or tangent feature information; based on normal feature information and target feature information, perform lighting rendering on the target object in the image to be processed, and obtain a lighting rendering image; output lighting rendering post image.
  • the terminal device not only obtains the normal feature information of the target object, but also can obtain the depth feature information and/or tangent feature information of the target object. It is related to the distribution of the light formed by the illumination on the object. Therefore, compared with the prior art, the terminal device can perform illumination rendering on the target object based on richer feature information of the target object, which can enrich the illumination rendering effect and improve the user experience.
  • FIG. 1 is a schematic flowchart 1 of an image processing method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic diagram of a scenario where the image processing method provided by the embodiment of the present disclosure is applied;
  • FIG. 3 is a schematic diagram 1 of a training process of a feature model provided by an embodiment of the present disclosure
  • FIG. 4 is a schematic structural diagram of an initial model provided by an embodiment of the present disclosure.
  • FIG. 5 is a second schematic diagram of a training process of a feature model provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of a sample image processing provided by an embodiment of the present disclosure.
  • FIG. 7 is a second schematic flowchart of an image processing method provided by an embodiment of the present disclosure.
  • FIG. 8 is a structural block diagram of an image processing apparatus provided by an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the embodiments of the present disclosure provide the following solution idea: when performing relighting processing on an object in an image, obtain normal feature information and target feature information of the object in the image,
  • the target feature information may include depth feature information and/or tangent feature information (tangent).
  • the normal feature information is related to the angle of illumination (such as a directional light source)
  • the depth feature information is related to the distance between the light and the object (or understood that the depth feature information is related to the light intensity of the light reaching the object)
  • the tangent feature information is related to the light on the object.
  • the distribution of the rays formed on it is related.
  • different lighting renderings for objects in an image can be implemented based on richer feature information of objects, such as rendering objects with different angles, different intensities and/or different lights. Distribution and other rendering, and then achieve the purpose of enriching the rendering effect of objects.
  • the objects in the image may be, but are not limited to, people, animals, and household appliances, such as tableware. It should be understood that the image, the to-be-processed image and the sample image involved in the following embodiments may be pictures or video frames in a video.
  • the embodiment of the present disclosure provides a scenario in which the image processing method is applied.
  • a user may perform lighting rendering on an object in an image through a terminal device.
  • the user can open a photo retouching application in the terminal device, select an image to be processed in the retouching application, the to-be-processed image can be displayed on the interface of the terminal device, and the to-be-processed image includes the user's face. and the user's hair.
  • a "light rendering” control may be displayed on the interface of the terminal device, and the user clicks on the "light rendering” control, and the terminal device may execute the image processing method provided by the embodiment of the present disclosure, so that the objects in the image to be processed (the user's face and the The user's hair) performs lighting rendering to obtain the image after lighting rendering.
  • lighting rendering may be performed from the side of the user with a lighting intensity of a.
  • the user's hair presents a light distribution in the shape of water ripples. It should be understood that the water ripple shape of the user's hair is merely an example.
  • a "light rendering" control may be displayed on the terminal device first, and after the user selects an image, the terminal device may perform lighting rendering on the objects in the image.
  • other image editing controls may also exist on the interface of the above-mentioned terminal device, and the manner in which the user triggers the terminal device to perform lighting rendering is not limited in the embodiments of the present disclosure.
  • the image processing method provided by the embodiment of the present disclosure can be applied to post-processing scenarios of various types of videos (eg, film and television works), for example, a terminal device can use a video frame in a video as an image to be processed , and then the objects in the image to be processed are illuminated and rendered.
  • the terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, tablet computers, notebook computers, speakers, wearable devices, smart screens, smart home appliances, internet of things (IoT) devices, camera devices, etc.
  • a device with image processing capabilities may also be a personal digital assistant (PDA), a handheld device with a wireless communication function, a computing device, a virtual reality (virtual reality, VR) terminal device, a drone device, an augmented reality (augmented reality (AR) terminal equipment, wireless terminals in industrial control, wireless terminals in smart homes, etc.
  • PDA personal digital assistant
  • a handheld device with a wireless communication function a computing device
  • a virtual reality (virtual reality, VR) terminal device a drone device
  • an augmented reality (augmented reality (AR) terminal equipment wireless terminals in industrial control, wireless terminals in smart homes, etc.
  • the form of the terminal device is not limited in the embodiments of the present disclosure.
  • FIG. 1 is a schematic flowchart 1 of an image processing method provided by an embodiment of the present disclosure.
  • the method of this embodiment may be applied to a terminal device, and the image processing method may include:
  • S201 Acquire a to-be-processed image, where the to-be-processed image includes a target object.
  • the image to be processed may be a local image stored in the terminal device.
  • the user can select at least one local image in the terminal device, and the terminal device uses the local image selected by the user as the image to be processed, as shown in interface 101 .
  • the image to be processed may be an image captured by a terminal device, or the image to be processed may be an image imported into the terminal device.
  • the image to be processed may be an image sent by another electronic device to the terminal device, that is, the other electronic device needs to perform illumination rendering on the image, and then sends the image to the terminal device for illumination rendering.
  • the image may be a picture, a group of pictures, or a video. Each video frame in the video can be thought of as an image.
  • the target object is included in the image to be processed.
  • the target objects can be, but are not limited to, people, animals, items, etc.
  • the items can be stainless steel tableware, ceramic tableware, plastic bags, and the like.
  • the target object is a person as an example for description.
  • S202 Using a feature model, acquire normal feature information and target feature information of the target object, where the target feature information includes: depth feature information and/or tangent feature information.
  • the feature model is used to extract normal feature information and target feature information of the target object in the image to be processed.
  • the target feature information includes: depth feature information and/or tangent feature information.
  • the normal feature information may be a normal map, a normal matrix, or a normal feature vector of the target object.
  • the depth feature information may be a depth map, a depth matrix, or a depth feature vector of the target object.
  • the tangent feature information may be a tangent map, a tangent matrix, or a tangent feature vector of the target object.
  • the feature model can be obtained by pre-training with sample data, and then the feature model is preset in the terminal device, so that the terminal device can use the feature model to obtain the normal feature information and the normal feature information of the target object in the image. target feature information.
  • the execution subject for training the feature model may be a terminal device or other electronic device.
  • the embodiment of the present disclosure does not limit the execution body of the feature model obtained by training.
  • the sample data may include a sample image and normal feature information of the sample image.
  • the sample data may include a sample image, normal feature information of a sample object in the sample image, and sample target feature information.
  • the sample target feature information may include sample depth feature information and/or sample tangent feature information. It should be noted that the type of sample target feature information included in the sample data is the same as the type of target feature information output based on the feature model. Exemplarily, if the sample target feature information included in the sample data is the sample tangent feature information, the target feature information output based on the feature model is also the tangent feature information.
  • the terminal device may input the image to be processed into a feature model, and the feature model may output normal feature information and target feature information of the target object in the image to be processed.
  • the use process of the feature model is described first, and for the specific training process of the feature model, reference may be made to the following related descriptions in FIG. 3 and FIG. 4 .
  • the types of the target object and the sample object may be the same. If the target object is a person, the sample object may be a person. Exemplarily, if the target object is a person with hair, the sample object may include a person with hair.
  • the terminal device can only perform a certain angle on the target based on the normal feature information. lighting rendering.
  • the terminal device can provide the target object with point light sources at different distances based on the depth feature information. lighting rendering. Because point light sources at different distances from the target object have different illumination intensities reaching the target object, it can also be understood that the terminal device can provide illumination rendering with different illumination intensities for the target object.
  • the target object has anisotropy
  • dishes such as hair, hair, stainless steel items exhibit anisotropy.
  • the target object is a user with hair, and the user's face and body are generally isotropic, tangent feature information is not required, and lighting rendering can be performed correctly.
  • the user's hair is anisotropic, and the light distribution on the hair should be in the shape of water ripples. If the terminal device does not obtain the tangent feature information of the hair, the light distribution on the hair is the same as that on the user's face.
  • the light distribution is the same as the body's light distribution, which is in the shape of a circle, which is different from the light distribution of hair under lighting in real life, resulting in poor rendering of hair.
  • the terminal device can obtain the tangent feature information of the target object, and can perform accurate light distribution rendering for both isotropic and anisotropic objects, the rendering effect is good, and the performance of anisotropic objects is improved. Rendering of objects of the opposite sex.
  • the more information contained in the target feature information the better the lighting rendering effect.
  • S203 Based on the normal feature information and the target feature information, perform illumination rendering on the target object in the image to be processed, to obtain an image after illumination rendering.
  • the terminal device may determine the illumination angle based on the normal feature information.
  • the terminal device determines the light intensity (or the distance between the point light source and the target object) based on the depth feature information.
  • the terminal device determines, based on the tangent feature information, the distribution of the light formed by the light on the target object, for example, the distribution of the light formed by the light on the object may include: the distribution of the light formed by the light on the face of the user is: Aperture shape, the distribution of hair in the shape of water ripples.
  • the terminal device may perform illumination rendering on the target object based on the illumination angle, the illumination intensity, and the distribution of light rays formed by the illumination on the object.
  • the lighting rendering can be: the terminal device converts the layer of the light angle (such as the light from the side of the user) and the layer corresponding to the light intensity (the light intensity is a), and the layer corresponding to the distribution of light formed by the light on the object ( The water ripple shape) is superimposed with the target object in the image to be processed to obtain the image after lighting rendering.
  • the terminal device may display the image rendered by the illumination.
  • the terminal device may output the image after illumination rendering to another electronic device, and the other electronic device may be a device that requests the terminal device to perform illumination rendering on the image, such as a smart wearable device, as shown in picture 2.
  • FIG. 2 is a schematic diagram of a scenario where the image processing method provided by the embodiment of the present disclosure is applied. Wherein, after receiving the image rendered by illumination, the other electronic device may display the image rendered by illumination.
  • the image processing method provided by the embodiment of the present disclosure includes: a terminal device obtains an image to be processed, the image to be processed includes a target object, and a feature model is used to obtain normal feature information and target feature information of the target object, and the target feature information includes: depth features Information and/or tangent feature information, based on normal feature information and target feature information, perform illumination rendering on the target object in the image to be processed, obtain an illuminated rendered image, and output the illuminated rendered image.
  • the terminal device when performing illumination rendering on the target object in the image, not only obtains the normal feature information of the target object, but also obtains the depth feature information and/or tangent feature information of the target object, because the depth feature information
  • the information is related to the distance of the light from the object
  • the tangent feature information is related to the distribution of the light formed by the light on the object. Therefore, compared with the prior art, the terminal device can perform more detailed information on the target object based on the richer feature information of the target object.
  • rich lighting rendering for example, lighting rendering with different angles, different lighting intensities and/or different light distributions is performed on the target object, thereby enriching lighting rendering effects and improving user experience.
  • the execution body of the training feature model may be a server, and the execution body of the training feature model may also be a terminal device or an electronic device with training capability.
  • the embodiment of the present disclosure does not limit the execution body of the training feature model.
  • the method for obtaining the feature model by training may include:
  • S401 Input a sample image including a sample object into an initial model.
  • the initial model may be an initial framework for training the feature model.
  • the initial model can be but is not limited to: Unet neural network based on fully convolutional neural network (FNC) framework, lightweight neural network shufflenet, convolutional neural network (CNN). Sample images can be fed into the initial model for training.
  • FNC fully convolutional neural network
  • CNN convolutional neural network
  • the sample data obtained by training the feature model may also include: sample normal feature information of the sample object in the sample image.
  • the sample data includes: a sample image, and sample normal feature information and sample target feature information of a sample object in the sample image.
  • the sample target feature information may include sample depth feature information and/or sample tangent feature information. It should be understood that the sample normal feature information, the sample depth feature information, and the sample tangent feature information may be regarded as accurate feature information of the sample object, and may be referred to as labeling information of the features of the sample object.
  • the sample normal feature information may include a sample normal map
  • the sample depth feature information may include a sample depth map
  • the sample tangent feature information may include a sample tangent map.
  • the sample normal feature information, depth feature information, and tangent feature information may also include other types of representations, such as feature vectors, feature matrices, etc., which are not limited in the present disclosure.
  • the (normal/depth/tangent) feature information is used as a (normal/depth/tangent) map for description.
  • S402 Acquire normal feature information, depth feature information, and tangent feature information of the sample object output by the initial model.
  • the processing process of a sample image in training is described.
  • the initial model can output normal feature information, depth feature information and tangent feature information of the sample object in the sample image.
  • FIG. 4 is a schematic structural diagram of an initial model provided by an embodiment of the present disclosure.
  • the initial model includes an input layer, a hidden layer, and an output layer
  • the hidden layer includes a normal feature extraction block, a depth feature extraction block, and a tangent feature extraction block.
  • the input layer is used to input sample images to the normal feature extraction block, the depth feature extraction block and the tangent feature extraction block.
  • the input layer and the output layer may use the Unet neural network
  • the hidden layer may use the shufflenet neural network.
  • the normal feature extraction block, the depth feature extraction block, and the tangent feature extraction block can also be referred to as three branches in the initial model.
  • the normal feature extraction block is used to extract information related to the normal map in the sample object of the sample image, obtain the normal map based on the information related to the normal map, and output the normal map of the sample object through the output layer.
  • the normal feature extraction block is also used to output information related to the normal map to the depth feature extraction block and the tangent feature extraction block.
  • the depth feature extraction block is used to extract the information related to the depth map in the sample object of the sample image, and obtain the depth map according to the information related to the depth map and the information related to the normal map, and output the sample object through the output layer. depth map.
  • the tangent feature extraction block is used to extract the information related to the tangent map in the sample object of the sample image, and obtain the tangent map according to the information related to the tangent map and the information related to the normal map, and output the sample object through the output layer.
  • the information related to the normal map may be normal features of the sample object, such as normal vectors.
  • the information related to the depth map may be the depth feature of the sample object, and the information related to the tangent map may be the tangent feature of the sample object.
  • the normal map of the sample object can be output. Because the accuracy of the normal map is high, in the embodiment of the present disclosure, the depth map and the tangent map can be obtained by combining the information related to the preset normal map, thereby improving the accuracy of the depth map and the tangent map.
  • S403 Obtain a first loss function of normal feature information, a second loss function of depth feature information, and a third loss function of tangent feature information.
  • the first loss function of the normal feature information and the second loss function of the depth feature information can be obtained.
  • the third loss function of the tangent feature information to update the weight between the input layer and the hidden layer in the initial model, as well as the weight between the hidden layer and the output layer, so that the normal map output by the initial model, Depth maps and tangent maps are more accurate.
  • the loss function represents the error between the feature information obtained by the initial model and the labeling information. The larger the loss function, the greater the error and the less accurate the training result of the initial model. Wherein, when the error between the feature information obtained by the initial model and the annotation information is less than the error threshold, the training result of the initial model is accurate.
  • the first loss function may be obtained based on the normal map and the sample normal map.
  • L1 is adopted.
  • Loss function formula obtain the first loss function based on the normal map and the sample normal map. Because there is no sample depth map and sample tangent map of the sample object in the sample data, the normal feature information can be obtained according to the depth map, and then the normal map corresponding to the depth map can be obtained, and then the L1 loss function formula can be used, based on the sample normal map.
  • the second loss function is obtained from the normal map corresponding to the depth map.
  • the tangent graph and the normal graph can be dot multiplied to obtain the dot multiplication result, and then the L1 loss function formula can be used to obtain the third loss function based on the dot multiplication result and 0.
  • the depth map is used to represent the depth of each pixel of the sample object.
  • the depth gradient of the pixel can be obtained based on the depth of the pixel and the depths of the pixels around the pixel, and the depth gradient of the pixel can be obtained based on the pixel.
  • the depth gradient of obtains the normal feature information of the pixel, such as the normal vector.
  • a normal map of the sample object that is, a normal map corresponding to the depth map of the sample object, can be obtained based on the normal feature information of each pixel of the sample object.
  • a binary image of the sample object may be obtained based on the depth map of the sample object, and the grayscale of each pixel on the binary image is proportional to the depth of the pixel.
  • the grayscale gradient of the pixel can be obtained based on the grayscale of the pixel and the grayscale of the pixels around the pixel, and the grayscale gradient of the pixel can be obtained based on the grayscale of the pixel.
  • a normal map of the sample object can be obtained based on the normal feature information of each pixel of the sample object. This embodiment of the present application does not limit the manner of obtaining the normal map of the sample object based on the depth map of the sample object.
  • the above method can be used to obtain the first loss function and the second loss function respectively. and the third loss function.
  • the L1 loss function formula can also be used to obtain the second loss function based on the depth map and the sample depth map, or the L1 loss function formula can be used to obtain the third loss function based on the tangent map and the sample tangent map.
  • the weight between each neuron in the input layer in the initial module and each neuron in the normal feature extraction block, and each neuron in the normal feature extraction block may be updated based on the first loss function
  • the weights between the neurons and each neuron in the output layer Based on the second loss function, the weights between each neuron in the input layer and each neuron in the depth feature extraction block, and between each neuron in the depth feature extraction block and each neuron in the output layer can be updated. the weight of.
  • the weights between each neuron in the input layer and each neuron in the tangent feature extraction block, and between each neuron in the tangent feature extraction block and each neuron in the output layer can be updated. the weight of.
  • the weights between the neurons in the initial model may be updated by means of back propagation.
  • the initial model can be continued to be trained based on the updated initial model and sample data, until the number of training times reaches the preset number of training times, and the training is terminated to obtain a feature model.
  • the first loss function output by the updated initial model (or initial model) is smaller than the first preset value
  • the second loss function is smaller than the second preset value
  • the third loss function is smaller than the third If the preset value is set, the training ends and the trained feature model is obtained.
  • a feature model can be obtained by training, and the feature model can be preset in the terminal device.
  • the sample images may be preprocessed, and the preprocessed sample images may be input to the initial model for training, so as to obtain a feature model with higher prediction accuracy.
  • S401 may include:
  • S405 Based on the sample image, obtain a mask image of the sample object in the sample image.
  • the mask image (mask) of the sample object which can be a binary image.
  • the pixel point of the sample object in the sample image can be set to 1
  • the pixel point of the non-sample object can be set to 0, that is, the sample object in the sample image appears white, and the non-sample image appears black.
  • the mask image of the sample object may be a grayscale image
  • the grayscale of the pixel point of the sample object may be 255
  • the grayscale of the pixel point of the non-sample object may be 0.
  • the mask image of the sample object is not limited in the embodiment of the present disclosure, as long as the pixel points of the sample object and the pixel points of the non-sample object in the sample image can be effectively distinguished.
  • the sample object in the sample image can be identified, and then the pixel point of the sample object in the sample image is set to 1, and the pixel point of the non-sample object is set to 0, so as to obtain the mask image of the sample object .
  • the mask image of the sample object is obtained by adopting a portrait matting technique.
  • FIG. 6 is a schematic diagram of a sample image processing provided by an embodiment of the present disclosure.
  • the sample image is an image A including the user's face and the user's hair, and the mask image of the sample image may be as shown in B in FIG. 6 .
  • the image A also includes a background part, such as the sun.
  • S406 Based on the sample image, obtain a mask image of a target portion of the sample object, where the target portion has anisotropy.
  • the target portion may be the entire area or a partial area of the sample object.
  • the target part has anisotropy, exemplarily, the sample object is the user (including the user's face and the user's hair), and the target part of the sample object can be the user's hair, wherein the hair has anisotropy, and correspondingly, it can be Based on image A, a mask image of the user's hair is obtained, as shown in D in FIG. 6 .
  • the mask image of the target portion may be pre-acquired or pre-set, which is not limited in the present disclosure.
  • a mask image of a target portion of the sample object may be used as sample data.
  • the size of the mask image of the sample object is the same as the size of the sample image, it is possible to convert the sample image into the sample image based on the correspondence between the mask image of the sample object and the pixels of the sample image Pixel points other than the sample object are adjusted to the preset value.
  • the pixels other than the user in the sample image can be adjusted to 0, so that the user in the sample image appears in color, except for the user.
  • the pixels of appear in black, as shown by C in Figure 6. It should be understood that the foreground portion of C in Figure 6 is actually colored, although it is represented in black and white in Figure 6.
  • S408 The mask image of the sample object and the processed sample image are cascaded and input to the initial model.
  • the sample image may be a color image.
  • the mask image of the sample object when the mask image of the sample object is obtained by segmenting the sample image, due to the influence of external factors such as light, the obtained mask image of the sample object also contains some non-identical objects.
  • the pixel point of the sample object In order to improve the prediction accuracy of the feature model, in this embodiment of the present disclosure, the mask image of the sample object and the processed sample image may be concatenated, and the mask image of the concatenated sample object and the processed sample image may be concatenated.
  • the resulting sample images are input to the initial model.
  • the processed sample image is obtained based on the mask image of the sample object
  • the mask image of the sample object also contains some non-sample object pixels
  • the foreground part of the processed sample image includes some pixels in the background area (ie, pixels of non-sample objects)
  • the sample object of the processed sample image is colored, and contains more information than the foreground part of the mask image of the sample object. Therefore, in the embodiment of the present disclosure, the mask image of the sample object and the processed sample image are cascaded and input to the initial model, so that in the process of obtaining the feature model through training, the sample object can be more accurately identified in The boundary of the sample image, thereby improving the prediction accuracy of the feature model.
  • the sample data may further include a sample tangent diagram of the target portion of the sample object.
  • the user s mask image and the processed sample image can be cascaded and input to the initial model.
  • the third loss function can be obtained as follows:
  • the calculation of the third loss function may be performed only for the target part.
  • the target part of the user is hair
  • the calculation of the third loss function can be performed for the user's hair.
  • the tangent map of the hair can be obtained from the user's tangent map output by the initial model.
  • the part of the tangent map corresponding to the pixel points of the hair may be obtained in the user's tangent map, that is, the tangent map of the hair.
  • the L1 loss function formula can be used to obtain the third loss function of the user's hair based on the tangent graph of the hair and the sample tangent graph of the hair, as shown in FIG. 6 .
  • the tangent graph of the target part of the sample object can be used to perform dot product, and then the L1 loss function formula can be used, based on The dot product result and 0 get the third loss function for the target part of the sample object.
  • the normal map of the target portion can be obtained from the normal map of the sample object based on the mask image of the target portion.
  • the sample image can be preprocessed, and the mask image of the sample object and the processed sample image are cascaded and input to the initial model to obtain a feature model through training, which can improve the The prediction accuracy of the feature model.
  • FIG. 7 is a second schematic flowchart of an image processing method provided by an embodiment of the present disclosure.
  • the image processing method provided by the embodiment of the present disclosure may include:
  • S801 Acquire a to-be-processed image, where the to-be-processed image includes a target object.
  • S802 Preprocess the image to be processed to obtain a processed image, and the pixels in the processed image other than the target object are preset values.
  • the terminal device may obtain a mask image of the target object based on the image to be processed, and then adjust the pixel points in the image to be processed other than the target object to a preset value based on the mask image. Based on the relevant description of the preprocessing of the sample image when the feature model is obtained by training, in this embodiment of the present disclosure, the terminal device can use the same method to preprocess the image to be processed, and for details, refer to the above relevant description.
  • the terminal device preprocesses the image to be processed, and the pixels in the processed image other than the target object are preset values, then the feature model is used to obtain the normal feature of the target object.
  • the feature model can completely focus on the target object in the foreground part, thereby improving the accuracy of the acquired normal feature information and target feature information of the target object.
  • the image to be processed may be as shown in A in FIG. 6 , the target object is a user, and the target part of the target object is the user's hair.
  • the terminal device may acquire the user's mask image B, and refer to the relevant description of S406 above.
  • the terminal device may, based on the user's mask image B, adjust the pixels in the image to be processed except the user to 0, that is, adjust the part of the image to be processed except the user to be black, thereby obtaining the processed image C.
  • S803 Input the processed image into the feature model to obtain normal feature information and target feature information.
  • the terminal device may input the processed image into the feature model, and then obtain normal feature information and target feature information of the target object output by the feature model.
  • the normal feature information may be a normal map, or a normal vector or a normal matrix, etc. The following description takes the normal feature information as a normal map as an example for description.
  • the feature model may include: an input layer, a hidden layer and an output layer.
  • the input layer is used to input the image to be processed to the hidden layer.
  • the hidden layer is used to obtain normal feature information and target feature information of the target object in the to-be-processed image based on the to-be-processed image, and output the normal feature information and target feature information of the target object through the output layer.
  • the hidden layer includes a normal feature extraction block and a target feature extraction block.
  • the normal feature extraction block is used to extract the information related to the normal map in the target object of the image to be processed, and obtain the normal map based on the information related to the normal map, and output the normal map of the target object through the output layer .
  • the normal feature extraction block is also used to output information related to the normal map to the target feature extraction block.
  • the target feature extraction block is used to extract the information related to the target feature information in the target object of the image to be processed, and obtain the target feature information based on the information related to the target feature information and the information related to the normal feature information, and pass the output layer. Output target feature information.
  • the target feature extraction block may include a depth feature extraction block and a tangent feature extraction block.
  • the input layer can be used to input the image to be processed to the normal feature extraction block, the depth feature extraction block and the tangent feature extraction block.
  • the normal feature extraction block is used to extract the information related to the normal map in the target object of the image to be processed, and obtain the normal map based on the information related to the normal map, and output the normal map of the target object through the output layer .
  • the normal feature extraction block is also used to output information related to the normal map to the depth feature extraction block and the tangent feature extraction block.
  • the depth feature extraction block is used to extract the information related to the depth map in the target object of the image to be processed, and obtain the depth map according to the information related to the depth map and the information related to the normal map, and output the target object through the output layer depth map.
  • the tangent feature extraction block is used to extract the information related to the tangent map in the target object of the image to be processed, and obtain the tangent map according to the information related to the tangent map and the information related to the normal map, and output the target object through the output layer tangent graph.
  • the information related to the normal map may be the normal feature of the target object, such as a normal vector, a normal matrix, and the like.
  • the information related to the depth map may be the depth feature of the target object, and the information related to the tangent map may be the tangent feature of the target object.
  • "information related to normal map” in S803 may be replaced with "information related to normal feature information”, and "information related to depth map” and “information related to tangent map” may be Replace with “information related to target feature information”.
  • “information related to depth map” can be replaced with “information related to depth feature information”
  • “information related to tangent map” can be replaced with "information related to tangent feature information”.
  • the terminal device inputs the processed image into the feature model, and can extract the information related to the target object and the normal feature information, as well as the information related to the target object and the target feature information, and then obtain the information based on the information related to the normal feature information.
  • the target feature information is acquired based on the information related to the normal feature information and the information related to the target feature information.
  • the normal feature information includes a normal map
  • the depth feature information includes a depth map
  • the tangent feature information includes a tangent map.
  • the normal feature extraction block in the feature model can extract information related to the normal map of the target object based on the processed image, and the depth feature extraction block can be based on the processed image.
  • extract the information related to the depth map of the target object, and the tangent feature extraction block can extract the information related to the tangent map of the target object based on the processed image.
  • the normal feature extraction block may obtain the normal map of the target object based on the information related to the normal map of the target object.
  • the depth feature extraction block may obtain the depth map of the target object based on the information related to the normal map of the target object and the information related to the depth map of the target object.
  • the tangent feature extraction block may obtain the tangent map of the target object based on the information related to the normal map of the target object and the information related to the tangent map of the target object. It should be understood that the information related to the normal map may be the normal feature of the target object, the information related to the depth map may be the depth feature of the target object, and the information related to the tangent map may be the tangent feature of the target object.
  • the terminal device may input the "processed image" whose parts other than the user are adjusted to be black into the feature model, thereby obtaining normal feature information of the user and target feature information.
  • the target feature information when the target feature information includes depth feature information, the depth feature information may be normal feature information of the user.
  • the tangent feature information may be the tangent feature information of the user or the tangent feature information of the user's hair.
  • the terminal device may obtain the tangent feature information of the user's hair from the tangent feature information of the user based on the mask image D of the user's hair, for example, set the tangent feature information of the user and the mask of the hair into the tangent feature information of the user.
  • the tangent feature information corresponding to the pixel points of the hair is used as the tangent feature information of the hair.
  • the above S803 can be replaced by: changing the mask image of the target object (as shown in FIG. 6 )
  • the image B) and the processed image (image C in Figure 6) are cascaded and input to the feature model.
  • the pixels in the processed image other than the target object are preset values, and cascading the mask image of the target object and the processed image can be understood as: combining the pixels of the mask image of the target object with the processed image
  • the pixels of the processed image are cascaded in one-to-one correspondence, and the cascaded mask image of the target object and the processed image are input to the feature model.
  • the terminal device can adjust the part other than the user to be a black "processed image", and input the user's mask image into the feature model after concatenating, so as to obtain the user's normal feature information, and
  • the target feature information can improve the accuracy of the normal feature information and the target feature information.
  • S804 Acquire a lighting rendering mode according to the normal feature information and the target feature information.
  • the terminal device stores lighting rendering modes corresponding to each normal feature information and each target feature information. That is to say, the normal feature information is different, and the lighting rendering mode is different. Similarly, the target feature information is different, and the lighting rendering mode is different. In the embodiment of the present disclosure, after obtaining the normal feature information and target feature information of the target object, the terminal device may obtain the lighting rendering mode of the target object based on the lighting rendering mode corresponding to each normal feature information and each target feature information.
  • the terminal device may use a table form, a database form, or an extensible markup language (Extensible markup language, XML) form to store the lighting rendering mode corresponding to each normal feature information and each target feature information.
  • XML extensible markup language
  • S805 Use the lighting rendering mode to perform lighting rendering on the target object.
  • the terminal device may perform illumination rendering on the target object based on the illumination rendering parameters corresponding to the illumination rendering mode.
  • the terminal device can obtain the normal feature information, the depth feature information, and the lighting rendering mode corresponding to the tangent feature information.
  • the terminal device adopts this lighting rendering mode, and can illuminate the user from the side of the user with the light intensity a, and the terminal device renders the user's hair, so that the user's hair presents a light distribution in the shape of water ripples.
  • the terminal device may preprocess the image to be processed, and cascade the mask image and the processed image of the target object and then input them into the feature model, so as to obtain more accurate normal feature information of the target object. and target feature information, which can improve the lighting rendering effect.
  • FIG. 8 is a structural block diagram of an image processing apparatus provided by an embodiment of the present disclosure.
  • the image processing apparatus may be the terminal device in the foregoing embodiment, or may be a chip or a processor in the terminal device. For convenience of explanation, only the parts related to the embodiments of the present disclosure are shown.
  • the image processing apparatus 900 includes: a processing module 901 and an output module 902 .
  • the processing module 901 is used to obtain an image to be processed, and the image to be processed includes a target object; adopt a feature model to obtain normal feature information and target feature information of the target object, and the target feature information includes: depth feature information and/or tangent feature information ; Based on normal feature information and target feature information, perform illumination rendering on the target object in the image to be processed, and obtain an image after illumination rendering.
  • the output module 902 is used for outputting the image after lighting rendering.
  • the feature model is specifically used to: extract the information related to the normal feature information of the target object and the information related to the target feature information of the target object; based on the information related to the normal feature information, obtain the method
  • the target feature information is obtained based on the information related to the normal feature information and the information related to the target feature information.
  • the processing module 901 is further configured to preprocess the image to be processed to obtain a processed image, and the pixels in the processed image other than the target object are preset values;
  • the image input feature model of obtains normal feature information and target feature information.
  • the processing module 901 is specifically configured to obtain a mask image of the target object based on the image to be processed; based on the mask image, adjust the pixels in the image to be processed other than the target object to the pre-processed image. Set value; the mask image and the processed image are concatenated and input to the feature model.
  • the processing module 901 is specifically configured to obtain a lighting rendering mode according to the normal feature information and the target feature information; and use the lighting rendering mode to perform lighting rendering on the target object.
  • the feature model is obtained by training the sample image, the sample normal feature information of the sample object in the sample image, and the sample target feature information of the sample object as sample data, and the sample target feature information includes the sample Depth feature information and/or sample tangent feature information.
  • the normal feature information is a normal map
  • the depth feature information is a depth map
  • the tangent feature information is a tangent map
  • the sample normal feature information is a sample normal map
  • the sample depth feature information is a sample depth Figure
  • the sample tangent feature information is the sample tangent graph.
  • the feature model in the training process is an initial feature model
  • the model parameters in the initial feature model related to the normal feature information output by the initial feature model are updated based on the first loss function
  • the first loss function is obtained based on the normal feature information of the sample and the normal feature information output by the initial feature model.
  • the model parameters in the initial feature model related to the depth feature information output by the initial feature model are updated based on a second loss function; wherein the second loss function is based on the The sample depth feature information and the depth feature information output by the initial feature model are obtained; and/or the second loss function is the normal feature information obtained based on the depth feature information output by the initial feature model, and the The normal characteristic information of the sample is obtained.
  • the model parameters in the initial feature model related to the tangent feature information output by the initial feature model are updated based on a third loss function; wherein the third loss function is based on the obtained from the tangent feature information output by the initial feature model and the normal feature information output by the initial feature model; and/or, the sample data further includes: the sample tangent feature information of the target part of the sample object and all The mask image of the target part, the target part has anisotropy, the third loss function is based on the sample feature information of the target part, the mask image of the target part and the output of the initial feature model Tangent feature information is obtained.
  • the feature model includes: an input layer, a hidden layer, and an output layer; the input layer is used to input the image to be processed into the hidden layer; the hidden layer is used to: based on the image to be processed, obtain Normal feature information and target feature information; output normal feature information and target feature information through the output layer.
  • the hidden layer includes: a normal feature extraction block and a target feature extraction block; a normal feature extraction block is used to: extract information related to normal feature information in the target object of the image to be processed ; Based on the information related to the normal feature information, obtain the normal feature information; output the normal feature information through the output layer, and output the information related to the normal feature information to the target feature extraction block.
  • the target feature extraction block is used to: extract the information related to the target feature information in the target object of the image to be processed; obtain the target feature information based on the information related to the target feature information and the information related to the normal feature information; through the output layer Output target feature information.
  • the target feature extraction block includes: a depth feature extraction block and a tangent feature extraction block.
  • the depth feature extraction block is used to: extract the information related to the depth feature information in the target object of the image to be processed; obtain the depth feature information based on the information related to the depth feature information and the information related to the normal feature information; pass the output layer Output depth feature information.
  • the tangent feature extraction block is used to: extract the information related to the tangent feature information in the target object of the image to be processed; obtain the tangent feature information based on the information related to the tangent feature information and the information related to the normal feature information; through the output layer Output tangent feature information.
  • the image processing apparatus provided in the embodiments of the present disclosure can be used to execute the steps of the device on the terminal in the above method embodiments, and the implementation principles and technical effects thereof are similar, and details are not described herein again in this embodiment.
  • the embodiments of the present disclosure further provide an electronic device.
  • FIG. 9 a schematic structural diagram of an electronic device 1000 suitable for implementing an embodiment of the present disclosure is shown.
  • the electronic device shown in FIG. 9 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.
  • the electronic device 1000 may include a processing device (such as a central processing unit, a graphics processor, etc.) 1001, which may be stored in a read only memory (Read Only Memory, ROM for short) 1002 according to a program or from a storage device 1008 is a program loaded into a random access memory (Random Access Memory, RAM for short) 1003 to perform various appropriate actions and processes.
  • a processing device such as a central processing unit, a graphics processor, etc.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004.
  • An input/output (I/O) interface 1005 is also connected to the bus 1004 . It should be understood that the processing apparatus 1001 may execute the steps executed by the processing module 901 shown in FIG. 8 above.
  • an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD for short) ), speaker, vibrator, etc. output device 1007; storage device 1008 including, eg, magnetic tape, hard disk, etc., and communication device 1009.
  • the communication means 1009 may allow the electronic device 1000 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 9 shows the electronic device 1000 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. It should be understood that the output device 1007 may perform the steps performed by the output module 902 shown in FIG. 8 above.
  • embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via the communication device 1009, or from the storage device 1008, or from the ROM 1002.
  • the processing apparatus 1001 the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .
  • the program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to: electric wire, optical cable, radio frequency (RF for short), etc., or any suitable combination of the above.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.
  • the aforementioned computer-readable medium carries one or more programs, and when the aforementioned one or more programs are executed by the electronic device, causes the electronic device to execute the methods shown in the foregoing embodiments.
  • Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, can be connected to an external A computer (eg using an internet service provider to connect via the internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner.
  • the name of the unit does not constitute a limitation of the unit itself under certain circumstances, for example, the first obtaining unit may also be described as "a unit that obtains at least two Internet Protocol addresses".
  • exemplary types of hardware logic components include: Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Products (Application Specific Standard Parts, referred to as ASSP), System on Chip (System on Chip, referred to as SOC), Complex Programmable Logic Device (Complex Programmable Logic Device, CPLD) and so on.
  • FPGA Field-Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • ASSP Application Specific Standard Parts
  • SOC System on Chip
  • Complex Programmable Logic Device Complex Programmable Logic Device, CPLD
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (Read-Only Memory) ROM), Erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, referred to as EPROM or Flash Memory), Optical Fiber, Portable Compact Disc Read-Only Memory (Compact Disc Read-Only Memory, referred to as CD-ROM), Optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM Random Access Memory
  • Read-Only Memory Read-Only Memory
  • EPROM Erasable Programmable Read-Only Memory
  • EPROM Erasable Programmable Read-Only Memory
  • Optical Fiber Portable Compact Disc Read-Only Memory
  • CD-ROM Compact Disc Read-Only Memory
  • Optical storage magnetic storage, or any suitable combination of the foregoing.
  • an image processing method comprising: acquiring an image to be processed, the image to be processed includes a target object; using a feature model to acquire an image of the target object normal feature information and target feature information, the target feature information includes: depth feature information and/or tangent feature information; based on the normal feature information and the target feature information, for the target object in the image to be processed Perform illumination rendering to obtain an image after illumination rendering; and output the image after illumination rendering.
  • the feature model is specifically configured to: extract information about the target object and the normal feature information, and information about the target object and the target feature information; Obtain the normal feature information based on the information related to the normal feature information; obtain the target based on the information related to the normal feature information and the target feature information characteristic information.
  • the method before acquiring the normal feature information and the target feature information of the target object by using a feature model, the method further includes: preprocessing the to-be-processed image to obtain a processed image.
  • image, pixel points other than the target object in the processed image are preset values;
  • the use of a feature model to obtain normal feature information and target feature information of the target object includes: The processed image is input into the feature model to obtain the normal feature information and the target feature information.
  • the preprocessing of the to-be-processed image to obtain a processed image includes: acquiring a mask image of the target object based on the to-be-processed image; the mask image, and adjusting the pixel points in the image to be processed except the target object to the preset value; the inputting the processed image into the feature model includes: The mask image and the processed image are concatenated and input to the feature model.
  • the performing illumination rendering on the target object in the to-be-processed image includes: obtaining an illumination rendering mode according to the normal feature information and the target feature information; using the The lighting rendering mode performs lighting rendering on the target object.
  • the feature model is obtained by training a sample image, sample normal feature information of a sample object in the sample image, and sample target feature information of the sample object as sample data
  • the sample target feature information includes sample depth feature information and/or sample tangent feature information.
  • the normal feature information is a normal map
  • the depth feature information is a depth map
  • the tangent feature information is a tangent map
  • the sample normal feature information is a sample method A graph
  • the sample depth feature information is a sample depth map
  • the sample tangent feature information is a sample tangent map
  • the feature model in the training process is an initial feature model
  • the model parameters in the initial feature model related to the normal feature information output by the initial feature model are based on the first loss function updated, and the first loss function is obtained based on the normal feature information of the sample and the normal feature information output by the initial feature model.
  • model parameters in the initial feature model related to the depth feature information output by the initial feature model are updated based on a second loss function; wherein the second loss function is Obtained based on the depth feature information of the sample and the depth feature information output by the initial feature model; and/or, the second loss function is normal feature information obtained based on the depth feature information output by the initial feature model, And the sample normal feature information is obtained.
  • model parameters in the initial feature model related to the tangent feature information output by the initial feature model are updated based on a third loss function; wherein the third loss function is Obtained based on the tangent feature information output by the initial feature model and the normal feature information output by the initial feature model; and/or, the sample data further includes: sample tangent feature information of the target portion of the sample object and the mask image of the target part, the target part has anisotropy, the third loss function is based on the sample feature information of the target part, the mask image of the target part and the initial feature model The output tangent feature information is obtained.
  • the feature model includes: an input layer, a hidden layer and an output layer; the input layer is used to input the image to be processed to the hidden layer; the hidden layer is used to: based on the image to be processed , obtain normal feature information and target feature information; output normal feature information and target feature information through the output layer.
  • the hidden layer includes: a normal feature extraction block and a target feature extraction block; the normal feature extraction block is used to: extract information related to normal features in the target object of the image to be processed based on the information related to the normal feature information, obtain the normal feature information; output the normal feature information through the output layer, and output the information related to the normal feature information to the target feature extraction block.
  • the target feature extraction block is used to: extract the information related to the target feature information in the target object of the image to be processed; obtain the target feature information based on the information related to the target feature information and the information related to the normal feature information; through the output layer Output target feature information.
  • the target feature extraction block includes: a depth feature extraction block and a tangent feature extraction block.
  • the depth feature extraction block is used to: extract the information related to the depth feature information in the target object of the image to be processed; obtain the depth feature information based on the information related to the depth feature information and the information related to the normal feature information; pass the output layer Output depth feature information.
  • the tangent feature extraction block is used to: extract the information related to the tangent feature information in the target object of the image to be processed; obtain the tangent feature information based on the information related to the tangent feature information and the information related to the normal feature information; through the output layer Output tangent feature information.
  • an image processing apparatus comprising: a processing module for acquiring an image to be processed, where the image to be processed includes a target object; using a feature model to acquire normal feature information and target feature information of the target object, the target feature information includes: depth feature information and/or tangent feature information; based on the normal feature information and the target feature information, The target object in the image is subjected to lighting rendering, and the image after lighting rendering is obtained.
  • An output module configured to output the image rendered by the lighting.
  • the feature model is specifically configured to: extract information about the target object and the normal feature information, and information about the target object and the target feature information; Obtain the normal feature information based on the information related to the normal feature information; obtain the target based on the information related to the normal feature information and the target feature information characteristic information.
  • the processing module is further configured to preprocess the image to be processed to obtain a processed image, in which pixels except for the target object are included in the processed image is a preset value; input the processed image into the feature model to obtain the normal feature information and the target feature information.
  • the processing module is specifically configured to obtain a mask image of the target object based on the image to be processed; and based on the mask image, remove all items from the image to be processed.
  • the pixel points other than the target object are adjusted to the preset value; the mask image and the processed image are cascaded and then input to the feature model.
  • a processing module is specifically configured to obtain a lighting rendering mode according to the normal feature information and the target feature information; and perform lighting rendering on the target object by using the lighting rendering mode .
  • the feature model is obtained by training a sample image, sample normal feature information of a sample object in the sample image, and sample target feature information of the sample object as sample data
  • the sample target feature information includes sample depth feature information and/or sample tangent feature information.
  • the normal feature information is a normal map
  • the depth feature information is a depth map
  • the tangent feature information is a tangent map
  • the sample normal feature information is a sample method A graph
  • the sample depth feature information is a sample depth map
  • the sample tangent feature information is a sample tangent map
  • the feature model in the training process is an initial feature model
  • the model parameters in the initial feature model related to the normal feature information output by the initial feature model are based on the first loss function updated, and the first loss function is obtained based on the normal feature information of the sample and the normal feature information output by the initial feature model.
  • model parameters in the initial feature model related to the depth feature information output by the initial feature model are updated based on a second loss function; wherein the second loss function is Obtained based on the depth feature information of the sample and the depth feature information output by the initial feature model; and/or, the second loss function is normal feature information obtained based on the depth feature information output by the initial feature model, And the sample normal feature information is obtained.
  • model parameters in the initial feature model related to the tangent feature information output by the initial feature model are updated based on a third loss function; wherein the third loss function is Obtained based on the tangent feature information output by the initial feature model and the normal feature information output by the initial feature model; and/or, the sample data further includes: sample tangent feature information of the target portion of the sample object and the mask image of the target part, the target part has anisotropy, the third loss function is based on the sample feature information of the target part, the mask image of the target part and the initial feature model The output tangent feature information is obtained.
  • the feature model includes: an input layer, a hidden layer and an output layer; the input layer is used to input the image to be processed to the hidden layer; the hidden layer is used to: based on the image to be processed , obtain normal feature information and target feature information; output normal feature information and target feature information through the output layer.
  • the hidden layer includes: a normal feature extraction block and a target feature extraction block; the normal feature extraction block is used to: extract information related to normal features in the target object of the image to be processed based on the information related to the normal feature information, obtain the normal feature information; output the normal feature information through the output layer, and output the information related to the normal feature information to the target feature extraction block.
  • the target feature extraction block is used to: extract the information related to the target feature information in the target object of the image to be processed; obtain the target feature information based on the information related to the target feature information and the information related to the normal feature information; through the output layer Output target feature information.
  • the target feature extraction block includes: a depth feature extraction block and a tangent feature extraction block.
  • the depth feature extraction block is used to: extract the information related to the depth feature information in the target object of the image to be processed; obtain the depth feature information based on the information related to the depth feature information and the information related to the normal feature information; pass the output layer Output depth feature information.
  • the tangent feature extraction block is used to: extract the information related to the tangent feature information in the target object of the image to be processed; obtain the tangent feature information based on the information related to the tangent feature information and the information related to the normal feature information; through the output layer Output tangent feature information.
  • an electronic device comprising: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory , causing the processor to perform the method described above in the first aspect and various possible designs of the first aspect.
  • a computer-readable storage medium where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, The methods described above in the first aspect and various possible designs of the first aspect are implemented.
  • a computer program product comprising a computer program or instructions, when the computer program or the instructions are executed by a processor, the first aspect and the first aspect above are implemented.
  • a computer program product comprising a computer program or instructions, when the computer program or the instructions are executed by a processor, the first aspect and the first aspect above are implemented.
  • a computer program that, when executed by a processor, performs the method described in the first aspect and various possible designs of the first aspect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Graphics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例提供一种图像处理方法、装置、电子设备以及可读存储介质，该方法包括：获取待处理图像，待处理图像中包括目标对象；采用特征模型，获取目标对象的法向特征信息和目标特征信息，目标特征信息包括：深度特征信息和/或切线特征信息；基于法向特征信息和目标特征信息，对待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；输出光照渲染后的图像。本公开相较于现有技术，终端设备可以基于目标对象更为丰富的特征信息，对目标对象进行光照渲染，能够丰富光照渲染效果。

Description

图像处理方法、装置、电子设备以及可读存储介质
相关申请的交叉引用
本申请要求于2021年4月28日提交的、申请号为202110468921.0、名称为“图像处理方法、装置、电子设备以及可读存储介质”的中国专利申请的优先权，其全部内容通过引用并入本文。
技术领域
本公开实施例涉及图像处理技术领域，尤其涉及一种图像处理方法、装置、电子设备以及可读存储介质。
背景技术
重光照(relighting)技术是改变图像拍摄时的光照得到新图像的技术。例如，脸部的重光照技术针对的对象是图像中的脸部，即对脸部进行重光照。重光照技术广泛应用在图像的后期处理中或者电影的后期制作中。示例性的，用户采用修图软件中的重光照功能，可以对图像中的脸部进行处理，以改变脸部的光影效果。
目前图像处理装置可以基于对象的法向信息，采用某一角度的光照对对象进行光照渲染，渲染效果单一。
发明内容
本公开实施例提供一种图像处理方法、装置、电子设备以及可读存储介质，可以丰富图像的光照渲染效果。
第一方面，本公开实施例提供一种图像处理方法，包括：获取待处理图像，所述待处理图像中包括目标对象；采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；输出所述光照渲染后的图像。
第二方面，本公开实施例提供一种图像处理装置，包括：
处理模块，用于获取待处理图像，所述待处理图像中包括目标对象；采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像。
输出模块，用于输出所述光照渲染后的图像。
第三方面，本公开实施例提供一种电子设备，包括：处理器和存储器；
所述存储器存储计算机执行指令；
所述处理器执行所述存储器存储的计算机执行指令，使得所述处理器执行如上第一方面 以及第一方面各种可能的设计所述的图像处理方法。
第四方面，本公开实施例提供一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如上第一方面以及第一方面各种可能的设计所述的图像处理方法。
第五方面，本公开实施例提供一种计算机程序产品，包括计算机程序或指令，所述计算机程序或所述指令被处理器执行时，实现如上第一方面以及第一方面各种可能的设计所述的图像处理方法。
第六方面，本公开实施例提供一种计算机程序，所述计算机程序被处理器执行时执行如上第一方面以及第一方面各种可能的设计所述的图像处理方法。
本实施例提供一种图像处理方法、装置、电子设备以及可读存储介质，该方法包括：获取待处理图像，待处理图像中包括目标对象；采用特征模型，获取目标对象的法向特征信息和目标特征信息，目标特征信息包括：深度特征信息和/或切线特征信息；基于法向特征信息和目标特征信息，对待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；输出光照渲染后的图像。本公开实施例中，终端设备不仅获取了目标对象的法向特征信息，还能够获取目标对象的深度特征信息和/或切线特征信息，因为深度特征信息与光照距离对象的远近相关，切线特征信息与光照在对象上形成的光线的分布相关，因此相较于现有技术，终端设备可以基于目标对象更为丰富的特征信息，对目标对象进行光照渲染，能够丰富光照渲染效果，提高用户体验。
附图说明
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的图像处理方法的流程示意图一；
图2为本公开实施例提供的图像处理方法应用的一种场景示意图；
图3为本公开实施例提供的特征模型的训练流程示意图一；
图4为本公开实施例提供的初始模型的结构示意图；
图5为本公开实施例提供的特征模型的训练流程示意图二；
图6为本公开实施例提供的一种样本图像处理的示意图；
图7为本公开实施例提供的图像处理方法的流程示意图二；
图8为本公开实施例提供的图像处理装置的结构框图；
图9为本公开实施例提供的电子设备的结构示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有 作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
为解决现有技术存在的技术问题，本公开实施例提供了以下解决思路：在对图像中的对象进行重光照(relighting)处理时，获取图像中的对象的法向特征信息和目标特征信息，该目标特征信息中可以包括深度特征信息和/或切线特征信息(tangent)。因为法向特征信息与光照的角度(如方向光源)相关，深度特征信息与光照距离对象的远近相关(或理解为深度特征信息与光照达到对象的光照强度相关)，切线特征信息与光照在对象上形成的光线的分布相关。因此相较于现有技术，本公开实施例中可以基于对象更为丰富的特征信息，实现对图像中的对象进行不同的光照渲染，如对对象进行不同角度、不同强度和/或不同光线的分布等的渲染，进而达到丰富对象的渲染效果的目的。在一种实施例中，图像中的对象可以但不限于为人物、动物、生活用具，生活用具如餐具。应理解，图像以及下述实施例中涉及的待处理图像、样本图像可以为图片或者视频中的视频帧。
本公开实施例提供了图像处理方法应用的一种场景。示例性的，用户可以通过终端设备对图像中的对象进行光照渲染。示例性的，用户可以打开终端设备中的修图应用程序，在修图应用程序中选择待处理图像，该待处理图像可以显示在终端设备的界面，并且该待处理图像中包括用户的脸部以及用户的头发。终端设备的界面上可以显示有“光照渲染”控件，用户点击该“光照渲染”控件，终端设备可以执行本公开实施例提供的图像处理方法，以对待处理图像中的对象(用户的脸部以及用户的头发)进行光照渲染，得到光照渲染后的图像。示例性的，光照渲染可以从用户的侧面、以光照强度为a进行光照，经光照渲染后，用户的头发呈现水波纹形状的光线分布。应理解，用户的头发的水波纹形状仅为示例。
示例性的，终端设备上可以先显示“光照渲染”控件，用户选择图像后，终端设备可以对该图像中的对象进行光照渲染。或者，上述终端设备的界面上还可以现有其他修图控件，本公开实施例中对用户如何触发终端设备执行光照渲染的方式不做限制。在一种实施例中，本公开实施例提供的图像处理方法可以应用于对各类视频(例如，影视作品)的后期处理场景中，如终端设备可以将视频中的视频帧做为待处理图像，进而待处理图像中的对象进行光照渲染。
本公开实施例中的终端设备可以包括但不限于为：手机、平板电脑、笔记本电脑、音箱、可穿戴设备、智慧屏、智能家用电器、物联网(internet of things，IoT)设备、摄像头设备等具有图像处理功能的设备。可选的，终端设备还可以为个人数字处理(personal digital assistant，PDA)、具有无线通信功能的手持设备、计算设备、虚拟现实(virtual reality，VR)终端设备、无人机设备、增强现实(augmented reality，AR)终端设备、工业控制(industrial control)中的无线终端、智慧家庭(smart home)中的无线终端等。本公开实施例中对终端设备的形态不做限制。
参考图1，图1为本公开实施例提供的图像处理方法的流程示意图一。本实施例的方法可以应用在终端设备中，该图像处理方法可以包括：
S201：获取待处理图像，待处理图像中包括目标对象。
待处理图像可以为终端设备中存储的本地图像，如用户可以在终端设备中选择至少一个本地图像，终端设备将用户选择的本地图像作为待处理图像，如界面101所示。在一种实施例中，待处理图像可以为终端设备拍摄的图像，或者待处理图像可以为导入终端设备的图像。在一种实施例中，待处理图像可以为另一电子设备发送给终端设备的图像，即该另一电子设 备需要对图像进行光照渲染，进而将该图像发送给终端设备进行光照渲染。其中，图像可以为一张图片，或者一组图片，或者视频。视频中的每个视频帧可以看做一个图像。
待处理图像中包括目标对象。目标对象可以但不限于为人物、动物、物品等，物品可以如不锈钢材质的餐具、陶瓷材质的餐具、塑料袋等。下述实施例中以目标对象为人物为例进行说明。
S202：采用特征模型，获取目标对象的法向特征信息和目标特征信息，目标特征信息包括：深度特征信息和/或切线特征信息。
特征模型用于提取待处理图像中的目标对象的法向特征信息和目标特征信息。其中，目标特征信息包括：深度特征信息和/或切线特征信息。在一种实施例中，法向特征信息可以为目标对象的法向图、法向矩阵、或者法向特征向量。深度特征信息可以为目标对象的深度图、深度矩阵、或者深度特征向量。切线特征信息可以为目标对象的切线图、切线矩阵、或者切线特征向量。
在一种实施例中，可以采用样本数据预先训练得到该特征模型，进而将该特征模型预置在终端设备中，以使终端设备可以采用该特征模型获取图像中目标对象的法向特征信息和目标特征信息。其中，训练该特征模型的执行主体可以为终端设备或其他电子设备。其他电子设备如模型训练机器、服务器等，本公开实施例对训练得到特征模型的执行主体不作限制。
在一种实施例中，样本数据可以包括样本图像、样本图像的法向特征信息。在一种实施例中，样本数据可以包括样本图像、样本图像中的样本对象的法向特征信息和样本目标特征信息。样本目标特征信息可以包括样本深度特征信息和/或样本切线特征信息。应注意的是，样本数据中包括的样本目标特征信息的类型与基于特征模型输出的目标特征信息的类型相同。示例性的，若样本数据中包括的样本目标特征信息为样本切线特征信息，则基于特征模型输出的目标特征信息也为切线特征信息。
在一种实施例中，终端设备在获取待处理图像后，可以将该待处理图像输入至特征模型，特征模型可以输出该待处理图像中目标对象的法向特征信息和目标特征信息。应理解，该实施例中先对特征模型的使用过程进行说明，特征模型的训练过程具体可以参照下述图3和图4的相关描述。应注意，目标对象和样本对象的类型可以相同，如目标对象为人物，则样本对象可以为人物。示例性的，如目标对象为留有头发的人物，样本对象可以包括留有头发的人物。
应注意的是，本公开实施例中之所以除了目标对象的法向特征信息之外，还要获取目标特征信息的原因在于：目前终端设备基于法向特征信息，仅能对目标进行某一角度的光照渲染。而若终端设备还获取了目标对象的深度特征信息，因为深度特征信息表征目标对象的每个像素点距离拍摄位置处的距离，因此终端设备可以基于深度特征信息为目标对象提供不同距离的点光源的光照渲染。因为距离目标对象不同距离的点光源，达到目标对象的光照强度不同，因此也可以理解为终端设备可以为目标对象提供不同光照强度的光照渲染。
另外，若目标对象具有各向异性，则还需要获取目标对象的切线特征信息对目标对象进行光线的分布的光照渲染。应理解，如头发、毛发、不锈钢制品的餐具均呈现各向异性。示例性的，如目标对象为留有头发的用户，用户的脸部和身体一般具有各向同性，则不需要切线特征信息，也能够正确地进行光照渲染。而用户的头发具有各向异性，光照在头发上的光线分布应该呈现水波纹的形状，而若终端设备未获取头发的切线特征信息，则光照在头发上 的光线分布与光照在用户的脸部和身体的光线分布相同，光线分布呈现光圈形状，这与实际生活中头发在光照下的光线的分布不同，导致对头发的渲染效果差。本公开实施例中，终端设备可以获取目标对象的切线特征信息，并可以对各向同性和各向异性的对象均能进行准确的光线的分布的渲染，渲染效果好，提升了对具有各向异性的对象的渲染效果。
在一种实施例中，目标特征信息中包含的信息越多，则光照渲染的效果越好。
S203：基于法向特征信息和目标特征信息，对待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像。
因为法向特征信息与光照的角度相关，深度特征信息与光照距离对象的远近相关，切线特征信息与光照在对象上形成的光线的分布相关。在一种实施例中，终端设备可以基于法向特征信息，确定光照角度。终端设备基于深度特征信息，确定光照强度(或点光源距离目标对象的距离)。以及终端设备基于切线特征信息，确定光照在目标对象上形成的光线的分布，示例性的，如光照在对象上形成的光线的分布可以包括：光照在用户的脸部上形成的光线的分布为光圈形状，在头发的分布为水波纹形状。
在一种实施例中，终端设备可以基于光照角度、光照强度和光照在对象上形成的光线的分布，对目标对象进行光照渲染。光照渲染可以为：终端设备将光照角度的图层(如来自用户侧面的光照)和光照强度对应的图层(光照强度为a)，以及光照在对象上形成的光线的分布对应的图层(水波纹形状)与待处理图像中的目标对象叠加在一起，得到光照渲染后的图像。
S204：输出光照渲染后的图像。
在一种实施例中，终端设备可以显示该光照渲染后的图像。或者，在一种实施例中，终端设备可以将该光照渲染后的图像输出至另一电子设备，该另一电子设备可以为向终端设备请求对图像进行光照渲染的设备，如智能穿戴设备，如图2所示。图2为本公开实施例提供的图像处理方法应用的一种场景示意图。其中，该另一电子设备接收到光照渲染后的图像后，可以显示该光照渲染后的图像。
本公开实施例提供的图像处理方法包括：终端设备获取待处理图像，待处理图像中包括目标对象，采用特征模型，获取目标对象的法向特征信息和目标特征信息，目标特征信息包括：深度特征信息和/或切线特征信息，基于法向特征信息和目标特征信息，对待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像，输出光照渲染后的图像。本公开实施例中，终端设备在对图像中的目标对象进行光照渲染时，不仅获取了目标对象的法向特征信息，还获取了目标对象的深度特征信息和/或切线特征信息，因为深度特征信息与光照距离对象的远近相关，切线特征信息与光照在对象上形成的光线的分布相关，因此相较于现有技术，终端设备可以基于目标对象更为丰富的特征信息，对目标对象进行更为丰富的光照渲染，如对目标对象进行不同角度、不同光照强度和/或不同的光线的分布的光照渲染，进而能够丰富光照渲染效果，提高用户体验。
为了更为清楚地对本公开实施例提供的图像处理方法进行介绍，下述结合图3对本公开实施例训练得到特征模型的过程进行说明。训练特征模型的执行主体可以为服务器，训练特征模型的执行主体还可以为终端设备或具备训练能力的电子设备，本公开实施例对训练特征模型的执行主体不作限制。参照图3，训练得到特征模型的方法可以包括：
S401：将包含有样本对象的样本图像输入至初始模型。
初始模型可以为训练特征模型的初始框架。初始模型可以但不限于为：基于全卷积神经网络(fully convolutional networks，FNC)框架的Unet神经网络、轻量化神经网络shufflenet、卷积神经网络(convolutional neural network，CNN)。可以将样本图像输入初始模型，以进行训练。
应注意的是，训练得到特征模型的样本数据除了样本图像之外，还可以包括：样本图像中样本对象的样本法向特征信息。在一种实施例中，样本数据中包括：样本图像，以及样本图像中样本对象的样本法向特征信息和样本目标特征信息。样本目标特征信息可以包括样本深度特征信息和/或样本切线特征信息。应理解，样本法向特征信息、样本深度特征信息和样本切线特征信息可以看做是样本对象准确的特征信息，可以称为样本对象的特征的标注信息。
在一种实施例中，样本法向特征信息可以包括样本法向图，样本深度特征信息可以包括样本深度图，样本切线特征信息可以包括样本切线图。此外，样本法向特征信息、深度特征信息、以及切线特征信息还可以包括其他类型的表现形式，如特征向量、特征矩阵等，本公开不对此进行限制。下述实施例中以(法向/深度/切线)特征信息为(法向/深度/切线)图进行说明。
S402：获取初始模型输出的样本对象的法向特征信息、深度特征信息和切线特征信息。
本公开实施例中以一个样本图像在训练中的处理过程进行说明。在训练过程中，将样本图像输入初始模型后，初始模型可以输出该样本图像中样本对象的法向特征信息、深度特征信息和切线特征信息。
图4为本公开实施例提供的初始模型的结构示意图。参照图4，初始模型包括输入层、隐含层和输出层，隐含层包括法向特征提取块、深度特征提取块以及切线特征提取块。输入层用于向法向特征提取块、深度特征提取块以及切线特征提取块输入样本图像。在一种实施例中，输入层和输出层可以采用Unet神经网络，隐含层可以采用shufflenet神经网络。法向特征提取块、深度特征提取块以及切线特征提取块也可以称为初始模型中的三个分支(branch)。
法向特征提取块，用于提取样本图像的样本对象中与法向图相关的信息，且基于该与法向图相关的信息获取法向图，且通过输出层输出样本对象的法向图。法向特征提取块，还用于向深度特征提取块和切线特征提取块输出与法向图相关的信息。深度特征提取块，用于提取样本图像的样本对象中与深度图相关的信息，且根据与深度图相关的信息和与法向图相关的信息，获取深度图，且通过输出层输出样本对象的深度图。切线特征提取块，用于提取样本图像的样本对象中与切线图相关的信息，且根据与切线图相关的信息和与法向图相关的信息，获取切线图，且通过输出层输出样本对象的切线图。其中，与法向图相关的信息可以为样本对象的法向特征，如法向向量等。与深度图相关的信息可以为样本对象的深度特征，与切线图相关的信息可以为样本对象的切线特征。
特征模型在训练过程中，可以输出样本对象的法向图。因为法向图的准确性高，本公开实施例中，可以结合与预设法向图相关的信息得到深度图和切线图，进而提高深度图和切线图的准确性。
S403：获取法向特征信息的第一损失函数、深度特征信息的第二损失函数，以及切线特征信息的第三损失函数。
针对初始模型输出的法向图、深度图和切线图，为了验证法向图、深度图和切线图的准确性，可以获取法向特征信息的第一损失函数、深度特征信息的第二损失函数，以及切线特 征信息的第三损失函数，以对初始模型中输入层和隐含层之间的权重，以及隐含层和输出层之间的权重进行更新，使得初始模型输出的法向图、深度图和切线图更为准确。应理解，损失函数表征初始模型得到的特征信息与标注信息之间的误差，损失函数越大，则误差越大，初始模型的训练结果越不准确。其中，当初始模型得到的特征信息与标注信息之间的误差在小于误差阈值时，该初始模型的训练结果准确。
在一种实施例中，针对样本数据中包括“样本图像和样本对象的样本法向图”的场景，可以基于法向图和样本法向图获取第一损失函数，示例性的，如采用L1损失函数公式，基于法向图和样本法向图获取第一损失函数。因为样本数据中没有样本对象的样本深度图和样本切线图，因此可以根据深度图获取法向特征信息，进而得到深度图对应的法向图，进而可以采用L1损失函数公式，基于样本法向图和深度图对应的法向图获取第二损失函数。同理的，可以对切线图和法向图进行点乘，获取点乘结果，进而采用L1损失函数公式，基于点乘结果和0获取第三损失函数。
应理解，深度图用于表征样本对象的各像素点的深度。在一种实施例中，以样本对象上的一个像素点为例，可以基于该像素点的深度，以及该像素点周围的像素点的深度，获取该像素点的深度梯度，可以基于该像素点的深度梯度，获取该像素点的法向特征信息，如法向向量。进而可以基于样本对象的各像素点的法向特征信息，得到样本对象的法向图，即样本对象的深度图对应的法向图。
或者，在一种实施例中，可以基于样本对象的深度图，得到样本对象的二值图，二值图上各像素点的灰度与像素点的深度呈正比关系。以样本对象上的一个像素点为例，可以基于该像素点的灰度，以及该像素点周围的像素点的灰度，获取该像素点的灰度梯度，可以基于该像素点的灰度梯度，获取该像素点的法向特征信息。进而可以基于样本对象的各像素点的法向特征信息，得到样本对象的法向图。本申请实施例对基于样本对象的深度图，得到样本对象的法向图的方式不做限制。
在一种实施例中，针对样本数据中包括“样本图像、样本对象的样本法向图、样本深度图和样本切线图”的场景，可以采用上述方法分别获取第一损失函数、第二损失函数和第三损失函数。也可以采用L1损失函数公式，基于深度图和样本深度图获取第二损失函数，也可以采用L1损失函数公式，基于切线图和样本切线图获取第三损失函数。
S404：基于第一损失函数、第二损失函数和第三损失函数，更新初始模型，以得到特征模型。
本公开实施例中，可以基于第一损失函数，更新初始模块中输入层中的各神经元和法向特征提取块中的各神经元之间的权重，以及法向特征提取块中的各神经元和输出层中的各神经元之间的权重。可以基于第二损失函数，更新输入层中的各神经元和深度特征提取块中的各神经元之间的权重，以及深度特征提取块中的各神经元和输出层中的各神经元之间的权重。可以基于第三损失函数，更新输入层中的各神经元和切线特征提取块中的各神经元之间的权重，以及切线特征提取块中的各神经元和输出层中的各神经元之间的权重。在一种实施例中，可以基于第一损失函数、第二损失函数和第三损失函数，采用反向传播(back propagation)方式，更新初始模型中各神经元之间的权重。
应理解，得到更新后的初始模型后，可以基于该更新后的初始模型和样本数据，继续训练初始模型，直至训练次数达到预设训练次数结束训练，得到特征模型。在一种实施例中， 若更新后的初始模型(或初始模型)输出的第一损失函数小于第一预设值，第二损失函数小于第二预设值，以及第三损失函数小于第三预设值，则结束训练，得到训练好的特征模型。
综上，可以训练得到特征模型，且将该特征模型预置在终端设备中。
在一种实施例中，可以对样本图像进行预处理，且将预处理后的样本图像输入至初始模型进行训练，以得到预测准确性更高的特征模型。
其中，参照图5，在上述S401之前可以包括：
S405：基于样本图像，获取样本图像中的样本对象的掩膜图像。
样本对象的掩膜图像(mask)，可以为二值图像。如可以将样本图像中的样本对象的像素点置为1，将非样本对象的像素点置为0，即样本图像中的样本对象呈现白色，非样本图像呈现黑色。在一种实施例中，样本对象的掩膜图像可以为灰度图，样本对象的像素点的灰度可以为255，非样本对象的像素点的灰度可以为0。本公开实施例中对样本对象的掩膜图像不做限制，只要能够有效区分样本图像中的样本对象的像素点和非样本对象的像素点即可。
在一种实施例中，可以识别样本图像中的样本对象，进而将样本图像中的样本对象的像素点置为1，将非样本对象的像素点置为0，以获取样本对象的掩膜图像。在一种实施例中，以采用人像分割(portrait matting)技术，获取样本对象的掩膜图像。
图6为本公开实施例提供的一种样本图像处理的示意图。参照图6，样本图像为：包括用户的脸部、用户的头发的图像A，样本图像的掩膜图像可以如图6中的B所示。其中，图像A中还包括背景部分，如太阳。
S406：基于样本图像，获取样本对象的目标部分的掩膜图像，目标部分具有各向异性。
目标部分可以为样本对象的全部区域或者部分区域。目标部分具有各向异性，示例性的，样本对象为用户(包括用户的脸部和用户的头发)，样本对象的目标部分可以为用户的头发，其中，头发具有各向异性，相应地，可以基于图像A，获取用户的头发的掩膜图像，如图6中的D所示。应理解，获取目标部分的掩膜图像的方法可以参照上述获取样本对象的掩膜图像的相关描述。此外，本领域技术人员应当理解的，目标部分的掩膜图像可以预先获取或者预先设置，本公开不对此进行限制。
在一种实施例中，可以将样本对象的目标部分的掩膜图像作为样本数据。
S407：基于样本对象的掩膜图像，将样本图像中除样本对象之外的像素点调整为预设值，以获取处理后的样本图像。
在获取样本对象的掩膜图像后，因为样本对象的掩膜图像的大小和样本图像的大小一致，因此，可以基于样本对象的掩膜图像和样本图像的像素点的对应关系，将样本图像中除样本对象之外的像素点调整为预设值。
示例性的，以预设值为0，样本对象为用户为例进行说明，可以将样本图像中除用户之外的像素点调整为0，使得样本图像中的用户呈现为彩色，除用户之外的像素点呈现为黑色，如图6中的C所示，应理解，图6中的C的前景部分在实际中是彩色的，尽管图6中以黑白色进行表征。
相应的，上述S401可以替换为如下S408：
S408：将样本对象的掩膜图像和处理后的样本图像进行级联后输入至初始模型。
样本图像可以为彩色图像，在一种实施例中，在样本图像中分割获取样本对象的掩膜图像时，因为光线等外界因素的影响，获取的样本对象的掩膜图像中还包含有部分非样本对象 的像素点。为了提高特征模型的预测准确性，本公开实施例中，可以将样本对象的掩膜图像和处理后的样本图像进行级联(concat)，且将级联后的样本对象的掩膜图像和处理后的样本图像输入至初始模型。
其中，因为处理后的样本图像是基于样本对象的掩膜图像得到的，若样本对象的掩膜图像中还包含有部分非样本对象的像素点，则处理后的样本图像的前景部分(彩色部分)包括部分背景区域的像素点(即非样本对象的像素点)，因为处理后的样本图像的样本对象为彩色，相较于样本对象的掩膜图像的前景部分，包含有更多的信息。因此，本公开实施例中，将样本对象的掩膜图像和处理后的样本图像进行级联后输入至初始模型，可以使得在训练得到特征模型的过程中，能够更为准确的识别样本对象在样本图像的边界，进而提高特征模型的预测准确性。
在一种实施例中，样本数据中还可以包括样本对象的目标部分的样本切线图。参照图6，可以将用户的掩膜图像和处理后的样本图像进行级联后输入至初始模型，相应的，在训练过程中，除了采用上述图3中的相关描述获取第三损失函数，还可以采用如下方式获取第三损失函数：
因为样本对象中只有目标部分(例如，头发)具有各向异性，因此本公开实施例中可以仅针对目标部分进行第三损失函数的计算。示例性的，用户的目标部分为头发，可以针对用户的头发进行第三损失函数的计算。如可以基于头发的掩膜图像，在初始模型输出的用户的切线图中，获取头发的切线图。示例性的，可以基于头发掩膜，在用户的切线图中获取头发的像素点对应的切线图的部分，即为头发的切线图。可以采用L1损失函数公式，基于头发的切线图和头发的样本切线图，获取用户的头发的第三损失函数，如图6所示。
在一种实施例中，若样本数据中未包括样本对象的目标部分的样本切线图，则可以采用目标部分的切线图和目标部分的法向图进行点乘，进而采用L1损失函数公式，基于点乘结果和0获取样本对象的目标部分的第三损失函数。应理解，可以基于目标部分的掩膜图像，在样本对象的法向图中获取目标部分的法向图。
综上所示，本公开实施例中，可以对样本图像进行预处理，且将样本对象的掩膜图像和处理后的样本图像进行级联后输入至初始模型，以训练得到特征模型，可以提高特征模型的预测准确性。
上述图6所示的特征模型，相较于上述图4的特征模型，增加了样本图像预处理的过程，可以提高特征模型的预测准确性。下述本公开实施例可以基于上述图6中的特征模型，获取图像的法向特征信息和目标特征信息，进而可以提高光照渲染的效果。图7为本公开实施例提供的图像处理方法的流程示意图二。参照图7，本公开实施例提供的图像处理方法可以包括：
S801：获取待处理图像，待处理图像中包括目标对象。
S802：对待处理图像进行预处理，得到处理后的图像，处理后的图像中除目标对象之外的像素点为预设值。
终端设备可以基于待处理图像，获取目标对象的掩膜图像，进而基于掩膜图像，将待处理图像中除目标对象之外的像素点调整为预设值。基于训练得到特征模型时对样本图像的预处理的相关描述，本公开实施例中，终端设备可以采用相同的方法对待处理图像进行预处理，具体可以参照上述的相关描述。
应理解，本公开实施例中，终端设备将待处理图像进行预处理，且处理后的图像中除目 标对象之外的像素点为预设值，则在采用特征模型获取目标对象的法向特征信息和目标特征信息时，特征模型可以完全集中于前景部分的目标对象，进而能够提高获取的目标对象的法向特征信息和目标特征信息的准确性。
示例性的，待处理图像可以为图6中的A所示，目标对象为用户，目标对象的目标部分为用户的头发。终端设备可以获取用户的掩膜图像B，参照上述S406的相关描述。终端设备可以基于用户的掩膜图像B，将待处理图像中除了用户之外的像素点调整为0，即待处理图像中除了用户之外的部分调整为黑色，从而得到处理后的图像C。
S803：将处理后的图像输入特征模型，得到法向特征信息和目标特征信息。
本公开实施例中，终端设备可以将处理后的图像输入特征模型，进而得到特征模型输出的目标对象的法向特征信息和目标特征信息。在一种实施例中，法向特征信息可以为法向图，或者法向向量或法向矩阵等，下述中以法向特征信息为法向图为例进行说明。
其中，与图4所示的初始模型的结构类似的，特征模型可以包括：输入层、隐含层和输出层。输入层用于向隐含层输入待处理图像。所述隐含层用于基于待处理图像，获取待处理图像中目标对象的法向特征信息和目标特征信息，且通过输出层输出目标对象的法向特征信息和目标特征信息。
在一种实施例中，隐含层包括法向特征提取块和目标特征提取块。法向特征提取块，用于提取待处理图像的目标对象中与法向图相关的信息，且基于该与法向图相关的信息获取法向图，且通过输出层输出目标对象的法向图。法向特征提取块，还用于向目标特征提取块输出与法向图相关的信息。目标特征提取块，用于提取待处理图像的目标对象中与目标特征信息相关的信息，且基于与目标特征信息相关的信息和与法向特征信息相关的信息，获取目标特征信息，通过输出层输出目标特征信息。
在一种实施例中，目标特征提取块可以包括深度特征提取块以及切线特征提取块。在该实施例中，输入层可以用于向法向特征提取块、深度特征提取块以及切线特征提取块输入待处理图像。
法向特征提取块，用于提取待处理图像的目标对象中与法向图相关的信息，且基于该与法向图相关的信息获取法向图，且通过输出层输出目标对象的法向图。法向特征提取块，还用于向深度特征提取块和切线特征提取块输出与法向图相关的信息。深度特征提取块，用于提取待处理图像的目标对象中与深度图相关的信息，且根据与深度图相关的信息和与法向图相关的信息，获取深度图，且通过输出层输出目标对象的深度图。切线特征提取块，用于提取待处理图像的目标对象中与切线图相关的信息，且根据与切线图相关的信息和与法向图相关的信息，获取切线图，且通过输出层输出目标对象的切线图。其中，与法向图相关的信息可以为目标对象的法向特征，如法向向量、法向矩阵等。同理的，与深度图相关的信息可以为目标对象的深度特征，与切线图相关的信息可以为目标对象的切线特征。
在一种实施例中，S803中“与法向图相关的信息”可以替换为“与法向特征信息相关的信息”，“与深度图相关的信息”和“与切线图相关的信息”可以替换为“与目标特征信息相关的信息”。其中，“与深度图相关的信息”可以替换为“与深度特征信息相关的信息”，“与切线图相关的信息”可以替换为“与切线特征信息相关的信息”。
据此，终端设备将处理后的图像输入特征模型，可以提取目标对象与法向特征信息相关的信息，以及目标对象与目标特征信息相关的信息，进而基于与法向特征信息相关的信息， 获取法向特征信息，基于与法向特征信息相关的信息和与目标特征信息相关的信息，获取目标特征信息。
在一种实施例中，法向特征信息包括法向图，深度特征信息包括深度图，切线特征信息包括切线图。当目标特征信息包括深度特征信息和切线特征信息时，特征模型中的法向特征提取块可以基于处理后的图像，提取与目标对象的法向图相关的信息，深度特征提取块可以基于处理后的图像，提取与目标对象的深度图相关的信息，以及切线特征提取块可以基于处理后的图像，提取与目标对象的切线图相关的信息。法向特征提取块可以基于与目标对象的法向图相关的信息，获取目标对象的法向图。深度特征提取块可以基于与目标对象的法向图相关的信息，以及与目标对象的深度图相关的信息，获取目标对象的深度图。切线特征提取块可以基于与目标对象的法向图相关的信息，以及与目标对象的切线图相关的信息，获取目标对象的切线图。应理解，与法向图相关的信息可以为目标对象的法向特征，与深度图相关的信息可以为目标对象的深度特征，与切线图相关的信息可以为目标对象的切线特征。
示例性的，终端设备可以将除了用户之外的部分调整为黑色的“处理后的图像”输入至特征模型，进而得到用户的法向特征信息，以及目标特征信息。在一种实施例中，目标特征信息包括深度特征信息时，深度特征信息可以为用户的法向特征信息。在一种实施例中，目标特征信息包括切线特征信息时，切线特征信息可以为用户的切线特征信息或用户的头发的切线特征信息。
在一种实施例中，终端设备可以基于用户的头发的掩膜图像D，在用户的切线特征信息中获取用户的头发的切线特征信息，如将用户的切线特征信息中，头发的掩膜中头发的像素点对应的切线特征信息，作为头发的切线特征信息。
在一种实施例中，为了提高深度模型获取目标对象的特征图(法向图、深度图、切线图)的准确性，上述S803可以替换为：将目标对象的掩膜图像(如图6中的图像B)和处理后的图像(如图6中的图像C)进行级联后输入至特征模型。其中，处理后的图像中除目标对象之外的像素点为预设值，将目标对象的掩膜图像和处理后的图像进行级联可以理解为：将目标对象的掩膜图像的像素点和处理后的图像的像素点一一对应级联，将级联后的目标对象的掩膜图像和处理后的图像，输入至特征模型。
示例性的，终端设备可以将除了用户之外的部分调整为黑色的“处理后的图像”，以及将用户的掩膜图像级联后输入至特征模型，进而得到用户的法向特征信息，以及目标特征信息，可以提高法向特征信息和目标特征信息的准确性。
S804：根据法向特征信息和目标特征信息，获取光照渲染模式。
在一种实施例中，终端设备中存储有各法向特征信息和各目标特征信息对应的光照渲染模式。也就是说，法向特征信息不同，光照渲染模式不同，同理的，目标特征信息不同，光照渲染模式不同。本公开实施例中，终端设备在获取目标对象的法向特征信息和目标特征信息后，可以基于由各法向特征信息和各目标特征信息对应的光照渲染模式，获取目标对象的光照渲染模式。
在一种实施例中，终端设备可以采用表格形式、或者数据库形式或者可扩展标记语言(extensible markup language，XML)形式，存储各法向特征信息和各目标特征信息对应的光照渲染模式，本公开实施例对此不作限制。
S805：采用光照渲染模式对目标对象进行光照渲染。
终端设备在确定目标对象的光照渲染模式，可以基于该光照渲染模式对应的光照渲染参数对目标对象进行光照渲染。
以目标特征信息包括深度特征信息和切线特征信息，且切线特征信息为用户的头发的切线特征信息为例，终端设备可以获取法向特征信息、深度特征信息以及切线特征信息对应的光照渲染模式。
示例性的，终端设备采用该光照渲染模式，可以从用户的侧面，且以光照强度为a对用户进行光照，且终端设备渲染用户的头发，使得用户的头发呈现水波纹形状的光线分布。
S806：输出光照渲染后的图像。
应理解，本公开实施例中的S801、S806可以参照上述实施例中S201、S204中的相关描述。
本公开实施例中，终端设备可以对待处理图像进行预处理，将目标对象的掩膜图像和处理后的图像进行级联后输入至特征模型，以获取更为准确的目标对象的法向特征信息和目标特征信息，进而可以提高光照渲染效果。
图8为本公开实施例提供的图像处理装置的结构框图。该图像处理装置可以为上述实施例中的终端设备，也可以为终端设备中的芯片或者处理器。为了便于说明，仅示出了与本公开实施例相关的部分。参照图8，图像处理装置900包括：处理模块901和输出模块902。
处理模块901，用于获取待处理图像，待处理图像中包括目标对象；采用特征模型，获取目标对象的法向特征信息和目标特征信息，目标特征信息包括：深度特征信息和/或切线特征信息；基于法向特征信息和目标特征信息，对待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像。
输出模块902，用于输出光照渲染后的图像。
在一种可能的实现方式中，特征模型具体用于：提取目标对象与法向特征信息相关的信息，以及目标对象与目标特征信息相关的信息；基于与法向特征信息相关的信息，获取法向特征信息；基于与法向特征信息相关的信息和与目标特征信息相关的信息，获取目标特征信息。
在一种可能的实现方式中，处理模块901，还用于对待处理图像进行预处理，得到处理后的图像，处理后的图像中除目标对象之外的像素点为预设值；将处理后的图像输入特征模型，得到法向特征信息和目标特征信息。
在一种可能的实现方式中，处理模块901，具体用于基于待处理图像，获取目标对象的掩膜图像；基于掩膜图像，将待处理图像中除目标对象之外的像素点调整为预设值；将掩膜图像和处理后的图像进行级联后输入至特征模型。
在一种可能的实现方式中，处理模块901，具体用于根据法向特征信息和目标特征信息，获取光照渲染模式；采用光照渲染模式对目标对象进行光照渲染。
在一种可能的实现方式中，特征模型是以样本图像、样本图像中的样本对象的样本法向特征信息，以及样本对象的样本目标特征信息为样本数据训练得到的，样本目标特征信息包括样本深度特征信息和/或样本切线特征信息。
在一种可能的实现方式中，法向特征信息为法向图，深度特征信息为深度图，切线特征信息为切线图；样本法向特征信息为样本法向图，样本深度特征信息为样本深度图，样本切线特征信息为样本切线图。
在一种可能的实现方式中，训练过程中的特征模型为初始特征模型，所述初始特征模型中与所述初始特征模型输出的法向特征信息相关的模型参数是基于第一损失函数更新的，且所述第一损失函数是基于所述样本法向特征信息和所述初始特征模型输出的法向特征信息得到的。
在一种可能的实现方式中，所述初始特征模型中与所述初始特征模型输出的深度特征信息相关的模型参数是基于第二损失函数更新的；其中，所述第二损失函数是基于所述样本深度特征信息和所述初始特征模型输出的深度特征信息得到的；和/或，所述第二损失函数是基于所述初始特征模型输出的深度特征信息得到的法向特征信息，以及所述样本法向特征信息得到的。
在一种可能的实现方式中，所述初始特征模型中与所述初始特征模型输出的切线特征信息相关的模型参数是基于第三损失函数更新的；其中，所述第三损失函数是基于所述初始特征模型输出的切线特征信息和所述初始特征模型输出的法向特征信息得到的；和/或，所述样本数据中还包括：所述样本对象的目标部分的样本切线特征信息和所述目标部分的掩膜图像，所述目标部分具备各向异性，所述第三损失函数是基于所述目标部分的样本特征信息、所述目标部分的掩膜图像和所述初始特征模型输出的切线特征信息得到的。
在一种可能的实现方式中，特征模型包括：输入层、隐含层和输出层；输入层，用于向隐含层输入待处理图像；隐含层，用于：基于待处理图像，获取法向特征信息和目标特征信息；通过输出层输出法向特征信息和目标特征信息。
在一种可能的实现方式中，隐含层包括：法向特征提取块和目标特征提取块；法向特征提取块，用于：提取待处理图像的目标对象中与法向特征信息相关的信息；基于与法向特征信息相关的信息，获取法向特征信息；通过输出层输出法向特征信息，且向目标特征提取块输出与法向特征信息相关的信息。
目标特征提取块，用于：提取待处理图像的目标对象中与目标特征信息相关的信息；基于与目标特征信息相关的信息和与法向特征信息相关的信息，获取目标特征信息；通过输出层输出目标特征信息。
在一种可能的实现方式中，目标特征提取块包括：深度特征提取块和切线特征提取块。深度特征提取块，用于：提取待处理图像的目标对象中与深度特征信息相关的信息；基于与深度特征信息相关的信息和与法向特征信息相关的信息，获取深度特征信息；通过输出层输出深度特征信息。
切线特征提取块，用于：提取待处理图像的目标对象中与切线特征信息相关的信息；基于与切线特征信息相关的信息和与法向特征信息相关的信息，获取切线特征信息；通过输出层输出切线特征信息。
本公开实施例提供的图像处理装置，可用于执行上述方法实施例中终端上设备的步骤，其实现原理和技术效果类似，本实施例此处不再赘述。
为了实现上述实施例，本公开实施例还提供了一种电子设备。
参考图9，其示出了适于用来实现本公开实施例的电子设备1000的结构示意图。图9示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图9所示，电子设备1000可以包括处理装置(例如中央处理器、图形处理器等)1001，其可以根据存储在只读存储器(Read Only Memory，简称ROM)1002中的程序或者从存储装 置1008加载到随机访问存储器(Random Access Memory，简称RAM)1003中的程序而执行各种适当的动作和处理。在RAM 1003中，还存储有电子设备1000操作所需的各种程序和数据。处理装置1001、ROM 1002以及RAM 1003通过总线1004彼此相连。输入/输出(I/O)接口1005也连接至总线1004。应理解，处理装置1001可以执行如上图8所示的处理模块901执行的步骤。
通常，以下装置可以连接至I/O接口1005：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置1006；包括例如液晶显示器(Liquid Crystal Display，简称LCD)、扬声器、振动器等的输出装置1007；包括例如磁带、硬盘等的存储装置1008，以及通信装置1009。通信装置1009可以允许电子设备1000与其他设备进行无线或有线通信以交换数据。虽然图9示出了具有各种装置的电子设备1000，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。应理解，输出装置1007可以执行如上图8所示的输出模块902执行的步骤。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置1009从网络上被下载和安装，或者从存储装置1008被安装，或者从ROM 1002被安装。在该计算机程序被处理装置1001执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，简称RF)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备执行上述实施例所示的方法。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常 规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(Local Area Network，简称LAN)或广域网(Wide Area Network，简称WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定，例如，第一获取单元还可以被描述为“获取至少两个网际协议地址的单元”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field-Programmable Gate Array，简称FPGA)、专用集成电路(Application Specific Integrated Circuit，简称ASIC)、专用标准产品(Application Specific Standard Parts，简称ASSP)、片上系统(System on Chip，简称SOC)、复杂可编程逻辑设备(Complex Programmable Logic Device，CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，简称RAM)、只读存储器(Read-Only Memory，简称ROM)、可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，简称EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(Compact Disc Read-Only Memory，简称CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
第一方面，根据本公开的一个或多个实施例，提供了一种图像处理方法，包括：获取待处理图像，所述待处理图像中包括目标对象；采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；输出所述光照渲染后的图像。
根据本公开的一个或多个实施例，所述特征模型具体用于：提取所述目标对象与所述法向特征信息相关的信息，以及所述目标对象与所述目标特征信息相关的信息；基于所述与所述法向特征信息相关的信息，获取所述法向特征信息；基于所述与所述法向特征信息相关的 信息和与所述目标特征信息相关的信息，获取所述目标特征信息。
根据本公开的一个或多个实施例，所述采用特征模型，获取所述目标对象的法向特征信息和目标特征信息之前，还包括：对所述待处理图像进行预处理，得到处理后的图像，所述处理后的图像中除所述目标对象之外的像素点为预设值；所述采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，包括：将所述处理后的图像输入所述特征模型，得到所述法向特征信息和所述目标特征信息。
根据本公开的一个或多个实施例，所述对所述待处理图像进行预处理，得到处理后的图像，包括：基于所述待处理图像，获取所述目标对象的掩膜图像；基于所述掩膜图像，将所述待处理图像中除所述目标对象之外的像素点调整为所述预设值；所述将所述处理后的图像输入所述特征模型，包括：将所述掩膜图像和所述处理后的图像进行级联后输入至所述特征模型。
根据本公开的一个或多个实施例，所述对所述待处理图像中的目标对象进行光照渲染，包括：根据所述法向特征信息和所述目标特征信息，获取光照渲染模式；采用所述光照渲染模式对所述目标对象进行光照渲染。
根据本公开的一个或多个实施例，所述特征模型是以样本图像、所述样本图像中的样本对象的样本法向特征信息，以及所述样本对象的样本目标特征信息为样本数据训练得到的，所述样本目标特征信息包括样本深度特征信息和/或样本切线特征信息。
根据本公开的一个或多个实施例，所述法向特征信息为法向图，所述深度特征信息为深度图，所述切线特征信息为切线图；所述样本法向特征信息为样本法向图，所述样本深度特征信息为样本深度图，所述样本切线特征信息为样本切线图。
根据本公开的一个或多个实施例，训练过程中的特征模型为初始特征模型，所述初始特征模型中与所述初始特征模型输出的法向特征信息相关的模型参数是基于第一损失函数更新的，且所述第一损失函数是基于所述样本法向特征信息和所述初始特征模型输出的法向特征信息得到的。
根据本公开的一个或多个实施例，所述初始特征模型中与所述初始特征模型输出的深度特征信息相关的模型参数是基于第二损失函数更新的；其中，所述第二损失函数是基于所述样本深度特征信息和所述初始特征模型输出的深度特征信息得到的；和/或，所述第二损失函数是基于所述初始特征模型输出的深度特征信息得到的法向特征信息，以及所述样本法向特征信息得到的。
根据本公开的一个或多个实施例，所述初始特征模型中与所述初始特征模型输出的切线特征信息相关的模型参数是基于第三损失函数更新的；其中，所述第三损失函数是基于所述初始特征模型输出的切线特征信息和所述初始特征模型输出的法向特征信息得到的；和/或，所述样本数据中还包括：所述样本对象的目标部分的样本切线特征信息和所述目标部分的掩膜图像，所述目标部分具备各向异性，所述第三损失函数是基于所述目标部分的样本特征信息、所述目标部分的掩膜图像和所述初始特征模型输出的切线特征信息得到的。
根据本公开的一个或多个实施例，特征模型包括：输入层、隐含层和输出层；输入层，用于向隐含层输入待处理图像；隐含层，用于：基于待处理图像，获取法向特征信息和目标特征信息；通过输出层输出法向特征信息和目标特征信息。
根据本公开的一个或多个实施例，隐含层包括：法向特征提取块和目标特征提取块；法 向特征提取块，用于：提取待处理图像的目标对象中与法向特征信息相关的信息；基于与法向特征信息相关的信息，获取法向特征信息；通过输出层输出法向特征信息，且向目标特征提取块输出与法向特征信息相关的信息。
目标特征提取块，用于：提取待处理图像的目标对象中与目标特征信息相关的信息；基于与目标特征信息相关的信息和与法向特征信息相关的信息，获取目标特征信息；通过输出层输出目标特征信息。
根据本公开的一个或多个实施例，目标特征提取块包括：深度特征提取块和切线特征提取块。深度特征提取块，用于：提取待处理图像的目标对象中与深度特征信息相关的信息；基于与深度特征信息相关的信息和与法向特征信息相关的信息，获取深度特征信息；通过输出层输出深度特征信息。
切线特征提取块，用于：提取待处理图像的目标对象中与切线特征信息相关的信息；基于与切线特征信息相关的信息和与法向特征信息相关的信息，获取切线特征信息；通过输出层输出切线特征信息。
第二方面，根据本公开的一个或多个实施例，提供了一种图像处理装置，包括：处理模块，用于获取待处理图像，所述待处理图像中包括目标对象；采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像。
输出模块，用于输出所述光照渲染后的图像。
根据本公开的一个或多个实施例，所述特征模型具体用于：提取所述目标对象与所述法向特征信息相关的信息，以及所述目标对象与所述目标特征信息相关的信息；基于所述与所述法向特征信息相关的信息，获取所述法向特征信息；基于所述与所述法向特征信息相关的信息和与所述目标特征信息相关的信息，获取所述目标特征信息。
根据本公开的一个或多个实施例，处理模块，还用于对所述待处理图像进行预处理，得到处理后的图像，所述处理后的图像中除所述目标对象之外的像素点为预设值；将所述处理后的图像输入所述特征模型，得到所述法向特征信息和所述目标特征信息。
根据本公开的一个或多个实施例，处理模块，具体用于基于所述待处理图像，获取所述目标对象的掩膜图像；基于所述掩膜图像，将所述待处理图像中除所述目标对象之外的像素点调整为所述预设值；将所述掩膜图像和所述处理后的图像进行级联后输入至所述特征模型。
根据本公开的一个或多个实施例，处理模块，具体用于根据所述法向特征信息和所述目标特征信息，获取光照渲染模式；采用所述光照渲染模式对所述目标对象进行光照渲染。
根据本公开的一个或多个实施例，所述特征模型是以样本图像、所述样本图像中的样本对象的样本法向特征信息，以及所述样本对象的样本目标特征信息为样本数据训练得到的，所述样本目标特征信息包括样本深度特征信息和/或样本切线特征信息。
根据本公开的一个或多个实施例，所述法向特征信息为法向图，所述深度特征信息为深度图，所述切线特征信息为切线图；所述样本法向特征信息为样本法向图，所述样本深度特征信息为样本深度图，所述样本切线特征信息为样本切线图。
根据本公开的一个或多个实施例，训练过程中的特征模型为初始特征模型，所述初始特征模型中与所述初始特征模型输出的法向特征信息相关的模型参数是基于第一损失函数更新 的，且所述第一损失函数是基于所述样本法向特征信息和所述初始特征模型输出的法向特征信息得到的。
根据本公开的一个或多个实施例，所述初始特征模型中与所述初始特征模型输出的深度特征信息相关的模型参数是基于第二损失函数更新的；其中，所述第二损失函数是基于所述样本深度特征信息和所述初始特征模型输出的深度特征信息得到的；和/或，所述第二损失函数是基于所述初始特征模型输出的深度特征信息得到的法向特征信息，以及所述样本法向特征信息得到的。
根据本公开的一个或多个实施例，所述初始特征模型中与所述初始特征模型输出的切线特征信息相关的模型参数是基于第三损失函数更新的；其中，所述第三损失函数是基于所述初始特征模型输出的切线特征信息和所述初始特征模型输出的法向特征信息得到的；和/或，所述样本数据中还包括：所述样本对象的目标部分的样本切线特征信息和所述目标部分的掩膜图像，所述目标部分具备各向异性，所述第三损失函数是基于所述目标部分的样本特征信息、所述目标部分的掩膜图像和所述初始特征模型输出的切线特征信息得到的。
根据本公开的一个或多个实施例，特征模型包括：输入层、隐含层和输出层；输入层，用于向隐含层输入待处理图像；隐含层，用于：基于待处理图像，获取法向特征信息和目标特征信息；通过输出层输出法向特征信息和目标特征信息。
根据本公开的一个或多个实施例，隐含层包括：法向特征提取块和目标特征提取块；法向特征提取块，用于：提取待处理图像的目标对象中与法向特征信息相关的信息；基于与法向特征信息相关的信息，获取法向特征信息；通过输出层输出法向特征信息，且向目标特征提取块输出与法向特征信息相关的信息。
目标特征提取块，用于：提取待处理图像的目标对象中与目标特征信息相关的信息；基于与目标特征信息相关的信息和与法向特征信息相关的信息，获取目标特征信息；通过输出层输出目标特征信息。
根据本公开的一个或多个实施例，目标特征提取块包括：深度特征提取块和切线特征提取块。深度特征提取块，用于：提取待处理图像的目标对象中与深度特征信息相关的信息；基于与深度特征信息相关的信息和与法向特征信息相关的信息，获取深度特征信息；通过输出层输出深度特征信息。
切线特征提取块，用于：提取待处理图像的目标对象中与切线特征信息相关的信息；基于与切线特征信息相关的信息和与法向特征信息相关的信息，获取切线特征信息；通过输出层输出切线特征信息。
第三方面，根据本公开的一个或多个实施例，提供了一种电子设备，包括：处理器和存储器；所述存储器存储计算机执行指令；所述处理器执行所述存储器存储的计算机执行指令，使得所述处理器执行如上第一方面以及第一方面各种可能的设计所述的方法。
第四方面，根据本公开的一个或多个实施例，提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如上第一方面以及第一方面各种可能的设计所述的方法。
第五方面，根据本公开的一个或多个实施例，提供了一种计算机程序产品，包括计算机程序或指令，所述计算机程序或所述指令被处理器执行时，实现如上第一方面以及第一方面各种可能的设计所述的方法。
第六方面，根据本公开的一个或多个实施例，提供了一种计算机程序，所述计算机程序被处理器执行时执行如上第一方面以及第一方面各种可能的设计所述的方法。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (18)

  1. 一种图像处理方法，其特征在于，包括：
    获取待处理图像，所述待处理图像中包括目标对象；
    采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；
    基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；
    输出所述光照渲染后的图像。
  2. 根据权利要求1所述的方法，其特征在于，所述特征模型具体用于：
    提取所述目标对象与所述法向特征信息相关的信息，以及所述目标对象与所述目标特征信息相关的信息；
    基于所述与所述法向特征信息相关的信息，获取所述法向特征信息；
    基于所述与所述法向特征信息相关的信息和与所述目标特征信息相关的信息，获取所述目标特征信息。
  3. 根据权利要求1或2所述的方法，其特征在于，所述采用特征模型，获取所述目标对象的法向特征信息和目标特征信息之前，还包括：
    对所述待处理图像进行预处理，得到处理后的图像，所述处理后的图像中除所述目标对象之外的像素点为预设值；
    所述采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，包括：
    将所述处理后的图像输入所述特征模型，得到所述法向特征信息和所述目标特征信息。
  4. 根据权利要求3所述的方法，其特征在于，所述对所述待处理图像进行预处理，得到处理后的图像，包括：
    基于所述待处理图像，获取所述目标对象的掩膜图像；
    基于所述掩膜图像，将所述待处理图像中除所述目标对象之外的像素点调整为所述预设值；
    所述将所述处理后的图像输入所述特征模型，包括：
    将所述掩膜图像和所述处理后的图像进行级联后输入至所述特征模型。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述对所述待处理图像中的目标对象进行光照渲染，包括：
    根据所述法向特征信息和所述目标特征信息，获取光照渲染模式；
    采用所述光照渲染模式对所述目标对象进行光照渲染。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，
    所述特征模型是以样本图像、所述样本图像中的样本对象的样本法向特征信息，以及所述样本对象的样本目标特征信息为样本数据训练得到的，所述样本目标特征信息包括样本深度特征信息和/或样本切线特征信息。
  7. 根据权利要求6所述的方法，其特征在于，所述法向特征信息包括法向图，所述深度特征信息包括深度图，所述切线特征信息包括切线图；所述样本法向特征信息包括样本法向图，所述样本深度特征信息包括样本深度图，所述样本切线特征信息包括样本 切线图。
  8. 根据权利要求1-6中任一项所述的方法，其特征在于，训练过程中的特征模型为初始特征模型，所述初始特征模型中与所述初始特征模型输出的法向特征信息相关的模型参数是基于第一损失函数更新的，且所述第一损失函数是基于所述样本法向特征信息和所述初始特征模型输出的法向特征信息得到的。
  9. 根据权利要求8所述的方法，其特征在于，所述初始特征模型中与所述初始特征模型输出的深度特征信息相关的模型参数是基于第二损失函数更新的；
    其中，所述第二损失函数是基于所述样本深度特征信息和所述初始特征模型输出的深度特征信息得到的；和/或，
    所述第二损失函数是基于所述初始特征模型输出的深度特征信息得到的法向特征信息，以及所述样本法向特征信息得到的。
  10. 根据权利要求8或9所述的方法，其特征在于，所述初始特征模型中与所述初始特征模型输出的切线特征信息相关的模型参数是基于第三损失函数更新的；
    其中，所述第三损失函数是基于所述初始特征模型输出的切线特征信息和所述初始特征模型输出的法向特征信息得到的；和/或，
    所述样本数据中还包括：所述样本对象的目标部分的样本切线特征信息和所述目标部分的掩膜图像，所述目标部分具备各向异性，所述第三损失函数是基于所述目标部分的样本特征信息、所述目标部分的掩膜图像和所述初始特征模型输出的切线特征信息得到的。
  11. 根据权利要求1或2所述的方法，其特征在于，所述特征模型包括：输入层、隐含层和输出层；
    所述输入层，用于向所述隐含层输入所述待处理图像；
    所述隐含层，用于：
    基于所述待处理图像，获取所述法向特征信息和所述目标特征信息；
    通过所述输出层输出所述法向特征信息和所述目标特征信息。
  12. 根据权利要求11所述的方法，其特征在于，所述隐含层包括：法向特征提取块和目标特征提取块；
    所述法向特征提取块，用于：
    提取所述待处理图像的所述目标对象中与法向特征信息相关的信息；
    基于所述与法向特征信息相关的信息，获取所述法向特征信息；
    通过所述输出层输出所述法向特征信息，且向所述目标特征提取块输出所述与法向特征信息相关的信息；
    所述目标特征提取块，用于：
    提取所述待处理图像的所述目标对象中与目标特征信息相关的信息；
    基于所述与目标特征信息相关的信息和所述与法向特征信息相关的信息，获取所述目标特征信息；
    通过所述输出层输出所述目标特征信息。
  13. 根据权利要求12所述的方法，其特征在于，所述目标特征提取块包括：深度特征提取块和切线特征提取块；
    所述深度特征提取块，用于：
    提取所述待处理图像的所述目标对象中与深度特征信息相关的信息；
    基于所述与深度特征信息相关的信息和所述与法向特征信息相关的信息，获取所述深度特征信息；
    通过所述输出层输出所述深度特征信息；
    所述切线特征提取块，用于：
    提取所述待处理图像的所述目标对象中与切线特征信息相关的信息；
    基于所述与切线特征信息相关的信息和所述与法向特征信息相关的信息，获取所述切线特征信息；
    通过所述输出层输出所述切线特征信息。
  14. 一种图像处理装置，其特征在于，包括：
    处理模块，用于获取待处理图像，所述待处理图像中包括目标对象；采用特征模型，获取所述目标对象的法向特征信息和目标特征信息，所述目标特征信息包括：深度特征信息和/或切线特征信息；基于所述法向特征信息和所述目标特征信息，对所述待处理图像中的目标对象进行光照渲染，得到光照渲染后的图像；
    输出模块，用于输出所述光照渲染后的图像。
  15. 一种电子设备，其特征在于，包括：处理器和存储器；
    所述存储器存储计算机执行指令；
    所述处理器执行所述存储器存储的计算机执行指令，使得所述处理器执行如权利要求1-13中任一项所述的方法。
  16. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如权利要求1-13中任一项所述的方法。
  17. 一种计算机程序产品，包括计算机程序或指令，其特征在于，所述计算机程序或所述指令被处理器执行时，实现权利要求1-13中任一项所述的方法。
  18. 一种计算机程序，其特征在于，所述计算机程序被处理器执行时执行如权利要求1-13中任一项所述的方法。
PCT/CN2022/083822 2021-04-28 2022-03-29 图像处理方法、装置、电子设备以及可读存储介质 WO2022227996A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/551,392 US20240177409A1 (en) 2021-04-28 2022-03-29 Image processing method and apparatus, electronic device, and readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110468921.0A CN115249215A (zh) 2021-04-28 2021-04-28 图像处理方法、装置、电子设备以及可读存储介质
CN202110468921.0 2021-04-28

Publications (1)

Publication Number Publication Date
WO2022227996A1 true WO2022227996A1 (zh) 2022-11-03

Family

ID=83697414

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/083822 WO2022227996A1 (zh) 2021-04-28 2022-03-29 图像处理方法、装置、电子设备以及可读存储介质

Country Status (3)

Country Link
US (1) US20240177409A1 (zh)
CN (1) CN115249215A (zh)
WO (1) WO2022227996A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115439616A (zh) * 2022-11-07 2022-12-06 成都索贝数码科技股份有限公司 基于多对象图像α叠加的异构对象表征方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150043806A1 (en) * 2013-08-08 2015-02-12 Adobe Systems Incorporated Automatic geometry and lighting inference for realistic image editing
US9905045B1 (en) * 2014-03-27 2018-02-27 Pixar Statistical hair scattering model
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
CN110853128A (zh) * 2019-11-12 2020-02-28 腾讯科技（深圳）有限公司 虚拟物体显示方法、装置、计算机设备及存储介质
CN111833430A (zh) * 2019-04-10 2020-10-27 上海科技大学 基于神经网络的光照数据预测方法、系统、终端及介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150043806A1 (en) * 2013-08-08 2015-02-12 Adobe Systems Incorporated Automatic geometry and lighting inference for realistic image editing
US9905045B1 (en) * 2014-03-27 2018-02-27 Pixar Statistical hair scattering model
CN111833430A (zh) * 2019-04-10 2020-10-27 上海科技大学 基于神经网络的光照数据预测方法、系统、终端及介质
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
CN110853128A (zh) * 2019-11-12 2020-02-28 腾讯科技（深圳）有限公司 虚拟物体显示方法、装置、计算机设备及存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115439616A (zh) * 2022-11-07 2022-12-06 成都索贝数码科技股份有限公司 基于多对象图像α叠加的异构对象表征方法
CN115439616B (zh) * 2022-11-07 2023-02-14 成都索贝数码科技股份有限公司 基于多对象图像α叠加的异构对象表征方法

Also Published As

Publication number Publication date
US20240177409A1 (en) 2024-05-30
CN115249215A (zh) 2022-10-28

Similar Documents

Publication Publication Date Title
US20200294249A1 (en) Network module and distribution method and apparatus, electronic device, and storage medium
US11734804B2 (en) Face image processing method and apparatus, electronic device, and storage medium
CN111243049B (zh) 人脸图像的处理方法、装置、可读介质和电子设备
WO2022042290A1 (zh) 一种虚拟模型处理方法、装置、电子设备和存储介质
CN110062157B (zh) 渲染图像的方法、装置、电子设备和计算机可读存储介质
US20230421716A1 (en) Video processing method and apparatus, electronic device and storage medium
WO2023207379A1 (zh) 图像处理方法、装置、设备及存储介质
CN109102484B (zh) 用于处理图像的方法和装置
CN110070499A (zh) 图像处理方法、装置和计算机可读存储介质
CN114445562A (zh) 三维重建方法及装置、电子设备和存储介质
WO2023226628A1 (zh) 图像展示方法、装置、电子设备及存储介质
WO2023138441A1 (zh) 视频生成方法、装置、设备及存储介质
CN112967193A (zh) 图像校准方法及装置、计算机可读介质和电子设备
WO2022227996A1 (zh) 图像处理方法、装置、电子设备以及可读存储介质
CN114331823A (zh) 图像处理方法、装置、电子设备及存储介质
CN112465940B (zh) 图像渲染方法、装置、电子设备及存储介质
CN110047126B (zh) 渲染图像的方法、装置、电子设备和计算机可读存储介质
CN110619602B (zh) 一种图像生成方法、装置、电子设备及存储介质
CN109816791B (zh) 用于生成信息的方法和装置
WO2022237435A1 (zh) 更换画面中的背景的方法、设备、存储介质及程序产品
CN113963000B (zh) 图像分割方法、装置、电子设备及程序产品
CN112053450B (zh) 文字的显示方法、装置、电子设备及存储介质
CN112950516B (zh) 图像局部对比度增强的方法及装置、存储介质及电子设备
CN114549607A (zh) 主体材质确定方法、装置、电子设备及存储介质
CN110807728B (zh) 对象的显示方法、装置、电子设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22794469

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18551392

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22794469

Country of ref document: EP

Kind code of ref document: A1