WO2021164289A1 - Procédé et appareil de traitement de portrait, et terminal - Google Patents

Procédé et appareil de traitement de portrait, et terminal Download PDF

Info

Publication number
WO2021164289A1
WO2021164289A1 PCT/CN2020/122767 CN2020122767W WO2021164289A1 WO 2021164289 A1 WO2021164289 A1 WO 2021164289A1 CN 2020122767 W CN2020122767 W CN 2020122767W WO 2021164289 A1 WO2021164289 A1 WO 2021164289A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
user
terminal
portrait processing
processor
Prior art date
Application number
PCT/CN2020/122767
Other languages
English (en)
Chinese (zh)
Inventor
田春长
汪亮
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021164289A1 publication Critical patent/WO2021164289A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting

Definitions

  • This application relates to the field of terminal technology, and more specifically, to a portrait processing method and device in the field of terminal technology, and a terminal.
  • Immersive video call means that both parties wear VR devices to make a video call.
  • Each caller is equipped with a camera.
  • the camera can capture the real scene of the caller during the video call, and transmit it to the other party's VR device to present it for simulation.
  • the call effect of real-time conversation with the other party.
  • the VR device when a user wears a VR device to make a video call, the VR device will cover part of the user’s face. For example, VR glasses will cover the user’s eyes. Therefore, when the user’s call is captured through the camera, the other party cannot When it comes to facial features such as the user's eye movements and expressions, the user experience is poor.
  • the present application provides a portrait processing method and device, and a terminal, which can restore the facial features of a user blocked by a VR device, thereby improving the user experience of the receiving end.
  • the present application provides a portrait processing method.
  • the method includes: acquiring a first image of a user, the first image includes the user's face, and a partial area of the user's face is captured by a virtual reality VR device. Occlusion, the partial area includes the area where the eyes of the user are located; the first image is input into the portrait processing model to obtain a second image, and the second image includes the complete face of the user, wherein the portrait
  • the processing model is obtained by training a sample training data set.
  • the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are at least Is collected by a sample user, the first original image in the plurality of original images includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the plurality of original images
  • the first restored image in the restored image includes the complete face of the first sample user, and the at least one sample user includes the first sample user; and the second image is sent to the receiving end.
  • Using the portrait processing method provided by the embodiment of the present application can restore the user's facial features hidden by the VR device, thereby improving the user experience at the receiving end.
  • the portrait processing apparatus may also obtain the first image in another manner, which is not limited in the embodiment of the present application.
  • the portrait processing device may receive a third image taken by the first camera device, the third image includes the user, and the portrait processing device obtains the third image from the third image. To intercept the first image.
  • the first camera device may be the camera device 120 as shown in FIG. 1.
  • the first camera device and the portrait processing device may be independent devices, or the first camera device and the portrait processing device may be integrated in the same device, which is not limited in the embodiment of the present application.
  • the third image may include at least the user. That is to say, the third image may also include the environment or scene where the user is located, which is not limited in the embodiment of the present application.
  • the embodiment of the present application only takes the partial area of the user's face as the area where the eyes are for introduction. It should be understood that this partial area may also include other areas of the user's face. There is no restriction on this.
  • the first image may include at least the face of the user, wherein a partial area of the face of the user is blocked by the VR device.
  • the first image may also include at least one of other parts of the user and the environment in which the user is located, or the first image may also include users other than the user, wherein, the method for processing the portrait of other users is similar to the method for processing the portrait of the user, and in order to avoid repetition, it will not be repeated here.
  • the portrait processing device may adopt a portrait similar to that of the user.
  • the processing method is similar to other users' portrait processing, in order to avoid repetition, it will not be repeated here.
  • the first image described in the embodiment of the present application may be a separate image, or may be a frame of image in a video stream, which is not limited in the embodiment of the present application.
  • the first image may be an image originally taken, or an image with higher definition obtained after basic image quality processing.
  • the basic image quality processing described in the embodiment of the present application may include at least one image processing step for improving image quality, such as denoising, sharpening, and brightness enhancement, etc.
  • image processing step for improving image quality such as denoising, sharpening, and brightness enhancement, etc. This embodiment of the present application does not do this. limited.
  • the portrait processing apparatus may obtain the portrait processing model in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device may be pre-configured with the portrait processing model when it leaves the factory.
  • the portrait processing apparatus may receive the portrait processing model from other devices, that is, the portrait processing model is trained by other devices.
  • the portrait processing model may be stored in a cloud server, and the portrait processing device may request the portrait processing model from the cloud server via the network.
  • the portrait processing apparatus may train the portrait processing model by itself.
  • the portrait processing device may obtain a sample training data set, and train and learn the sample training data set to obtain the portrait processing model.
  • the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the multiple original images
  • the first original image in the image includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image in the plurality of restored images includes the first A complete face of the sample user, and the at least one sample user includes the first sample user.
  • the portrait processing apparatus may train and learn the sample training data set through a variety of methods to obtain the portrait processing model, which is not limited in the embodiment of the present application.
  • the portrait processing device may train and learn the sample training data set through a neural network model to obtain the portrait processing model.
  • the neural network model may be a generative adversarial nets (GAN) model.
  • the GAN may include a conditional GAN, a deep convolution GAN, and so on.
  • the facial features of the user hidden by the VR device in the first image can be restored through the portrait processing model, thereby improving the user experience at the receiving end.
  • the portrait processing device may input the first image and feature reference information into the portrait processing model to obtain the second image.
  • the feature reference information includes at least one of eye feature parameters and head feature parameters
  • the eye feature parameters include at least one of position information, size information, and viewing angle information
  • the position information is used for Indicates the position of each of the two eyes of the user
  • the size information is used to indicate the size of each eye
  • the viewing angle information is used to indicate the gaze angle of each eye
  • the characteristic parameters include the three-axis posture angle and acceleration of the user's head.
  • the portrait processing method provided by the embodiments of the present application, combining the first image and feature reference information, and inputting the portrait processing device, by providing more portrait-related features, the degree of restoration and authenticity of the occluded area can be improved.
  • the portrait processing apparatus may obtain the feature reference information in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device may receive a fourth image captured by a second camera device, the second camera device is a camera device built in the VR device, and the fourth image includes all The two eyes of the user; the portrait processing device may extract the eye feature parameters from the fourth image.
  • the second camera device may be a camera device built in the VR device, for example, a built-in infrared (IR) camera.
  • IR infrared
  • the embodiment of the present application only takes the second camera device as a built-in camera device on the VR device as an example.
  • a camera device for real images of the facial area blocked by the VR device which is not limited in the embodiment of the present application.
  • the portrait processing apparatus may extract the eye feature parameters from multiple images of the user stored locally, where the multiple images include the eyes of the user.
  • the multiple images may be photos containing faces taken daily by the user, for example, selfies.
  • the portrait processing device may establish a facial feature database of the user through photos taken daily by the user.
  • the facial feature database includes feature parameters of various facial organs, such as eye feature parameters,
  • the portrait processing device can retrieve the eye feature parameters from the facial feature database.
  • the portrait processing device may receive the eye characteristic parameters from other measuring devices, such as an eye tracking sensor.
  • the multiple images and the facial feature database may also be stored in the cloud, and the portrait processing device may request the cloud through the network, which is not limited in the embodiment of the present application.
  • the VR device when restoring the occluded content in the first image through the portrait processing model, since the VR device mainly occludes the user's eye area, in combination with the user's eye feature parameters, It can improve the degree of restoration and authenticity of the blocked content.
  • the portrait processing device may receive the head characteristic parameters measured by an inertial measurement device.
  • an inertial measurement device can also be an inertial measurement unit (IMU), which is a device that measures the three-axis attitude angle (or angular velocity) and acceleration of an object.
  • IMU inertial measurement unit
  • an IMU contains three single-axis accelerometers and three single-axis gyroscopes.
  • the accelerometer detects the acceleration signal of the object in the independent three-axis coordinate system of the carrier, while the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Measure the angular velocity and acceleration of the object in the three-dimensional space, and calculate the posture of the object based on this.
  • the inertial measurement device may be an independent measurement device fixed on the user's head, or the inertial measurement device may be integrated in the VR device, which is not limited in the embodiment of the present application.
  • the user's head may be in a motion state, that is, it has a motion speed and a rotation angle, it is combined with the user
  • the characteristic parameters of the head can further improve the restoration and authenticity of the occluded content.
  • the feature reference information may also include other feature parameters, such as nose feature parameters, face feature parameters, hairstyle feature parameters, and other parameters that can describe the user's portrait feature, which is not limited in the embodiment of the present application.
  • the portrait processing device may send the second image to the receiving end, or the portrait processing device may send a target image to the receiving end, and the target image is a result of performing a process on the second image.
  • the synthesized or spliced image, or the portrait processing device may send a video stream to the receiving end, and the video stream includes the second image or the target image, which is not limited in this embodiment of the application.
  • the portrait processing device may send the second image, or the target image or video stream containing the second image, to the receiving end.
  • the portrait processing apparatus may generate the video stream in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device sending the second image to the receiving end includes: the portrait processing device splicing the second image and the fifth image to obtain the target image, wherein: The fifth image is an image obtained by intercepting the first image from the fourth image; the portrait processing device sends the target image to the receiving end.
  • the portrait processing device sending the second image to the receiving end includes: the portrait processing device synthesizes the second image and the fourth image to obtain a target image, Wherein, the second image covers the upper layer of the first image in the fourth image; the portrait processing device sends the target image to the receiving end.
  • the portrait processing apparatus may generate the video stream in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing apparatus may perform video encoding on the second image to obtain a video image; obtain the video stream according to the video image, and the video stream includes the video image .
  • the restored second image restores the user's eye area blocked by the VR device
  • the actual situation is that the user's face is wearing the VR device
  • the second image is different from the actual situation. There are certain differences between.
  • the portrait processing device may superimpose an eye mask layer on the eye area of the user in the second image, wherein the eye mask layer undergoes a perspective processing of the first transparency to simulate the wear of the user With the VR device, the authenticity of the restoration can be improved.
  • the value of the first transparency should satisfy that the user can see that the user is wearing the VR device without affecting the display of the obscured eye area under the VR device.
  • the specific value of is not limited.
  • an embodiment of the present application provides a terminal, including: a processor and a transceiver coupled with the processor,
  • the processor is configured to control the transceiver to obtain a first image of the user, the first image includes the user's face, and a partial area of the user's face is blocked by a virtual reality VR device;
  • the image is input into a portrait processing model to obtain a second image, the second image includes the complete face of the user, wherein the portrait processing model is obtained by training on a sample training data set, and the sample training data set includes multiple Original images and a plurality of restored images corresponding to the plurality of original images, wherein the plurality of original images are collected from at least one sample user, and the first original image of the plurality of original images includes the first original image
  • the first restored image in the plurality of restored images includes the complete face of the first sample user, the At least one sample user includes the first sample user; and controls the transceiver to send the second image to the receiving end.
  • the processor is specifically configured to: input the first image and feature reference information into the portrait processing model to obtain the second image, wherein the feature reference information includes eye At least one of the head feature parameter and the head feature parameter, the eye feature parameter includes at least one of position information, size information, and angle of view information, and the position information is used to indicate whether the user’s two eyes The position of each eye, the size information is used to indicate the size of each eye, the viewing angle information is used to indicate the gaze angle of each eye, and the head feature parameter includes the user's head The three-axis attitude angle and acceleration of the part.
  • the feature reference information includes the eye feature parameters
  • the processor is further configured to input the first image and feature reference information into the portrait processing model to obtain Before the second image, control the transceiver to receive a third image of the user captured by the first camera device, the third image including the two eyes of the user, wherein the first camera Is a built-in camera device of the VR device; the processor is used to extract the eye feature parameters from the second image.
  • the feature reference information includes the head feature parameters
  • the processor is further configured to input the first image and feature reference information into the portrait processing model to obtain Before the second image, control the transceiver to receive the head characteristic parameter measured by the inertial measurement device.
  • the processor is specifically configured to control the transceiver to receive the first image taken by the second camera.
  • the processor is specifically configured to: control the transceiver to receive a third image taken by a second camera, the third image including the user; and to intercept the third image The first image.
  • the processor is further configured to: stitch the second image and the fourth image to obtain a target image, wherein the fourth image is a cut from the third image The image after the first image; controlling the transceiver to send the target image to the receiving end.
  • the processor is further configured to: synthesize the second image and the third image to obtain a target image, wherein the second image is overlaid on the third image The upper layer of the first image in; controlling the transceiver to send the target image to the receiving end.
  • the processor is further configured to superimpose an eye mask layer on the area where the eyes of the user in the second image are located, and the eye mask layer is subjected to a perspective processing of the first transparency.
  • the portrait processing model is obtained by training the sample training data set by generating a confrontation network model.
  • the present application also provides a portrait processing device, which is used to execute the foregoing first aspect or any possible implementation of the first aspect.
  • the portrait processing apparatus may include a unit for executing the above-mentioned first aspect or any possible implementation of the first aspect.
  • an embodiment of the present application also provides a chip device, including: a communication interface and a processor, the communication interface and the processor communicate with each other through an internal connection path, and the processor is used to implement the first Aspect or any of its possible implementations.
  • an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, the computer program including instructions for implementing the method in the first aspect or any possible implementation manner thereof.
  • the embodiments of the present application also provide a computer program product.
  • the computer program product contains instructions. When the instructions are run on a computer, the computer can implement the first aspect or any of its possible implementations. Methods.
  • FIG. 1 provides a schematic diagram of an application scenario 100 of an embodiment of the present application
  • Figure 2 provides a schematic diagram of another application scenario of an embodiment of the present application.
  • FIG. 3 provides a schematic flowchart of a portrait processing method 200 according to an embodiment of the present application
  • FIG. 4 provides a schematic block diagram of the VR device 110 according to an embodiment of the present application.
  • FIG. 5 provides a schematic diagram of the receiving end display interface of the embodiment of the present application.
  • FIG. 6 provides a schematic block diagram of a portrait processing apparatus 300 according to an embodiment of the present application.
  • FIG. 7 provides a schematic block diagram of a mobile phone 400 according to an embodiment of the present application.
  • FIG. 8 provides a schematic block diagram of a portrait processing system 500 according to an embodiment of the present application.
  • Fig. 1 shows a schematic diagram of an application scenario provided by an embodiment of the present application. As shown in FIG. 1, the user wears the VR device 110.
  • the camera device 120 is used to take a scene image of the user and send the scene image to the portrait processing device 130, where the scene image includes the user's face, and a partial area of the user's face is blocked by the VR device 110.
  • the portrait processing device 130 is used for restoring the area of the scene image that is blocked by the VR device by the user through a portrait processing method to obtain a target image, wherein the target image includes a complete face of the user.
  • the portrait processing device 130 is also used to transmit the target image to the receiving end 140 for display.
  • FIG. 1 only schematically shows a situation where the scene image includes the face of the user, but the embodiment of the present application is not limited to this.
  • the scene image may also include at least one of other parts of the user and the environment in which the user is located, or the scene image may also include users other than the user.
  • the embodiment does not limit this.
  • Figure 2 shows a situation where the scene image includes the user's whole body.
  • the portrait processing device 130 may use the same The portrait processing method is similar to the portrait processing of other users. In order to avoid repetition, it will not be repeated here.
  • the scene image described in the embodiment of the present application may be a separate image, or may be a frame of image in a video stream, which is not limited in the embodiment of the present application.
  • the scene image of the user is displayed through the display device (receiving end) of the opposite user.
  • the receiving end described in the embodiment of the present application may be any display device that can receive and display the target image sent by the terminal, which is not limited in the embodiment of the present application.
  • a scene image of the user is displayed through a display device (receiving end).
  • the receiving end may also be a terminal with a display screen, such as a VR device, a mobile phone, or a smart TV, which is not limited in the embodiment of the present application.
  • the camera 120 and the portrait processing device 130 in FIG. 1 may be independent devices, or the camera 120 and the portrait processing device 130 may be integrated in the same device, which is not limited in the embodiment of the present application.
  • the VR device described in the embodiments of the present application may be a wearable device capable of implementing virtual reality functions.
  • wearable devices are also called wearable smart devices, which are the general term for the application of wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, helmets, and masks.
  • a wearable device is a portable device that is worn directly on the body or integrated into the user's accessories.
  • Wearable devices are not only a kind of hardware device, but also realize powerful functions through software support, data interaction, and cloud interaction.
  • wearable smart devices include full-featured, large-sized, complete or partial functions that can be achieved without relying on smart phones, such as smart helmets or smart glasses, and only focus on a certain type of application function, and need to be used in conjunction with other devices such as smart phones.
  • various smart bracelets, smart jewelry, patches, etc. capable of portrait processing and/or image display are not limited in the embodiments of the present application.
  • the portrait processing device 130 and the receiving end 140 may communicate in a wired or wireless manner, which is not limited in the embodiment of the present application.
  • wired manner may be connected through a data line or through an internal bus connection to achieve communication.
  • the above-mentioned wireless method may be implemented through a communication network
  • the communication network may be a local area network, or a wide area network switched by a relay device, or include a local area network and a wide area network.
  • the communication network may be a short-distance communication network such as a wifi hotspot network, a wifi P2P network, a Bluetooth network, a zigbee network, or a near field communication (NFC) network.
  • the communication network is a wide area network, for example, the communication network may be a 3rd-generation wireless telephone technology (3G) network, or the 4th generation mobile communication technology (4G). ) Network, 5th-generation mobile communication technology (5G) network, public land mobile network (PLMN) or the Internet, etc., which are not limited in the embodiment of the present application.
  • FIG. 3 shows a schematic flowchart of a portrait processing method 200 provided by an embodiment of the present application.
  • the method 200 may be applied to the application scenario described in FIG. 1 and executed by the portrait processing apparatus 130.
  • the first camera device takes a first image of the user, the first image includes the user's face, and a partial area of the user's face is blocked by the virtual reality VR device, and the partial area includes the user's face.
  • the first camera device sends a first image of the user to a portrait processing device; correspondingly, the portrait processing device receives the first image sent by the first camera device.
  • the portrait processing apparatus may also obtain the first image in another manner, which is not limited in the embodiment of the present application.
  • the portrait processing device may receive a third image taken by the first camera device, the third image includes the user, and the portrait processing device obtains the third image from the third image. To intercept the first image.
  • the first camera device may be the camera device 120 as shown in FIG. 1.
  • the first camera device and the portrait processing device may be independent devices.
  • the portrait processing device is a terminal and the first camera device is a camera; or the first camera device and the portrait processing device may be integrated in the same
  • a device, such as a terminal, is not limited in this embodiment of the present application.
  • the third image may include at least the user. That is to say, the third image may also include the environment or scene where the user is located, which is not limited in the embodiment of the present application.
  • the embodiment of the present application only takes the partial area of the user's face as the area where the eyes are for introduction. It should be understood that this partial area may also include other areas of the user's face. There is no restriction on this.
  • the first image may include at least the face of the user, wherein a partial area of the face of the user is blocked by the VR device.
  • the first image may also include at least one of other parts of the user and the environment in which the user is located, or the first image may also include users other than the user, wherein, the method for processing the portrait of other users is similar to the method for processing the portrait of the user, and in order to avoid repetition, it will not be repeated here.
  • the portrait processing device may adopt a portrait similar to that of the user.
  • the processing method is similar to other users' portrait processing, in order to avoid repetition, it will not be repeated here.
  • the first image described in the embodiment of the present application may be a separate image, or may be a frame of image in a video stream, which is not limited in the embodiment of the present application.
  • the first image may be an image originally taken, or an image with higher definition obtained after basic image quality processing.
  • the basic image quality processing described in the embodiment of the present application may include at least one image processing step for improving image quality, such as denoising, sharpening, and brightness enhancement, etc.
  • image processing step for improving image quality such as denoising, sharpening, and brightness enhancement, etc. This embodiment of the present application does not do this. limited.
  • the portrait processing device inputs the first image into a portrait processing model to obtain a second image, where the second image includes a complete face of the user, wherein the portrait processing model is obtained by training a sample data set Obtained by training, the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and
  • the first original image in the plurality of original images includes the first sample user, and the partial area of the face of the first sample user is occluded by the VR device, and the first restored image in the plurality of restored images includes all The complete face of the first sample user, and the at least one sample user includes the first sample user.
  • the portrait processing apparatus may first obtain the portrait processing model.
  • the portrait processing apparatus may obtain the portrait processing model in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device may be pre-configured with the portrait processing model when it leaves the factory.
  • the portrait processing apparatus may receive the portrait processing model from other devices, that is, the portrait processing model is trained by other devices.
  • the portrait processing model may be stored in a cloud server, and the portrait processing device may request the portrait processing model from the cloud server via the network.
  • the portrait processing apparatus may train the portrait processing model by itself.
  • the following will take the portrait processing device itself training the portrait processing model as an example to introduce the training process of the portrait processing model.
  • the process of training the portrait processing model by other equipment is similar to the process of training the portrait processing device itself, in order to avoid Repeat, I won’t repeat it here.
  • the portrait processing device may obtain a sample training data set, and train and learn the sample training data set to obtain the portrait processing model.
  • the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the multiple original images
  • the first original image in the image includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image in the plurality of restored images includes the first A complete face of the sample user, and the at least one sample user includes the first sample user.
  • the portrait processing apparatus may train and learn the sample training data set through a variety of methods to obtain the portrait processing model, which is not limited in the embodiment of the present application.
  • the portrait processing device may train and learn the sample training data set through a neural network model to obtain the portrait processing model.
  • the neural network model may be a generative adversarial nets (GAN) model, and the GAN may include a conditional GAN, a deep convolution GAN, and so on.
  • GAN generative adversarial nets
  • the facial features of the user hidden by the VR device in the first image can be restored through the portrait processing model, thereby improving the user experience at the receiving end.
  • the portrait processing apparatus may input the first image and feature reference information into the portrait processing model to obtain the second image.
  • the feature reference information includes at least one of eye feature parameters and head feature parameters
  • the eye feature parameters include at least one of position information, size information, and viewing angle information
  • the position information is used for Indicates the position of each of the two eyes of the user
  • the size information is used to indicate the size of each eye
  • the viewing angle information is used to indicate the gaze angle of each eye
  • the head The characteristic parameters include the three-axis posture angle and acceleration of the user's head.
  • the sample training data set used when training the portrait processing model may include multiple original images, multiple restored images corresponding to the multiple original images, and at least one feature reference information, wherein the multiple The original image is collected from at least one sample user, the first original image of the plurality of original images includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device ,
  • the first restored image in the plurality of restored images includes the complete face of the first sample user, the at least one sample user includes the first sample user, and the at least one feature reference information includes the at least The feature reference information of each sample user in a sample user.
  • the portrait processing method provided by the embodiments of the present application, combining the first image and feature reference information, and inputting the portrait processing device, by providing more portrait-related features, the degree of restoration and authenticity of the occluded area can be improved.
  • the portrait processing apparatus may obtain the feature reference information in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device may receive a fourth image captured by a second camera device, the second camera device is a camera device built in the VR device, and the fourth image includes all The two eyes of the user; the portrait processing device may extract the eye feature parameters from the fourth image.
  • the second camera device may be a camera device built in the VR device, for example, a built-in infrared (IR) camera.
  • IR infrared
  • the second camera device may be a camera device 150 built in the VR device 110 as shown in FIG. 1.
  • the embodiment of the present application only takes the second camera device as a built-in camera device on the VR device as an example.
  • a camera device for real images of the facial area blocked by the VR device which is not limited in the embodiment of the present application.
  • the portrait processing apparatus may extract the eye feature parameters from multiple images of the user stored locally, where the multiple images include the eyes of the user.
  • the multiple images may be photos containing faces taken daily by the user, for example, selfies.
  • the portrait processing device may establish a facial feature database of the user through photos taken daily by the user.
  • the facial feature database includes feature parameters of various facial organs, such as eye feature parameters,
  • the portrait processing device can retrieve the eye feature parameters from the facial feature database.
  • the portrait processing device may receive the eye characteristic parameters from other measuring devices, such as an eye tracking sensor.
  • the multiple images and the facial feature database may also be stored in the cloud, and the portrait processing device may request the cloud through the network, which is not limited in the embodiment of the present application.
  • the VR device when restoring the occluded content in the first image through the portrait processing model, since the VR device mainly occludes the user's eye area, in combination with the user's eye feature parameters, It can improve the degree of restoration and authenticity of the blocked content.
  • the portrait processing device may receive the head characteristic parameters measured by an inertial measurement device.
  • an inertial measurement device can also be an inertial measurement unit (IMU), which is a device that measures the three-axis attitude angle (or angular velocity) and acceleration of an object.
  • IMU inertial measurement unit
  • an IMU contains three single-axis accelerometers and three single-axis gyroscopes.
  • the accelerometer detects the acceleration signal of the object in the independent three-axis coordinate system of the carrier, while the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Measure the angular velocity and acceleration of the object in the three-dimensional space, and calculate the posture of the object based on this.
  • the inertial measurement device may be an independent measurement device fixed on the user's head, or the inertial measurement device may be integrated in the VR device, which is not limited in the embodiment of the present application.
  • the user's head may be in a motion state, that is, it has a motion speed and a rotation angle, it is combined with the user
  • the characteristic parameters of the head can further improve the restoration and authenticity of the occluded content.
  • the feature reference information may also include other feature parameters, such as nose feature parameters, face feature parameters, hairstyle feature parameters, and other parameters that can describe the user's portrait feature, which is not limited in the embodiment of the present application.
  • the portrait processing device sends the second image to the receiving end; correspondingly, the receiving end receives the second image sent by the portrait processing device.
  • the portrait processing apparatus may send the second image to the receiving end, or the portrait processing apparatus may send a target image to the receiving end, and the target image is a response to the first image.
  • Two images are synthesized or spliced, or the portrait processing device may send a video stream to the receiving end, and the video stream includes the second image or the target image. This embodiment of the application does not do this limited.
  • the portrait processing device may send the second image, or the target image or video stream containing the second image, to the receiving end.
  • the portrait processing apparatus may generate the target image in a variety of ways, which is not limited in the embodiment of the present application.
  • the portrait processing device may splice the second image and the fifth image to obtain a target image, wherein the fifth image is captured from the fourth image. Describe the image after the first image.
  • the portrait processing device may synthesize the second image and the fourth image to obtain a target image, wherein the second image is overlaid on the fourth image The upper layer of the first image.
  • the portrait processing apparatus may generate the video stream in a variety of ways, which is not limited in this embodiment of the application.
  • the portrait processing apparatus may perform video encoding on the second image to obtain a video image; obtain the video stream according to the video image, and the video stream includes the video image .
  • the restored second image restores the user's eye area blocked by the VR device
  • the actual situation is that the user's face is wearing the VR device
  • the second image is different from the actual situation. There are certain differences between.
  • the portrait processing device may superimpose an eye mask layer on the eye area of the user in the second image, wherein the eye mask layer passes through the second image.
  • a perspective treatment of transparency may be superimposed on the eye area of the user in the second image, wherein the eye mask layer passes through the second image.
  • the value of the first transparency should satisfy that the user can see that the user is wearing the VR device without affecting the display of the obscured eye area under the VR device.
  • the specific value of is not limited.
  • the VR device worn by the user can be simulated, thereby improving the restoration Authenticity.
  • the above-mentioned portrait processing apparatus may be a terminal, and in order to realize the above-mentioned functions, the terminal includes hardware and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Those skilled in the art can use different methods for each specific application in combination with the embodiments to implement the described functions, but such implementation should not be considered as going beyond the scope of the present application.
  • the terminal may be divided into functional modules according to the foregoing method examples.
  • each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 6 shows a schematic diagram of a possible composition of the portrait processing apparatus 300 involved in the foregoing embodiment.
  • the apparatus 300 may include: a transceiver unit 310 and processing unit 320.
  • the processing unit 320 may control the transceiver unit 310 to implement the method described in the above method 200 embodiment, and/or other processes used in the technology described herein.
  • the apparatus 300 provided in this embodiment is used to execute the above-mentioned portrait processing method, and therefore can achieve the same effect as the above-mentioned implementation method.
  • the device 300 may include a processing module, a storage module, and a communication module.
  • the processing module can be used to control and manage the actions of the device 300, for example, can be used to support the device 200 to execute the steps performed by the above-mentioned units.
  • the storage module can be used to support the device 300 to execute and store program codes and data.
  • the communication module can be used for communication between the device 300 and other devices.
  • the processing module may be a processor or a controller. It can implement or execute various exemplary logical blocks, modules, and circuits described in conjunction with the disclosure of this application.
  • the processor may also be a combination of computing functions, for example, a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on.
  • the storage module may be a memory.
  • the communication module may specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, and other devices that interact with other terminals.
  • the apparatus 300 involved in the embodiment of the present application may be a terminal.
  • the terminal described in the embodiments of this application may be mobile or fixed.
  • the terminal may be a mobile phone, a camera, a video camera, a tablet personal computer, a smart TV, or a laptop computer.
  • personal digital assistants personal digital assistants, PDAs
  • personal computers personal computers
  • wearable devices such as smart watches, etc., which are not limited in the embodiments of the present application.
  • FIG. 7 shows a schematic structural diagram of a mobile phone 400.
  • the mobile phone 400 may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, and a battery 442, Antenna 1, antenna 2, mobile communication module 450, wireless communication module 460, audio module 470, speaker 470A, receiver 470B, microphone 470C, earphone interface 470D, sensor module 480, buttons 490, motor 491, indicator 492, camera 493, A display screen 494, and a subscriber identification module (SIM) card interface 495, etc.
  • SIM subscriber identification module
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 400.
  • the mobile phone 400 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 410 may include one or more processing units.
  • the processor 410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc.
  • AP application processor
  • GPU graphics processing unit
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • different processing units may be independent components, or may be integrated in one or more processors.
  • the mobile phone 400 may also include one or more processors 410.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
  • a memory may be provided in the processor 410 for storing instructions and data.
  • the memory in the processor 410 may be a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 410. If the processor 410 needs to use the instruction or data again, it can be directly called from the memory. In this way, repeated accesses are avoided, the waiting time of the processor 410 is reduced, and the efficiency of the mobile phone 400 in processing data or executing instructions is improved.
  • the processor 410 may include one or more interfaces.
  • the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, and a universal asynchronous transceiver (universal asynchronous transceiver) interface.
  • asynchronous receiver/transmitter, UART) interface mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, SIM card interface, and/or USB interface, etc.
  • the USB interface 440 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on.
  • the USB interface 440 can be used to connect a charger to charge the mobile phone 400, and can also be used to transfer data between the mobile phone 400 and peripheral devices.
  • the USB interface 440 can also be used to connect earphones and play audio through the earphones.
  • the interface connection relationship between the modules illustrated in the embodiment of the present application is merely a schematic description, and does not constitute a structural limitation of the mobile phone 400.
  • the mobile phone 400 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.
  • the charging management module 440 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 440 may receive the charging input of the wired charger through the USB interface 430.
  • the charging management module 440 may receive the wireless charging input through the wireless charging coil of the mobile phone 400. While the charging management module 440 charges the battery 442, it can also supply power to the mobile phone through the power management module 441.
  • the power management module 441 is used to connect the battery 442, the charging management module 440 and the processor 410.
  • the power management module 441 receives input from the battery 442 and/or the charge management module 440, and supplies power to the processor 410, the internal memory 421, the external memory, the display screen 494, the camera 494, and the wireless communication module 460.
  • the power management module 441 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 441 may also be provided in the processor 410.
  • the power management module 441 and the charging management module 440 may also be provided in the same device.
  • the wireless communication function of the mobile phone 400 can be implemented by the antenna 1, the antenna 2, the mobile communication module 450, the wireless communication module 460, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in the mobile phone 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • the antenna 4 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.
  • the mobile communication module 450 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied on the mobile phone 400.
  • the mobile communication module 450 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
  • the mobile communication module 450 may receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation.
  • the mobile communication module 450 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 450 may be provided in the processor 410.
  • at least part of the functional modules of the mobile communication module 450 and at least part of the modules of the processor 410 may be provided in the same device.
  • the wireless communication module 460 can provide applications on the mobile phone 400 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellite systems ( Global navigation satellite system, GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • Wi-Fi wireless fidelity
  • BT bluetooth
  • GNSS global navigation satellite systems
  • FM frequency modulation
  • NFC near field communication
  • infrared technology infrared, IR
  • the wireless communication module 460 may be one or more devices that integrate at least one communication processing module, where one communication processing module may correspond to a network interface, and the network interface may be set in different service function modes. Network interfaces in different modes can establish a network connection corresponding to that mode. .
  • a network connection supporting P2P function can be established through a network interface in P2P function mode
  • a network connection supporting STA function can be established through a network interface in STA function mode
  • a network supporting AP function can be established through a network interface in AP mode connect.
  • the wireless communication module 460 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 410.
  • the wireless communication module 460 may also receive a signal to be sent from the processor 410, perform frequency modulation, amplify, and convert it into electromagnetic waves through the antenna 2 and radiate it out.
  • the mobile phone 400 implements a display function through a GPU, a display screen 494, and an application processor.
  • the GPU is an image processing microprocessor, which connects the display screen 494 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations and is used for graphics rendering.
  • the processor 410 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 494 is used to display images, videos, and the like.
  • the display screen 494 includes a display panel.
  • the display panel can use liquid crystal display (LCD), organic light-emitting diode (organic light-emitting diode, OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED organic light-emitting diode
  • FLED flexible light-emitting diode
  • Miniled MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the mobile phone 400 may include one or more display screens 494.
  • the above-mentioned display screen 494 in FIG. 7 may be bent.
  • the above-mentioned display screen 494 can be bent means that the display screen can be bent at any position to any angle, and can be maintained at that angle.
  • the display screen 494 can be folded in half from the middle. You can also fold up and down from the middle.
  • a display screen that can be bent is referred to as a foldable display screen.
  • the touch display screen may be one screen, or a display screen formed by patching together multiple screens, which is not limited here.
  • the display screen 494 of the mobile phone 400 may be a flexible screen.
  • the flexible screen has attracted much attention due to its unique characteristics and great potential.
  • flexible screens have the characteristics of strong flexibility and bendability, and can provide users with new interactive methods based on bendable characteristics, which can meet users' more needs for mobile phones.
  • the foldable display on the mobile phone can be switched between a small screen in a folded configuration and a large screen in an unfolded configuration at any time. Therefore, users use the split screen function on mobile phones equipped with foldable display screens more and more frequently.
  • the mobile phone 400 can realize a shooting function through an ISP, a camera 493, a video codec, a GPU, a display screen 494, and an application processor.
  • the ISP is used to process the data fed back by the camera 494. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 494.
  • the camera 493 is used to capture still images or videos.
  • the object generates an optical image through the lens and is projected to the photosensitive element.
  • the photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor
  • the mobile phone 400 may include one or more cameras 493.
  • CMOS complementary metal-oxide-semiconductor
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the mobile phone 400 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the mobile phone 400 may support one or more video codecs. In this way, the mobile phone 400 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG4, MPEG4, and so on.
  • MPEG moving picture experts group
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • the transfer of business functions between human brain neurons can be used to quickly process input information, and it can also continuously self-learn.
  • applications such as intelligent cognition of the mobile phone 400 can be realized, such as image recognition, face recognition, voice recognition, text understanding, and so on.
  • the external memory interface 420 may be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the mobile phone 400.
  • the external memory card communicates with the processor 410 through the external memory interface 420 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 421 may be used to store one or more computer programs, and the one or more computer programs include instructions.
  • the processor 410 can execute the above-mentioned instructions stored in the internal memory 421 to enable the mobile phone 400 to execute the off-screen display method provided in some embodiments of the present application, as well as various applications and data processing.
  • the internal memory 421 may include a program storage area and a data storage area.
  • the storage program area can store the operating system; the storage program area can also store one or more applications (such as photo galleries, contacts, etc.).
  • the data storage area can store data (such as photos, contacts, etc.) created during the use of the mobile phone 400.
  • the internal memory 421 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), and the like.
  • the processor 410 may execute the instructions stored in the internal memory 421 and/or the instructions stored in the memory provided in the processor 410 to cause the mobile phone 400 to execute the instructions provided in the embodiments of the present application. Screen display methods, as well as other applications and data processing.
  • the mobile phone 400 can implement audio functions through the audio module 470, the speaker 470A, the receiver 470B, the microphone 470C, the earphone interface 470D, and the application processor. For example, music playback, recording, etc.
  • the sensor module 480 may include a pressure sensor 480A, a gyroscope sensor 480B, an air pressure sensor 480C, a magnetic sensor 480D, an acceleration sensor 480E, a distance sensor 480F, a proximity light sensor 480G, a fingerprint sensor 480H, a temperature sensor 480J, a touch sensor 480K, and an ambient light sensor 480L, bone conduction sensor 480M, etc.
  • the pressure sensor 480A is used to sense a pressure signal, and can convert the pressure signal into an electrical signal.
  • the pressure sensor 480A may be provided on the display screen 494.
  • the capacitive pressure sensor may include at least two parallel plates with conductive materials.
  • the mobile phone 400 determines the intensity of the pressure according to the change in capacitance.
  • the mobile phone 400 detects the intensity of the touch operation according to the pressure sensor 480A.
  • the mobile phone 400 can also calculate the touched position based on the detection signal of the pressure sensor 480A.
  • touch operations that act on the same touch position but have different touch operation strengths may correspond to different operation instructions. For example: when a touch operation whose intensity of the touch operation is less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
  • the gyro sensor 480B can be used to determine the movement posture of the mobile phone 400.
  • the angular velocity of the mobile phone 400 around three axes ie, X, Y, and Z axes
  • the gyro sensor 480B can be used for image stabilization.
  • the gyroscope sensor 480B detects the shake angle of the mobile phone 400, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the mobile phone 400 through reverse movement to achieve anti-shake.
  • the gyro sensor 480B can also be used for navigation and somatosensory game scenes.
  • the acceleration sensor 480E can detect the magnitude of the acceleration of the mobile phone 400 in various directions (generally three axes). When the mobile phone 400 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify mobile phone gestures, switch between horizontal and vertical screens, pedometers and other applications.
  • the ambient light sensor 480L is used to sense the brightness of the ambient light.
  • the mobile phone 400 can adaptively adjust the brightness of the display 494 according to the perceived brightness of the ambient light.
  • the ambient light sensor 480L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 480L can also cooperate with the proximity light sensor 480G to detect whether the mobile phone 400 is in the pocket to prevent accidental touch.
  • the fingerprint sensor 480H is used to collect fingerprints.
  • the mobile phone 400 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, and so on.
  • the temperature sensor 480J is used to detect temperature.
  • the mobile phone 400 uses the temperature detected by the temperature sensor 480J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 480J exceeds the threshold, the mobile phone 400 performs a reduction in the performance of the processor located near the temperature sensor 480J, so as to reduce power consumption and implement thermal protection.
  • the mobile phone 400 when the temperature is lower than another threshold, the mobile phone 400 heats the battery 442 to avoid abnormal shutdown of the mobile phone 400 due to low temperature.
  • the mobile phone 400 boosts the output voltage of the battery 442 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 480K also called “touch panel”.
  • the touch sensor 480K may be arranged on the display screen 494, and the touch screen is composed of the touch sensor 480K and the display screen 494, which is also called a “touch screen”.
  • the touch sensor 480K is used to detect touch operations acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • the visual output related to the touch operation can be provided through the display screen 494.
  • the touch sensor 480K may also be arranged on the surface of the mobile phone 400, which is different from the position of the display screen 494.
  • the button 490 includes a power-on button, a volume button, and so on.
  • the button 490 may be a mechanical button. It can also be a touch button.
  • the mobile phone 400 can receive key input, and generate key signal input related to user settings and function control of the mobile phone 400.
  • the related functions implemented by the processing unit 320 in FIG. 6 may be implemented by the processor 410 in FIG. 7, and the related functions implemented by the transceiving unit 310 in FIG. 6 may be implemented by the processor 410 in FIG.
  • the antenna 450 or the antenna 460 is controlled to implement, or the related functions implemented by the transceiver unit 310 in FIG. 6 may be implemented by the processor 410 in FIG. 7 controlling other components of the mobile phone 400 through an internal interface.
  • FIG. 8 shows a portrait processing system 500 provided by an embodiment of the present application.
  • the system 500 includes a mobile phone 510, a VR device 520, and a receiving end 530. There is a communication interface between the mobile phone 510 and the VR device 520. There is a communication interface between.
  • the mobile phone 510 includes a first camera 511, a first image processing module 512, a second image processing module 513, a portrait processing module 514, a storage module 515, an image synthesis module 516, a video encoding module 517, and the VR device 520 includes a display screen 521, a second camera 522, and an inertial measurement device 523.
  • the inertial measurement device 523 may be integrated in the VR device 520, or may be an independent device, which is not limited in the embodiment of the present application.
  • the first camera 511 may be a built-in camera of the mobile phone 510, or may be an independent camera, which is not limited in the embodiment of the present application.
  • the display screen 521 is used to play VR resources obtained from the mobile phone 510 through the interface.
  • the first camera 511 is used to take a third image of the user and send it to the first image processing module 512, where the third image includes the user and the scene in which the user is located, and a partial area of the user’s face is
  • the VR device 520 blocks, and the partial area includes the area where the eyes of the user are located.
  • the first image processing module 512 is configured to receive the third image sent by the first camera 511; perform basic image processing such as enhancement effects on the third image; intercept the first image from the processed third image, so The first image includes the face of the user, and the first image is sent to the portrait processing module 514.
  • the second camera 522 is a built-in camera of the VR device 520, which is used to take a fourth image of the user and send it to the second image processing module 513, where the fourth image includes two eyes of the user.
  • the second image processing module 513 is configured to receive the fourth image sent by the second camera 522, perform basic image processing such as enhancement effects on the fourth image, and send the processed fourth image to the portrait processing module 514 .
  • the inertial measurement device 523 is used to obtain the user's head characteristic parameters, which include the three-axis posture and acceleration of the user's head, and send them to the portrait processing module 514.
  • the portrait processing module 514 is configured to extract the user's eye characteristic parameters from the received fourth image; input the first image, the eye characteristic parameters, and the head characteristic parameters into the storage module 515
  • the second image is obtained by the portrait processing model stored in the image processing model; the second image is sent to the image synthesis module 516, and the second image includes the complete face of the user.
  • the image synthesis module 516 is configured to synthesize or splice the second image to obtain a target image, and send the target to the video encoding module 517.
  • the video encoding module 517 is configured to encode the target image to obtain a video stream, and send the video stream to the receiving end 530.
  • the portrait processing module 514 in FIG. 8 may be a computing unit such as NPU, DSP, GPU, etc., as described in FIG. 7, which is not limited in the embodiment of the present application.
  • the first camera 511 in FIG. 8 may be the camera 494 described in FIG. 7; the first image processing module 512 and the second image processing module 513 may belong to the ISP described in FIG. 7, or The first image processing module 512 and the second image processing module can be different ISPs; the portrait processing module 514 can be the NPU, DSP, GPU and other computing units described in FIG. 7; the storage module 515 can be the one described in FIG. 7
  • the image synthesis module 516 may be the GPU described in FIG. 7; the video encoding module 517 may be the video encoder described in FIG.
  • each module in the aforementioned mobile phone 500 may also be implemented by other devices in FIG. 7 that can implement the functions implemented by each module, which is not limited in the embodiment of the present application.
  • system 500 provided by the embodiment of the present application is described above in only one possible implementation manner, and the embodiment of the present application does not limit this, and the system 500 should be able to implement the other steps described in the foregoing method embodiment 200. To avoid repetition, I won’t repeat it here.
  • This embodiment also provides a computer storage medium that stores computer instructions that, when the computer instructions run on an electronic device, cause the electronic device to execute the above-mentioned related method steps to implement the portrait processing method in the above-mentioned embodiment.
  • This embodiment also provides a computer program product, which when the computer program product runs on a computer, causes the computer to execute the above-mentioned related steps, so as to realize the portrait processing method in the above-mentioned embodiment.
  • the embodiments of the present application also provide a device.
  • the device may specifically be a chip, component or module.
  • the device may include a processor and a memory connected to each other.
  • the memory is used to store computer execution instructions.
  • the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the portrait processing method in the foregoing method embodiments.
  • the server, terminal, computer storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding method provided above. The beneficial effects of the method are not repeated here.
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un procédé et un appareil de traitement de portrait, et un terminal, qui peuvent restaurer des caractéristiques faciales d'un utilisateur bloqué par un dispositif de réalité virtuelle de façon à améliorer l'expérience d'utilisateur d'une extrémité de réception. Le procédé comprend les étapes suivantes : un terminal acquiert une première image d'un utilisateur, la première image comprenant le visage de l'utilisateur, une zone partielle du visage de l'utilisateur étant bloquée par un appareil de réalité virtuelle, et la zone partielle comprenant la zone où se trouvent les yeux de l'utilisateur ; le terminal entre la première image dans un modèle de traitement de portrait afin d'obtenir une seconde image, la seconde image comprenant le visage complet de l'utilisateur ; et le terminal envoie la seconde image à une extrémité de réception.
PCT/CN2020/122767 2020-02-18 2020-10-22 Procédé et appareil de traitement de portrait, et terminal WO2021164289A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010100149.2A CN111385514B (zh) 2020-02-18 2020-02-18 人像处理方法和装置以及终端
CN202010100149.2 2020-02-18

Publications (1)

Publication Number Publication Date
WO2021164289A1 true WO2021164289A1 (fr) 2021-08-26

Family

ID=71219765

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/122767 WO2021164289A1 (fr) 2020-02-18 2020-10-22 Procédé et appareil de traitement de portrait, et terminal

Country Status (2)

Country Link
CN (1) CN111385514B (fr)
WO (1) WO2021164289A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111385514B (zh) * 2020-02-18 2021-06-29 华为技术有限公司 人像处理方法和装置以及终端
CN114594851B (zh) * 2020-11-30 2024-06-25 华为技术有限公司 图像处理方法、服务器和虚拟现实设备
CN112558906A (zh) * 2020-12-11 2021-03-26 上海影创信息科技有限公司 具有成像距离的显示控制方法和系统、存储介质及其vr设备
CN116503289B (zh) * 2023-06-20 2024-01-09 北京天工异彩影视科技有限公司 一种视觉特效应用处理方法和系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103888710A (zh) * 2012-12-21 2014-06-25 深圳市捷视飞通科技有限公司 一种视频会议系统和方法
US20180063484A1 (en) * 2016-01-20 2018-03-01 Gerard Dirk Smits Holographic video capture and telepresence system
CN109785369A (zh) * 2017-11-10 2019-05-21 中国移动通信有限公司研究院 一种虚拟现实人像采集方法及装置
CN111385514A (zh) * 2020-02-18 2020-07-07 华为技术有限公司 人像处理方法和装置以及终端

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7257237B1 (en) * 2003-03-07 2007-08-14 Sandia Corporation Real time markerless motion tracking using linked kinematic chains
CN106372603A (zh) * 2016-08-31 2017-02-01 重庆大学 遮挡人脸识别方法及装置
CN106529409B (zh) * 2016-10-10 2019-08-09 中山大学 一种基于头部姿态的眼睛注视视角测定方法
CN109831622B (zh) * 2019-01-03 2021-06-22 华为技术有限公司 一种拍摄方法及电子设备
CN109886216B (zh) * 2019-02-26 2023-07-18 华南理工大学 基于vr情景人脸图像复原的表情识别方法、设备及介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103888710A (zh) * 2012-12-21 2014-06-25 深圳市捷视飞通科技有限公司 一种视频会议系统和方法
US20180063484A1 (en) * 2016-01-20 2018-03-01 Gerard Dirk Smits Holographic video capture and telepresence system
CN109785369A (zh) * 2017-11-10 2019-05-21 中国移动通信有限公司研究院 一种虚拟现实人像采集方法及装置
CN111385514A (zh) * 2020-02-18 2020-07-07 华为技术有限公司 人像处理方法和装置以及终端

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
WANG, XUDONG: "Research on Occluded Face Detection and De-occlusion Based on Deep Learning", CHINA MASTER’S THESES FULL-TEXT DATABASE, no. 2, 15 February 2020 (2020-02-15), pages 1 - 73, XP055839595, ISSN: 1674-0246 *

Also Published As

Publication number Publication date
CN111385514A (zh) 2020-07-07
CN111385514B (zh) 2021-06-29

Similar Documents

Publication Publication Date Title
WO2021213120A1 (fr) Procédé et appareil de projection d'écran et dispositif électronique
WO2020168970A1 (fr) Procédé de commande d'affichage d'écran et dispositif électronique
WO2020211532A1 (fr) Procédé de commande d'affichage et appareil associé
WO2020168965A1 (fr) Procédé de commande d'un dispositif électronique à écran pliant et dispositif électronique
WO2021164289A1 (fr) Procédé et appareil de traitement de portrait, et terminal
WO2021104008A1 (fr) Procédé d'affichage d'un écran pliable et appareil associé
WO2021027725A1 (fr) Procédé d'affichage d'élément de page et dispositif électronique
WO2021052279A1 (fr) Procédé d'affichage sur écran pliable, et dispositif électronique
CN110798568B (zh) 具有折叠屏的电子设备的显示控制方法及电子设备
WO2021169394A1 (fr) Procédé d'embellissement d'une image du corps humain sur la base de la profondeur et dispositif électronique
WO2022262313A1 (fr) Procédé de traitement d'image à base d'incrustation d'image, dispositif, support de stockage, et produit de programme
WO2021104104A1 (fr) Procédé de traitement d'affichage écoénergétique, et appareil
WO2021208723A1 (fr) Procédé et appareil d'affichage plein écran, et dispositif électronique
CN110401768B (zh) 调节电子设备的工作状态的方法和装置
WO2020249076A1 (fr) Procédé d'étalonnage de visage et dispositif électronique
CN113797530B (zh) 图像的预测方法、电子设备和存储介质
WO2022027972A1 (fr) Procédé de recherche de dispositif et dispositif électronique
WO2021052407A1 (fr) Procédé de commande de dispositif électronique et dispositif électronique
CN108848405B (zh) 图像处理方法和装置
CN113741681A (zh) 一种图像校正方法与电子设备
WO2021031745A1 (fr) Procédé d'ouverture d'application et dispositif électronique
CN114579016A (zh) 一种共享输入设备的方法、电子设备及系统
CN114257920B (zh) 一种音频播放方法、系统和电子设备
CN115150542B (zh) 一种视频防抖方法及相关设备
WO2020056694A1 (fr) Procédé de communication à réalité augmentée, et dispositifs électroniques

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20919714

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20919714

Country of ref document: EP

Kind code of ref document: A1