WO2019128508A1 - Method and apparatus for processing image, storage medium, and electronic device - Google Patents

Method and apparatus for processing image, storage medium, and electronic device Download PDF

Info

Publication number
WO2019128508A1
WO2019128508A1 PCT/CN2018/115470 CN2018115470W WO2019128508A1 WO 2019128508 A1 WO2019128508 A1 WO 2019128508A1 CN 2018115470 W CN2018115470 W CN 2018115470W WO 2019128508 A1 WO2019128508 A1 WO 2019128508A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
face
original image
target face
location information
Prior art date
Application number
PCT/CN2018/115470
Other languages
French (fr)
Chinese (zh)
Inventor
陈岩
刘耀勇
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2019128508A1 publication Critical patent/WO2019128508A1/en

Links

Images

Classifications

    • G06T3/04
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/02
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168Feature extraction; Face representation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30196Human being; Person
    • G06T2207/30201Face

Definitions

  • the present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, storage medium, and electronic device.
  • the embodiment of the present application provides an image processing method, device, storage medium, and electronic device, which can lighten image stitching and improve image synthesis.
  • an embodiment of the present application provides an image processing method, which is applied to an electronic device, and includes:
  • the corrected target face image is merged with the original image.
  • an embodiment of the present application provides an image processing apparatus, which is applied to an electronic device, and includes:
  • a location obtaining module configured to acquire first location information of a face key point in the original image
  • An alignment module configured to match a corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image;
  • a correction module for correcting the target face image based on the trained convolutional neural network model and the face region in the original image
  • a fusion module for merging the corrected target face image with the original image.
  • the embodiment of the present application further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to perform the following steps:
  • the corrected target face image is merged with the original image.
  • an embodiment of the present application further provides an electronic device, including a processor and a memory, where the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is configured to perform the following steps. :
  • parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
  • FIG. 1 is a schematic diagram of a scenario structure of an electronic device for implementing deep learning according to an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application.
  • FIG. 4 is a schematic partial structural diagram of a convolutional neural network provided by an embodiment of the present application.
  • FIG. 5 is another application scenario diagram of an image processing method provided by an embodiment of the present application.
  • FIG. 6 is another schematic flowchart of an image processing apparatus according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 8 is another schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 9 is still another schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of still another structure of an image processing apparatus according to an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • FIG. 12 is another schematic structural diagram of an electronic device according to an embodiment of the present application.
  • the embodiment of the present application provides an image processing method, device, storage medium, and electronic device. The details will be described separately below.
  • FIG. 1 is a schematic diagram of a scenario in which an electronic device implements deep learning according to an embodiment of the present disclosure.
  • the electronic device can record the input and output data during the processing.
  • the electronic device may include a data collection and statistics system and a prediction system with feedback adjustment.
  • the electronic device can acquire a large amount of image classification result data of the user through the data acquisition system, make corresponding statistics, and extract image features of the image, and analyze and process the extracted image features based on machine depth learning, and the loss to the convolutional neural network.
  • the function performs parameter training.
  • the electronic device predicts the classification result of the image through the prediction system.
  • the prediction system reversely reciprocates the weights of the weighting items according to the final result of the user behavior. After a number of iterative corrections, the weights of the weighting items of the prediction system are finally converged to form a learned database.
  • the electronic device may be a mobile terminal, such as a mobile phone, a tablet computer, or the like, or may be a conventional PC (Personal Computer), etc., which is not limited in this embodiment of the present application.
  • a mobile terminal such as a mobile phone, a tablet computer, or the like
  • PC Personal Computer
  • An embodiment of the present application provides an image processing method, which is applied to an electronic device, where the image processing method includes:
  • the corrected target face image is merged with the original image.
  • the method before acquiring the first location information of the face key in the original image, the method further includes:
  • parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
  • the step of correcting the target face image based on the trained convolutional neural network model and the face region in the original image includes:
  • the step of matching the corresponding target face image from the preset face database according to the location information, and aligning the target face image with the face in the original image includes:
  • the target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  • the method further includes:
  • the original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
  • the step of fusing the corrected target face image with the original image comprises:
  • the corrected target face image is merged with the background image in the original image by using the face mask.
  • the acquiring first location information of a face key point in the original image includes:
  • the face key point detection is performed on the face area to obtain the first position information of the face key point.
  • an image processing method is provided. As shown in FIG. 2, the flow may be as follows:
  • the original image includes at least one human face.
  • the original image may be an image captured by the electronic device through the camera, or may be an image directly obtained by the electronic device from a storage area of a server or other external device.
  • the acquired image may be an image obtained by self-timer of the front device head, or may be an image with a face collected by the rear camera.
  • the camera can be a digital camera or an analog camera.
  • the digital camera converts the analog image signal generated by the image acquisition device into a digital signal, which is then stored in a computer.
  • the image signal captured by the analog camera must be converted to a digital mode by a specific image capture card and compressed before being converted to a computer for use.
  • the digital camera captures the image directly and transmits it to the computer via a serial, parallel or USB interface.
  • the electronic device generally adopts a digital camera to convert the collected image into data in real time and display it on the display interface of the electronic device (ie, the preview frame of the camera).
  • the face image is first detected on the original image, the face region is determined, and then the face key point is detected from the face region to obtain the location information of the face key point. That is, in some embodiments, the step of "acquiring first location information of a face key point in the original image” may include the following process:
  • the face key point detection is performed on the face area to obtain the first position information of the face key point.
  • FIG. 3 is a schematic diagram of calibration results of detected key points of a face.
  • the chin is calibrated by 17 key points of the face
  • the left and right eyebrows are each calibrated by 5 key points of the face
  • the nose is calibrated by 9 key points of the face
  • the left and right eyes are respectively calibrated by 6 key points of the face.
  • the mouth is calibrated by 20 key points of the face, with a total of 68 face key points.
  • the original image since the original image may be affected by the shooting angle at the time of shooting, the image is deformed, which seriously affects the recognition of the image. Therefore, it is necessary to take certain measures to transform the image and perform a certain degree of correction to facilitate the identification and registration of the machine. Therefore, in the embodiment of the present application, it is necessary to construct a face database in advance, and store an image of a reference face with different postures for performing identity replacement, in particular, to obtain photos of different angles, so that the original image can be in the face database. Refer to the face image as the target identity, and perform matching and face changing operations.
  • the manner in which the target face image is aligned with the face in the original image may be various.
  • the target face image may be aligned with the face in the original image by using an affine transformation. That is, in some embodiments, the step of “matching the corresponding target face image from the preset face database according to the first location information and aligning the target face image with the face in the original image” may include the following process. :
  • the target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  • the first location information is matched with the second location information of the sample face image in the face database, and the greater the matching degree, the closer the posture and size of the two images are, so that the subsequent affine transformation process can be reduced.
  • the computational difficulty of the medium transformation matrix In order to improve the matching degree between the face image and the sample face image, a large number of face images with different postures can be obtained to increase the density of the deflection angle of the sample face image in the face database, and reduce the interval value between the deflection angles.
  • affine transformation also known as affine mapping
  • affine mapping means that in geometry, a vector space undergoes a linear transformation and is connected to a translation, transforming into another vector space.
  • Affine transformations include: translation, rotation, scaling, beveling, and so on.
  • To perform an affine transformation you must first obtain the transformation matrix.
  • To obtain the transformation matrix you must first obtain the coordinates of the feature points, angles, etc., such as geometric matching, bolb and other methods can obtain the feature point coordinates, angle information.
  • first, coordinate and angle information (ie, second position information) of key points in the target face image are acquired, and then an affine transformation matrix is calculated according to the acquired second position information and the first position information.
  • the target face image is affine transformed according to the calculated transformation matrix, and the target face image is mapped to the face position of the original image.
  • the method may further include the following steps:
  • the constructed convolutional neural network is trained to adjust the content loss function, the illumination loss function, and the smoothing loss function parameters to obtain the trained convolutional neural network model.
  • the constructed convolutional neural network is a multi-scale architecture with branches, and the branches perform operations on different sampling versions according to the size of the input test image. Small images are automatically upsampled to a size of 2 after being convolved, and then channeled to a large image. Each such branch has a zero-filled convolution module followed by a linear rectification. These branches are then combined by a nearest-neighbor upsampling that differs by one and a concatenation along the channel axis.
  • the training sample is first input, the parameter initialization is performed, and after the convolution and sampling process, the full connection layer is reached, and the affine transformation and parameter calculation are performed, and the processed image is outputted, and the loss is obtained by logistic regression analysis.
  • the weight of the function is continuously feedback to correct the parameter settings of each loss function by artificially judging whether it meets the expectations.
  • "correcting the target face image based on the trained convolutional neural network model and the face region in the original image” may include the following steps:
  • the adjustment parameter is generated under the constraint of the content loss function, the illumination loss function, and the smoothing loss function
  • the target face image is corrected according to the adjustment parameters.
  • the face exchange can be described as a problem of style migration.
  • the goal of style migration is to render an image into the style of another image. Based on this, the pose and expression of the face in the original image are taken as the content, and the target face image is used as the style, and a loss function that allows the convolutional neural network to generate a high image realism result is designed.
  • the loss function of the image is based on a feature map in an already trained neural network.
  • the nearest neighbor method can be used, that is, the image at a certain position in the original image is replaced with the most similar segment in the target image.
  • the search domain is restricted according to the key points extracted from the face. That is, for a certain part of the face in the original image, a similar tile search is performed only near a certain part of the mesh image.
  • multiple images of the target face ie multiple style images
  • the loss is limited in the image area, but the search can be performed on the pieces extracted by the plurality of images, so that a variety of expressions can be reproduced.
  • the algorithm trained a convolutional neural network classifier for illumination. For two images that are invariant except for illumination, the classifier determines whether the pair of images has undergone illumination transformation, and uses the feature map obtained from the network to calculate the illumination loss.
  • the corrected target face image is merged with the original image.
  • One type is a region-based algorithm, which refers to a parameter that uses the relationship between two images to determine the coordinate change between images, including a space-based pixel registration algorithm and a frequency domain-based algorithm.
  • the other type is an algorithm based on feature splicing, which uses the obvious features (points, lines, edges, contours, corner points) in the image to calculate the transformation between images.
  • the third type is based on the splicing of the maximum mutual information, and the splicing work is shifted from the spatial domain to the small domain wave, and the wavelet reconstruction can obtain a complete image.
  • the following steps may be further included:
  • the original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
  • the pattern is cut along the position of the detected edge feature point. Since the edge feature point is a face edge feature point, the face region can be finally separated from the original image, and finally the background image can be obtained.
  • step "merging the corrected target face image with the original image” may include the following process:
  • the face mask is used to fuse the corrected target face image with the background image in the original image.
  • the position information of the edge feature points may be relative position information between the edge feature points.
  • a closed pattern is formed based on the edge feature points, and a region other than the closed pattern is used as a segmentation mask.
  • the face mask is superimposed on the target face image mapped on the face region of the original image, and aligned with the face region of the original image, and the region of the target face image that is not blocked by the face mask is displayed. The occluded area is not displayed.
  • a is the target face image
  • b is the original image
  • c is a face mask generated based on the face region in the original image b
  • d is obtained by affine transformation of the target image a.
  • the image is finally output as the fused image e after changing the face.
  • the processed target face image may be merged with the target image based on the Poisson fusion technique to cover the original face image in the target image.
  • the face image in the target picture is replaced with the processed target face image.
  • Poisson fusion technology can better eliminate the boundary between the target face image and the target image, making the picture more natural and unobtrusive, achieving seamless splicing.
  • the first location information of the face key point in the original image is obtained; the corresponding target face image is matched from the preset face database according to the first location information, and the target face image is in the original image. Face alignment; based on the trained convolutional neural network model and the face region in the original image, the target face image is corrected; and the corrected target face image is merged with the original image.
  • the program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
  • another image processing method is also provided. As shown in FIG. 6, the flow may be as follows:
  • the constructed convolutional neural network is a multi-scale architecture with branches, and the branches are executed on different sampling versions according to the size of the input test image. Operation. Small images are automatically upsampled to a size of 2 after being convolved, and then channeled to a large image. Each such branch has a zero-filled convolution module followed by a linear correction. These branches are then combined by a nearest neighbor upsampling that differs by one and a cascade along the channel axis.
  • parameter training of the loss function in the convolutional neural network is performed.
  • the face exchange can be described as a problem of style migration.
  • the goal of style migration is to render an image into the style of another image. Based on this, the pose and expression of the face in the original image are taken as the content, and the target face image is used as the style, and a loss function that allows the convolutional neural network to generate a high image realism result is designed.
  • images of multiple angles of the same face may be acquired as training samples.
  • the training sample is first input, the parameter initialization is performed, and after the convolution and sampling process, the full connection layer is reached, and the affine transformation and parameter calculation are performed, and the processed image is outputted, and the loss is obtained by logistic regression analysis.
  • the weight of the function is continuously feedback to correct the parameter settings of each loss function by artificially judging whether it meets the expectations.
  • the face key point detection acquires first position information of the face key point in the original image and second position information of the face key point of the target face image.
  • the original image includes at least one human face.
  • the original image may be an image captured by the electronic device through the camera, or may be an image directly obtained by the electronic device from a storage area of a server or other external device.
  • the target face image is a reference face for the identity replacement of the face in the original image.
  • a face database in advance, and store an image of a reference face with different postures for performing identity replacement, in particular, to obtain photos of different angles, so that the original image can be in the face database.
  • the face image as the target identity, and perform matching and face changing operations.
  • the original image is firstly detected by the face, the face area is determined, and then the face key point is detected from the face area to obtain the position information of the face key point.
  • the original image may be affected by the shooting angle when shooting, the image is deformed, which seriously affects the recognition of the image. Therefore, it is necessary to take certain measures to transform the image and perform a certain degree of correction to facilitate the identification and registration of the machine.
  • the first location information is matched with the two location information, and a matching relationship between the original image and the face key points in the target face image is established one by one.
  • the affine transformation matrix of the two images is obtained through the face key point pair, and the target face image is mapped to the face region of the original image based on the affine transformation matrix, so that the target face image and the original image are in the original image.
  • the faces are aligned.
  • the content feature, the illumination feature, and the smooth feature are extracted from the face region of the original image based on the trained convolutional neural network model; the content loss function, the illumination loss function, according to the content feature, the illumination feature, and the smooth feature, Under the constraint of the smoothing loss function, the corresponding adjustment parameters are generated, and the target face image is corrected according to the adjustment parameters.
  • the trained convolutional neural network model to adjust the parameters, it is better to keep the facial expression, skin color, illumination and other features in the original image unchanged, making the face after changing face more natural.
  • the original image needs to be face-detected, and then the third position information of the edge feature points of the face region is calculated based on the correlation edge algorithm.
  • the pattern is cut along the position of the detected edge feature point. Since the edge feature point is a face edge feature point, the face region can be finally separated from the original image, and finally the background image can be obtained.
  • the position information of the edge feature points may be relative position information between the edge feature points.
  • a closed pattern is formed based on the edge feature points, and an area other than the closed pattern is used as a face mask.
  • the face mask is superimposed on the target face image mapped on the face region of the original image, and aligned with the face region of the original image, and the region of the target face image that is not blocked by the face mask is displayed. The occluded area is not displayed.
  • the processed target face image may be merged with the target image based on the Poisson fusion technique to cover the original face image in the target image.
  • the face image in the target picture is replaced with the processed target face image.
  • Poisson fusion technology can better eliminate the boundary between the target face image and the target image, making the picture more natural and unobtrusive, achieving seamless splicing.
  • the image processing method obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and Aligning the target face image with the face in the original image; correcting the target face image based on the trained convolutional neural network model and the face region in the original image; correcting the target face image and original Image fusion.
  • the program can better maintain the expression, skin color and illumination of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
  • an image processing apparatus is further provided, which may be integrated in an electronic device in the form of software or hardware, and the electronic device may specifically include a mobile phone, a tablet computer, a notebook computer, and the like.
  • the image processing apparatus 30 may include a location acquisition module 31, an alignment module 32, a correction module 33, and a fusion module 34, where:
  • a location obtaining module 31 configured to acquire first location information of a face key point in the original image
  • the aligning module 32 is configured to match the corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image;
  • the correction module 33 is configured to correct the target face image based on the trained convolutional neural network model and the face region in the original image;
  • the fusion module 34 is configured to fuse the corrected target face image with the original image.
  • the image processing apparatus 30 may further include:
  • a building module 35 configured to construct a convolutional neural network before acquiring first location information of a face key point in the original image
  • a sample obtaining module 36 configured to acquire an image of a plurality of angles of a face in the original image as a training sample
  • the training module 37 is configured to perform parameter training on the constructed convolutional neural network based on the training samples to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
  • the correction module 33 can include:
  • the extraction sub-module 331 is configured to extract a content feature, an illumination feature, and a smooth feature from the face region of the original image based on the trained convolutional neural network model;
  • the generating submodule 332 is configured to generate an adjustment parameter according to the content feature, the illumination feature, and the smoothing feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
  • the correction sub-module 333 is configured to correct the target facial image according to the adjustment parameter.
  • the correction module 32 can include:
  • the obtaining sub-module 321 is configured to acquire second location information of a face key point of the plurality of sample face images in the preset face database;
  • the selecting sub-module 323 is configured to select, from the sample face image, the sample face image with the largest matching degree as the target face image;
  • the mapping sub-module 324 is configured to map the target face image to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  • the image processing device 30 can further include:
  • the edge feature point obtaining module 38 is configured to: after aligning the target face image with the face in the original image, and performing edge feature points on the face region in the original image before merging the corrected target face image with the original image Detecting, and acquiring third position information of edge feature points;
  • the segmentation module 39 is configured to perform image segmentation processing on the original image according to the third position information to remove the face region and use the remaining image region as the background image.
  • the fusion module 34 can be used to:
  • the corrected target face image is merged with the background image in the original image by using the face mask.
  • the location acquisition module 31 can be used to:
  • the face key point detection is performed on the face area to obtain the first position information of the face key point.
  • the image processing apparatus obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and Aligning the target face image with the face in the original image; correcting the target face image based on the trained convolutional neural network model and the face region in the original image; correcting the target face image and original Image fusion.
  • the program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
  • an electronic device is further provided, and the electronic device may be a device such as a smart phone or a tablet computer.
  • the electronic device 400 includes a processor 401 and a memory 402.
  • the processor 401 is electrically connected to the memory 402.
  • the processor 401 is a control center of the electronic device 400, which connects various parts of the entire electronic device using various interfaces and lines, executes the electronic by running or loading an application stored in the memory 402, and calling data stored in the memory 402.
  • the various functions and processing data of the device enable overall monitoring of the electronic device.
  • the processor 401 in the electronic device 400 loads the instructions corresponding to the process of one or more applications into the memory 402 according to the following steps, and is stored and stored in the memory 402 by the processor 401.
  • the application thus implementing various functions:
  • the corrected target face image is merged with the original image.
  • the processor 401 before acquiring the first location information of the face key in the original image, the processor 401 is configured to perform the following steps:
  • the constructed convolutional neural network is trained to adjust the content loss function, the illumination loss function, and the smoothing loss function parameters to obtain the trained convolutional neural network model.
  • the processor 401 is further configured to perform the following steps:
  • the adjustment parameter is generated under the constraint of the content loss function, the illumination loss function, and the smoothing loss function
  • the target face image is corrected according to the adjustment parameters.
  • the processor 401 is further configured to perform the following steps:
  • the target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  • the processor 401 after aligning the target face image with the face in the original image, before the merged target face image is merged with the original image, the processor 401 further performs the following steps:
  • the original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
  • the processor 401 is further configured to perform the following steps:
  • the corrected target face image is merged with the background image in the original image using a face mask.
  • Memory 402 can be used to store applications and data.
  • the application stored in the memory 402 contains instructions that can be executed in the processor.
  • Applications can form various functional modules.
  • the processor 401 executes various functional applications and data processing by running an application stored in the memory 402.
  • the electronic device 400 further includes a display screen 403, a control circuit 404, a radio frequency circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power source 409.
  • the processor 401 is electrically connected to the display screen 403, the control circuit 404, the radio frequency circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power source 409, respectively.
  • the display screen 403 can be used to display information entered by the user or information provided to the user as well as various graphical user interfaces of the electronic device, which can be composed of images, text, icons, video, and any combination thereof.
  • the display screen 403 can be used as a screen in the embodiment of the present application for displaying information.
  • the control circuit 404 is electrically connected to the display screen 403 for controlling the display screen 403 to display information.
  • the radio frequency circuit 405 is configured to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices through wireless communication, and to transmit and receive signals with network devices or other electronic devices.
  • the input unit 406 can be configured to receive input digits, character information, or user characteristic information (eg, fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls.
  • the input unit 406 can include a fingerprint identification module.
  • the audio circuit 407 can provide an audio interface between the user and the electronic device through a speaker and a microphone.
  • Sensor 408 is used to collect external environmental information.
  • Sensor 408 can include ambient brightness sensors, acceleration sensors, light sensors, motion sensors, and other sensors.
  • Power source 409 is used to power various components of electronic device 400.
  • the power supply 409 can be logically coupled to the processor 401 through a power management system to enable functions such as managing charging, discharging, and power management through the power management system.
  • the camera 410 is used for collecting external images, and can be a digital camera or an analog camera. In some embodiments, camera 410 may convert the acquired external picture into data for transmission to processor 401 to perform image processing operations.
  • the electronic device 400 may further include a Bluetooth module or the like, and details are not described herein again.
  • the electronic device obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and The target face image is aligned with the face in the original image; the target face image is corrected based on the trained convolutional neural network model and the face region in the original image; the corrected target face image and the original image are corrected Fusion.
  • the program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
  • a further embodiment of the present application further provides a storage medium having stored therein a plurality of instructions adapted to be loaded by a processor to perform the steps of any of the image processing methods described above.
  • the program may be stored in a computer readable storage medium, and the storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.
  • ROM Read Only Memory
  • RAM Random Access Memory

Abstract

A method and apparatus for processing an image, a storage medium, and an electronic device. The method comprises: matching, according to position information of a face key point in an original image, the corresponding target face image from a preset face database, and aligning the target face image with the face in the original image; revising the target face image on the basis of the trained convolutional neural network model and a face area in the original image; and fusing the revised target face image with the original image.

Description

图像处理方法、装置、存储介质及电子设备Image processing method, device, storage medium and electronic device
本申请要求于2017年12月28日提交中国专利局、申请号为201711466358.3、发明名称为“图像处理方法、装置、存储介质及电子设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。The present application claims priority to Chinese Patent Application No. JP-A No. No. No. No. No. No. No. No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No In this application.
技术领域Technical field
本申请涉及图像处理技术领域,尤其涉及一种图像处理方法、装置、存储介质及电子设备。The present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, storage medium, and electronic device.
背景技术Background technique
现有的电子设备一般具有拍照、摄影功能。随着智能电子设备和计算机视觉技术的高速发展,用户对于智能电子设备的摄像头的需求不仅仅局限在传统的拍照、摄影,而更多倾向于图像处理功能,如智能美颜、风格迁移等技术被越来越多的智能电子设备所普及。Existing electronic devices generally have a photographing and photographing function. With the rapid development of intelligent electronic devices and computer vision technology, users' demand for smart electronic device cameras is not limited to traditional photography and photography, but more inclined to image processing functions, such as smart beauty, style migration and other technologies. It is popularized by more and more intelligent electronic devices.
发明内容Summary of the invention
本申请实施例提供一种图像处理方法、装置、存储介质及电子设备,可以淡化图像拼接痕迹,提升图像合成效果。The embodiment of the present application provides an image processing method, device, storage medium, and electronic device, which can lighten image stitching and improve image synthesis.
第一方面,本申请实施例提供一种图像处理方法,应用于电子设备,包括:In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and includes:
获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
第二方面,本申请实施例提供了一种图像处理装置,应用于电子设备,包括:In a second aspect, an embodiment of the present application provides an image processing apparatus, which is applied to an electronic device, and includes:
位置获取模块,用于获取原始图像中人脸关键点的第一位置信息;a location obtaining module, configured to acquire first location information of a face key point in the original image;
对齐模块,用于根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;An alignment module, configured to match a corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image;
修正模块,用于基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;a correction module for correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
融合模块,用于将修正后的目标人脸图像与原始图像融合。A fusion module for merging the corrected target face image with the original image.
第三方面,本申请实施例还提供了一种存储介质,所述存储介质中存储有多条指令,所述指令适于由处理器加载以执行以下步骤:In a third aspect, the embodiment of the present application further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to perform the following steps:
获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
第四方面,本申请实施例还提供了一种电子设备,包括处理器、存储器,所述处理器与所述存储器电性连接,所述存储器用于存储指令和数据;处理器用于执行以下步骤:In a fourth aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, where the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is configured to perform the following steps. :
构建卷积神经网络;Construct a convolutional neural network;
获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
基于所述训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
附图说明DRAWINGS
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的 附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present application. Other drawings can also be obtained from those skilled in the art based on these drawings without paying any creative effort.
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present application. Other drawings can also be obtained from those skilled in the art based on these drawings without paying any creative effort.
图1是本申请实施例提供的电子设备实现深度学习的场景构架示意图。FIG. 1 is a schematic diagram of a scenario structure of an electronic device for implementing deep learning according to an embodiment of the present application.
图2是本申请实施例提供的图像处理方法的一种流程示意图。FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
图3是本申请实施例提供的图像处理方法的一种应用场景图。FIG. 3 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application.
图4是本申请实施例提供的卷积神经网络的局部结构示意图。4 is a schematic partial structural diagram of a convolutional neural network provided by an embodiment of the present application.
图5是本申请实施例提供的图像处理方法的另一种应用场景图。FIG. 5 is another application scenario diagram of an image processing method provided by an embodiment of the present application.
图6是本申请实施例提供的图像处理装置的另一种流程示意图。FIG. 6 is another schematic flowchart of an image processing apparatus according to an embodiment of the present application.
图7是本申请实施例提供的图像处理装置的一种结构示意图。FIG. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
图8是本申请实施例提供的图像处理装置的另一种结构示意图。FIG. 8 is another schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
图9是本申请实施例提供的图像处理装置的又一种结构示意图。FIG. 9 is still another schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
图10是本申请实施例提供的图像处理装置的再一种结构示意图。FIG. 10 is a schematic diagram of still another structure of an image processing apparatus according to an embodiment of the present application.
图11是本申请实施例提供的电子设备的一种结构示意图。FIG. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
图12是本申请实施例提供的电子设备的另一种结构示意图。FIG. 12 is another schematic structural diagram of an electronic device according to an embodiment of the present application.
具体实施方式Detailed ways
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the drawings in the embodiments of the present application. It is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present application without creative efforts are within the scope of the present application.
本申请实施例提供一种图像处理方法、装置、存储介质及电子设备。以下将分别进行详细说明。The embodiment of the present application provides an image processing method, device, storage medium, and electronic device. The details will be described separately below.
请参阅图1,图1为本申请实施例提供的电子设备实现深度学习的场景示意图。Referring to FIG. 1 , FIG. 1 is a schematic diagram of a scenario in which an electronic device implements deep learning according to an embodiment of the present disclosure.
当用户通过电子设备中的图像处理功能对图像进行处理时,电子设备可记录处理过程中的输入输出数据。其中,电子设备中可以包括数据采集统计系统与带回馈调整的预测系统。电子设备可通过数据采集系统获取用户大量的图像分类结果数据,作出相应的统计,并提取图像的图像特征,基于机器深度学习对所提取到的图像特征进行分析处理,对卷积神经网络的损失函数进行参数训练。在输入图像时,电子设备通过预测系统预测图像的分类结果。在用户做出最终选择行为后,所述预测系统根据用户行为的最终结果,反向回馈调整各权重项的权值。经过多次的迭代更正以后,使得所述预测系统的各个权重项的权值最终收敛,形成学习得到的数据库。When the user processes the image through the image processing function in the electronic device, the electronic device can record the input and output data during the processing. The electronic device may include a data collection and statistics system and a prediction system with feedback adjustment. The electronic device can acquire a large amount of image classification result data of the user through the data acquisition system, make corresponding statistics, and extract image features of the image, and analyze and process the extracted image features based on machine depth learning, and the loss to the convolutional neural network. The function performs parameter training. When an image is input, the electronic device predicts the classification result of the image through the prediction system. After the user makes the final selection behavior, the prediction system reversely reciprocates the weights of the weighting items according to the final result of the user behavior. After a number of iterative corrections, the weights of the weighting items of the prediction system are finally converged to form a learned database.
电子设备可以为移动终端,如手机、平板电脑等,也可以为传统的PC(Personal Computer,个人电脑)等,本申请实施例对此不进行限定。The electronic device may be a mobile terminal, such as a mobile phone, a tablet computer, or the like, or may be a conventional PC (Personal Computer), etc., which is not limited in this embodiment of the present application.
本申请实施例提供一种图像处理方法,应用于电子设备,所述图像处理方法包括:An embodiment of the present application provides an image processing method, which is applied to an electronic device, where the image processing method includes:
获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
在一些实施例中,在获取原始图像中人脸关键点的第一位置信息之前,所述方法还包括:In some embodiments, before acquiring the first location information of the face key in the original image, the method further includes:
构建卷积神经网络;Construct a convolutional neural network;
获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
基于所述训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
在一些实施例中,基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正的步骤,包括:In some embodiments, the step of correcting the target face image based on the trained convolutional neural network model and the face region in the original image includes:
基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting content features, illumination features, and smoothing features from the face regions of the original image based on the trained convolutional neural network model;
根据所述内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;And generating an adjustment parameter according to the content feature, the illumination feature, and the smooth feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
根据所述调整参数对目标人脸图像进行修正。Correcting the target face image according to the adjustment parameter.
在一些实施例中,根据所述位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐的步骤,包括:In some embodiments, the step of matching the corresponding target face image from the preset face database according to the location information, and aligning the target face image with the face in the original image includes:
获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining second location information of a face key point of the plurality of sample face images in the preset face database;
将第一位置信息与第二位置信息进行匹配;Matching the first location information with the second location information;
从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sample face image with the largest matching degree from the sample face image as the target face image;
通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
在一些实施例中,在将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,所述方法还包括:In some embodiments, after the target face image is aligned with the face in the original image, and the corrected target face image is merged with the original image, the method further includes:
对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;Performing edge feature point detection on the face region in the original image, and acquiring third position information of the edge feature point;
根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
在一些实施例中,将修正后的目标人脸图像与原始图像融合的步骤,包括:In some embodiments, the step of fusing the corrected target face image with the original image comprises:
根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
利用所述人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image by using the face mask.
在一些实施例中,所述获取原始图像中人脸关键点的第一位置信息,包括:In some embodiments, the acquiring first location information of a face key point in the original image includes:
提取原始图像的图像特征;Extracting image features of the original image;
根据所述图像特征确定原始图像中的人脸区域;Determining a face region in the original image according to the image feature;
对人脸区域进行人脸关键点检测,以获取人脸关键点的第一位置信息。The face key point detection is performed on the face area to obtain the first position information of the face key point.
在一实施例中,提供一种图像处理方法,如图2所示,流程可以如下:In an embodiment, an image processing method is provided. As shown in FIG. 2, the flow may be as follows:
101、获取原始图像中人脸关键点的第一位置信息。101. Acquire first location information of a key point of the face in the original image.
本申请实施例中,该原始图像中包括有至少一个人脸。其中,该原始图像具体可以为电子设备通过摄像头采集到的图像,也可以是电子设备从服务器或其他外接设备的存储区中直接获取得图像。In the embodiment of the present application, the original image includes at least one human face. The original image may be an image captured by the electronic device through the camera, or may be an image directly obtained by the electronic device from a storage area of a server or other external device.
若电子设备通过自带摄像头采集图像,则所采集的图像可以是通过前置设备头自拍得到的图像,也可以是通过后置摄像头所采集到的带有人脸的图像。在一些实施例中,该摄像头可以为数字摄像头,也可为模拟摄像头。数字摄像头可以将图像采集设备产生的模拟图像信号转换成数字信号,进而将其储存在计算机里。模拟摄像头捕捉到的图像信号必须经过特定的图像捕捉卡将模拟信号转换成数字模式,并加以压缩后才可以转换到计算机上运用。数字摄像头可以直接捕捉影像,然后通过串、并口或者USB接口传到计算机里。本申请实施例中,电子设备一般采用数字摄像头,以实时将所采集的画面转换成数据在电子设备的显示界面(即相机的预览框)上实时显示。If the electronic device collects an image through its own camera, the acquired image may be an image obtained by self-timer of the front device head, or may be an image with a face collected by the rear camera. In some embodiments, the camera can be a digital camera or an analog camera. The digital camera converts the analog image signal generated by the image acquisition device into a digital signal, which is then stored in a computer. The image signal captured by the analog camera must be converted to a digital mode by a specific image capture card and compressed before being converted to a computer for use. The digital camera captures the image directly and transmits it to the computer via a serial, parallel or USB interface. In the embodiment of the present application, the electronic device generally adopts a digital camera to convert the collected image into data in real time and display it on the display interface of the electronic device (ie, the preview frame of the camera).
本申请实施例中,首先需对原始图像进行人脸检测,确定出人脸区域,然后再从人脸区域中进行人脸关键点的检测,以获取人脸关键点的位置信息。也即,在一些实施例中, 步骤“获取原始图像中人脸关键点的第一位置信息”可以包括以下流程:In the embodiment of the present application, the face image is first detected on the original image, the face region is determined, and then the face key point is detected from the face region to obtain the location information of the face key point. That is, in some embodiments, the step of "acquiring first location information of a face key point in the original image" may include the following process:
提取原始图像的图像特征;Extracting image features of the original image;
根据所述图像特征确定原始图像中的人脸区域;Determining a face region in the original image according to the image feature;
对人脸区域进行人脸关键点检测,以获取人脸关键点的第一位置信息。The face key point detection is performed on the face area to obtain the first position information of the face key point.
具体地,可以采用Dlib进行人脸关键点的检测,Dlib是一个机器学习的C++库,包含了许多机器学习常用的算法。参考图3,图3为所检测到的人脸关键点的标定结果示意图。其中,下巴由17个人脸关键点标定而成,左右眉毛各由5个人脸关键点标定而成,鼻子由9个人脸关键点标定而成,左右眼各自由6个人脸关键点标定而成,嘴巴由20个人脸关键点标定而成,共计有68个人脸关键点。Specifically, Dlib can be used to detect face key points. Dlib is a machine learning C++ library that contains many algorithms commonly used in machine learning. Referring to FIG. 3, FIG. 3 is a schematic diagram of calibration results of detected key points of a face. Among them, the chin is calibrated by 17 key points of the face, the left and right eyebrows are each calibrated by 5 key points of the face, the nose is calibrated by 9 key points of the face, and the left and right eyes are respectively calibrated by 6 key points of the face. The mouth is calibrated by 20 key points of the face, with a total of 68 face key points.
102、根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐。102. Match the corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image.
具体地,由于原始图像在拍摄时可能会受拍摄角度的影响,导致图像发生形变,严重影响到图像的识别。因此,需采取一定措施对图像进行变换,对其进行一定程度的校正以方便机器的识别和配准。因此,在本申请实施例中,需要预先构建一个人脸数据库,存储用于进行身份替换的参考人脸不同姿态的图像,具体可以使获取不同角度的照片,以便原始图像可以人脸数据库中的参考人脸图像作为目标身份,进行匹配、换脸操作。Specifically, since the original image may be affected by the shooting angle at the time of shooting, the image is deformed, which seriously affects the recognition of the image. Therefore, it is necessary to take certain measures to transform the image and perform a certain degree of correction to facilitate the identification and registration of the machine. Therefore, in the embodiment of the present application, it is necessary to construct a face database in advance, and store an image of a reference face with different postures for performing identity replacement, in particular, to obtain photos of different angles, so that the original image can be in the face database. Refer to the face image as the target identity, and perform matching and face changing operations.
本申请实施例中,将目标人脸图像与原始图像中的人脸对齐的方式可以有多种,比如,可以采用仿射变换的方式将目标人脸图像与原始图像中的人脸对齐。也即,在一些实施例中,步骤“根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐”可以包括以下流程:In the embodiment of the present application, the manner in which the target face image is aligned with the face in the original image may be various. For example, the target face image may be aligned with the face in the original image by using an affine transformation. That is, in some embodiments, the step of “matching the corresponding target face image from the preset face database according to the first location information and aligning the target face image with the face in the original image” may include the following process. :
获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining second location information of a face key point of the plurality of sample face images in the preset face database;
将第一位置信息与第二位置信息进行匹配;Matching the first location information with the second location information;
从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sample face image with the largest matching degree from the sample face image as the target face image;
通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
具体地,将第一位置信息与人脸数据库中样本人脸图像的第二位置信息进行匹配,匹配度越大,则表示两个图像的姿态、大小越接近,从而可降低后续仿射变换过程中变换矩阵的计算难度。为了提升人脸图像与样本人脸图像的匹配度,可以获取大量不同姿态的人脸图像,以增加人脸数据库中样本人脸图像偏转角度的密度,减小偏转角度之间的间隔值。Specifically, the first location information is matched with the second location information of the sample face image in the face database, and the greater the matching degree, the closer the posture and size of the two images are, so that the subsequent affine transformation process can be reduced. The computational difficulty of the medium transformation matrix. In order to improve the matching degree between the face image and the sample face image, a large number of face images with different postures can be obtained to increase the density of the deflection angle of the sample face image in the face database, and reduce the interval value between the deflection angles.
其中,仿射变换,又称仿射映射,是指在几何中,一个向量空间进行一次线性变换并接上一个平移,变换为另一个向量空间。仿射变换有:平移、旋转、缩放、斜切等。要进行仿射变换,必须先获取变换矩阵。要获取变换矩阵,必须先获取特征点坐标、角度等信息,如几何匹配、bolb等方法都可获取特征点坐标、角度信息。Among them, affine transformation, also known as affine mapping, means that in geometry, a vector space undergoes a linear transformation and is connected to a translation, transforming into another vector space. Affine transformations include: translation, rotation, scaling, beveling, and so on. To perform an affine transformation, you must first obtain the transformation matrix. To obtain the transformation matrix, you must first obtain the coordinates of the feature points, angles, etc., such as geometric matching, bolb and other methods can obtain the feature point coordinates, angle information.
在具体实施时,首先要获取目标人脸图像中关键点的坐标、角度信息(即第二位置信息),然后根据获取的第二位置信息和第一位置信息计算出仿射变换矩阵。根据所计算出的变换矩阵对目标人脸图像进行仿射变换,将目标人脸图像映射到原始图像的人脸位置。In a specific implementation, first, coordinate and angle information (ie, second position information) of key points in the target face image are acquired, and then an affine transformation matrix is calculated according to the acquired second position information and the first position information. The target face image is affine transformed according to the calculated transformation matrix, and the target face image is mapped to the face position of the original image.
103、基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正。103. Correct the target face image based on the trained convolutional neural network model and the face region in the original image.
在一些实施例中,在获取原始图像中人脸关键点的第一位置信息之前,该方法还可以包括以下步骤:In some embodiments, before acquiring the first location information of the face key point in the original image, the method may further include the following steps:
构建卷积神经网络;Construct a convolutional neural network;
获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
基于训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, the constructed convolutional neural network is trained to adjust the content loss function, the illumination loss function, and the smoothing loss function parameters to obtain the trained convolutional neural network model.
参考图4,在本申请实施例中,所构建的卷积神经网络为一个带有分支的多尺度架构,这些分支根据所输入测试图像的尺寸的不同,在不同的采样版本上执行运算。小尺寸的图像经过卷积后自动上采样为2倍大小,然后再和大尺寸的图像进行通道连接。每一个这样的分支都有零填充的卷积模块,其后还跟着线性修正(linear rectification)。这些分支再通过相差一倍的最近邻上采样(nearest-neighbor upsampling)和沿信道轴的级联(concatenation along the channel axis)组合起来。Referring to FIG. 4, in the embodiment of the present application, the constructed convolutional neural network is a multi-scale architecture with branches, and the branches perform operations on different sampling versions according to the size of the input test image. Small images are automatically upsampled to a size of 2 after being convolved, and then channeled to a large image. Each such branch has a zero-filled convolution module followed by a linear rectification. These branches are then combined by a nearest-neighbor upsampling that differs by one and a concatenation along the channel axis.
在具体训练过程中,首先输入训练样本,执行参数初始化,经过卷积和采样过程后到达全连接层,并进行仿射变换和参数计算,输出处理后的图像,通过逻辑回归分析,得到各损失函数的权重,通过人为判断是否符合期望来不断反馈修正各损失函数的参数设置。In the specific training process, the training sample is first input, the parameter initialization is performed, and after the convolution and sampling process, the full connection layer is reached, and the affine transformation and parameter calculation are performed, and the processed image is outputted, and the loss is obtained by logistic regression analysis. The weight of the function is continuously feedback to correct the parameter settings of each loss function by artificially judging whether it meets the expectations.
在一些实施例中,“基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正”可以包括以下步骤:In some embodiments, "correcting the target face image based on the trained convolutional neural network model and the face region in the original image" may include the following steps:
基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting content features, illumination features, and smoothing features from the face regions of the original image based on the trained convolutional neural network model;
根据内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;According to the content feature, the illumination feature, and the smooth feature, the adjustment parameter is generated under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
根据调整参数对目标人脸图像进行修正。The target face image is corrected according to the adjustment parameters.
在本申请实施例中,可将人脸交换描述成一种风格迁移的问题而实现。而风格迁移的目标是将一张图像渲染成另一张图像的风格。有基于此,将原始图像中人脸的姿势和表情作为内容,目标人脸图像作为风格,设计一种能让该卷积神经网络生成高图像真实度结果的损失函数。图像的损失函数是基于一个已经训练好的神经网络里的特征图。In the embodiment of the present application, the face exchange can be described as a problem of style migration. The goal of style migration is to render an image into the style of another image. Based on this, the pose and expression of the face in the original image are taken as the content, and the target face image is used as the style, and a loss function that allows the convolutional neural network to generate a high image realism result is designed. The loss function of the image is based on a feature map in an already trained neural network.
针对风格损失函数的问题,可采用最近邻方法,即原图中的某个位置的图像用目标图中最相似的片段进行替换。根据人脸中提取的人脸关键点来对搜索域进行限制。即对原始图像中人脸的某个部分,只在目图像中的某个部分附近进行相似片区搜索。For the problem of the style loss function, the nearest neighbor method can be used, that is, the image at a certain position in the original image is replaced with the most similar segment in the target image. The search domain is restricted according to the key points extracted from the face. That is, for a certain part of the face in the original image, a similar tile search is performed only near a certain part of the mesh image.
在一些实施例中,需要目标人脸的多张图像,即多张风格图像。在相似片区搜索时,损失在图像区域上有所限制,但是可以在多张图像提取的片区上进行搜索,这样,可以保证能够复现多种多样的表情。In some embodiments, multiple images of the target face, ie multiple style images, are required. In the case of similar slice search, the loss is limited in the image area, but the search can be performed on the pieces extracted by the plurality of images, so that a variety of expressions can be reproduced.
本实施例中,为了保持换脸过程中光照保持不变,需要对光照上的变换进行惩罚。而为了提取光照变化,算法针对光照训练了一个卷积神经网络分类器。针对两张除了光照外其他都不变的图像,分类器判断这一对图像是否发生了光照变换,并使用从这个网络中得到的特征图进行光照损失的计算。In this embodiment, in order to keep the illumination unchanged during the face changing process, it is necessary to punish the transformation on the illumination. In order to extract the illumination changes, the algorithm trained a convolutional neural network classifier for illumination. For two images that are invariant except for illumination, the classifier determines whether the pair of images has undergone illumination transformation, and uses the feature map obtained from the network to calculate the illumination loss.
104、将修正后的目标人脸图像与原始图像融合。104. The corrected target face image is merged with the original image.
本申请实施例中,将修正后的目标人脸图像与原始图像融合的方式可以有多种。一类是基于区域的算法,是指利用两张图像间灰度的关系来确定图像间坐标变化的参数,其中包括基于空间的像素配准算法和基于频域的算法。另一类是基于特征拼接的算法,是利用图像中的明显特征(点、线、边缘、轮廓、角点)来计算图像之间的变换。第三类是基于最大互信息的拼接,将拼接工作由空间域转向小域波,进行小波重构即可获得完整的图像。In the embodiment of the present application, there may be multiple ways to fuse the corrected target face image with the original image. One type is a region-based algorithm, which refers to a parameter that uses the relationship between two images to determine the coordinate change between images, including a space-based pixel registration algorithm and a frequency domain-based algorithm. The other type is an algorithm based on feature splicing, which uses the obvious features (points, lines, edges, contours, corner points) in the image to calculate the transformation between images. The third type is based on the splicing of the maximum mutual information, and the splicing work is shifted from the spatial domain to the small domain wave, and the wavelet reconstruction can obtain a complete image.
在一些实施例中,在将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,还可以包括以下步骤:In some embodiments, after the target face image is aligned with the face in the original image, before the corrected target face image is merged with the original image, the following steps may be further included:
对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;Performing edge feature point detection on the face region in the original image, and acquiring third position information of the edge feature point;
根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
具体地,沿着所检测到的边缘特征点的位置切割图案,由于边缘特征点为人脸边缘特征点,因此,最终可以将人脸区域从原始图像中分割开来,最终可得到背景图像。Specifically, the pattern is cut along the position of the detected edge feature point. Since the edge feature point is a face edge feature point, the face region can be finally separated from the original image, and finally the background image can be obtained.
则步骤“将修正后的目标人脸图像与原始图像融合”可以包括以下流程:Then the step "merging the corrected target face image with the original image" may include the following process:
根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
利用该人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The face mask is used to fuse the corrected target face image with the background image in the original image.
其中,边缘特征点的位置信息可以为这些边缘特征点相互之间的相对位置信息。基于这些边缘特征点构成一闭合图案,将该闭合图案以外的区域作为人脸掩膜(segmentation mask)。将人脸掩膜加持在映射在原始图像上人脸区域的目标人脸图像上,并与原始图像的人脸区域对齐,将目标人脸图像中未被该人脸掩膜遮挡的区域进行显示,而被遮挡的区域则不显示。比如,参考图5,其中,a为目标人脸图像,b为原始图像,c为基于原始图像b中的人脸区域生成的人脸掩膜,d为目标图像a经仿射变换后得到的图像,最终输出换脸后的融合图像e。The position information of the edge feature points may be relative position information between the edge feature points. A closed pattern is formed based on the edge feature points, and a region other than the closed pattern is used as a segmentation mask. The face mask is superimposed on the target face image mapped on the face region of the original image, and aligned with the face region of the original image, and the region of the target face image that is not blocked by the face mask is displayed. The occluded area is not displayed. For example, referring to FIG. 5, where a is the target face image, b is the original image, c is a face mask generated based on the face region in the original image b, and d is obtained by affine transformation of the target image a. The image is finally output as the fused image e after changing the face.
在将目标画面中人脸图像替换为处理后的目标人脸图像时,可通基于泊松融合技术,将处理后的目标人脸图像与目标画面融合,覆盖目标画面中原有的人脸图像,从而实现将目标画面中人脸图像替换为处理后的目标人脸图像。其中,泊松融合技术可以较好地消除目标人脸图像与目标画面的交界,使得画面更加自然且不突兀,实现无缝拼接。When the face image in the target picture is replaced with the processed target face image, the processed target face image may be merged with the target image based on the Poisson fusion technique to cover the original face image in the target image. Thereby, the face image in the target picture is replaced with the processed target face image. Among them, Poisson fusion technology can better eliminate the boundary between the target face image and the target image, making the picture more natural and unobtrusive, achieving seamless splicing.
由上可知,通过获取原始图像中人脸关键点的第一位置信息;根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;将修正后的目标人脸图像与原始图像融合。该方案可通过深度学习技术更好的保持原图的某些特征不变,同时可淡化图像拼接痕迹,提升图像合成效果。It can be seen that the first location information of the face key point in the original image is obtained; the corresponding target face image is matched from the preset face database according to the first location information, and the target face image is in the original image. Face alignment; based on the trained convolutional neural network model and the face region in the original image, the target face image is corrected; and the corrected target face image is merged with the original image. The program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
在一实施例中,还提供另一种图像处理方法,如图6所示,流程可以如下:In an embodiment, another image processing method is also provided. As shown in FIG. 6, the flow may be as follows:
201、构建卷积神经网络。201. Construct a convolutional neural network.
在本申请实施例中,在本申请实施例中,所构建的卷积神经网络为一个带有分支的多尺度架构,这些分支根据所输入测试图像的尺寸的不同,在不同的采样版本上执行运算。小尺寸的图像经过卷积后自动上采样为2倍大小,然后再和大尺寸的图像进行通道连接。每一个这样的分支都有零填充的卷积模块,其后还跟着线性修正。这些分支再通过相差一倍的最近邻上采样和沿信道轴的级联组合起来。In the embodiment of the present application, in the embodiment of the present application, the constructed convolutional neural network is a multi-scale architecture with branches, and the branches are executed on different sampling versions according to the size of the input test image. Operation. Small images are automatically upsampled to a size of 2 after being convolved, and then channeled to a large image. Each such branch has a zero-filled convolution module followed by a linear correction. These branches are then combined by a nearest neighbor upsampling that differs by one and a cascade along the channel axis.
202、基于训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。202. Perform parameter training on the constructed convolutional neural network based on the training samples to adjust the content loss function, the illumination loss function, and the parameter setting of the smoothing loss function to obtain a trained convolutional neural network model.
在本申请实施例中,为了保证后续换脸后原图中的某些特征不会损失掉,需对卷积神经网络中的损失函数进行参数训练。In the embodiment of the present application, in order to ensure that certain features in the original image are not lost after the subsequent face changing, parameter training of the loss function in the convolutional neural network is performed.
具体地,可将人脸交换描述成一种风格迁移的问题而实现。而风格迁移的目标是将一张图像渲染成另一张图像的风格。有基于此,将原始图像中人脸的姿势和表情作为内容,目标人脸图像作为风格,设计一种能让该卷积神经网络生成高图像真实度结果的损失函数。Specifically, the face exchange can be described as a problem of style migration. The goal of style migration is to render an image into the style of another image. Based on this, the pose and expression of the face in the original image are taken as the content, and the target face image is used as the style, and a loss function that allows the convolutional neural network to generate a high image realism result is designed.
在一些实施例中,可以获取同一人脸的多个角度的图像作为训练样本。在具体训练过程中,首先输入训练样本,执行参数初始化,经过卷积和采样过程后到达全连接层,并进行仿射变换和参数计算,输出处理后的图像,通过逻辑回归分析,得到各损失函数的权重,通过人为判断是否符合期望来不断反馈修正各损失函数的参数设置。In some embodiments, images of multiple angles of the same face may be acquired as training samples. In the specific training process, the training sample is first input, the parameter initialization is performed, and after the convolution and sampling process, the full connection layer is reached, and the affine transformation and parameter calculation are performed, and the processed image is outputted, and the loss is obtained by logistic regression analysis. The weight of the function is continuously feedback to correct the parameter settings of each loss function by artificially judging whether it meets the expectations.
203、人脸关键点检测,获取原始图像中人脸关键点的第一位置信息、以及目标人脸图像的人脸关键点的第二位置信息。203. The face key point detection acquires first position information of the face key point in the original image and second position information of the face key point of the target face image.
本申请实施例中,该原始图像中包括有至少一个人脸。其中,该原始图像具体可以为电子设备通过摄像头采集到的图像,也可以是电子设备从服务器或其他外接设备的存储区中直接获取得图像。而目标人脸图像则是用于对原始图像中人脸进行身份替换的参考人脸。In the embodiment of the present application, the original image includes at least one human face. The original image may be an image captured by the electronic device through the camera, or may be an image directly obtained by the electronic device from a storage area of a server or other external device. The target face image is a reference face for the identity replacement of the face in the original image.
因此,在本申请实施例中,需要预先构建一个人脸数据库,存储用于进行身份替换的参考人脸不同姿态的图像,具体可以使获取不同角度的照片,以便原始图像可以人脸数据库中的参考人脸图像作为目标身份,进行匹配、换脸操作。Therefore, in the embodiment of the present application, it is necessary to construct a face database in advance, and store an image of a reference face with different postures for performing identity replacement, in particular, to obtain photos of different angles, so that the original image can be in the face database. Refer to the face image as the target identity, and perform matching and face changing operations.
具体实施时,首先需对原始图像进行人脸检测,确定出人脸区域,然后再从人脸区域中进行人脸关键点的检测,以获取人脸关键点的位置信息。In the specific implementation, the original image is firstly detected by the face, the face area is determined, and then the face key point is detected from the face area to obtain the position information of the face key point.
204、根据第一位置信息和第二位置信息计算仿射变换矩阵,基于仿射变换矩阵将目标人脸图像映射到原始图像的人脸区域。204. Calculate an affine transformation matrix according to the first location information and the second location information, and map the target face image to the face region of the original image based on the affine transformation matrix.
由于原始图像在拍摄时可能会受拍摄角度的影响,导致图像发生形变,严重影响到图像的识别。因此,需采取一定措施对图像进行变换,对其进行一定程度的校正以方便机器的识别和配准。Since the original image may be affected by the shooting angle when shooting, the image is deformed, which seriously affects the recognition of the image. Therefore, it is necessary to take certain measures to transform the image and perform a certain degree of correction to facilitate the identification and registration of the machine.
具体地,将第一位置信息与二位置信息进行匹配,并建立起原始图像与目标人脸图像中人脸关键点一一对应的匹配关系。然后通过人脸关键点对求出两副图像的仿射变换矩阵,并基于仿射变换矩阵将目标人脸图像映射到原始图像的人脸区域,以使将目标人脸图像与原始图像中的人脸对齐。Specifically, the first location information is matched with the two location information, and a matching relationship between the original image and the face key points in the target face image is established one by one. Then, the affine transformation matrix of the two images is obtained through the face key point pair, and the target face image is mapped to the face region of the original image based on the affine transformation matrix, so that the target face image and the original image are in the original image. The faces are aligned.
205、基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正。205. Correct the target face image based on the trained convolutional neural network model and the face region in the original image.
具体地,基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;根据内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成相应的调整参数,根据调整参数对目标人脸图像进行修正。通过训练好的卷积神经网络模型调参,更好的保持原始图像中人脸的表情、肤色、光照等特征不变,使得换脸后的人脸更加自然。Specifically, the content feature, the illumination feature, and the smooth feature are extracted from the face region of the original image based on the trained convolutional neural network model; the content loss function, the illumination loss function, according to the content feature, the illumination feature, and the smooth feature, Under the constraint of the smoothing loss function, the corresponding adjustment parameters are generated, and the target face image is corrected according to the adjustment parameters. Through the trained convolutional neural network model to adjust the parameters, it is better to keep the facial expression, skin color, illumination and other features in the original image unchanged, making the face after changing face more natural.
206、对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息。206. Perform edge feature point detection on the face region in the original image, and obtain third location information of the edge feature point.
同样的,首先需对原始图像进行人脸检测,再基于相关边缘算法,计算获取到人脸区域边缘特征点的第三位置信息。Similarly, the original image needs to be face-detected, and then the third position information of the edge feature points of the face region is calculated based on the correlation edge algorithm.
207、根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。207. Perform image segmentation processing on the original image according to the third location information to remove the face region, and use the remaining image region as the background image.
具体地,沿着所检测到的边缘特征点的位置切割图案,由于边缘特征点为人脸边缘特征点,因此,最终可以将人脸区域从原始图像中分割开来,最终可得到背景图像。Specifically, the pattern is cut along the position of the detected edge feature point. Since the edge feature point is a face edge feature point, the face region can be finally separated from the original image, and finally the background image can be obtained.
208、根据第三位置信息生成人脸掩膜,并利用该人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。208. Generate a face mask according to the third location information, and use the face mask to fuse the corrected target face image with the background image in the original image.
其中,边缘特征点的位置信息可以为这些边缘特征点相互之间的相对位置信息。基于这些边缘特征点构成一闭合图案,将该闭合图案以外的区域作为人脸掩膜。将人脸掩膜加持在映射在原始图像上人脸区域的目标人脸图像上,并与原始图像的人脸区域对齐,将目标人脸图像中未被该人脸掩膜遮挡的区域进行显示,而被遮挡的区域则不显示。The position information of the edge feature points may be relative position information between the edge feature points. A closed pattern is formed based on the edge feature points, and an area other than the closed pattern is used as a face mask. The face mask is superimposed on the target face image mapped on the face region of the original image, and aligned with the face region of the original image, and the region of the target face image that is not blocked by the face mask is displayed. The occluded area is not displayed.
在将目标画面中人脸图像替换为处理后的目标人脸图像时,可通基于泊松融合技术,将处理后的目标人脸图像与目标画面融合,覆盖目标画面中原有的人脸图像,从而实现将目标画面中人脸图像替换为处理后的目标人脸图像。其中,泊松融合技术可以较好地消除目标人脸图像与目标画面的交界,使得画面更加自然且不突兀,实现无缝拼接。When the face image in the target picture is replaced with the processed target face image, the processed target face image may be merged with the target image based on the Poisson fusion technique to cover the original face image in the target image. Thereby, the face image in the target picture is replaced with the processed target face image. Among them, Poisson fusion technology can better eliminate the boundary between the target face image and the target image, making the picture more natural and unobtrusive, achieving seamless splicing.
由上可知,本申请实施例提供的图像处理方法,通过获取原始图像中人脸关键点的第一位置信息;根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;将修正后的目标人脸图像与原始图像融合。该方案可通过深度学习技术可以更好的保持原图的表情、肤色、光照不变,同时可淡化图像拼接痕迹,提升图像合成效果。It can be seen that the image processing method provided by the embodiment of the present application obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and Aligning the target face image with the face in the original image; correcting the target face image based on the trained convolutional neural network model and the face region in the original image; correcting the target face image and original Image fusion. The program can better maintain the expression, skin color and illumination of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
在本申请又一实施例中,还提供一种图像处理装置,该图像处理装置可以软件或硬件的形式集成在电子设备中,该电子设备具体可以包括手机、平板电脑、笔记本电脑等设备。 如图7所示,该图像处理装置30可以包括位置获取模块31、对齐模块32、修正模块33、以及融合模块34,其中:In another embodiment of the present application, an image processing apparatus is further provided, which may be integrated in an electronic device in the form of software or hardware, and the electronic device may specifically include a mobile phone, a tablet computer, a notebook computer, and the like. As shown in FIG. 7, the image processing apparatus 30 may include a location acquisition module 31, an alignment module 32, a correction module 33, and a fusion module 34, where:
位置获取模块31,用于获取原始图像中人脸关键点的第一位置信息;a location obtaining module 31, configured to acquire first location information of a face key point in the original image;
对齐模块32,用于根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;The aligning module 32 is configured to match the corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image;
修正模块33,用于基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;The correction module 33 is configured to correct the target face image based on the trained convolutional neural network model and the face region in the original image;
融合模块34,用于将修正后的目标人脸图像与原始图像融合。The fusion module 34 is configured to fuse the corrected target face image with the original image.
在一些实施例中,参考图8,该图像处理装置30还可以包括:In some embodiments, referring to FIG. 8, the image processing apparatus 30 may further include:
构建模块35,用于在获取原始图像中人脸关键点的第一位置信息之前,构建卷积神经网络;a building module 35, configured to construct a convolutional neural network before acquiring first location information of a face key point in the original image;
样本获取模块36,用于获取原始图像中人脸的多个角度的图像作为训练样本;a sample obtaining module 36, configured to acquire an image of a plurality of angles of a face in the original image as a training sample;
训练模块37,用于基于训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。The training module 37 is configured to perform parameter training on the constructed convolutional neural network based on the training samples to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
在一些实施例中,参考图9,修正模块33可以包括:In some embodiments, referring to FIG. 9, the correction module 33 can include:
提取子模块331,用于基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;The extraction sub-module 331 is configured to extract a content feature, an illumination feature, and a smooth feature from the face region of the original image based on the trained convolutional neural network model;
生成子模块332,用于根据内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;The generating submodule 332 is configured to generate an adjustment parameter according to the content feature, the illumination feature, and the smoothing feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
修正子模块333,用于根据调整参数对目标人脸图像进行修正。The correction sub-module 333 is configured to correct the target facial image according to the adjustment parameter.
在一些实施例中,参考图10,修正模块32可以包括:In some embodiments, referring to FIG. 10, the correction module 32 can include:
获取子模块321,用于获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;The obtaining sub-module 321 is configured to acquire second location information of a face key point of the plurality of sample face images in the preset face database;
匹配子模块322,用于将第一位置信息与第二位置信息进行匹配;a matching sub-module 322, configured to match the first location information with the second location information;
选取子模块323,用于从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;The selecting sub-module 323 is configured to select, from the sample face image, the sample face image with the largest matching degree as the target face image;
映射子模块324,用于通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The mapping sub-module 324 is configured to map the target face image to the face region of the original image by affine transformation to align the target face image with the face in the original image.
在一些实施例中,继续参考图10,该图像处理装,30还可以包括:In some embodiments, with continued reference to FIG. 10, the image processing device 30 can further include:
边缘特征点获取模块38,用于将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;The edge feature point obtaining module 38 is configured to: after aligning the target face image with the face in the original image, and performing edge feature points on the face region in the original image before merging the corrected target face image with the original image Detecting, and acquiring third position information of edge feature points;
分割模块39,用于根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The segmentation module 39 is configured to perform image segmentation processing on the original image according to the third position information to remove the face region and use the remaining image region as the background image.
在一些实施例中,融合模块34可以用于:In some embodiments, the fusion module 34 can be used to:
根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
利用所述人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image by using the face mask.
在一些实施例中,位置获取模块31可以用于:In some embodiments, the location acquisition module 31 can be used to:
提取原始图像的图像特征;Extracting image features of the original image;
根据所述图像特征确定原始图像中的人脸区域;Determining a face region in the original image according to the image feature;
对人脸区域进行人脸关键点检测,以获取人脸关键点的第一位置信息。The face key point detection is performed on the face area to obtain the first position information of the face key point.
由上可知,本申请实施例提供的图像处理装置,通过获取原始图像中人脸关键点的第一位置信息;根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;基于训练好的卷积神经网络模型和原始图像中的人脸 区域,对目标人脸图像进行修正;将修正后的目标人脸图像与原始图像融合。该方案可通过深度学习技术更好的保持原图的某些特征不变,同时可淡化图像拼接痕迹,提升图像合成效果。It can be seen that the image processing apparatus provided by the embodiment of the present application obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and Aligning the target face image with the face in the original image; correcting the target face image based on the trained convolutional neural network model and the face region in the original image; correcting the target face image and original Image fusion. The program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
在本申请又一实施例中还提供一种电子设备,该电子设备可以是智能手机、平板电脑等设备。如图11所示,电子设备400包括处理器401、存储器402。其中,处理器401与存储器402电性连接。In another embodiment of the present application, an electronic device is further provided, and the electronic device may be a device such as a smart phone or a tablet computer. As shown in FIG. 11, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.
处理器401是电子设备400的控制中心,利用各种接口和线路连接整个电子设备的各个部分,通过运行或加载存储在存储器402内的应用程序,以及调用存储在存储器402内的数据,执行电子设备的各种功能和处理数据,从而对电子设备进行整体监控。The processor 401 is a control center of the electronic device 400, which connects various parts of the entire electronic device using various interfaces and lines, executes the electronic by running or loading an application stored in the memory 402, and calling data stored in the memory 402. The various functions and processing data of the device enable overall monitoring of the electronic device.
在本实施例中,电子设备400中的处理器401会按照如下的步骤,将一个或一个以上的应用程序的进程对应的指令加载到存储器402中,并由处理器401来运行存储在存储器402中的应用程序,从而实现各种功能:In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the process of one or more applications into the memory 402 according to the following steps, and is stored and stored in the memory 402 by the processor 401. In the application, thus implementing various functions:
获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
在一些实施例中,在获取原始图像中人脸关键点的第一位置信息之前,处理器401用于执行以下步骤:In some embodiments, before acquiring the first location information of the face key in the original image, the processor 401 is configured to perform the following steps:
构建卷积神经网络;Construct a convolutional neural network;
获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
基于训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, the constructed convolutional neural network is trained to adjust the content loss function, the illumination loss function, and the smoothing loss function parameters to obtain the trained convolutional neural network model.
在一些实施例中,处理器401还进一步用于执行以下步骤:In some embodiments, the processor 401 is further configured to perform the following steps:
基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting content features, illumination features, and smoothing features from the face regions of the original image based on the trained convolutional neural network model;
根据内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;According to the content feature, the illumination feature, and the smooth feature, the adjustment parameter is generated under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
根据调整参数对目标人脸图像进行修正。The target face image is corrected according to the adjustment parameters.
在一些实施例中,处理器401还进一步用于执行以下步骤:In some embodiments, the processor 401 is further configured to perform the following steps:
获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining second location information of a face key point of the plurality of sample face images in the preset face database;
将第一位置信息与第二位置信息进行匹配;Matching the first location information with the second location information;
从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sample face image with the largest matching degree from the sample face image as the target face image;
通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
在一些实施例中,在将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,处理器401还执行以下步骤:In some embodiments, after aligning the target face image with the face in the original image, before the merged target face image is merged with the original image, the processor 401 further performs the following steps:
对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;Performing edge feature point detection on the face region in the original image, and acquiring third position information of the edge feature point;
根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
在一些实施例中,处理器401还进一步用于执行以下步骤:In some embodiments, the processor 401 is further configured to perform the following steps:
根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
利用人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image using a face mask.
存储器402可用于存储应用程序和数据。存储器402存储的应用程序中包含有可在处理 器中执行的指令。应用程序可以组成各种功能模块。处理器401通过运行存储在存储器402的应用程序,从而执行各种功能应用以及数据处理。 Memory 402 can be used to store applications and data. The application stored in the memory 402 contains instructions that can be executed in the processor. Applications can form various functional modules. The processor 401 executes various functional applications and data processing by running an application stored in the memory 402.
在一些实施例中,如图12所示,电子设备400还包括:显示屏403、控制电路404、射频电路405、输入单元406、音频电路407、传感器408以及电源409。其中,处理器401分别与显示屏403、控制电路404、射频电路405、输入单元406、音频电路407、传感器408以及电源409电性连接。In some embodiments, as shown in FIG. 12, the electronic device 400 further includes a display screen 403, a control circuit 404, a radio frequency circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power source 409. The processor 401 is electrically connected to the display screen 403, the control circuit 404, the radio frequency circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power source 409, respectively.
显示屏403可用于显示由用户输入的信息或提供给用户的信息以及电子设备的各种图形用户接口,这些图形用户接口可以由图像、文本、图标、视频和其任意组合来构成。其中,该显示屏403可以作为本申请实施例中的屏幕,用于显示信息。The display screen 403 can be used to display information entered by the user or information provided to the user as well as various graphical user interfaces of the electronic device, which can be composed of images, text, icons, video, and any combination thereof. The display screen 403 can be used as a screen in the embodiment of the present application for displaying information.
控制电路404与显示屏403电性连接,用于控制显示屏403显示信息。The control circuit 404 is electrically connected to the display screen 403 for controlling the display screen 403 to display information.
射频电路405用于收发射频信号,以通过无线通信与网络设备或其他电子设备建立无线通讯,与网络设备或其他电子设备之间收发信号。The radio frequency circuit 405 is configured to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices through wireless communication, and to transmit and receive signals with network devices or other electronic devices.
输入单元406可用于接收输入的数字、字符信息或用户特征信息(例如指纹),以及产生与用户设置以及功能控制有关的键盘、鼠标、操作杆、光学或者轨迹球信号输入。其中,输入单元406可以包括指纹识别模组。The input unit 406 can be configured to receive input digits, character information, or user characteristic information (eg, fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls. The input unit 406 can include a fingerprint identification module.
音频电路407可通过扬声器、传声器提供用户与电子设备之间的音频接口。The audio circuit 407 can provide an audio interface between the user and the electronic device through a speaker and a microphone.
传感器408用于采集外部环境信息。传感器408可以包括环境亮度传感器、加速度传感器、光传感器、运动传感器、以及其他传感器。 Sensor 408 is used to collect external environmental information. Sensor 408 can include ambient brightness sensors, acceleration sensors, light sensors, motion sensors, and other sensors.
电源409用于给电子设备400的各个部件供电。在一些实施例中,电源409可以通过电源管理系统与处理器401逻辑相连,从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。 Power source 409 is used to power various components of electronic device 400. In some embodiments, the power supply 409 can be logically coupled to the processor 401 through a power management system to enable functions such as managing charging, discharging, and power management through the power management system.
摄像头410用于采集外界画面,可以使数字摄像头,也可以为模拟摄像头。在一些实施例中,摄像头410可将采集到的外界画面转换成数据发送给处理器401以执行图像处理操作。The camera 410 is used for collecting external images, and can be a digital camera or an analog camera. In some embodiments, camera 410 may convert the acquired external picture into data for transmission to processor 401 to perform image processing operations.
尽管图12中未示出,电子设备400还可以包括蓝牙模块等,在此不再赘述。Although not shown in FIG. 12, the electronic device 400 may further include a Bluetooth module or the like, and details are not described herein again.
由上可知,本申请实施例提供的电子设备,通过获取原始图像中人脸关键点的第一位置信息;根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;将修正后的目标人脸图像与原始图像融合。该方案可通过深度学习技术更好的保持原图的某些特征不变,同时可淡化图像拼接痕迹,提升图像合成效果。It can be seen that the electronic device provided by the embodiment of the present application obtains the first location information of the face key point in the original image, and matches the corresponding target face image from the preset face database according to the first location information, and The target face image is aligned with the face in the original image; the target face image is corrected based on the trained convolutional neural network model and the face region in the original image; the corrected target face image and the original image are corrected Fusion. The program can better maintain certain features of the original image through deep learning technology, and at the same time, it can dilute the image stitching and improve the image synthesis effect.
本申请又一实施例中还提供一种存储介质,该存储介质中存储有多条指令,该指令适于由处理器加载以执行上述任一图像处理方法的步骤。A further embodiment of the present application further provides a storage medium having stored therein a plurality of instructions adapted to be loaded by a processor to perform the steps of any of the image processing methods described above.
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成,该程序可以存储于一计算机可读存储介质中,存储介质可以包括:只读存储器(ROM,Read Only Memory)、随机存取记忆体(RAM,Random Access Memory)、磁盘或光盘等。A person skilled in the art may understand that all or part of the various steps of the foregoing embodiments may be performed by a program to instruct related hardware. The program may be stored in a computer readable storage medium, and the storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.
在描述本申请的概念的过程中使用了术语“一”和“所述”以及类似的词语(尤其是在所附的权利要求书中),应该将这些术语解释为既涵盖单数又涵盖复数。此外,除非本文中另有说明,否则在本文中叙述数值范围时仅仅是通过快捷方法来指代属于相关范围的每个独立的值,而每个独立的值都并入本说明书中,就像这些值在本文中单独进行了陈述一样。另外,除非本文中另有指明或上下文有明确的相反提示,否则本文中所述的所有方法的步骤都可以按任何适当次序加以执行。本申请的改变并不限于描述的步骤顺序。除非另外主张,否则使用本文中所提供的任何以及所有实例或示例性语言(例如,“例如”)都仅仅为了更好地说明本申请的概念,而并非对本申请的概念的范围加以限制。在不脱离精神和范围的情况 下,所属领域的技术人员将易于明白多种修改和适应。The terms "a", "an", "the", and "the" In addition, unless otherwise stated herein, the recitation of numerical ranges herein is merely referring to each of the individual These values are stated separately in this article. In addition, the steps of all methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise indicated. Changes to the application are not limited to the sequence of steps described. The use of any and all examples or exemplary language, such as "a" Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope of the invention.
以上对本申请实施例所提供的一种图像处理方法、装置、存储介质及电子设备进行了详细介绍,本文中应用程序了具体个例对本申请的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本申请的方法及其核心思想;同时,对于本领域的技术人员,依据本申请的思想,在具体实施方式及应用程序范围上均会有改变之处,综上所述,本说明书内容不应理解为对本申请的限制。The image processing method, the device, the storage medium and the electronic device provided by the embodiments of the present application are described in detail. In the present application, the principle and the implementation manner of the application are described in the specific examples, and the description of the above embodiments is provided. It is only used to help understand the method of the present application and its core ideas; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the scope of specific implementations and applications, in summary, The contents of this specification are not to be construed as limiting the application.

Claims (20)

  1. 一种图像处理方法,应用于电子设备,其中,包括:An image processing method is applied to an electronic device, including:
    获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
    根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
    基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
    将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
  2. 如权利要求1所述的图像处理方法,其中,在获取原始图像中人脸关键点的第一位置信息之前,所述方法还包括:The image processing method according to claim 1, wherein the method further comprises: before acquiring the first location information of the face key point in the original image, the method further comprising:
    构建卷积神经网络;Construct a convolutional neural network;
    获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
    基于所述训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
  3. 如权利要求2所述的图像处理方法,其中,基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正的步骤,包括:The image processing method according to claim 2, wherein the step of correcting the target face image based on the trained convolutional neural network model and the face region in the original image comprises:
    基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting content features, illumination features, and smoothing features from the face regions of the original image based on the trained convolutional neural network model;
    根据所述内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;And generating an adjustment parameter according to the content feature, the illumination feature, and the smooth feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
    根据所述调整参数对目标人脸图像进行修正。Correcting the target face image according to the adjustment parameter.
  4. 如权利要求1所述的图像处理方法,其中,根据所述位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐的步骤,包括:The image processing method according to claim 1, wherein the step of matching the corresponding target face image from the preset face database according to the position information, and aligning the target face image with the face in the original image, include:
    获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining second location information of a face key point of the plurality of sample face images in the preset face database;
    将第一位置信息与第二位置信息进行匹配;Matching the first location information with the second location information;
    从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sample face image with the largest matching degree from the sample face image as the target face image;
    通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  5. 如权利要求1所述的图像处理方法,其中,在将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,所述方法还包括:The image processing method according to claim 1, wherein the method further comprises: after aligning the target face image with the face in the original image, and merging the corrected target face image with the original image, the method further comprising:
    对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;Performing edge feature point detection on the face region in the original image, and acquiring third position information of the edge feature point;
    根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
  6. 如权利要求5所述的图像处理方法,其中,将修正后的目标人脸图像与原始图像融合的步骤,包括:The image processing method according to claim 5, wherein the step of fusing the corrected target face image with the original image comprises:
    根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
    利用所述人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image by using the face mask.
  7. 如权利要求1所述的图像处理方法,其中,所述获取原始图像中人脸关键点的第一位置信息,包括:The image processing method according to claim 1, wherein the obtaining the first location information of the face key point in the original image comprises:
    提取原始图像的图像特征;Extracting image features of the original image;
    根据所述图像特征确定原始图像中的人脸区域;Determining a face region in the original image according to the image feature;
    对人脸区域进行人脸关键点检测,以获取人脸关键点的第一位置信息。The face key point detection is performed on the face area to obtain the first position information of the face key point.
  8. 一种图像处理装置,其中,所述装置包括:An image processing apparatus, wherein the apparatus comprises:
    位置获取模块,用于获取原始图像中人脸关键点的第一位置信息;a location obtaining module, configured to acquire first location information of a face key point in the original image;
    对齐模块,用于根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并 将目标人脸图像与原始图像中的人脸对齐;An alignment module, configured to match a corresponding target face image from the preset face database according to the first location information, and align the target face image with the face in the original image;
    修正模块,用于基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;a correction module for correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
    融合模块,用于将修正后的目标人脸图像与原始图像融合。A fusion module for merging the corrected target face image with the original image.
  9. 如权利要求8所述的图像处理装置,其中,所述装置还包括:The image processing device of claim 8, wherein the device further comprises:
    构建模块,用于在获取原始图像中人脸关键点的第一位置信息之前,构建卷积神经网络;a building module, configured to construct a convolutional neural network before acquiring first position information of a face key point in the original image;
    样本获取模块,用于获取原始图像中人脸的多个角度的图像作为训练样本;a sample obtaining module, configured to acquire an image of a plurality of angles of a face in the original image as a training sample;
    训练模块,用于基于所述训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。The training module is configured to perform parameter training on the constructed convolutional neural network based on the training sample to adjust a parameter of a content loss function, an illumination loss function, and a smoothing loss function to obtain a trained convolutional neural network model.
  10. 如权利要求9所述的图像处理装置,其中,所述修正模块包括:The image processing device according to claim 9, wherein said correction module comprises:
    提取子模块,用于基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting a sub-module for extracting content features, illumination features, and smoothing features from a face region of the original image based on the trained convolutional neural network model;
    生成子模块,用于根据所述内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;Generating a submodule, configured to generate an adjustment parameter according to the content feature, the illumination feature, and the smooth feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
    修正子模块,用于根据所述调整参数对目标人脸图像进行修正。The correction submodule is configured to correct the target facial image according to the adjustment parameter.
  11. 如权利要求8所述的图像处理装置,其中,所述对齐模块包括:The image processing device of claim 8, wherein the alignment module comprises:
    获取子模块,用于获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining a sub-module, configured to acquire second location information of a face key point of the plurality of sample face images in the preset face database;
    匹配子模块,用于将第一位置信息与第二位置信息进行匹配;a matching submodule, configured to match the first location information with the second location information;
    选取子模块,用于从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sub-module for selecting a sample face image with the largest matching degree from the sample face image as the target face image;
    映射子模块,用于通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。a mapping sub-module for mapping the target face image to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  12. 如权利要求8所述的图像处理装置,其中,还包括:The image processing device according to claim 8, further comprising:
    边缘特征点获取模块,用于将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;The edge feature point acquiring module is configured to perform edge feature point detection on the face region in the original image after the target face image is aligned with the face in the original image, and the corrected target face image is merged with the original image. And obtaining third position information of the edge feature points;
    分割模块,用于根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The segmentation module is configured to perform image segmentation processing on the original image according to the third position information to remove the face region and use the remaining image region as the background image.
  13. 如权利要求12所述的图像处理装置,其中,所述融合模块用于:The image processing device according to claim 12, wherein said fusion module is configured to:
    根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
    利用所述人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image by using the face mask.
  14. 一种存储介质,其中,所述存储介质中存储有多条指令,所述指令适于由处理器加载以执行以下步骤:A storage medium, wherein the storage medium stores a plurality of instructions adapted to be loaded by a processor to perform the following steps:
    获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
    根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
    基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
    将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
  15. 一种电子设备,其中,包括处理器和存储器,所述处理器与所述存储器电性连接,所述存储器用于存储指令和数据;所述处理器用于执行以下步骤:An electronic device, comprising a processor and a memory, the processor being electrically connected to the memory, the memory for storing instructions and data; the processor for performing the following steps:
    获取原始图像中人脸关键点的第一位置信息;Obtaining first position information of a face key point in the original image;
    根据第一位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐;Matching the corresponding target face image from the preset face database according to the first location information, and aligning the target face image with the face in the original image;
    基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正;Correcting the target face image based on the trained convolutional neural network model and the face region in the original image;
    将修正后的目标人脸图像与原始图像融合。The corrected target face image is merged with the original image.
    .
  16. 如权利要求15所述的电子设备,其中,在获取原始图像中人脸关键点的第一位置信息之前,所述处理器用于执行以下步骤:The electronic device of claim 15, wherein the processor is configured to perform the following steps before acquiring the first location information of the face key in the original image:
    构建卷积神经网络;Construct a convolutional neural network;
    获取原始图像中人脸的多个角度的图像作为训练样本;Obtaining images of multiple angles of the face in the original image as training samples;
    基于所述训练样本对所构建的卷积神经网络进行参数训练,以调整内容损失函数、光照损失函数、及平滑损失函数的参数设置,得到训练后的卷积神经网络模型。Based on the training samples, parameter training is performed on the constructed convolutional neural network to adjust the parameter settings of the content loss function, the illumination loss function, and the smoothing loss function to obtain a trained convolutional neural network model.
  17. 如权利要求16所述的电子设备,其中,在基于训练好的卷积神经网络模型和原始图像中的人脸区域,对目标人脸图像进行修正时,所述处理器用于执行以下步骤:The electronic device of claim 16, wherein the processor is configured to perform the following steps when the target face image is corrected based on the trained convolutional neural network model and the face region in the original image:
    基于训练好的卷积神经网络模型从原始图像的人脸区域提取内容特征、光照特征、及平滑特征;Extracting content features, illumination features, and smoothing features from the face regions of the original image based on the trained convolutional neural network model;
    根据所述内容特征、光照特征、及平滑特征,在内容损失函数、光照损失函数、及平滑损失函数的约束下,生成调整参数;And generating an adjustment parameter according to the content feature, the illumination feature, and the smooth feature, under the constraint of the content loss function, the illumination loss function, and the smoothing loss function;
    根据所述调整参数对目标人脸图像进行修正。Correcting the target face image according to the adjustment parameter.
  18. 如权利要求15所述的电子设备,其中,在根据所述位置信息从预设人脸数据库中匹配对应的目标人脸图像,并将目标人脸图像与原始图像中的人脸对齐时,所述处理器用于执行以下步骤:The electronic device according to claim 15, wherein when the corresponding target face image is matched from the preset face database according to the position information, and the target face image is aligned with the face in the original image, The processor is used to perform the following steps:
    获取预设人脸数据库中多个样本人脸图像的人脸关键点的第二位置信息;Obtaining second location information of a face key point of the plurality of sample face images in the preset face database;
    将第一位置信息与第二位置信息进行匹配;Matching the first location information with the second location information;
    从样本人脸图像中选取匹配度最大的样本人脸图像,作为目标人脸图像;Selecting a sample face image with the largest matching degree from the sample face image as the target face image;
    通过仿射变换将目标人脸图像映射到原始图像的人脸区域,以使目标人脸图像与原始图像中的人脸对齐。The target face image is mapped to the face region of the original image by affine transformation to align the target face image with the face in the original image.
  19. 如权利要求15所述的电子设备,其中,在将目标人脸图像与原始图像中的人脸对齐之后,将修正后的目标人脸图像与原始图像融合之前,所述处理器用于执行以下步骤:The electronic device of claim 15, wherein the processor is configured to perform the following steps before merging the corrected target face image with the original image after aligning the target face image with the face in the original image :
    对原始图像中的人脸区域进行边缘特征点检测,并获取边缘特征点的第三位置信息;Performing edge feature point detection on the face region in the original image, and acquiring third position information of the edge feature point;
    根据第三位置信息对原始图像进行图像分割处理,以将人脸区域去除,将剩余图像区域作为背景图像。The original image is subjected to image segmentation processing according to the third position information to remove the face region, and the remaining image region is used as the background image.
  20. 如权利要求19所述的电子设备,其中,在将修正后的目标人脸图像与原始图像融合时,所述处理器用于执行以下步骤:The electronic device of claim 19, wherein the processor is configured to perform the following steps when the corrected target face image is merged with the original image:
    根据第三位置信息生成人脸掩膜;Generating a face mask according to the third location information;
    利用所述人脸掩膜,将修正后的目标人脸图像与原始图像中的背景图像融合。The corrected target face image is merged with the background image in the original image by using the face mask.
PCT/CN2018/115470 2017-12-28 2018-11-14 Method and apparatus for processing image, storage medium, and electronic device WO2019128508A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711466358.3 2017-12-28
CN201711466358.3A CN109978754A (en) 2017-12-28 2017-12-28 Image processing method, device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
WO2019128508A1 true WO2019128508A1 (en) 2019-07-04

Family

ID=67066431

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/115470 WO2019128508A1 (en) 2017-12-28 2018-11-14 Method and apparatus for processing image, storage medium, and electronic device

Country Status (2)

Country Link
CN (1) CN109978754A (en)
WO (1) WO2019128508A1 (en)

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110321849A (en) * 2019-07-05 2019-10-11 腾讯科技(深圳)有限公司 Image processing method, device and computer readable storage medium
CN110517185A (en) * 2019-07-23 2019-11-29 北京达佳互联信息技术有限公司 Image processing method, device, electronic equipment and storage medium
CN110517214A (en) * 2019-08-28 2019-11-29 北京百度网讯科技有限公司 Method and apparatus for generating image
CN110766645A (en) * 2019-10-24 2020-02-07 西安电子科技大学 Target person reproduction graph generation method based on person identification and segmentation
CN110826395A (en) * 2019-09-18 2020-02-21 平安科技(深圳)有限公司 Method and device for generating face rotation model, computer equipment and storage medium
CN111008935A (en) * 2019-11-01 2020-04-14 北京迈格威科技有限公司 Face image enhancement method, device, system and storage medium
CN111008927A (en) * 2019-08-07 2020-04-14 深圳华侨城文化旅游科技集团有限公司 Face replacement method, storage medium and terminal equipment
CN111028142A (en) * 2019-11-25 2020-04-17 泰康保险集团股份有限公司 Image processing method, apparatus and storage medium
CN111080747A (en) * 2019-12-26 2020-04-28 维沃移动通信有限公司 Face image processing method and electronic equipment
CN111160350A (en) * 2019-12-23 2020-05-15 Oppo广东移动通信有限公司 Portrait segmentation method, model training method, device, medium and electronic equipment
CN111179285A (en) * 2019-12-31 2020-05-19 珠海方图智能科技有限公司 Image processing method, system and storage medium
CN111402181A (en) * 2020-03-13 2020-07-10 北京奇艺世纪科技有限公司 Image fusion method and device and computer readable storage medium
CN111445437A (en) * 2020-02-25 2020-07-24 杭州火烧云科技有限公司 Method, system and equipment for processing image by skin processing model constructed based on convolutional neural network
CN111583159A (en) * 2020-05-29 2020-08-25 北京金山云网络技术有限公司 Image completion method and device and electronic equipment
CN111666974A (en) * 2020-04-29 2020-09-15 平安科技(深圳)有限公司 Image matching method and device, computer equipment and storage medium
CN111739046A (en) * 2020-06-19 2020-10-02 百度在线网络技术(北京)有限公司 Method, apparatus, device and medium for model update and image detection
CN111784611A (en) * 2020-07-03 2020-10-16 厦门美图之家科技有限公司 Portrait whitening method, portrait whitening device, electronic equipment and readable storage medium
CN111783608A (en) * 2020-06-24 2020-10-16 南京烽火星空通信发展有限公司 Face changing video detection method
CN111797797A (en) * 2020-07-13 2020-10-20 深圳大学 Face image processing method based on grid deformation optimization, terminal and storage medium
CN111815534A (en) * 2020-07-14 2020-10-23 厦门美图之家科技有限公司 Real-time skin makeup migration method, device, electronic device and readable storage medium
CN111833240A (en) * 2020-06-03 2020-10-27 北京百度网讯科技有限公司 Face image conversion method and device, electronic equipment and storage medium
CN111914630A (en) * 2020-06-19 2020-11-10 北京百度网讯科技有限公司 Method, apparatus, device and storage medium for generating training data for face recognition
CN112001262A (en) * 2020-07-28 2020-11-27 山东师范大学 Method for generating accessory capable of influencing face authentication
CN112001859A (en) * 2020-08-10 2020-11-27 深思考人工智能科技(上海)有限公司 Method and system for repairing face image
CN112102200A (en) * 2020-09-21 2020-12-18 腾讯科技(深圳)有限公司 Image completion model initialization method, training method and image completion method
CN112116565A (en) * 2020-09-03 2020-12-22 深圳大学 Countermeasure sample generation method and device for tampered copied image and storage medium
CN112241716A (en) * 2020-10-23 2021-01-19 北京百度网讯科技有限公司 Training sample generation method and device
CN112287772A (en) * 2020-10-10 2021-01-29 深圳市中达瑞和科技有限公司 Fingerprint trace detection method, fingerprint detection device and computer readable storage medium
CN112288665A (en) * 2020-09-30 2021-01-29 北京大米科技有限公司 Image fusion method and device, storage medium and electronic equipment
CN112330526A (en) * 2019-08-05 2021-02-05 深圳Tcl新技术有限公司 Training method of face conversion model, storage medium and terminal equipment
CN112419328A (en) * 2019-08-22 2021-02-26 北京市商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN112541477A (en) * 2020-12-24 2021-03-23 北京百度网讯科技有限公司 Expression package generation method and device, electronic equipment and storage medium
CN112766215A (en) * 2021-01-29 2021-05-07 北京字跳网络技术有限公司 Face fusion method and device, electronic equipment and storage medium
CN112991191A (en) * 2019-12-13 2021-06-18 北京金山云网络技术有限公司 Face image enhancement method and device and electronic equipment
CN113095134A (en) * 2021-03-08 2021-07-09 北京达佳互联信息技术有限公司 Facial expression extraction model generation method and device, and facial image generation method and device
CN113129208A (en) * 2019-12-31 2021-07-16 深圳云天励飞技术有限公司 Human face image generation method based on sketch and related product
CN113128277A (en) * 2019-12-31 2021-07-16 Tcl集团股份有限公司 Generation method of face key point detection model and related equipment
CN113128304A (en) * 2019-12-31 2021-07-16 深圳云天励飞技术有限公司 Image processing method and electronic equipment
CN113469041A (en) * 2021-06-30 2021-10-01 北京市商汤科技开发有限公司 Image processing method and device, computer equipment and storage medium
CN113569790A (en) * 2019-07-30 2021-10-29 北京市商汤科技开发有限公司 Image processing method and device, processor, electronic device and storage medium
CN113706428A (en) * 2021-07-02 2021-11-26 杭州海康威视数字技术股份有限公司 Image generation method and device
CN113705466A (en) * 2021-08-30 2021-11-26 浙江中正智能科技有限公司 Human face facial feature occlusion detection method used for occlusion scene, especially under high-imitation occlusion
WO2022100680A1 (en) * 2020-11-13 2022-05-19 北京字跳网络技术有限公司 Mixed-race face image generation method, mixed-race face image generation model training method and apparatus, and device
CN115908119A (en) * 2023-01-05 2023-04-04 广州佰锐网络科技有限公司 Facial image beauty processing method and system based on artificial intelligence
CN116758124A (en) * 2023-06-16 2023-09-15 北京代码空间科技有限公司 3D model correction method and terminal equipment
CN110517185B (en) * 2019-07-23 2024-02-09 北京达佳互联信息技术有限公司 Image processing method, device, electronic equipment and storage medium

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110458781B (en) * 2019-08-14 2022-07-19 北京百度网讯科技有限公司 Method and apparatus for processing image
CN110602403A (en) * 2019-09-23 2019-12-20 华为技术有限公司 Method for taking pictures under dark light and electronic equipment
CN110838084B (en) * 2019-09-24 2023-10-17 咪咕文化科技有限公司 Method and device for transferring style of image, electronic equipment and storage medium
CN110879983B (en) * 2019-11-18 2023-07-25 讯飞幻境(北京)科技有限公司 Face feature key point extraction method and face image synthesis method
CN111127378A (en) * 2019-12-23 2020-05-08 Oppo广东移动通信有限公司 Image processing method, image processing device, computer equipment and storage medium
CN111209823B (en) * 2019-12-30 2023-07-18 南京华图信息技术有限公司 Infrared face alignment method
CN111476709B (en) * 2020-04-09 2023-04-07 广州方硅信息技术有限公司 Face image processing method and device and electronic equipment
CN111598818B (en) * 2020-04-17 2023-04-28 北京百度网讯科技有限公司 Training method and device for face fusion model and electronic equipment
CN111553865B (en) * 2020-04-30 2023-08-22 深圳市商汤科技有限公司 Image restoration method and device, electronic equipment and storage medium
CN111553864B (en) * 2020-04-30 2023-11-28 深圳市商汤科技有限公司 Image restoration method and device, electronic equipment and storage medium
CN111741214A (en) * 2020-05-13 2020-10-02 北京迈格威科技有限公司 Image processing method and device and electronic equipment
CN111783519A (en) * 2020-05-15 2020-10-16 北京迈格威科技有限公司 Image processing method, image processing device, electronic equipment and storage medium
CN111709878B (en) * 2020-06-17 2023-06-23 北京百度网讯科技有限公司 Face super-resolution implementation method and device, electronic equipment and storage medium
CN111797753A (en) * 2020-06-29 2020-10-20 北京灵汐科技有限公司 Training method, device, equipment and medium of image driving model, and image generation method, device and medium
CN111754396B (en) * 2020-07-27 2024-01-09 腾讯科技(深圳)有限公司 Face image processing method, device, computer equipment and storage medium
CN112750071B (en) * 2020-11-04 2023-11-24 上海序言泽网络科技有限公司 User-defined expression making method and system
CN112381749A (en) * 2020-11-24 2021-02-19 维沃移动通信有限公司 Image processing method, image processing device and electronic equipment
CN113469903A (en) * 2021-06-11 2021-10-01 维沃移动通信有限公司 Image processing method and device, electronic equipment and readable storage medium
CN113658125A (en) * 2021-08-11 2021-11-16 全芯智造技术有限公司 Method, device and storage medium for evaluating layout hot spot
CN115082298A (en) * 2022-07-15 2022-09-20 北京百度网讯科技有限公司 Image generation method, image generation device, electronic device, and storage medium
CN115861042B (en) * 2023-02-08 2023-07-21 荣耀终端有限公司 Image processing method, electronic equipment and medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104657974A (en) * 2013-11-25 2015-05-27 腾讯科技(上海)有限公司 Image processing method and device
CN105741229A (en) * 2016-02-01 2016-07-06 成都通甲优博科技有限责任公司 Method for realizing quick fusion of face image
CN106412547A (en) * 2016-08-29 2017-02-15 厦门美图之家科技有限公司 Image white balance method and device based on convolutional neural network, and computing device
CN106815566A (en) * 2016-12-29 2017-06-09 天津中科智能识别产业技术研究院有限公司 A kind of face retrieval method based on multitask convolutional neural networks

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104123749A (en) * 2014-07-23 2014-10-29 邢小月 Picture processing method and system
TWI553565B (en) * 2014-09-22 2016-10-11 銘傳大學 Utilizing two-dimensional image to estimate its three-dimensional face angle method, and its database establishment of face replacement and face image replacement method
CN105095856B (en) * 2015-06-26 2019-03-22 上海交通大学 Face identification method is blocked based on mask
CN106023063A (en) * 2016-05-09 2016-10-12 西安北升信息科技有限公司 Video transplantation face changing method
CN107330408B (en) * 2017-06-30 2021-04-20 北京乐蜜科技有限责任公司 Video processing method and device, electronic equipment and storage medium
CN107507216B (en) * 2017-08-17 2020-06-09 北京觅己科技有限公司 Method and device for replacing local area in image and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104657974A (en) * 2013-11-25 2015-05-27 腾讯科技(上海)有限公司 Image processing method and device
CN105741229A (en) * 2016-02-01 2016-07-06 成都通甲优博科技有限责任公司 Method for realizing quick fusion of face image
CN106412547A (en) * 2016-08-29 2017-02-15 厦门美图之家科技有限公司 Image white balance method and device based on convolutional neural network, and computing device
CN106815566A (en) * 2016-12-29 2017-06-09 天津中科智能识别产业技术研究院有限公司 A kind of face retrieval method based on multitask convolutional neural networks

Cited By (68)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110321849A (en) * 2019-07-05 2019-10-11 腾讯科技(深圳)有限公司 Image processing method, device and computer readable storage medium
CN110321849B (en) * 2019-07-05 2023-12-22 腾讯科技(深圳)有限公司 Image data processing method, device and computer readable storage medium
CN110517185A (en) * 2019-07-23 2019-11-29 北京达佳互联信息技术有限公司 Image processing method, device, electronic equipment and storage medium
CN110517185B (en) * 2019-07-23 2024-02-09 北京达佳互联信息技术有限公司 Image processing method, device, electronic equipment and storage medium
CN113569790A (en) * 2019-07-30 2021-10-29 北京市商汤科技开发有限公司 Image processing method and device, processor, electronic device and storage medium
CN113569790B (en) * 2019-07-30 2022-07-29 北京市商汤科技开发有限公司 Image processing method and device, processor, electronic device and storage medium
CN112330526A (en) * 2019-08-05 2021-02-05 深圳Tcl新技术有限公司 Training method of face conversion model, storage medium and terminal equipment
CN112330526B (en) * 2019-08-05 2024-02-09 深圳Tcl新技术有限公司 Training method of face conversion model, storage medium and terminal equipment
CN111008927A (en) * 2019-08-07 2020-04-14 深圳华侨城文化旅游科技集团有限公司 Face replacement method, storage medium and terminal equipment
CN111008927B (en) * 2019-08-07 2023-10-31 深圳华侨城文化旅游科技集团有限公司 Face replacement method, storage medium and terminal equipment
CN112419328A (en) * 2019-08-22 2021-02-26 北京市商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN110517214A (en) * 2019-08-28 2019-11-29 北京百度网讯科技有限公司 Method and apparatus for generating image
CN110517214B (en) * 2019-08-28 2022-04-12 北京百度网讯科技有限公司 Method and apparatus for generating image
CN110826395B (en) * 2019-09-18 2023-10-31 平安科技(深圳)有限公司 Face rotation model generation method and device, computer equipment and storage medium
CN110826395A (en) * 2019-09-18 2020-02-21 平安科技(深圳)有限公司 Method and device for generating face rotation model, computer equipment and storage medium
CN110766645A (en) * 2019-10-24 2020-02-07 西安电子科技大学 Target person reproduction graph generation method based on person identification and segmentation
CN111008935A (en) * 2019-11-01 2020-04-14 北京迈格威科技有限公司 Face image enhancement method, device, system and storage medium
CN111008935B (en) * 2019-11-01 2023-12-12 北京迈格威科技有限公司 Face image enhancement method, device, system and storage medium
CN111028142B (en) * 2019-11-25 2024-02-13 泰康保险集团股份有限公司 Image processing method, device and storage medium
CN111028142A (en) * 2019-11-25 2020-04-17 泰康保险集团股份有限公司 Image processing method, apparatus and storage medium
CN112991191A (en) * 2019-12-13 2021-06-18 北京金山云网络技术有限公司 Face image enhancement method and device and electronic equipment
CN111160350A (en) * 2019-12-23 2020-05-15 Oppo广东移动通信有限公司 Portrait segmentation method, model training method, device, medium and electronic equipment
CN111160350B (en) * 2019-12-23 2023-05-16 Oppo广东移动通信有限公司 Portrait segmentation method, model training method, device, medium and electronic equipment
CN111080747A (en) * 2019-12-26 2020-04-28 维沃移动通信有限公司 Face image processing method and electronic equipment
CN111080747B (en) * 2019-12-26 2023-04-07 维沃移动通信有限公司 Face image processing method and electronic equipment
CN111179285A (en) * 2019-12-31 2020-05-19 珠海方图智能科技有限公司 Image processing method, system and storage medium
CN113129208A (en) * 2019-12-31 2021-07-16 深圳云天励飞技术有限公司 Human face image generation method based on sketch and related product
CN113128304B (en) * 2019-12-31 2024-01-05 深圳云天励飞技术有限公司 Image processing method and electronic equipment
CN113128304A (en) * 2019-12-31 2021-07-16 深圳云天励飞技术有限公司 Image processing method and electronic equipment
CN113128277A (en) * 2019-12-31 2021-07-16 Tcl集团股份有限公司 Generation method of face key point detection model and related equipment
CN111445437A (en) * 2020-02-25 2020-07-24 杭州火烧云科技有限公司 Method, system and equipment for processing image by skin processing model constructed based on convolutional neural network
CN111402181A (en) * 2020-03-13 2020-07-10 北京奇艺世纪科技有限公司 Image fusion method and device and computer readable storage medium
CN111666974A (en) * 2020-04-29 2020-09-15 平安科技(深圳)有限公司 Image matching method and device, computer equipment and storage medium
CN111583159B (en) * 2020-05-29 2024-01-05 北京金山云网络技术有限公司 Image complement method and device and electronic equipment
CN111583159A (en) * 2020-05-29 2020-08-25 北京金山云网络技术有限公司 Image completion method and device and electronic equipment
CN111833240A (en) * 2020-06-03 2020-10-27 北京百度网讯科技有限公司 Face image conversion method and device, electronic equipment and storage medium
CN111833240B (en) * 2020-06-03 2023-07-25 北京百度网讯科技有限公司 Face image conversion method and device, electronic equipment and storage medium
CN111739046A (en) * 2020-06-19 2020-10-02 百度在线网络技术(北京)有限公司 Method, apparatus, device and medium for model update and image detection
CN111914630A (en) * 2020-06-19 2020-11-10 北京百度网讯科技有限公司 Method, apparatus, device and storage medium for generating training data for face recognition
CN111783608A (en) * 2020-06-24 2020-10-16 南京烽火星空通信发展有限公司 Face changing video detection method
CN111784611A (en) * 2020-07-03 2020-10-16 厦门美图之家科技有限公司 Portrait whitening method, portrait whitening device, electronic equipment and readable storage medium
CN111784611B (en) * 2020-07-03 2023-11-03 厦门美图之家科技有限公司 Portrait whitening method, device, electronic equipment and readable storage medium
CN111797797B (en) * 2020-07-13 2023-09-15 深圳大学 Face image processing method, terminal and storage medium based on grid deformation optimization
CN111797797A (en) * 2020-07-13 2020-10-20 深圳大学 Face image processing method based on grid deformation optimization, terminal and storage medium
CN111815534A (en) * 2020-07-14 2020-10-23 厦门美图之家科技有限公司 Real-time skin makeup migration method, device, electronic device and readable storage medium
CN111815534B (en) * 2020-07-14 2023-12-19 厦门美图之家科技有限公司 Real-time skin makeup migration method, device, electronic equipment and readable storage medium
CN112001262A (en) * 2020-07-28 2020-11-27 山东师范大学 Method for generating accessory capable of influencing face authentication
CN112001262B (en) * 2020-07-28 2022-07-29 山东师范大学 Method for generating accessory capable of influencing face authentication
CN112001859A (en) * 2020-08-10 2020-11-27 深思考人工智能科技(上海)有限公司 Method and system for repairing face image
CN112116565A (en) * 2020-09-03 2020-12-22 深圳大学 Countermeasure sample generation method and device for tampered copied image and storage medium
CN112116565B (en) * 2020-09-03 2023-12-05 深圳大学 Method, apparatus and storage medium for generating countersamples for falsifying a flip image
CN112102200A (en) * 2020-09-21 2020-12-18 腾讯科技(深圳)有限公司 Image completion model initialization method, training method and image completion method
CN112288665A (en) * 2020-09-30 2021-01-29 北京大米科技有限公司 Image fusion method and device, storage medium and electronic equipment
CN112287772A (en) * 2020-10-10 2021-01-29 深圳市中达瑞和科技有限公司 Fingerprint trace detection method, fingerprint detection device and computer readable storage medium
CN112241716A (en) * 2020-10-23 2021-01-19 北京百度网讯科技有限公司 Training sample generation method and device
CN112241716B (en) * 2020-10-23 2023-06-20 北京百度网讯科技有限公司 Training sample generation method and device
WO2022100680A1 (en) * 2020-11-13 2022-05-19 北京字跳网络技术有限公司 Mixed-race face image generation method, mixed-race face image generation model training method and apparatus, and device
US11875601B2 (en) 2020-12-24 2024-01-16 Beijing Baidu Netcom Science and Technology Co., Ltd Meme generation method, electronic device and storage medium
CN112541477A (en) * 2020-12-24 2021-03-23 北京百度网讯科技有限公司 Expression package generation method and device, electronic equipment and storage medium
CN112766215A (en) * 2021-01-29 2021-05-07 北京字跳网络技术有限公司 Face fusion method and device, electronic equipment and storage medium
CN113095134A (en) * 2021-03-08 2021-07-09 北京达佳互联信息技术有限公司 Facial expression extraction model generation method and device, and facial image generation method and device
CN113469041A (en) * 2021-06-30 2021-10-01 北京市商汤科技开发有限公司 Image processing method and device, computer equipment and storage medium
CN113706428B (en) * 2021-07-02 2024-01-05 杭州海康威视数字技术股份有限公司 Image generation method and device
CN113706428A (en) * 2021-07-02 2021-11-26 杭州海康威视数字技术股份有限公司 Image generation method and device
CN113705466A (en) * 2021-08-30 2021-11-26 浙江中正智能科技有限公司 Human face facial feature occlusion detection method used for occlusion scene, especially under high-imitation occlusion
CN113705466B (en) * 2021-08-30 2024-02-09 浙江中正智能科技有限公司 Face five sense organ shielding detection method for shielding scene, especially under high imitation shielding
CN115908119A (en) * 2023-01-05 2023-04-04 广州佰锐网络科技有限公司 Facial image beauty processing method and system based on artificial intelligence
CN116758124A (en) * 2023-06-16 2023-09-15 北京代码空间科技有限公司 3D model correction method and terminal equipment

Also Published As

Publication number Publication date
CN109978754A (en) 2019-07-05

Similar Documents

Publication Publication Date Title
WO2019128508A1 (en) Method and apparatus for processing image, storage medium, and electronic device
WO2019128507A1 (en) Image processing method and apparatus, storage medium and electronic device
CN108229369B (en) Image shooting method and device, storage medium and electronic equipment
WO2019137131A1 (en) Image processing method, apparatus, storage medium, and electronic device
CN109684924B (en) Face living body detection method and device
CN110532984B (en) Key point detection method, gesture recognition method, device and system
US11163978B2 (en) Method and device for face image processing, storage medium, and electronic device
KR102174595B1 (en) System and method for identifying faces in unconstrained media
WO2021036059A1 (en) Image conversion model training method, heterogeneous face recognition method, device and apparatus
CN109952594B (en) Image processing method, device, terminal and storage medium
AU2012219026B2 (en) Image quality assessment
CN106056064B (en) A kind of face identification method and face identification device
CN108198130B (en) Image processing method, image processing device, storage medium and electronic equipment
US9443325B2 (en) Image processing apparatus, image processing method, and computer program
JP4597391B2 (en) Facial region detection apparatus and method, and computer-readable recording medium
CN111989689A (en) Method for identifying objects within an image and mobile device for performing the method
JP2012530994A (en) Method and apparatus for half-face detection
CN109937434B (en) Image processing method, device, terminal and storage medium
US8774519B2 (en) Landmark detection in digital images
CN111382648A (en) Method, device and equipment for detecting dynamic facial expression and storage medium
CN111898571A (en) Action recognition system and method
CN109087240B (en) Image processing method, image processing apparatus, and storage medium
CN112183200B (en) Eye movement tracking method and system based on video image
WO2022063321A1 (en) Image processing method and apparatus, device and storage medium
CN112101275B (en) Human face detection method, device, equipment and medium for multi-view camera

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18893459

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18893459

Country of ref document: EP

Kind code of ref document: A1