WO2022258013A1 - Image processing method and apparatus, electronic device and readable storage medium - Google Patents
Image processing method and apparatus, electronic device and readable storage medium Download PDFInfo
- Publication number
- WO2022258013A1 WO2022258013A1 PCT/CN2022/097859 CN2022097859W WO2022258013A1 WO 2022258013 A1 WO2022258013 A1 WO 2022258013A1 CN 2022097859 W CN2022097859 W CN 2022097859W WO 2022258013 A1 WO2022258013 A1 WO 2022258013A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- sub
- images
- face
- target
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 133
- 238000000034 method Methods 0.000 claims abstract description 45
- 230000036548 skin texture Effects 0.000 claims abstract description 21
- 230000001815 facial effect Effects 0.000 claims description 32
- 230000009466 transformation Effects 0.000 claims description 28
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 21
- 230000004927 fusion Effects 0.000 claims description 14
- 238000004891 communication Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000007667 floating Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims 1
- 230000008569 process Effects 0.000 description 15
- 238000003384 imaging method Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 239000000284 extract Substances 0.000 description 5
- 238000009499 grossing Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000013508 migration Methods 0.000 description 4
- 230000005012 migration Effects 0.000 description 4
- 230000007704 transition Effects 0.000 description 4
- 208000002874 Acne Vulgaris Diseases 0.000 description 3
- 206010000496 acne Diseases 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 3
- 229940021231 clearskin Drugs 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000037303 wrinkles Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 208000003351 Melanosis Diseases 0.000 description 1
- 241000699670 Mus sp. Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000011982 device technology Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- the embodiments of the present application relate to the field of image processing, and in particular, to an image processing method, device, electronic equipment, and readable storage medium.
- the camera takes photos of portraits under different lighting conditions, and is affected by various degradation problems such as noise, motion blur, highlights, post-beauty denoising, etc., and the imaged faces lack good skin quality and details.
- the internal blemishes such as acne marks
- wrinkles and excessive noise will greatly affect the skin feel and aesthetics of the human face after imaging.
- the purpose of the embodiment of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can solve the problem of poor skin quality in human face imaging.
- the embodiment of the present application provides an image processing method, the method comprising: acquiring a target mask image of the face area in the first image; based on a binarized face mask image of the target mask image, acquiring A reference face image matched with the target mask image in the reference face image collection; image processing is performed on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images; The image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image to generate a second image corresponding to the first image; wherein, the set of reference face images includes a plurality of processed human faces Face mask image; the face skin quality value of the target area is greater than the face skin quality value of the area corresponding to the target area in the target mask image.
- the embodiment of the present application also provides an image processing device, the device including: an acquisition module and an image processing module; an acquisition module, configured to acquire a target mask image of a face area in the first image; an acquisition module , is also used for the binarized face mask image based on the target mask image, and obtains the reference face image matching the target mask image in the reference face image set; the acquisition module is also used for the reference face image and the target mask image for image processing to obtain N reference face sub-images and N target mask sub-images; the image processing module is used to obtain the image of the target area of the reference face sub-image obtained by the acquisition module and the image obtained by the acquisition module The images of the corresponding areas of the target mask sub-image are fused to generate a second image corresponding to the first image; wherein, the reference face image set includes a plurality of face mask images processed through skin texture; the face of the target area The skin quality value is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
- the embodiment of the present application provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and operable on the processor.
- the program or instruction is executed by the processor, the The steps of the image processing method as described in the first aspect.
- an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .
- the embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, so as to implement the first aspect the method described.
- the target mask image of the human face area in the first image is acquired, and based on the binarized human face mask image of the target mask image, Obtain the reference face image matching the target mask image in the reference face image set.
- image processing is carried out for the reference face image and the target mask image, and N reference face sub-images and N target mask sub-images are obtained, and the image of the target area of the reference face sub-image and the image of the target mask sub-image
- the images of the corresponding areas are fused to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain a second image with better skin quality, which greatly improves the quality of the facial skin after imaging.
- FIG. 1 is a schematic diagram of an interface to which an image processing method provided in an embodiment of the present application is applied;
- Fig. 2 is a schematic structural diagram of an image pyramid provided by an embodiment of the present application.
- FIG. 3 is a schematic structural diagram of an image processing device provided in an embodiment of the present application.
- FIG. 4 is one of the structural schematic diagrams of an electronic device provided in an embodiment of the present application.
- FIG. 5 is a second schematic structural diagram of an electronic device provided by an embodiment of the present application.
- the image processing method provided in the embodiment of the present application may be applied to a scene of beautifying an image including a human face.
- the human face and skin with blemishes in the captured image are combined with the image with better skin quality Image fusion can effectively remove the poor texture and transition of the face of the portrait, making the skin of the face after imaging delicate and clear, and greatly improving the skin quality of the face after imaging.
- an image processing method provided by the embodiment of the present application may include the following steps 201 to 204:
- Step 201 The image processing apparatus acquires a target mask image of a face area in a first image.
- the above-mentioned first image may be an image captured by the electronic device, or may be an image stored in the electronic device read by the electronic device.
- the image processing apparatus acquires an image of the face area of the first image in a red green blue (red green blue, RGB) color space. And in the obtained face area, a mask image of the face area, that is, the above-mentioned target mask image, is generated through a face parsing algorithm.
- RGB red green blue
- target mask image can be understood as, after the contour of the human face contained in the first image is obtained, all images outside the range of the contour of the human face are covered, for example, set to the same color, so that the image processing device can only recognize the image of the face area.
- the purpose of acquiring only the image of the face area of the first image is to eliminate the interference of images of other areas, so as to facilitate the optimization of the image of the face area.
- Step 202 the image processing device acquires a reference face image matching the target mask image in the reference face image set based on the binarized face mask image of the target mask image.
- image binarization is the process of setting the gray value of the pixel on the image to 0 or 255, that is, the process of presenting an obvious black-and-white effect to the entire image.
- the above binarized face mask image can be understood as a black and white image of a face image. That is, if the facial features areas of the above target mask image are all set to black, and all non-facial features areas are all set to white.
- the above binarized face image is an image including only facial features, that is, the binarized face image is an image including eyes, nose, eyebrows, mouth and other areas of the face area.
- the binarized face mask image is mainly used for matching with the face images in the reference face image collection.
- the matching algorithm used in the above-mentioned step 202 is an E-log-Hua template matching algorithm.
- Step 203 the image processing device performs image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images.
- the above-mentioned image processing for the above-mentioned reference face image and the above-mentioned target mask image may include performing formatting processing on the reference face image, and then performing degradation processing and processing on the formatted reference face image.
- Scaling operation to generate N reference face sub-images wherein, the scaling ratio between every two adjacent reference face sub-images is the same, and the image with a smaller resolution is an image obtained after degrading the image with a larger resolution.
- the processing method of obtaining N target mask sub-images after processing the target mask image is similar to the above-mentioned processing method for the reference face image, and the target image can be processed based on the above-mentioned processing method for the reference face image.
- the mask image is processed to obtain N target mask sub-images.
- N reference face sub-images there is a one-to-one correspondence between the N reference face sub-images and the N target mask sub-images.
- N for example, taking the above-mentioned N as 5 as an example, there is a corresponding relationship between the five reference face sub-images numbered 0-4 and the five target mask sub-images numbered 0-4. Wherein, as the number increases, the resolution of the image gradually decreases.
- Step 204 The image processing device fuses the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image to generate a second image corresponding to the first image.
- the aforementioned collection of reference face images includes a plurality of face mask images that have undergone skin texture processing.
- the face skin quality value of the above target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
- the image processing device finds a matching reference face image from the collection of reference face images based on the above binarized face mask image. Afterwards, the image of the poor skin quality area in the target mask image may be fused with the image with better skin quality in the corresponding area in the reference face image, so as to obtain a second image with better skin quality.
- the image processing device may generate the first image pyramid according to the reference face image, and generate the second image pyramid according to the target mask image, and then, based on the first image pyramid and the second image pyramid, The target mask image is processed, and the skin quality of the reference face image is transferred to obtain a face image with better skin quality in the first image.
- the target mask image of the face area in the first image is obtained, and the reference face image is obtained based on the binarized face mask image of the target mask image A set of reference face images that match the target mask image.
- image processing is carried out for the reference face image and the target mask image, and N reference face sub-images and N target mask sub-images are obtained, and the image of the target area of the reference face sub-image and the image of the target mask sub-image.
- the images of the corresponding areas are fused to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain a second image with better skin quality, which greatly improves the quality of the facial skin after imaging.
- the image processing device may realize the migration of the image with better skin quality in the reference face image to the region with poorer skin quality in the first image based on the image pyramid.
- the image processing method provided in the embodiment of the present application may further include the following steps 201a1 to 202a3:
- Step 202a1 the image processing device acquires N face images that have undergone skin texture processing, where N is a positive integer.
- N human face images with better skin quality that have undergone image-level professional image processing (including skin color adjustment, freckle and acne removal, skin smoothing, enhancement, etc.) can be obtained, and the set includes N human face images.
- step 202a2 the image processing device extracts facial features information of each of the above N facial images, and constructs a binarized mask according to the facial features information.
- a face image corresponds to a binarization mask.
- the facial features information includes various information about the facial features in the face image, for example, the area where the facial features are located, specific location coordinates, and the like.
- the image processing device uses the face analysis model to decompose the pixel-by-pixel segmentation mask image of each part of the face area of each face image, and only An image of the facial features of a human face. Afterwards, the image processing device constructs a binarized mask image of the mask image based on the mask image. Each face object includes a corresponding binarized mask image.
- step 202a3 the image processing device generates the set of reference face images based on the above N face images that have undergone skin texture processing and a binarized mask corresponding to each face image.
- the aforementioned set of reference human face images includes N human face images that have undergone skin texture processing, and N binary mask images corresponding to the human face images.
- the above-mentioned human face image processed with skin quality is mainly used for image fusion with an image of a region with poor skin quality in the target mask image.
- the main user of the above-mentioned binary mask image adjusts the position of the facial features of the above-mentioned reference face image to make it closer to the position of the facial features in the target mask image, so that the appearance of the person in the reference face image can be adjusted as much as possible after adjustment. It is possible to keep the appearance of the face in the target mask image consistent, so as to facilitate the subsequent migration of skin quality.
- the image processing device can construct a set of reference face images based on the reference face image processed by the skin texture and the corresponding binarized mask image, so that after the image processing device obtains an image that needs to be processed, it can based on This collection performs processing on images.
- the image processing device after the image processing device obtains the aforementioned set of reference face images, it can process the acquired first image based on the set, and the specific processing process needs to be completed by using an image pyramid.
- step 203 may also include the following steps 203a1 and 203a2:
- step 203a1 the image processing device constructs a first image pyramid based on the aforementioned reference face image.
- step 203a2 the image processing device constructs a second image pyramid based on the target mask image.
- the first image pyramid includes the N reference face sub-images
- the second image pyramid includes the N target mask sub-images
- the image pyramid is a kind of multi-scale representation of images, and it is an effective but conceptually simple structure to explain images at multiple resolutions.
- the pyramid of an image is a series of images arranged in a pyramid shape with gradually reduced resolution and derived from the same original image. It is obtained by down-sampling in steps, and the sampling is stopped until a certain termination condition is reached.
- the image processing device needs to construct an image pyramid of each face image in the reference face image set.
- the first image pyramid may be a Laplacian pyramid.
- the bottom layer (level 0) of the first image pyramid may be the above-mentioned reference face image, or may be a formatted image of the reference face image.
- the image size of the 0th layer of the pyramid constructed by each face image in the reference face image set is the same, and the scaling ratio between layers is also the same.
- FIG. 2 it is a schematic structural diagram of an image pyramid, which includes five layers (L0 to L4), each layer contains an image, and the images between layers are scaled according to a preset ratio Zoomed out.
- the image processing apparatus may process the target mask image based on the reference face image and feature points of the target mask image, so as to obtain the second image.
- step 204 may include the following steps 204a1 to 204a4:
- step 204a1 the image processing device extracts the first feature point of the reference face image and the second feature point of the target mask image.
- Step 204a2 the image processing device obtains the binarized face mask image of each of the N reference face sub-images based on the first feature point, and the binary face mask image contained in each of the above images Vertex coordinates of each triangle block in the M triangle blocks.
- Step 204a3 The image processing device obtains the vertex coordinates of each of the K triangle blocks contained in the binarized face mask image of each of the N target mask sub-images based on the second feature point.
- step 204a4 the image processing device fuses the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image based on the vertex coordinates.
- the image processing device successfully After constructing the above-mentioned first image pyramid and the second image pyramid, the above-mentioned first image pyramid and the second image pyramid can be triangulated,
- step 204a1 to step 204a4 may include the following steps from step 204b1 to step 204b3:
- Step 204b1 The image processing device extracts the first feature point of the reference face image, and performs triangulation on the reference face image based on the first feature point to obtain M triangular blocks.
- one first feature point corresponds to one triangular block, and the circumscribed circle of each triangular block does not include other first feature points, and M is a positive integer.
- the image processing device may extract a plurality of first feature points from the above-mentioned reference face image, and then, the image processing device performs triangulation (also referred to as image triangulation) based on each feature point, so that the generated The circumscribed circle of each triangular block does not include any other first feature points.
- triangulation also referred to as image triangulation
- image triangulation can be understood as dividing the image into several triangular fragments, each of which is a triangle, and any two triangles on the image either do not intersect, or intersect exactly on a common side (two cannot intersect at the same time). strip or two or more sides).
- Step 204b2 the image processing device acquires the binarized face mask image of each layer image of the first image pyramid.
- Step 204b3 The image processing device determines the binarization of each layer image based on the binarized face mask image corresponding to each layer image of the first image pyramid, and the scaling ratio between the first target layer and the second target layer Vertex coordinates of each triangle block in the M triangle blocks contained in the face mask image.
- the first target layer and the second target layer are two adjacent layers of the above-mentioned first image pyramid.
- the image processing device constructs the first image pyramid of the above-mentioned reference face image, it also needs to generate a binary face mask image corresponding to each layer image based on the images contained in each layer of the first image pyramid.
- the image processing device can determine the M triangular blocks of each layer image of the first image pyramid based on the vertex coordinates of each of the M triangular blocks Vertex coordinates of each triangle block in .
- each triangular block of the reference face image can find a corresponding triangular block in each layer image.
- the vertex coordinates of each triangular block of each layer image may be recalculated based on the scaling ratio.
- the image processing device obtains the vertex coordinates of each triangular block of each layer image of the first image pyramid, it can adjust the image of each layer based on the triangular block of each layer image, so that the appearance of the characters contained in it is closer to the target The appearance of the character contained in the mask image.
- the image processing device may construct the second image pyramid of the target mask image according to this method.
- steps 204a1 to 204a4 may specifically include the following steps 204c1 to 204c4:
- step 204c1 the image processing device constructs a second image pyramid based on the target mask image.
- the 0th layer of the second image pyramid is constructed from the target mask image or the image obtained after the target mask image is formatted.
- each layer image of the first image pyramid is the same as that of each layer image of the second image pyramid. And the size of the images used to construct the first image pyramid and the second image pyramid is also the same.
- the size of the image may be represented by resolution or inches, which is not limited in the embodiment of the present application.
- Step 204c2 The image processing device extracts the second feature points of the target mask image, and performs triangulation on the target mask image based on the second feature points to obtain K triangular blocks.
- one second feature point corresponds to one triangular block, and the circumscribed circle of each triangular block does not include other second feature points, and K is a positive integer.
- step 204c3 the image processing device acquires the binarized face mask image of each layer image of the second image pyramid.
- Step 204c4 the image processing device determines the image of each layer of the second image pyramid based on the binarized face mask image of the image of each layer of the second image pyramid and the scaling ratio between the third target layer and the fourth target layer The vertex coordinates of each triangular block in the K triangular blocks contained in the binarized face mask image of .
- the third target layer and the fourth target layer are adjacent layers of the second image pyramid.
- steps 204c1 to 204c4 are similar to steps 204b1 to 204b3, for comparison with the explanations of steps 204c1 to 204c4, you can refer to the above explanations of steps 204b1 to 204b3.
- steps 204a1 to 204a4 you can refer to the description of the processing of the first image pyramid and the second image pyramid, in order to prevent repetition, in This will not be repeated here.
- the image processing device acquires the image pyramid of the reference face image and the image pyramid of the target mask image, it can migrate the image of the region with better skin quality in the reference face image to the target mask image based on the image pyramid. Images in areas of poor quality.
- the image processing apparatus may improve the face skin quality of the face area in the first image based on the above N reference face sub-images and N target mask sub-images.
- step 204a4 may include the following steps 204d1 to 204d3:
- Step 204d1 the image processing device performs affine transformation on the vertex coordinates of the M triangular blocks based on the vertex coordinates of the K triangular blocks.
- Step 204d2 the image processing device combines the first target area image of the first reference face sub-image in the N reference face sub-images after affine transformation with the first target mask sub-image in the above N target mask sub-images The image of the second target area is fused to obtain N processed target mask sub-images.
- the image processing device reconstructs the above N processed target mask sub-images to generate the above second image.
- the above-mentioned first reference face sub-image is: any one of the above-mentioned N reference face sub-images; the above-mentioned first target mask sub-image is corresponding to the above-mentioned first reference face sub-image among the above-mentioned N target mask sub-images
- the target mask sub-image; the first target area image is the image of the first target area of the first reference face sub-image, and the first target area corresponds to the second target area of the first target mask sub-image image area.
- the aforementioned steps 204d1 to 204d3 may include the following steps 204e1 to step 204e3:
- Step 204e1 The image processing device performs affine transformation on the vertex coordinates of the M triangular blocks in each layer of the image in the first image pyramid based on the vertex coordinates of the K triangular blocks in each layer of the image in the second image pyramid.
- the image processing device needs to process each layer of the image in the second image pyramid. That is, the mechanical energy affine transformation of each layer of image.
- affine transformation also known as affine mapping
- affine mapping means that in geometry, a vector space is transformed into another vector space by performing a linear transformation followed by a translation.
- Affine transformation is geometrically defined as an affine transformation between two vector spaces or an affine mapping consists of a non-singular linear transformation (transformation using a function) followed by a translation transformation.
- the image processing device may perform affine transformation on each layer image of the first image pyramid based on the obtained transformation matrix.
- Step 204e2 the image processing device performs image fusion on the image of the first target area of the fifth target layer of the first image pyramid after the affine transformation, and the image of the second target area of the sixth target layer of the second image pyramid , to obtain the processed second image pyramid.
- each layer of the first image pyramid corresponds to each layer of the second image pyramid
- layer 0 of the first image pyramid corresponds to layer 0 of the second image pyramid
- the first image The nth level of the pyramid corresponds to the nth level of the second image pyramid. Therefore, the image processing device may perform image fusion on the image of the first target area of the first image pyramid and the image of the corresponding area (ie, the second target area) of the second image pyramid to obtain the processed second image pyramid.
- step 204e3 the image processing device reconstructs the processed second image pyramid to generate the second image.
- the above-mentioned fifth target layer is: any layer of the above-mentioned first image pyramid; the above-mentioned sixth target layer is the layer corresponding to the above-mentioned fifth target layer in the above-mentioned second image pyramid; The image of the first target area of the five target layers, and the image area corresponding to the first target area and the second target area of the sixth target layer; the number of layers of the first image pyramid and the second image pyramid is the same, And the scaling ratio of each layer is also the same.
- the sixth target layer is the layer corresponding to the fifth target layer in the second image pyramid. It can be understood that the number of layers of the sixth target layer in the second image pyramid is the same as that of the fifth target layer in the first The number of layers in the image pyramid is the same, that is, the same layer.
- the image processing device reconstructs the Laplacian pyramid after the texture migration of the reference image, that is, the above-mentioned second image pyramid, to obtain a final result.
- the image processing device needs to construct a Gaussian pyramid before constructing the Laplacian pyramid of the target mask image or the reference face image.
- the original image is used as the bottom image G0 (the 0th layer of the Gaussian pyramid), and it is convolved with a Gaussian kernel (5*5), and then the convolved image is down-sampled (removing even rows and columns) to get The previous layer image G1.
- this image is used as an input, and the convolution and downsampling operations are repeated to obtain a higher-level image, and iterated multiple times to form a pyramid-shaped image data structure, that is, a Gaussian pyramid.
- Laplacian Pyramid Laplacian Pyramid
- the image processing device can restore the corresponding Gaussian pyramid from the top layer of the Laplacian pyramid after image fusion, and finally obtain the original image G0. It is the method of using interpolation from the highest level.
- the image processing device may first perform skin smoothing on each layer of the image in the second image pyramid.
- the image processing method provided in the embodiment of the present application may also include the following step 203b:
- step 203b the image processing device performs guided filtering and microdermabrasion on each layer of the image in the second image pyramid according to the preset radius and the relative precision of the floating point.
- the image processing device may also perform guided filter skin smoothing on each of the above N target mask sub-images according to a preset radius and a floating-point relative precision.
- the image processing device can set a reasonable radius radius and eps floating-point relative precision for each layer of the Laplacian pyramid, and perform guided filtering for dermabrasion to alleviate problems such as acne marks, wrinkles, and excessive unevenness on the face.
- the image processing device optimizes the face and skin quality in the first image according to the first image pyramid constructed based on the reference face image and the second image pyramid constructed based on the target mask image, and obtains the second image pyramid with better skin quality. Two images.
- the image processing method provided in the embodiment of the present application through the face and skin quality migration method based on multi-layer image pyramid fusion, performs image fusion on the human face skin with blemishes in the captured image and the image with better skin quality, which can effectively Remove the poor texture and transition of the face of the portrait, making the skin of the face after imaging delicate and clear, and greatly improving the skin quality of the face after imaging.
- the image processing method provided in the embodiment of the present application may be executed by an image processing device, or a control module in the image processing device for executing the image processing method.
- the image processing device executed by the image processing device is taken as an example to describe the image processing device provided in the embodiment of the present application.
- FIG. 3 is a schematic diagram of a possible structure of an image processing device provided by an embodiment of the present application.
- the image processing device 300 includes: an acquisition module 301 and an image processing module 302; The target mask image of the face area in an image; the acquisition module 301 is also used to obtain the target mask image in the reference face image set that matches the target mask image based on the binarized face mask image of the target mask image.
- Reference face image is also used for image processing for reference face image and target mask image, obtains N reference face sub-images and N target mask sub-images;
- Image processing module 302 is used for The image of the target area of the reference face sub-image acquired by the acquisition module 301 is fused with the image of the corresponding area of the target mask sub-image acquired by the acquisition module 301 to generate a second image corresponding to the first image; wherein, the reference face image
- the collection includes a plurality of face mask images processed by skin texture; the face skin quality value of the target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
- the device 300 also includes: a generation module 303; an acquisition module 301, which is also used to acquire N face images processed by skin texture, where N is a positive integer; and an acquisition module 301, which is also used to extract N facial images
- a generation processing module 303 is used to obtain the N processed skins based on the acquisition module 301
- the qualitatively processed face images, and the binarized mask image corresponding to each face image acquired by the acquisition module 301 generate a set of reference face images.
- the device 300 also includes: a construction module 304; the construction module 304 is used to construct a first image pyramid based on the reference face image, and the first image pyramid includes N reference face sub-images; the construction module 304 is also used to Based on the target mask image, a second image pyramid is constructed, and the second image pyramid includes N target mask sub-images.
- the obtaining module 301 is also used to extract the first feature point of the reference face image and the second feature point of the target mask image; the obtaining module 301 is also used to obtain N reference face sub-points based on the first feature point The binarized face mask image of each image of the image, and the vertex coordinates of each triangle block in the M triangle blocks contained in the binarized face mask image of each image; the acquisition module 301 is also used for Acquire the vertex coordinates of each triangle block in the K triangle blocks contained in the binarized face mask image of each image of N target mask sub-images based on the second feature point; the image processing module 302 is specifically used based on Vertex coordinates, the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image.
- the device 300 further includes: a transformation module 305; the transformation module 305 is configured to perform affine transformation on the vertex coordinates of the M triangle blocks acquired by the acquisition module 301 based on the vertex coordinates of the K triangle blocks acquired by the acquisition module 301
- the image processing module 302 is specifically used to combine the first target area image of the first reference face sub-image in the N reference face sub-images after affine transformation with the first target mask in the N target mask sub-images Image fusion is performed on the second target area image of the sub-images to obtain N processed target mask sub-images; the image processing module 302 is also specifically used to reconstruct the N processed target mask sub-images to generate the first N target mask sub-images.
- the first reference human face sub-image is: any one of N reference human face sub-images;
- the first target mask sub-image is the target corresponding to the first reference human face sub-image in N target mask sub-images
- the first target area image is an image of the first target area of the first reference face sub-image, and the first target area is an image area corresponding to the second target area of the first target mask sub-image.
- the image processing module 302 is further configured to perform guided filter skin smoothing on each layer of the image of the second image pyramid constructed by the construction module 304 according to the preset radius and relative precision of the floating point.
- the image processing apparatus in the embodiment of the present application may be a device, or may be a component, an integrated circuit, or a chip in a terminal.
- the device may be a mobile electronic device or a non-mobile electronic device.
- the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant).
- non-mobile electronic devices can be servers, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., this application Examples are not specifically limited.
- Network Attached Storage NAS
- personal computer personal computer, PC
- television television
- teller machine or self-service machine etc.
- the image processing device in the embodiment of the present application may be a device with an operating system.
- the operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.
- the image processing device provided in the embodiment of the present application can implement various processes implemented by the image processing device in the method embodiments shown in FIG. 1 to FIG. 2 , and details are not repeated here to avoid repetition.
- the image processing device provided in the embodiment of the present application, through the image processing method, performs image fusion of the human face skin with defects in the captured image and the image with better skin quality, which can effectively remove the bad texture and transition of the human face , making the skin of the human face after imaging delicate and clear, and greatly improving the skin quality of the human face after imaging.
- the embodiment of the present application further provides an electronic device M00, including a processor M01, a memory M02, and programs or instructions stored in the memory M02 and operable on the processor M01,
- an electronic device M00 including a processor M01, a memory M02, and programs or instructions stored in the memory M02 and operable on the processor M01,
- the program or instruction is executed by the processor M01, each process of the above-mentioned image processing method embodiment can be achieved, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.
- the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
- FIG. 5 is a schematic diagram of a hardware structure of an electronic device implementing various embodiments of the present application.
- the electronic device 100 includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110, etc. part.
- the electronic device 100 can also include a power supply (such as a battery) for supplying power to various components, and the power supply can be logically connected to the processor 110 through the power management system, so that the management of charging, discharging, and function can be realized through the power management system. Consumption management and other functions.
- a power supply such as a battery
- the structure of the electronic device shown in FIG. 5 does not constitute a limitation to the electronic device.
- the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange different components, and details will not be repeated here. .
- the input unit 104 is used to obtain the target mask image of the face area in the first image; the processor 110 is also used to obtain the reference face mask image based on the binarized face mask image of the target mask image. A reference face image matched with the target mask image in the image collection; Processor 110 is also used to perform image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target masks sub-image; the processor 110 is used to fuse the image of the target area of the acquired reference face sub-image with the image of the corresponding area of the acquired target mask sub-image to generate a second image corresponding to the first image; wherein, the reference The face image set includes a plurality of face mask images processed by skin texture; the face skin quality value of the target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
- the target mask image of the face area in the first image is obtained, and the reference face image is obtained based on the binarized face mask image of the target mask image A set of reference face images that match the target mask image.
- the image of the target area of the reference face image is fused with the image of the corresponding area of the target mask image to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain the second image with better skin quality. image, which greatly improves the quality of human face skin after imaging.
- the input unit 104 is also used to acquire N facial images processed through skin quality, where N is a positive integer; the processor 110 is also used to extract facial features information of each facial image in the N facial images, and Construct a binarized mask image according to facial features information; a face image corresponds to a binarized mask image; processor 110 is used to obtain N processed face images based on the input unit 104, and obtain The binarized mask image corresponding to each face image in the image is used to generate a set of reference face images.
- the image processing device can construct a set of reference face images based on the reference face image processed by the skin texture and the corresponding binarized mask image, so that after the image processing device obtains an image that needs to be processed, it can based on This collection performs processing on images.
- the processor 110 is configured to construct a first image pyramid based on a reference face image, and the first image pyramid includes N reference face sub-images; the processor 110 is also configured to construct a second image pyramid based on a target mask image. An image pyramid, the second image pyramid includes N target mask sub-images.
- the processor 110 is also used to extract the first feature point of the reference face image and the second feature point of the target mask image; the processor 110 is also used to obtain N reference face sub-points based on the first feature point The binarized face mask image of each image of the image, and the vertex coordinates of each triangle block in the M triangle blocks contained in the binarized face mask image of each image; the processor 110 is also used for Acquire the vertex coordinates of each of the K triangular blocks contained in the binarized face mask image of each of the N target mask sub-images based on the second feature point; the processor 110 is specifically used to Coordinates, the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image.
- the image processing device obtains the vertex coordinates of each triangular block of each layer image of the first image pyramid, it can adjust the image of each layer based on the triangular block of each layer image, so that the appearance of the characters contained in it is closer to the target The appearance of the character contained in the mask image.
- the processor 110 is configured to perform affine transformation on the acquired apex coordinates of the M triangular blocks based on the acquired apex coordinates of the K triangular blocks; the processor 110 is specifically configured to affinely transform the Image fusion of the first target area image of the first reference face sub-image in the N reference face sub-images with the second target area image of the first target mask sub-image in the N target mask sub-images to obtain N processing The processed target mask sub-image; the processor 110 is also specifically configured to reconstruct N processed target mask sub-images to generate a second image; wherein, the first reference face sub-image is: N reference people Any one of the face sub-images; the first target mask sub-image is the target mask sub-image corresponding to the first reference face sub-image in the N target mask sub-images; the first target area image is the first reference face sub-image The image of the first target area of , and the first target area is an image area corresponding to the second target area of the first target mask sub-image.
- the image processing device can perform affine transformation on the triangulated N reference face images based on the image feature points to obtain an image similar to the person in the target reference face image, and then perform skin texture transfer to obtain Second image with better skin quality.
- the image processing processor 110 is further configured to perform guided filtering skin smoothing on each layer of the image of the second image pyramid constructed by the structure processor 110 according to the preset radius and the relative precision of the floating point.
- the image processing device optimizes the face and skin quality in the first image according to the first image pyramid constructed based on the reference face image and the second image pyramid constructed based on the target mask image, and obtains the second image pyramid with better skin quality. Two images.
- the electronic device provided in the embodiment of the present application can effectively remove the bad texture and transition of the face of the portrait by image processing method to fuse the flawed human face skin in the captured image with the image with better skin quality, It makes the skin of the human face after imaging delicate and clear, and greatly improves the skin quality of the human face after imaging.
- the input unit 104 may include a graphics processing unit (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processing unit 1041 is used by the image capturing device (such as the image data of the still picture or video obtained by the camera) for processing.
- the display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like.
- the user input unit 107 includes a touch panel 1071 and other input devices 1072 .
- the touch panel 1071 is also called a touch screen.
- the touch panel 1071 may include two parts, a touch detection device and a touch controller.
- Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.
- Memory 109 may be used to store software programs as well as various data, including but not limited to application programs and operating systems.
- the processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 110 .
- the embodiment of the present application also provides a readable storage medium, the readable storage medium stores a program or an instruction, and when the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is realized, and can achieve the same To avoid repetition, the technical effects will not be repeated here.
- the processor is the processor in the electronic device described in the above embodiments.
- the readable storage medium includes computer readable storage medium, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.
- the embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above image processing method embodiment Each process can achieve the same technical effect, so in order to avoid repetition, it will not be repeated here.
- chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.
- the term “comprising”, “comprising” or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase “comprising a " does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.
- the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The present application belongs to the field of image processing, and discloses an image processing method and apparatus, an electronic device and a readable storage medium. The method comprises: acquiring a target mask image of a face region in a first image; on the basis of a binarized face mask image of the target mask image, acquiring a reference face image in a reference face image set that matches the target mask image; performing image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images; and fusing images of a target region of the reference face sub-images with images of a corresponding region of the target mask sub-images to generate a second image corresponding to the first image, wherein the reference face image set comprises a plurality of face mask images which have undergone skin texture processing, and a face skin texture value of the target region is greater than a face skin texture value of a corresponding region of the target region in the target mask image.
Description
相关申请的交叉引用Cross References to Related Applications
本申请主张在2021年06月11日在中国提交的中国专利申请号202110653986.2的优先权,其全部内容通过引用包含于此。This application claims priority to Chinese Patent Application No. 202110653986.2 filed in China on June 11, 2021, the entire contents of which are hereby incorporated by reference.
本申请实施例涉及图像处理领域,尤其涉及一种图像处理方法、装置、电子设备及可读存储介质。The embodiments of the present application relate to the field of image processing, and in particular, to an image processing method, device, electronic equipment, and readable storage medium.
随着电子设备技术的发展,用户使用电子设备进行拍摄的频率越来越高,用户对电子设备拍摄图像的质量的要求也越来越高。With the development of electronic device technology, users use electronic devices to shoot more and more frequently, and users have higher and higher requirements on the quality of images captured by electronic devices.
在相关技术中,相机对人像拍照在不同光照成像条件下,受到噪声、运动模糊、高光、后期美颜去噪等多种退化问题影响,成像的人脸缺失良好的肤质和细节,同时脸部的瑕疵(如痘印)、皱纹和噪声过度不均,极大影响成像后的人脸肤感和美观度。In related technologies, the camera takes photos of portraits under different lighting conditions, and is affected by various degradation problems such as noise, motion blur, highlights, post-beauty denoising, etc., and the imaged faces lack good skin quality and details. The internal blemishes (such as acne marks), wrinkles and excessive noise will greatly affect the skin feel and aesthetics of the human face after imaging.
发明内容Contents of the invention
本申请实施例的目的是提供一种图像处理方法、装置、电子设备及可读存储介质,能够解决人脸成像肤质较差的问题。The purpose of the embodiment of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can solve the problem of poor skin quality in human face imaging.
为了解决上述技术问题,本申请是这样实现的:In order to solve the above-mentioned technical problems, the application is implemented as follows:
第一方面,本申请实施例提供一种图像处理方法,该方法包括:获取第一图像中的人脸区域的目标蒙版图像;基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像;针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;将参考人脸子图像的目标区域的图像与目标蒙版子图像的对应区域的图像进行融合,生成与第一图像对应的第二图像;其中,参考人脸图像集合包括多个经过肤质处理的人脸蒙版图像;目标区域的人脸肤质值大于目标蒙版图像中与目标区域对应区域的人脸肤质值。In the first aspect, the embodiment of the present application provides an image processing method, the method comprising: acquiring a target mask image of the face area in the first image; based on a binarized face mask image of the target mask image, acquiring A reference face image matched with the target mask image in the reference face image collection; image processing is performed on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images; The image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image to generate a second image corresponding to the first image; wherein, the set of reference face images includes a plurality of processed human faces Face mask image; the face skin quality value of the target area is greater than the face skin quality value of the area corresponding to the target area in the target mask image.
第二方面,本申请实施例还提供了一种图像处理装置,该装置包括:获取模块和 图像处理模块;获取模块,用于获取第一图像中的人脸区域的目标蒙版图像;获取模块,还用于基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像;获取模块,还用于针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;图像处理模块,用于将获取模块获取的参考人脸子图像的目标区域的图像与获取模块获取的目标蒙版子图像的对应区域的图像进行融合,生成与第一图像对应的第二图像;其中,参考人脸图像集合包括多个经过肤质处理的人脸蒙版图像;目标区域的人脸肤质值高于目标蒙版图像中与目标区域对应区域的人脸肤质值。In a second aspect, the embodiment of the present application also provides an image processing device, the device including: an acquisition module and an image processing module; an acquisition module, configured to acquire a target mask image of a face area in the first image; an acquisition module , is also used for the binarized face mask image based on the target mask image, and obtains the reference face image matching the target mask image in the reference face image set; the acquisition module is also used for the reference face image and the target mask image for image processing to obtain N reference face sub-images and N target mask sub-images; the image processing module is used to obtain the image of the target area of the reference face sub-image obtained by the acquisition module and the image obtained by the acquisition module The images of the corresponding areas of the target mask sub-image are fused to generate a second image corresponding to the first image; wherein, the reference face image set includes a plurality of face mask images processed through skin texture; the face of the target area The skin quality value is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
第三方面,本申请实施例提供了一种电子设备,包括处理器、存储器及存储在该存储器上并可在该处理器上运行的程序或指令,该程序或指令被该处理器执行时实现如第一方面所述的图像处理方法的步骤。In the third aspect, the embodiment of the present application provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and operable on the processor. When the program or instruction is executed by the processor, the The steps of the image processing method as described in the first aspect.
第四方面,本申请实施例提供了一种可读存储介质,所述可读存储介质上存储程序或指令,所述程序或指令被处理器执行时实现如第一方面所述的方法的步骤。In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .
第五方面,本申请实施例提供了一种芯片,所述芯片包括处理器和通信接口,所述通信接口和所述处理器耦合,所述处理器用于运行程序或指令,实现如第一方面所述的方法。In the fifth aspect, the embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, so as to implement the first aspect the method described.
在本申请实施例中,在获取到包含人脸的第一图像之后,获取第一图像中的人脸区域的目标蒙版图像,并基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像。之后,针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像,并将参考人脸子图像的目标区域的图像与目标蒙版子图像的对应区域的图像进行融合,去除脸部不佳纹理和过度,恢复细腻、清晰的皮肤质感,得到肤质更好的第二图像,极大提升了成像后人脸皮肤的质量。In the embodiment of the present application, after the first image containing the human face is acquired, the target mask image of the human face area in the first image is acquired, and based on the binarized human face mask image of the target mask image, Obtain the reference face image matching the target mask image in the reference face image set. Afterwards, image processing is carried out for the reference face image and the target mask image, and N reference face sub-images and N target mask sub-images are obtained, and the image of the target area of the reference face sub-image and the image of the target mask sub-image The images of the corresponding areas are fused to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain a second image with better skin quality, which greatly improves the quality of the facial skin after imaging.
图1是本申请实施例提供的一种图像处理方法所应用的界面的示意图;FIG. 1 is a schematic diagram of an interface to which an image processing method provided in an embodiment of the present application is applied;
图2是本申请实施例提供的一种图像金字塔的结构示意图;Fig. 2 is a schematic structural diagram of an image pyramid provided by an embodiment of the present application;
图3是本申请实施例提供的一种图像处理装置结构示意图;FIG. 3 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;
图4是本申请实施例提供的一种电子设备的结构示意图之一;FIG. 4 is one of the structural schematic diagrams of an electronic device provided in an embodiment of the present application;
图5是本申请实施例提供的一种电子设备的结构示意图之二。FIG. 5 is a second schematic structural diagram of an electronic device provided by an embodiment of the present application.
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚地描述,显然,所描述的实施例是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员获得的所有其他实施例,都属于本申请保护的范围。The following will clearly describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in this application belong to the protection scope of this application.
本申请的说明书和权利要求书中的术语“第一”、“第二”等是用于区别类似的对象,而不用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换,以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施,且“第一”、“第二”等所区分的对象通常为一类,并不限定对象的个数,例如第一对象可以是一个,也可以是多个。此外,说明书以及权利要求中“和/或”表示所连接对象的至少其中之一,字符“/”,一般表示前后关联对象是一种“或”的关系。The terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific sequence or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application can be practiced in sequences other than those illustrated or described herein, and that references to "first," "second," etc. distinguish Objects are generally of one type, and the number of objects is not limited. For example, there may be one or more first objects. In addition, "and/or" in the specification and claims means at least one of the connected objects, and the character "/" generally means that the related objects are an "or" relationship.
本申请实施例提供的图像处理方法可以应用于对包含人脸的图像进行美化的场景中。The image processing method provided in the embodiment of the present application may be applied to a scene of beautifying an image including a human face.
示例性的,针对对包含人脸的图像进行美化的场景,在相关技术中,电子设备在成像时,由于受到噪声、运动模糊、高光、后期美颜去噪等多种退化问题影响,成像的人脸缺失良好的肤质和细节,同时脸部的瑕疵、皱纹和噪声造成过度不均,极度影响成像的人脸肤感和美观度。Exemplarily, for the scene of beautifying an image containing a human face, in related technologies, when an electronic device is imaging, it is affected by various degradation problems such as noise, motion blur, highlight, and post-beautification denoising. The human face lacks good skin texture and details, while the blemishes, wrinkles and noise on the face cause excessive unevenness, which greatly affects the skin feel and aesthetics of the imaged human face.
针对这一问题,在本申请实施例提供的技术方案中,通过基于多层图像金字塔融合的人脸肤质迁移方法,将拍摄好的图像中存在瑕疵的人脸皮肤与肤质较好的图像进行图像融合,可以有效去除人像脸部的不佳纹理和过渡,使得成像后的人脸皮肤细腻、清晰,极大的提升了成像后的人脸的肤质。To solve this problem, in the technical solution provided by the embodiment of the present application, through the face and skin quality transfer method based on multi-layer image pyramid fusion, the human face and skin with blemishes in the captured image are combined with the image with better skin quality Image fusion can effectively remove the poor texture and transition of the face of the portrait, making the skin of the face after imaging delicate and clear, and greatly improving the skin quality of the face after imaging.
下面结合附图,通过具体的实施例及其应用场景对本申请实施例提供的图像处理方法进行详细地说明。The image processing method provided by the embodiment of the present application will be described in detail below through specific embodiments and application scenarios with reference to the accompanying drawings.
如图1所示,本申请实施例提供的一种图像处理方法,该方法可以包括下述步骤201至步骤204:As shown in Figure 1, an image processing method provided by the embodiment of the present application may include the following steps 201 to 204:
步骤201、图像处理装置获取第一图像中的人脸区域的目标蒙版图像。 Step 201. The image processing apparatus acquires a target mask image of a face area in a first image.
示例性地,上述第一图像可以是电子设备拍摄得到的图像,也可以是电子设备读取的电子设备中存储的图像。Exemplarily, the above-mentioned first image may be an image captured by the electronic device, or may be an image stored in the electronic device read by the electronic device.
示例性地,图像处理装置在获取到上述第一图像之后,在红绿蓝(red green blue,RGB)颜色空间中获取该第一图像的人脸区域的图像。并在获取到的人脸区域内通过人脸解析算法,生成人脸区域的蒙版图像,即上述目标蒙版图像。Exemplarily, after acquiring the above-mentioned first image, the image processing apparatus acquires an image of the face area of the first image in a red green blue (red green blue, RGB) color space. And in the obtained face area, a mask image of the face area, that is, the above-mentioned target mask image, is generated through a face parsing algorithm.
需要说明的是,上述目标蒙版图像可以理解为,在获取到第一图像中包含的人脸的轮廓之后,将该人脸轮廓范围之外的所有图像全部进行遮盖,例如,设置为同一种颜色,进而使图像处理装置只能识别出人脸区域的图像。It should be noted that the above-mentioned target mask image can be understood as, after the contour of the human face contained in the first image is obtained, all images outside the range of the contour of the human face are covered, for example, set to the same color, so that the image processing device can only recognize the image of the face area.
可以理解的是,仅获取第一图像的人脸区域的图像,是为了排除其他区域图像的干扰,方便对人脸区域的图像进行优化。It can be understood that the purpose of acquiring only the image of the face area of the first image is to eliminate the interference of images of other areas, so as to facilitate the optimization of the image of the face area.
步骤202、图像处理装置基于上述目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与该目标蒙版图像相匹配的参考人脸图像。 Step 202 , the image processing device acquires a reference face image matching the target mask image in the reference face image set based on the binarized face mask image of the target mask image.
示例性地,图像二值化(image binarization)就是将图像上的像素点的灰度值设置为0或255,也就是将整个图像呈现出明显的黑白效果的过程。上述二值化人脸蒙版图像可以理解为人脸图像的黑白图像。即若上述目标蒙版图像的五官区域全部设置为黑色,而所有的非五官区域全部设置为白色。Exemplarily, image binarization (image binarization) is the process of setting the gray value of the pixel on the image to 0 or 255, that is, the process of presenting an obvious black-and-white effect to the entire image. The above binarized face mask image can be understood as a black and white image of a face image. That is, if the facial features areas of the above target mask image are all set to black, and all non-facial features areas are all set to white.
示例性地,上述二值化人脸图像为仅包括五官的图像,即该二值化人脸图像为包括人脸区域的眼睛、鼻子、眉毛、嘴巴等区域的图像。该二值化人脸蒙版图像主要用于与参考人脸图像合集中的人脸图像进行匹配。Exemplarily, the above binarized face image is an image including only facial features, that is, the binarized face image is an image including eyes, nose, eyebrows, mouth and other areas of the face area. The binarized face mask image is mainly used for matching with the face images in the reference face image collection.
示例性地,上述步骤202中使用的匹配算法为额日志华模板匹配算法。Exemplarily, the matching algorithm used in the above-mentioned step 202 is an E-log-Hua template matching algorithm.
步骤203、图像处理装置针对上述参考人脸图像以及上述目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像。 Step 203 , the image processing device performs image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images.
示例性地,上述针对上述参考人脸图像以及上述目标蒙版图像进行图像处理,可以包括对参考人脸图像进行格式化处理,之后,对经过格式化处理后的参考人脸图像进行退化处理及缩放操作,生成N个参考人脸子图像。其中,每两个相邻的参考人脸子图像之间的缩放比例相同,且分辨率较小的图像是分辨率较大的图像经过退化处理后得到的图像。Exemplarily, the above-mentioned image processing for the above-mentioned reference face image and the above-mentioned target mask image may include performing formatting processing on the reference face image, and then performing degradation processing and processing on the formatted reference face image. Scaling operation to generate N reference face sub-images. Wherein, the scaling ratio between every two adjacent reference face sub-images is the same, and the image with a smaller resolution is an image obtained after degrading the image with a larger resolution.
需要说明的是,目标蒙版图像经过处理后得到N个目标蒙版子图像的处理方式与上述针对参考人脸图像进行处理的方式相似,可以基于上述针对参考人脸图像进行处理的方式对目标蒙版图像进行处理,得到N个目标蒙版子图像。It should be noted that the processing method of obtaining N target mask sub-images after processing the target mask image is similar to the above-mentioned processing method for the reference face image, and the target image can be processed based on the above-mentioned processing method for the reference face image. The mask image is processed to obtain N target mask sub-images.
示例性地,上述N个参考人脸子图像与上述N个目标蒙版子图像存在一一对应的关系。例如,以上述N为5为例,5个编号为0~4的参考人脸子图像,与5个编号为0~4的目标蒙版子图像之间,编号相同的图像存在对应关系。其中,随着编号的增加,图像的分辨率逐渐降低。Exemplarily, there is a one-to-one correspondence between the N reference face sub-images and the N target mask sub-images. For example, taking the above-mentioned N as 5 as an example, there is a corresponding relationship between the five reference face sub-images numbered 0-4 and the five target mask sub-images numbered 0-4. Wherein, as the number increases, the resolution of the image gradually decreases.
步骤204、图像处理装置将上述参考人脸子图像的目标区域的图像与目标蒙版子图像 的对应区域的图像进行融合,生成与第一图像对应的第二图像。 Step 204. The image processing device fuses the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image to generate a second image corresponding to the first image.
其中,上述参考人脸图像合集包括多个经过肤质处理的人脸蒙版图像。上述目标区域的人脸肤质值高于目标蒙版图像中与目标区域对应区域的人脸肤质值。Wherein, the aforementioned collection of reference face images includes a plurality of face mask images that have undergone skin texture processing. The face skin quality value of the above target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
示例性地,图像处理装置基于上述二值化人脸蒙版图像,从参考人脸图像合集中找到匹配的参考人脸图像。之后,可以将目标蒙版图像中肤质较差区域的图像与该参考人脸图像中对应区域的肤质较好的图像进行图像融合,进而得到肤质较好的第二图像。Exemplarily, the image processing device finds a matching reference face image from the collection of reference face images based on the above binarized face mask image. Afterwards, the image of the poor skin quality area in the target mask image may be fused with the image with better skin quality in the corresponding area in the reference face image, so as to obtain a second image with better skin quality.
在一种可能的实现方式中,图像处理装置可以根据参考人脸图像生成第一图像金字塔,以及根据目标蒙版图像生成第二图像金字塔,之后,基于该第一图像金字塔以及第二图像金字塔,对目标蒙版图像进行处理,将参考人脸图像的皮肤进行肤质迁移,得到第一图像肤质较好的人脸图像。In a possible implementation manner, the image processing device may generate the first image pyramid according to the reference face image, and generate the second image pyramid according to the target mask image, and then, based on the first image pyramid and the second image pyramid, The target mask image is processed, and the skin quality of the reference face image is transferred to obtain a face image with better skin quality in the first image.
如此,在获取到包含人脸的第一图像之后,获取第一图像中的人脸区域的目标蒙版图像,并基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像。之后,针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像,并将参考人脸子图像的目标区域的图像与目标蒙版子图像的对应区域的图像进行融合,去除脸部不佳纹理和过度,恢复细腻、清晰的皮肤质感,得到肤质更好的第二图像,极大提升了成像后人脸皮肤的质量。In this way, after the first image containing the face is obtained, the target mask image of the face area in the first image is obtained, and the reference face image is obtained based on the binarized face mask image of the target mask image A set of reference face images that match the target mask image. Afterwards, image processing is carried out for the reference face image and the target mask image, and N reference face sub-images and N target mask sub-images are obtained, and the image of the target area of the reference face sub-image and the image of the target mask sub-image The images of the corresponding areas are fused to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain a second image with better skin quality, which greatly improves the quality of the facial skin after imaging.
可选地,在本申请实施例中,图像处理装置可以基于图像金字塔来实现将参考人脸图像中的肤质较好的图像迁移到第一图像的肤质较差的区域中。Optionally, in the embodiment of the present application, the image processing device may realize the migration of the image with better skin quality in the reference face image to the region with poorer skin quality in the first image based on the image pyramid.
示例性地,上述步骤202之前,本申请实施例提供的图像处理方法,还可以包括以下步骤201a1至步骤202a3:Exemplarily, before the above step 202, the image processing method provided in the embodiment of the present application may further include the following steps 201a1 to 202a3:
步骤202a1、图像处理装置获取N个经过肤质处理的人脸图像,N为正整数。Step 202a1, the image processing device acquires N face images that have undergone skin texture processing, where N is a positive integer.
示例性地,图像处理装置在获取与目标蒙版图像匹配的参考人脸图像之前,还需要创建参考人脸图像集合。可以获取N个经过影像级专业图像处理(包括肤色调整、祛斑祛痘、磨皮、增强等)的肤质较好的人脸图像,该集合中包括N个人脸图像。Exemplarily, before the image processing apparatus acquires the reference face image matching the target mask image, it also needs to create a set of reference face images. N human face images with better skin quality that have undergone image-level professional image processing (including skin color adjustment, freckle and acne removal, skin smoothing, enhancement, etc.) can be obtained, and the set includes N human face images.
步骤202a2、图像处理装置提取上述N个人脸图像中每个人脸图像的五官信息,并根据所述五官信息构建二值化蒙版。In step 202a2, the image processing device extracts facial features information of each of the above N facial images, and constructs a binarized mask according to the facial features information.
其中,一个人脸图像对应一个二值化蒙版。Among them, a face image corresponds to a binarization mask.
示例性地,上述五官信息包括人脸图像中人脸五官的各种信息,例如、五官所处的区域、具体位置坐标等。Exemplarily, the facial features information includes various information about the facial features in the face image, for example, the area where the facial features are located, specific location coordinates, and the like.
示例性地,图像处理装置在获取到上述N个人脸图像后,通过人脸解析模型,分解出每个人脸图像的人脸区域各个部件的逐像素分割蒙版图像,该蒙版图像中仅保留人脸的五官图像。之后,图像处理装置基于该蒙版图像,构建蒙版图像的二值化蒙版图像。每个人脸对象均包括对应的二值化蒙版图像。Exemplarily, after the image processing device acquires the aforementioned N face images, it uses the face analysis model to decompose the pixel-by-pixel segmentation mask image of each part of the face area of each face image, and only An image of the facial features of a human face. Afterwards, the image processing device constructs a binarized mask image of the mask image based on the mask image. Each face object includes a corresponding binarized mask image.
步骤202a3、图像处理装置基于上述N个经过肤质处理的人脸图像,以及与每个人脸图像对应的二值化蒙版,生成所述参考人脸图像集合。In step 202a3, the image processing device generates the set of reference face images based on the above N face images that have undergone skin texture processing and a binarized mask corresponding to each face image.
示例性地,上述参考人脸图像集合中包括N个经过肤质处理的人脸图像,以及与人脸图像对应的N个二值化蒙版图像。Exemplarily, the aforementioned set of reference human face images includes N human face images that have undergone skin texture processing, and N binary mask images corresponding to the human face images.
示例性地,上述经过肤质处理的人脸图像主要用于与目标蒙版图像中肤质较差区域的图像进行图像融合。上述二值化蒙版图像主要用户调整上述参考人脸图像的五官的位置,使其更接近目标蒙版图像中人脸五官的位置,使得参考人脸图像的人物长相在经过调整后,能够尽可能的与目标蒙版图像中人脸的人物长相保持一致,方便后续肤质的迁移。Exemplarily, the above-mentioned human face image processed with skin quality is mainly used for image fusion with an image of a region with poor skin quality in the target mask image. The main user of the above-mentioned binary mask image adjusts the position of the facial features of the above-mentioned reference face image to make it closer to the position of the facial features in the target mask image, so that the appearance of the person in the reference face image can be adjusted as much as possible after adjustment. It is possible to keep the appearance of the face in the target mask image consistent, so as to facilitate the subsequent migration of skin quality.
如此,图像处理装置可以基于经过肤质处理的参考人脸图像以及与其对应的二值化蒙版图像构建参考人脸图像集合,使得图像处理装置在获取到一张需要处理的图像后,能够基于该集合对图像进行处理。In this way, the image processing device can construct a set of reference face images based on the reference face image processed by the skin texture and the corresponding binarized mask image, so that after the image processing device obtains an image that needs to be processed, it can based on This collection performs processing on images.
可选地,在本申请实施例中,图像处理装置在得到上述参考人脸图像集合之后,就可以基于该集合对获取到的第一图像进行处理,具体处理过程需要利用图像金字塔来完成。Optionally, in the embodiment of the present application, after the image processing device obtains the aforementioned set of reference face images, it can process the acquired first image based on the set, and the specific processing process needs to be completed by using an image pyramid.
示例性地,上述步骤203,还可以包括以下步骤203a1和步骤203a2:Exemplarily, the above step 203 may also include the following steps 203a1 and 203a2:
步骤203a1、图像处理装置基于上述参考人脸图像,构建第一图像金字塔。In step 203a1, the image processing device constructs a first image pyramid based on the aforementioned reference face image.
步骤203a2、图像处理装置基于上述目标蒙版图像,构建第二图像金字塔。In step 203a2, the image processing device constructs a second image pyramid based on the target mask image.
其中,上述第一图像金子塔包括上述N个参考人脸子图像,上述第二图像金字塔包括上述N个目标蒙版子图像。Wherein, the first image pyramid includes the N reference face sub-images, and the second image pyramid includes the N target mask sub-images.
需要说明的是,图像金字塔是图像多尺度表达的一种,是一种以多分辨率来解释图像的有效但概念简单的结构。一幅图像的金字塔是一系列以金字塔形状排列的分辨率逐步降低,且来源于同一张原始图的图像集合。其通过梯次向下采样获得,直到达到某个终止条件才停止采样。我们将一层一层的图像比喻成金字塔,层级越高,则图像越小,分辨率越低。It should be noted that the image pyramid is a kind of multi-scale representation of images, and it is an effective but conceptually simple structure to explain images at multiple resolutions. The pyramid of an image is a series of images arranged in a pyramid shape with gradually reduced resolution and derived from the same original image. It is obtained by down-sampling in steps, and the sampling is stopped until a certain termination condition is reached. We compare layer-by-layer images to a pyramid. The higher the level, the smaller the image and the lower the resolution.
示例性地,图像处理装置需要构建上述参考人脸图像集合中的每个人脸图像的图像金字塔。Exemplarily, the image processing device needs to construct an image pyramid of each face image in the reference face image set.
示例性地,该第一图像金字塔可以为拉普拉斯金字塔。该第一图像金字塔的底层(第0层)可以为上述参考人脸图像,也可以为该参考人脸图像经过格式化处理后的图像。使得参考人脸图像集合中每个人脸图像构建的金字塔的第0层的图像尺寸均相同,且层与层之间的缩放比例也相同。Exemplarily, the first image pyramid may be a Laplacian pyramid. The bottom layer (level 0) of the first image pyramid may be the above-mentioned reference face image, or may be a formatted image of the reference face image. The image size of the 0th layer of the pyramid constructed by each face image in the reference face image set is the same, and the scaling ratio between layers is also the same.
举例说明,如图2所示,为一种图像金字塔的结构示意图,该图像金字塔包括五层(L0至L4),每层均包含一个图像,层与层之间的图像按照预设缩放比例进行了缩放。For example, as shown in Figure 2, it is a schematic structural diagram of an image pyramid, which includes five layers (L0 to L4), each layer contains an image, and the images between layers are scaled according to a preset ratio Zoomed out.
可选地,在本申请实施例中,图像处理装置可以通过基于参考人脸图像以及目标蒙版图像的特征点,对目标蒙版图像进行处理,进而得到第二图像。Optionally, in the embodiment of the present application, the image processing apparatus may process the target mask image based on the reference face image and feature points of the target mask image, so as to obtain the second image.
示例性地,上述步骤204可以包括以下步骤204a1至步骤204a4:Exemplarily, the above step 204 may include the following steps 204a1 to 204a4:
步骤204a1、图像处理装置提取上述参考人脸图像的第一特征点和上述目标蒙版图像的第二特征点。In step 204a1, the image processing device extracts the first feature point of the reference face image and the second feature point of the target mask image.
步骤204a2、图像处理装置基于上述第一特征点获取上述N个参考人脸子图像每张图像的二值化人脸蒙版图像,以及上述每张图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的顶点坐标。Step 204a2, the image processing device obtains the binarized face mask image of each of the N reference face sub-images based on the first feature point, and the binary face mask image contained in each of the above images Vertex coordinates of each triangle block in the M triangle blocks.
步骤204a3、图像处理装置基于上述第二特征点获取上述N个目标蒙版子图像每张图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标。Step 204a3: The image processing device obtains the vertex coordinates of each of the K triangle blocks contained in the binarized face mask image of each of the N target mask sub-images based on the second feature point.
步骤204a4、图像处理装置基于上述顶点坐标,将上述参考人脸子图像的目标区域的图像与上述目标蒙版子图像的对应区域的图像进行融合。In step 204a4, the image processing device fuses the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image based on the vertex coordinates.
示例性地,示例性地,以上述N个参考人脸子图像为第一图像金字塔中的图像、上述N个目标蒙版子图像为上述第二图像金字塔中的图像为例,图像处理装置在成功构建上述第一图像金字塔和第二图像金字塔之后,可以对上述第一图像金字塔和第二图像金字塔进行三角化处理,Exemplarily, exemplarily, taking the aforementioned N reference face sub-images as images in the first image pyramid, and the aforementioned N target mask sub-images as images in the aforementioned second image pyramid as an example, the image processing device successfully After constructing the above-mentioned first image pyramid and the second image pyramid, the above-mentioned first image pyramid and the second image pyramid can be triangulated,
示例性地,上述步骤204a1至步骤204a4具体处理步骤可以包括以下步骤204b1至步骤204b3:Exemplarily, the above-mentioned specific processing steps from step 204a1 to step 204a4 may include the following steps from step 204b1 to step 204b3:
步骤204b1、图像处理装置提取上述参考人脸图像的第一特征点,并基于该第一特征点对所述参考人脸图像进行三角剖分,得到M个三角块。Step 204b1. The image processing device extracts the first feature point of the reference face image, and performs triangulation on the reference face image based on the first feature point to obtain M triangular blocks.
其中,一个第一特征点对应一个三角块、且每个三角块的外接圆的范围内均不包括其他第一特征点,M为正整数。Wherein, one first feature point corresponds to one triangular block, and the circumscribed circle of each triangular block does not include other first feature points, and M is a positive integer.
示例性地,图像处理装置可以从上述参考人脸图像中提取到多个第一特征点,之后, 图像处理装置基于每个特征点进行三角剖分(也可称为图像三角化),使得生成的每个三角块的外接圆的范围内均不包括其他任何第一特征点。Exemplarily, the image processing device may extract a plurality of first feature points from the above-mentioned reference face image, and then, the image processing device performs triangulation (also referred to as image triangulation) based on each feature point, so that the generated The circumscribed circle of each triangular block does not include any other first feature points.
可以理解的是,若图像处理装置提取的某个特征点不能满足每个三角块的外接圆的范围内均不包括其他特征点,则该特征点不能作为第一特征点。It can be understood that if a certain feature point extracted by the image processing device cannot meet the requirement that no other feature points are included within the circumscribed circle of each triangular block, this feature point cannot be used as the first feature point.
需要说明的是,图像三角化可以理解为将图像分成若干个三角形碎片,每块碎片都是三角形,图像上上任何两个三角形,要么不相交,要么恰好相交于一条公共边(不能同时交两条或两条以上的边)。It should be noted that image triangulation can be understood as dividing the image into several triangular fragments, each of which is a triangle, and any two triangles on the image either do not intersect, or intersect exactly on a common side (two cannot intersect at the same time). strip or two or more sides).
步骤204b2、图像处理装置获取该第一图像金字塔每层图像的二值化人脸蒙版图像。Step 204b2, the image processing device acquires the binarized face mask image of each layer image of the first image pyramid.
步骤204b3、图像处理装置基于该第一图像金字塔每层图像对应的二值化人脸蒙版图像,以及第一目标层与第二目标层之间的缩放比例,确定每层图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的顶点坐标。Step 204b3: The image processing device determines the binarization of each layer image based on the binarized face mask image corresponding to each layer image of the first image pyramid, and the scaling ratio between the first target layer and the second target layer Vertex coordinates of each triangle block in the M triangle blocks contained in the face mask image.
其中,第一目标层与第二目标层为上述第一图像金字塔的两个相邻层。Wherein, the first target layer and the second target layer are two adjacent layers of the above-mentioned first image pyramid.
示例性地,图像处理装置在构建上述参考人脸图像的第一图像金字塔之后,还需要基于该第一图像金字塔每层所包含的图像生成每层图像对应的二值化人脸蒙版图像。Exemplarily, after the image processing device constructs the first image pyramid of the above-mentioned reference face image, it also needs to generate a binary face mask image corresponding to each layer image based on the images contained in each layer of the first image pyramid.
示例性地,图像处理装置在获取到参考人脸图像的M个三角块之后,可以基于该M个三角块中每个三角块的顶点坐标,确定第一图像金字塔每层图像的M个三角块中每个三角块的顶点坐标。Exemplarily, after the image processing device acquires the M triangular blocks of the reference face image, it can determine the M triangular blocks of each layer image of the first image pyramid based on the vertex coordinates of each of the M triangular blocks Vertex coordinates of each triangle block in .
需要说明的是,由于第一图像金字塔中每层图像均为基于上述参考人脸图像得到的,因此,参考人脸图像的每个三角块均可以在每层图像中找到对应的三角块。并且,由于第一图像金字塔层与层之间存在图像缩放,因此,可以基于缩放比例重新计算每层图像的每个三角块的顶点坐标。It should be noted that, since each layer image in the first image pyramid is obtained based on the above-mentioned reference face image, each triangular block of the reference face image can find a corresponding triangular block in each layer image. Moreover, since image scaling exists between layers of the first image pyramid, the vertex coordinates of each triangular block of each layer image may be recalculated based on the scaling ratio.
如此,图像处理装置在获取到第一图像金字塔每层图像的每个三角块的顶点坐标之后,可以基于每层图像的三角块对每层图像进行调整,使其包含的人物外观更加接近于目标蒙版图像所包含的人物外观。In this way, after the image processing device obtains the vertex coordinates of each triangular block of each layer image of the first image pyramid, it can adjust the image of each layer based on the triangular block of each layer image, so that the appearance of the characters contained in it is closer to the target The appearance of the character contained in the mask image.
示例性地,与上述构建参考人脸图像的第一图像金字塔的方法类似,图像处理装置可以依据此方法构建目标蒙版图像的第二图像金字塔。Exemplarily, similar to the above method of constructing the first image pyramid of the reference face image, the image processing device may construct the second image pyramid of the target mask image according to this method.
示例性地,上述步骤204a1至步骤204a4,具体还可以包括以下步骤204c1至步骤204c4:Exemplarily, the above steps 204a1 to 204a4 may specifically include the following steps 204c1 to 204c4:
步骤204c1、图像处理装置基于上述目标蒙版图像构建第二图像金字塔。In step 204c1, the image processing device constructs a second image pyramid based on the target mask image.
示例性地,与上述第一图像金字塔类似,该第二图像金字塔的第0层为目标蒙版图像或者目标蒙版图像进过格式化处理后得到的图像构建的。Exemplarily, similar to the above-mentioned first image pyramid, the 0th layer of the second image pyramid is constructed from the target mask image or the image obtained after the target mask image is formatted.
需要说明的是,第一图像金字塔的每层图像与第二图像金字塔的每层图像的尺寸均相同。且构建第一图像金字塔与构建第二图像金字塔的图像的尺寸也相同。It should be noted that the size of each layer image of the first image pyramid is the same as that of each layer image of the second image pyramid. And the size of the images used to construct the first image pyramid and the second image pyramid is also the same.
可以理解的是,本申请实施例中图像的尺寸可以使用分辨率来表示,也可以使用英寸来表示,本申请实施例对此不做限定。It can be understood that, in the embodiment of the present application, the size of the image may be represented by resolution or inches, which is not limited in the embodiment of the present application.
步骤204c2、图像处理装置提取所述目标蒙版图像的第二特征点,并基于所述第二特征点对所述目标蒙版图像进行三角剖分,得到K个三角块。Step 204c2. The image processing device extracts the second feature points of the target mask image, and performs triangulation on the target mask image based on the second feature points to obtain K triangular blocks.
其中,一个第二特征点对应一个三角块、且每个三角块的外接圆的范围内均不包括其他第二特征点,K为正整数。Wherein, one second feature point corresponds to one triangular block, and the circumscribed circle of each triangular block does not include other second feature points, and K is a positive integer.
步骤204c3、图像处理装置获取上述第二图像金字塔每层图像的二值化人脸蒙版图像。In step 204c3, the image processing device acquires the binarized face mask image of each layer image of the second image pyramid.
步骤204c4、图像处理装置基于上述第二图像金字塔每层图像的二值化人脸蒙版图像,以及第三目标层与第四目标层之间的缩放比例,确定上述第二图像金字塔每层图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标。Step 204c4, the image processing device determines the image of each layer of the second image pyramid based on the binarized face mask image of the image of each layer of the second image pyramid and the scaling ratio between the third target layer and the fourth target layer The vertex coordinates of each triangular block in the K triangular blocks contained in the binarized face mask image of .
其中,所述第三目标层与所述第四目标层为所述第二图像金字塔的相邻层。Wherein, the third target layer and the fourth target layer are adjacent layers of the second image pyramid.
需要说明的是,由于上述步骤204c1至步骤204c4与步骤204b1至步骤204b3相似,对比步骤204c1至步骤204c4的解释说明,可以参照上述步骤204b1至步骤204b3的解释说明。上述步骤204a1至步骤204a4中对N个参考人脸子图像及N个目标蒙版子图像的具体处理过程,可以参考对第一图像金字塔和第二图像金字塔的处理过程的描述,为了防止重复,在此不再赘述。It should be noted that, since the above steps 204c1 to 204c4 are similar to steps 204b1 to 204b3, for comparison with the explanations of steps 204c1 to 204c4, you can refer to the above explanations of steps 204b1 to 204b3. For the specific processing of the N reference face sub-images and the N target mask sub-images in the above steps 204a1 to 204a4, you can refer to the description of the processing of the first image pyramid and the second image pyramid, in order to prevent repetition, in This will not be repeated here.
如此,图像处理装置在获取到参考人脸图像的图像金字塔以及目标蒙版图像的图像金字塔之后,可以基于图像金字塔将参考人脸图像中肤质较好区域的图像迁移到目标蒙版图像中肤质较差区域的图像。In this way, after the image processing device acquires the image pyramid of the reference face image and the image pyramid of the target mask image, it can migrate the image of the region with better skin quality in the reference face image to the target mask image based on the image pyramid. Images in areas of poor quality.
进一步可选地,在本申请实施例中,图像处理装置可以基于上述N个参考人脸子图像以及N个目标蒙版子图像来改善第一图像中人脸区域的人脸肤质。Further optionally, in the embodiment of the present application, the image processing apparatus may improve the face skin quality of the face area in the first image based on the above N reference face sub-images and N target mask sub-images.
示例性地,上述步骤204a4可以包括以下步骤204d1至步骤204d3:Exemplarily, the above step 204a4 may include the following steps 204d1 to 204d3:
步骤204d1、图像处理装置基于K个三角块的顶点坐标,对M个三角块的顶点坐标进行仿射变换。Step 204d1, the image processing device performs affine transformation on the vertex coordinates of the M triangular blocks based on the vertex coordinates of the K triangular blocks.
步骤204d2、图像处理装置将经过仿射变换后的N个参考人脸子图像中第一参考人脸 子图像的第一目标区域图像,与上述N个目标蒙版子图像中第一目标蒙版子图像的第二目标区域图像进行图像融合,得到N个处理后的目标蒙版子图像.Step 204d2, the image processing device combines the first target area image of the first reference face sub-image in the N reference face sub-images after affine transformation with the first target mask sub-image in the above N target mask sub-images The image of the second target area is fused to obtain N processed target mask sub-images.
步骤204d3、图像处理装置对上述N个经过处理后的目标蒙版子图像进行重建,生成上述第二图像.Step 204d3. The image processing device reconstructs the above N processed target mask sub-images to generate the above second image.
其中,上述第一参考人脸子图像为:上述N个参考人脸子图像中的任一个;上述第一目标蒙版子图像为上述N个目标蒙版子图像中与上述第一参考人脸子图像对应的目标蒙版子图像;上述第一目标区域图像为上述第一参考人脸子图像的第一目标区域的图像、且上述第一目标区域与上述第一目标蒙版子图像的第二目标区域对应的图像区域。Wherein, the above-mentioned first reference face sub-image is: any one of the above-mentioned N reference face sub-images; the above-mentioned first target mask sub-image is corresponding to the above-mentioned first reference face sub-image among the above-mentioned N target mask sub-images The target mask sub-image; the first target area image is the image of the first target area of the first reference face sub-image, and the first target area corresponds to the second target area of the first target mask sub-image image area.
示例性地,以上述N个参考人脸子图像为第一图像金字塔中的图像、上述N个目标蒙版子图像为第二图像金字塔中的图像为例,上述步骤204d1至步骤204d3可以包括以下步骤204e1至步骤204e3:Exemplarily, taking the aforementioned N reference face sub-images as images in the first image pyramid, and the aforementioned N target mask sub-images as images in the second image pyramid as an example, the aforementioned steps 204d1 to 204d3 may include the following steps 204e1 to step 204e3:
步骤204e1、图像处理装置基于上述第二图像金字塔的每层图像的K个三角块的顶点坐标,对上述第一图像金字塔的每层图像的M个三角块的顶点坐标进行仿射变换。Step 204e1: The image processing device performs affine transformation on the vertex coordinates of the M triangular blocks in each layer of the image in the first image pyramid based on the vertex coordinates of the K triangular blocks in each layer of the image in the second image pyramid.
示例性地,为了使参考人脸图像中的人物的外形长相更加接近与第一图像中的人物外形长相,因此,图像处理装置需要对第二图像金字塔中每层图像进行处理。即对每层图像机械能仿射变换。Exemplarily, in order to make the appearance of the person in the reference face image closer to the appearance of the person in the first image, the image processing device needs to process each layer of the image in the second image pyramid. That is, the mechanical energy affine transformation of each layer of image.
需要说明的是,仿射变换,又称仿射映射,是指在几何中,一个向量空间进行一次线性变换并接上一个平移,变换为另一个向量空间。仿射变换是在几何上定义为两个向量空间之间的一个仿射变换或者仿射映射由一个非奇异的线性变换(运用一次函数进行的变换)接上一个平移变换组成。It should be noted that affine transformation, also known as affine mapping, means that in geometry, a vector space is transformed into another vector space by performing a linear transformation followed by a translation. Affine transformation is geometrically defined as an affine transformation between two vector spaces or an affine mapping consists of a non-singular linear transformation (transformation using a function) followed by a translation transformation.
示例性地,在对第一图像金字塔的每层图像进行仿射变换之前,需要按照目标蒙版图像所包含的三角块的坐标,计算对应的参考人脸图像的每个三角块到目标蒙版图像的每个三角块的仿射变换矩阵。图像处理装置可以基于得到的变换矩阵对第一图像金字塔的每层图像进行仿射变换。Exemplarily, before performing affine transformation on each layer image of the first image pyramid, it is necessary to calculate the coordinates of each triangle block of the corresponding reference face image to the target mask according to the coordinates of the triangle blocks contained in the target mask image The affine transformation matrix for each triangular block of the image. The image processing device may perform affine transformation on each layer image of the first image pyramid based on the obtained transformation matrix.
步骤204e2、图像处理装置将经过仿射变换后的第一图像金字塔的第五目标层的第一目标区域图像,与所述第二图像金字塔的第六目标层的第二目标区域图像进行图像融合,得到处理后的第二图像金字塔。Step 204e2, the image processing device performs image fusion on the image of the first target area of the fifth target layer of the first image pyramid after the affine transformation, and the image of the second target area of the sixth target layer of the second image pyramid , to obtain the processed second image pyramid.
示例性地,由于第一图像金字塔的每一层均与第二图像金字塔的每一层向对应,例如,第一图像金字塔的第0层与第二图像金字塔的第0层对应,第一图像金字塔的第n层与第 二图像金字塔的第n层对应。因此,图像处理装置可以将第一图像金字塔的第一目标区域的图像与第二图像金字塔的对应区域(即第二目标区域)的图像进行图像融合,得到经过处理后的第二图像金字塔。Exemplarily, since each layer of the first image pyramid corresponds to each layer of the second image pyramid, for example, layer 0 of the first image pyramid corresponds to layer 0 of the second image pyramid, the first image The nth level of the pyramid corresponds to the nth level of the second image pyramid. Therefore, the image processing device may perform image fusion on the image of the first target area of the first image pyramid and the image of the corresponding area (ie, the second target area) of the second image pyramid to obtain the processed second image pyramid.
步骤204e3、图像处理装置对经过处理后的第二图像金字塔进行重建,生成所述第二图像。In step 204e3, the image processing device reconstructs the processed second image pyramid to generate the second image.
其中,上述第五目标层为:上述第一图像金字塔的任一层;上述第六目标层为上述第二图像金字塔中与上述第五目标层对应的层;上述第一目标区域图像为上述第五目标层的第一目标区域的图像、且上述第一目标区域与上述第六目标层的第二目标区域对应的图像区域;上述第一图像金字塔与上述第二图像金字塔的层数形同、且每层的缩放比例也相同。Wherein, the above-mentioned fifth target layer is: any layer of the above-mentioned first image pyramid; the above-mentioned sixth target layer is the layer corresponding to the above-mentioned fifth target layer in the above-mentioned second image pyramid; The image of the first target area of the five target layers, and the image area corresponding to the first target area and the second target area of the sixth target layer; the number of layers of the first image pyramid and the second image pyramid is the same, And the scaling ratio of each layer is also the same.
可以理解的是,第六目标层为上述第二图像金字塔中与上述第五目标层对应的层可以理解为,第六目标层在第二图像金字塔中的层数与第五目标层在第一图像金字塔中的层数相同,即同一层。It can be understood that the sixth target layer is the layer corresponding to the fifth target layer in the second image pyramid. It can be understood that the number of layers of the sixth target layer in the second image pyramid is the same as that of the fifth target layer in the first The number of layers in the image pyramid is the same, that is, the same layer.
示例性地,图像处理装置对经过参考图像纹理迁移后的拉普拉斯金字塔,即上述第二图像金字塔,进行重建,得到最终的结果。Exemplarily, the image processing device reconstructs the Laplacian pyramid after the texture migration of the reference image, that is, the above-mentioned second image pyramid, to obtain a final result.
需要说明的是,图像处理装置在构建目标蒙版图像或参考人脸图像的拉普拉斯金字塔之前,需要构建高斯金字塔。首先将原图像作为最底层图像G0(高斯金字塔的第0层),利用高斯核(5*5)对其进行卷积,然后对卷积后的图像进行下采样(去除偶数行和列)得到上一层图像G1。之后,将此图像作为输入,重复卷积和下采样操作得到更上一层图像,反复迭代多次,形成一个金字塔形的图像数据结构,即高斯金字塔。It should be noted that the image processing device needs to construct a Gaussian pyramid before constructing the Laplacian pyramid of the target mask image or the reference face image. First, the original image is used as the bottom image G0 (the 0th layer of the Gaussian pyramid), and it is convolved with a Gaussian kernel (5*5), and then the convolved image is down-sampled (removing even rows and columns) to get The previous layer image G1. Afterwards, this image is used as an input, and the convolution and downsampling operations are repeated to obtain a higher-level image, and iterated multiple times to form a pyramid-shaped image data structure, that is, a Gaussian pyramid.
在高斯金字塔的运算过程中,图像经过卷积和下采样操作会丢失部分高频细节信息。为描述这些高频信息,人们定义了拉普拉斯金字塔(Laplacian Pyramid,LP)。用高斯金字塔的每一层图像减去其上一层图像上采样并高斯卷积之后的预测图像,得到一系列的差值图像即为LP分解图像。During the operation of the Gaussian pyramid, some high-frequency detail information will be lost after the image undergoes convolution and downsampling operations. To describe these high-frequency information, people define the Laplacian Pyramid (Laplacian Pyramid, LP). The predicted image after upsampling and Gaussian convolution is subtracted from each layer image of the Gaussian pyramid, and a series of difference images are obtained, which are LP decomposition images.
示例性地,图像处理装置对进行图像融合后的拉普拉斯金字塔,从其顶层开始逐层从上至下按下式进行递推,可以恢复其对应的高斯金字塔,并最终可得到原图像G0。就是从最高层开始使用内插的方法。Exemplarily, the image processing device can restore the corresponding Gaussian pyramid from the top layer of the Laplacian pyramid after image fusion, and finally obtain the original image G0. It is the method of using interpolation from the highest level.
在一种可能的实现方式中,图像处理装置在进行图像融合之前,还可以先对第二图像金字塔的每层图像进行磨皮处理。In a possible implementation manner, before performing image fusion, the image processing device may first perform skin smoothing on each layer of the image in the second image pyramid.
示例性地,上述步骤203a2之后,本申请实施例提供的图像处理方法,还可以包括以 下步骤203b:Exemplarily, after the above step 203a2, the image processing method provided in the embodiment of the present application may also include the following step 203b:
步骤203b、图像处理装置按照预设半径及浮点相对精度,对上述第二图像金字塔的每层图像进行导向滤波磨皮处理。In step 203b, the image processing device performs guided filtering and microdermabrasion on each layer of the image in the second image pyramid according to the preset radius and the relative precision of the floating point.
示例性地,图像处理装置也可以按照预设半径及浮点相对精度,对上述N个目标蒙版子图像中的每个图像进行导向滤波磨皮处理。Exemplarily, the image processing device may also perform guided filter skin smoothing on each of the above N target mask sub-images according to a preset radius and a floating-point relative precision.
示例性地,图像处理装置可以对拉普拉斯金字塔的每一层设置合理的radius半径和eps浮点相对精度,并进行导向滤波磨皮,减轻脸部痘印、皱纹和过度不均等问题。Exemplarily, the image processing device can set a reasonable radius radius and eps floating-point relative precision for each layer of the Laplacian pyramid, and perform guided filtering for dermabrasion to alleviate problems such as acne marks, wrinkles, and excessive unevenness on the face.
如此,图像处理装置根据基于参考人脸图像构建的第一图像金字塔以及基于目标蒙版图像构建的第二图像金字塔,对第一图像中的人脸肤质进行优化,得到肤质较好的第二图像。In this way, the image processing device optimizes the face and skin quality in the first image according to the first image pyramid constructed based on the reference face image and the second image pyramid constructed based on the target mask image, and obtains the second image pyramid with better skin quality. Two images.
本申请实施例提供的图像处理方法,通过基于多层图像金字塔融合的人脸肤质迁移方法,将拍摄好的图像中存在瑕疵的人脸皮肤与肤质较好的图像进行图像融合,可以有效去除人像脸部的不佳纹理和过渡,使得成像后的人脸皮肤细腻、清晰,极大的提升了成像后的人脸的肤质。The image processing method provided in the embodiment of the present application, through the face and skin quality migration method based on multi-layer image pyramid fusion, performs image fusion on the human face skin with blemishes in the captured image and the image with better skin quality, which can effectively Remove the poor texture and transition of the face of the portrait, making the skin of the face after imaging delicate and clear, and greatly improving the skin quality of the face after imaging.
需要说明的是,本申请实施例提供的图像处理方法,执行主体可以为图像处理装置,或者该图像处理装置中的用于执行图像处理方法的控制模块。本申请实施例中以图像处理装置执行图像处理方法为例,说明本申请实施例提供的图像处理装置。It should be noted that, the image processing method provided in the embodiment of the present application may be executed by an image processing device, or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device executed by the image processing device is taken as an example to describe the image processing device provided in the embodiment of the present application.
需要说明的是,本申请实施例中,上述各个方法附图所示的。图像处理方法均是以结合本申请实施例中的一个附图为例示例性的说明的。具体实现时,上述各个方法附图所示的图像处理方法还可以结合上述实施例中示意的其它可以结合的任意附图实现,此处不再赘述。It should be noted that, in the embodiment of the present application, the above-mentioned methods are shown in the drawings. The image processing methods are described by way of example in conjunction with a drawing in the embodiment of the present application. During specific implementation, the image processing methods shown in the drawings of the above methods can also be implemented in combination with any other drawings shown in the above embodiments that can be combined, and will not be repeated here.
图3为实现本申请实施例提供的一种图像处理装置的可能的结构示意图,如图3所示,图像处理装置300包括:获取模块301和图像处理模块302;获取模块301,用于获取第一图像中的人脸区域的目标蒙版图像;获取模块301,还用于基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像;获取模块301,还用于针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;图像处理模块302,用于将获取模块301获取的参考人脸子图像的目标区域的图像与获取模块301获取的目标蒙版子图像的对应区域的图像进行融合,生成与第一图像对应的第二图像;其中,参考人脸图像集合包括多个经过肤质处理的 人脸蒙版图像;目标区域的人脸肤质值高于目标蒙版图像中与目标区域对应区域的人脸肤质值。FIG. 3 is a schematic diagram of a possible structure of an image processing device provided by an embodiment of the present application. As shown in FIG. 3 , the image processing device 300 includes: an acquisition module 301 and an image processing module 302; The target mask image of the face area in an image; the acquisition module 301 is also used to obtain the target mask image in the reference face image set that matches the target mask image based on the binarized face mask image of the target mask image. Reference face image; Acquisition module 301 is also used for image processing for reference face image and target mask image, obtains N reference face sub-images and N target mask sub-images; Image processing module 302 is used for The image of the target area of the reference face sub-image acquired by the acquisition module 301 is fused with the image of the corresponding area of the target mask sub-image acquired by the acquisition module 301 to generate a second image corresponding to the first image; wherein, the reference face image The collection includes a plurality of face mask images processed by skin texture; the face skin quality value of the target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
可选地,装置300还包括:生成模块303;获取模块301,还用于获取N个经过肤质处理的人脸图像,N为正整数;获取模块301,还用于提取N个人脸图像中每个人脸图像的五官信息,并根据五官信息构建二值化蒙版图像;一个人脸图像对应一个二值化蒙版图像;生成处理模块303,用于基于获取模块301获取的N个经过肤质处理的人脸图像,以及与获取模块301获取的每个人脸图像对应的二值化蒙版图像,生成参考人脸图像集合。Optionally, the device 300 also includes: a generation module 303; an acquisition module 301, which is also used to acquire N face images processed by skin texture, where N is a positive integer; and an acquisition module 301, which is also used to extract N facial images The facial features information of each face image, and construct a binarized mask image according to the facial features information; a face image corresponds to a binarized mask image; a generation processing module 303 is used to obtain the N processed skins based on the acquisition module 301 The qualitatively processed face images, and the binarized mask image corresponding to each face image acquired by the acquisition module 301, generate a set of reference face images.
可选地,装置300还包括:构建模块304;构建模块304,用于基于参考人脸图像,构建第一图像金字塔,第一图像金字塔包括N个参考人脸子图像;构建模块304,还用于基于目标蒙版图像,构建第二图像金字塔,第二图像金字塔包括N个目标蒙版子图像。Optionally, the device 300 also includes: a construction module 304; the construction module 304 is used to construct a first image pyramid based on the reference face image, and the first image pyramid includes N reference face sub-images; the construction module 304 is also used to Based on the target mask image, a second image pyramid is constructed, and the second image pyramid includes N target mask sub-images.
可选地,获取模块301,还用于提取参考人脸图像的第一特征点和目标蒙版图像的第二特征点;获取模块301,还用于基于第一特征点获取N个参考人脸子图像每张图像的二值化人脸蒙版图像,以及每张图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的顶点坐标;获取模块301,还用于基于第二特征点获取N个目标蒙版子图像每张图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标;图像处理模块302,具体用于基于顶点坐标,将参考人脸子图像的目标区域的图像与目标蒙版子图像的对应区域的图像进行融合。Optionally, the obtaining module 301 is also used to extract the first feature point of the reference face image and the second feature point of the target mask image; the obtaining module 301 is also used to obtain N reference face sub-points based on the first feature point The binarized face mask image of each image of the image, and the vertex coordinates of each triangle block in the M triangle blocks contained in the binarized face mask image of each image; the acquisition module 301 is also used for Acquire the vertex coordinates of each triangle block in the K triangle blocks contained in the binarized face mask image of each image of N target mask sub-images based on the second feature point; the image processing module 302 is specifically used based on Vertex coordinates, the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image.
可选地,装置还300包括:变换模块305;变换模块305,用于基于获取模块301获取的K个三角块的顶点坐标,对获取模块301获取的M个三角块的顶点坐标进行仿射变换;图像处理模块302,具体用于将经过仿射变换后的N个参考人脸子图像中第一参考人脸子图像的第一目标区域图像,与N个目标蒙版子图像中第一目标蒙版子图像的第二目标区域图像进行图像融合,得到N个处理后的目标蒙版子图像;图像处理模块302,具体还用于对N个经过处理后的目标蒙版子图像进行重建,生成第二图像;其中,第一参考人脸子图像为:N个参考人脸子图像中的任一个;第一目标蒙版子图像为N个目标蒙版子图像中与第一参考人脸子图像对应的目标蒙版子图像;第一目标区域图像为第一参考人脸子图像的第一目标区域的图像、且第一目标区域与第一目标蒙版子图像的第二目标区域对应的图像区域。Optionally, the device 300 further includes: a transformation module 305; the transformation module 305 is configured to perform affine transformation on the vertex coordinates of the M triangle blocks acquired by the acquisition module 301 based on the vertex coordinates of the K triangle blocks acquired by the acquisition module 301 The image processing module 302 is specifically used to combine the first target area image of the first reference face sub-image in the N reference face sub-images after affine transformation with the first target mask in the N target mask sub-images Image fusion is performed on the second target area image of the sub-images to obtain N processed target mask sub-images; the image processing module 302 is also specifically used to reconstruct the N processed target mask sub-images to generate the first N target mask sub-images. Two images; Wherein, the first reference human face sub-image is: any one of N reference human face sub-images; the first target mask sub-image is the target corresponding to the first reference human face sub-image in N target mask sub-images The mask sub-image; the first target area image is an image of the first target area of the first reference face sub-image, and the first target area is an image area corresponding to the second target area of the first target mask sub-image.
可选地,图像处理模块302,还用于按照预设半径及浮点相对精度,对构建模块304构建的第二图像金字塔的每层图像进行导向滤波磨皮处理。Optionally, the image processing module 302 is further configured to perform guided filter skin smoothing on each layer of the image of the second image pyramid constructed by the construction module 304 according to the preset radius and relative precision of the floating point.
本申请实施例中的图像处理装置可以是装置,也可以是终端中的部件、集成电路、或芯片。该装置可以是移动电子设备,也可以为非移动电子设备。示例性的,移动电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、可穿戴设备、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本或者个人数字助理(personal digital assistant,PDA)等,非移动电子设备可以为服务器、网络附属存储器(Network Attached Storage,NAS)、个人计算机(personal computer,PC)、电视机(television,TV)、柜员机或者自助机等,本申请实施例不作具体限定。The image processing apparatus in the embodiment of the present application may be a device, or may be a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant). assistant, PDA), etc., non-mobile electronic devices can be servers, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., this application Examples are not specifically limited.
本申请实施例中的图像处理装置可以为具有操作系统的装置。该操作系统可以为安卓(Android)操作系统,可以为iOS操作系统,还可以为其他可能的操作系统,本申请实施例不作具体限定。The image processing device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.
本申请实施例提供的图像处理装置能够实现图1至图2的方法实施例中图像处理装置实现的各个过程,为避免重复,这里不再赘述。The image processing device provided in the embodiment of the present application can implement various processes implemented by the image processing device in the method embodiments shown in FIG. 1 to FIG. 2 , and details are not repeated here to avoid repetition.
本实施例中各种实现方式具有的有益效果具体可以参见上述方法实施例中相应实现方式所具有的有益效果,为避免重复,此处不再赘述。For the beneficial effects of the various implementations in this embodiment, refer to the beneficial effects of the corresponding implementations in the foregoing method embodiments. To avoid repetition, details are not repeated here.
本申请实施例提供的图像处理装置,通过图像处理方法,将拍摄好的图像中存在瑕疵的人脸皮肤与肤质较好的图像进行图像融合,可以有效去除人像脸部的不佳纹理和过渡,使得成像后的人脸皮肤细腻、清晰,极大的提升了成像后的人脸的肤质。The image processing device provided in the embodiment of the present application, through the image processing method, performs image fusion of the human face skin with defects in the captured image and the image with better skin quality, which can effectively remove the bad texture and transition of the human face , making the skin of the human face after imaging delicate and clear, and greatly improving the skin quality of the human face after imaging.
可选地,如图4所示,本申请实施例还提供一种电子设备M00,包括处理器M01,存储器M02,存储在存储器M02上并可在所述处理器M01上运行的程序或指令,该程序或指令被处理器M01执行时实现上述图像处理方法实施例的各个过程,且能达到相同的技术效果,为避免重复,这里不再赘述。Optionally, as shown in FIG. 4 , the embodiment of the present application further provides an electronic device M00, including a processor M01, a memory M02, and programs or instructions stored in the memory M02 and operable on the processor M01, When the program or instruction is executed by the processor M01, each process of the above-mentioned image processing method embodiment can be achieved, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.
需要注意的是,本申请实施例中的电子设备包括上述所述的移动电子设备和非移动电子设备。It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
图5为实现本申请各个实施例的一种电子设备的硬件结构示意图。FIG. 5 is a schematic diagram of a hardware structure of an electronic device implementing various embodiments of the present application.
该电子设备100包括但不限于:射频单元101、网络模块102、音频输出单元103、输入单元104、传感器105、显示单元106、用户输入单元107、接口单元108、存储器109、以及处理器110等部件。The electronic device 100 includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110, etc. part.
本领域技术人员可以理解,电子设备100还可以包括给各个部件供电的电源(比如电池),电源可以通过电源管理系统与处理器110逻辑相连,从而通过电源管理系统实现管 理充电、放电、以及功耗管理等功能。图5中示出的电子设备结构并不构成对电子设备的限定,电子设备可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置,在此不再赘述。Those skilled in the art can understand that the electronic device 100 can also include a power supply (such as a battery) for supplying power to various components, and the power supply can be logically connected to the processor 110 through the power management system, so that the management of charging, discharging, and function can be realized through the power management system. Consumption management and other functions. The structure of the electronic device shown in FIG. 5 does not constitute a limitation to the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange different components, and details will not be repeated here. .
其中,输入单元104,用于获取第一图像中的人脸区域的目标蒙版图像;获处理器110,还用于基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像;处理器110,还用于针对参考人脸图像以及目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;处理器110用于将获取的参考人脸子图像的目标区域的图像与获取的目标蒙版子图像的对应区域的图像进行融合,生成与第一图像对应的第二图像;其中,参考人脸图像集合包括多个经过肤质处理的人脸蒙版图像;目标区域的人脸肤质值高于目标蒙版图像中与目标区域对应区域的人脸肤质值。Wherein, the input unit 104 is used to obtain the target mask image of the face area in the first image; the processor 110 is also used to obtain the reference face mask image based on the binarized face mask image of the target mask image. A reference face image matched with the target mask image in the image collection; Processor 110 is also used to perform image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target masks sub-image; the processor 110 is used to fuse the image of the target area of the acquired reference face sub-image with the image of the corresponding area of the acquired target mask sub-image to generate a second image corresponding to the first image; wherein, the reference The face image set includes a plurality of face mask images processed by skin texture; the face skin quality value of the target area is higher than the face skin quality value of the area corresponding to the target area in the target mask image.
如此,在获取到包含人脸的第一图像之后,获取第一图像中的人脸区域的目标蒙版图像,并基于目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与目标蒙版图像相匹配的参考人脸图像。之后,将参考人脸图像的目标区域的图像与目标蒙版图像的对应区域的图像进行融合,去除脸部不佳纹理和过度,恢复细腻、清晰的皮肤质感,得到肤质更好的第二图像,极大提升了成像后人脸皮肤的质量。In this way, after the first image containing the face is obtained, the target mask image of the face area in the first image is obtained, and the reference face image is obtained based on the binarized face mask image of the target mask image A set of reference face images that match the target mask image. After that, the image of the target area of the reference face image is fused with the image of the corresponding area of the target mask image to remove the bad texture and excess of the face, restore the delicate and clear skin texture, and obtain the second image with better skin quality. image, which greatly improves the quality of human face skin after imaging.
可选地,输入单元104,还用于获取N个经过肤质处理的人脸图像,N为正整数;处理器110,还用于提取N个人脸图像中每个人脸图像的五官信息,并根据五官信息构建二值化蒙版图像;一个人脸图像对应一个二值化蒙版图像;处理器110,用于基于输入单元104获取的N个经过肤质处理的人脸图像,以及与获取的每个人脸图像对应的二值化蒙版图像,生成参考人脸图像集合。Optionally, the input unit 104 is also used to acquire N facial images processed through skin quality, where N is a positive integer; the processor 110 is also used to extract facial features information of each facial image in the N facial images, and Construct a binarized mask image according to facial features information; a face image corresponds to a binarized mask image; processor 110 is used to obtain N processed face images based on the input unit 104, and obtain The binarized mask image corresponding to each face image in the image is used to generate a set of reference face images.
如此,图像处理装置可以基于经过肤质处理的参考人脸图像以及与其对应的二值化蒙版图像构建参考人脸图像集合,使得图像处理装置在获取到一张需要处理的图像后,能够基于该集合对图像进行处理。In this way, the image processing device can construct a set of reference face images based on the reference face image processed by the skin texture and the corresponding binarized mask image, so that after the image processing device obtains an image that needs to be processed, it can based on This collection performs processing on images.
可选地,处理器110,用于基于参考人脸图像,构建第一图像金字塔,第一图像金字塔包括N个参考人脸子图像;处理器110,还用于基于目标蒙版图像,构建第二图像金字塔,第二图像金字塔包括N个目标蒙版子图像。Optionally, the processor 110 is configured to construct a first image pyramid based on a reference face image, and the first image pyramid includes N reference face sub-images; the processor 110 is also configured to construct a second image pyramid based on a target mask image. An image pyramid, the second image pyramid includes N target mask sub-images.
可选地,处理器110,还用于提取参考人脸图像的第一特征点和目标蒙版图像的第二特征点;处理器110,还用于基于第一特征点获取N个参考人脸子图像每张图像的二值化人脸蒙版图像,以及每张图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的 顶点坐标;处理器110,还用于基于第二特征点获取N个目标蒙版子图像每张图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标;处理器110,具体用于基于顶点坐标,将参考人脸子图像的目标区域的图像与目标蒙版子图像的对应区域的图像进行融合。Optionally, the processor 110 is also used to extract the first feature point of the reference face image and the second feature point of the target mask image; the processor 110 is also used to obtain N reference face sub-points based on the first feature point The binarized face mask image of each image of the image, and the vertex coordinates of each triangle block in the M triangle blocks contained in the binarized face mask image of each image; the processor 110 is also used for Acquire the vertex coordinates of each of the K triangular blocks contained in the binarized face mask image of each of the N target mask sub-images based on the second feature point; the processor 110 is specifically used to Coordinates, the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image.
如此,图像处理装置在获取到第一图像金字塔每层图像的每个三角块的顶点坐标之后,可以基于每层图像的三角块对每层图像进行调整,使其包含的人物外观更加接近于目标蒙版图像所包含的人物外观。In this way, after the image processing device obtains the vertex coordinates of each triangular block of each layer image of the first image pyramid, it can adjust the image of each layer based on the triangular block of each layer image, so that the appearance of the characters contained in it is closer to the target The appearance of the character contained in the mask image.
可选地,处理器110,用于基于获取的K个三角块的顶点坐标,对获取的M个三角块的顶点坐标进行仿射变换;处理器110,具体用于将经过仿射变换后的N个参考人脸子图像中第一参考人脸子图像的第一目标区域图像,与N个目标蒙版子图像中第一目标蒙版子图像的第二目标区域图像进行图像融合,得到N个处理后的目标蒙版子图像;处理器110,具体还用于对N个经过处理后的目标蒙版子图像进行重建,生成第二图像;其中,第一参考人脸子图像为:N个参考人脸子图像中的任一个;第一目标蒙版子图像为N个目标蒙版子图像中与第一参考人脸子图像对应的目标蒙版子图像;第一目标区域图像为第一参考人脸子图像的第一目标区域的图像、且第一目标区域与第一目标蒙版子图像的第二目标区域对应的图像区域。Optionally, the processor 110 is configured to perform affine transformation on the acquired apex coordinates of the M triangular blocks based on the acquired apex coordinates of the K triangular blocks; the processor 110 is specifically configured to affinely transform the Image fusion of the first target area image of the first reference face sub-image in the N reference face sub-images with the second target area image of the first target mask sub-image in the N target mask sub-images to obtain N processing The processed target mask sub-image; the processor 110 is also specifically configured to reconstruct N processed target mask sub-images to generate a second image; wherein, the first reference face sub-image is: N reference people Any one of the face sub-images; the first target mask sub-image is the target mask sub-image corresponding to the first reference face sub-image in the N target mask sub-images; the first target area image is the first reference face sub-image The image of the first target area of , and the first target area is an image area corresponding to the second target area of the first target mask sub-image.
如此,图像处理装置可以基于图像特征点,对三角化后的N个参考人脸图像进行仿射变换,得到与目标参考人脸图像的人物相似的图像,之后,再进行肤质迁移,进而得到肤质较好的第二图像。In this way, the image processing device can perform affine transformation on the triangulated N reference face images based on the image feature points to obtain an image similar to the person in the target reference face image, and then perform skin texture transfer to obtain Second image with better skin quality.
可选地,图像处处理器110,还用于按照预设半径及浮点相对精度,对构处理器110构建的第二图像金字塔的每层图像进行导向滤波磨皮处理。Optionally, the image processing processor 110 is further configured to perform guided filtering skin smoothing on each layer of the image of the second image pyramid constructed by the structure processor 110 according to the preset radius and the relative precision of the floating point.
如此,图像处理装置根据基于参考人脸图像构建的第一图像金字塔以及基于目标蒙版图像构建的第二图像金字塔,对第一图像中的人脸肤质进行优化,得到肤质较好的第二图像。In this way, the image processing device optimizes the face and skin quality in the first image according to the first image pyramid constructed based on the reference face image and the second image pyramid constructed based on the target mask image, and obtains the second image pyramid with better skin quality. Two images.
本申请实施例提供的电子设备,通过图像处理方法,将拍摄好的图像中存在瑕疵的人脸皮肤与肤质较好的图像进行图像融合,可以有效去除人像脸部的不佳纹理和过渡,使得成像后的人脸皮肤细腻、清晰,极大的提升了成像后的人脸的肤质。The electronic device provided in the embodiment of the present application can effectively remove the bad texture and transition of the face of the portrait by image processing method to fuse the flawed human face skin in the captured image with the image with better skin quality, It makes the skin of the human face after imaging delicate and clear, and greatly improves the skin quality of the human face after imaging.
应理解的是,本申请实施例中,输入单元104可以包括图形处理器(Graphics Processing Unit,GPU)1041和麦克风1042,图形处理器1041对在视频捕获模式或图像捕获模式中 由图像捕获装置(如摄像头)获得的静态图片或视频的图像数据进行处理。显示单元106可包括显示面板1061,可以采用液晶显示器、有机发光二极管等形式来配置显示面板1061。用户输入单元107包括触控面板1071以及其他输入设备1072。触控面板1071,也称为触摸屏。触控面板1071可包括触摸检测装置和触摸控制器两个部分。其他输入设备1072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆,在此不再赘述。存储器109可用于存储软件程序以及各种数据,包括但不限于应用程序和操作系统。处理器110可集成应用处理器和调制解调处理器,其中,应用处理器主要处理操作系统、用户界面和应用程序等,调制解调处理器主要处理无线通信。可以理解的是,上述调制解调处理器也可以不集成到处理器110中。It should be understood that, in the embodiment of the present application, the input unit 104 may include a graphics processing unit (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processing unit 1041 is used by the image capturing device ( Such as the image data of the still picture or video obtained by the camera) for processing. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 107 includes a touch panel 1071 and other input devices 1072 . The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts, a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here. Memory 109 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 110 .
本申请实施例还提供一种可读存储介质,所述可读存储介质上存储有程序或指令,该程序或指令被处理器执行时实现上述图像处理方法实施例的各个过程,且能达到相同的技术效果,为避免重复,这里不再赘述。The embodiment of the present application also provides a readable storage medium, the readable storage medium stores a program or an instruction, and when the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is realized, and can achieve the same To avoid repetition, the technical effects will not be repeated here.
其中,所述处理器为上述实施例中所述的电子设备中的处理器。所述可读存储介质,包括计算机可读存储介质,如计算机只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、磁碟或者光盘等。Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer readable storage medium, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.
本申请实施例另提供了一种芯片,所述芯片包括处理器和通信接口,所述通信接口和所述处理器耦合,所述处理器用于运行程序或指令,实现上述图像处理方法实施例的各个过程,且能达到相同的技术效果,为避免重复,这里不再赘述。The embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above image processing method embodiment Each process can achieve the same technical effect, so in order to avoid repetition, it will not be repeated here.
应理解,本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。It should be understood that the chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.
需要说明的是,在本文中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外,需要指出的是,本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能,还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能,例如,可以按不同于所描述的次序来执行所描述的方法,并且还可以添加、省去、或组合各种步骤。另外,参照某些示例所描述的特征可在其他示例中被组合。It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台电子设备(可以是手机,计算机,服务器,或者网络设备等)执行本申请各个实施例所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of computer software products, which are stored in a storage medium (such as ROM/RAM, magnetic disk, etc.) , optical disc), including several instructions to make an electronic device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.
上面结合附图对本申请的实施例进行了描述,但是本申请并不局限于上述的具体实施方式,上述的具体实施方式仅仅是示意性的,而不是限制性的,本领域的普通技术人员在本申请的启示下,在不脱离本申请宗旨和权利要求所保护的范围情况下,还可做出很多形式,均属于本申请的保护之内。The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.
Claims (17)
- 一种图像处理方法,所述方法包括:An image processing method, the method comprising:获取第一图像中的人脸区域的目标蒙版图像;Obtain the target mask image of the face area in the first image;基于所述目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与所述目标蒙版图像相匹配的参考人脸图像;Based on the binarized face mask image of the target mask image, obtain a reference face image matching the target mask image in the set of reference face images;针对所述参考人脸图像以及所述目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;Perform image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images;将所述参考人脸子图像的目标区域的图像与所述目标蒙版子图像的对应区域的图像进行融合,生成与所述第一图像对应的第二图像;Fusing the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image to generate a second image corresponding to the first image;其中,所述参考人脸图像集合包括多个经过肤质处理的人脸蒙版图像;所述目标区域的人脸肤质值大于所述目标蒙版图像中与所述目标区域对应区域的人脸肤质值。Wherein, the reference face image set includes a plurality of face mask images processed by skin texture; the face skin quality value of the target area is greater than that of the people in the target mask image corresponding to the target area Face value.
- 根据权利要求1所述的方法,其中,所述基于所述目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与所述目标蒙版图像相匹配的参考人脸图像之前,所述方法还包括:The method according to claim 1, wherein said binary face mask image based on said target mask image acquires a reference face matching said target mask image in a set of reference face images Before the image, the method also includes:获取N个经过肤质处理的人脸图像,N为正整数;Obtain N face images processed by skin quality, where N is a positive integer;提取所述N个人脸图像中每个人脸图像的五官信息,并根据所述五官信息构建二值化蒙版图像;一个人脸图像对应一个二值化蒙版图像;extracting facial features information of each facial image in the N facial images, and constructing a binarized mask image according to the facial features information; one facial image corresponds to a binarized mask image;基于所述N个经过肤质处理的人脸图像,以及与每个人脸图像对应的二值化蒙版图像,生成所述参考人脸图像集合。The set of reference human face images is generated based on the N human face images subjected to skin texture processing and a binarized mask image corresponding to each human face image.
- 根据权利要求1所述的方法,其中,所述针对所述参考人脸图像以及所述目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像,包括:The method according to claim 1, wherein said image processing is performed on said reference face image and said target mask image to obtain N reference face sub-images and N target mask sub-images, comprising:基于所述参考人脸图像,构建第一图像金字塔,所述第一图像金字塔包括N个参考人脸子图像;Based on the reference face image, construct the first image pyramid, the first image pyramid includes N reference face sub-images;基于所述目标蒙版图像,构建第二图像金字塔,所述第二图像金字塔包括N个目标蒙版子图像。Based on the target mask image, construct a second image pyramid, where the second image pyramid includes N target mask sub-images.
- 根据权利要求1所述的方法,其中,所述将所述参考人脸子图像的目标区域的图像与所述目标蒙版子图像的对应区域的图像进行融合,生成与所述第一图像对应的第二图像,包括:The method according to claim 1, wherein the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image to generate an image corresponding to the first image. Second image, including:提取所述参考人脸图像的第一特征点和所述目标蒙版图像的第二特征点;extracting the first feature point of the reference face image and the second feature point of the target mask image;基于所述第一特征点获取所述N个参考人脸子图像每张图像的二值化人脸蒙版图像,以及所述每张图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的顶点坐标;Obtain the binarized face mask image of each of the N reference face sub-images based on the first feature point, and the M triangles contained in the binarized face mask image of each image Vertex coordinates of each triangle block in the block;基于所述第二特征点获取所述N个目标蒙版子图像每张图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标;Obtain the vertex coordinates of each triangle block in the K triangle blocks contained in the binarized face mask image of each image of the N target mask sub-images based on the second feature point;基于所述顶点坐标,将所述参考人脸子图像的目标区域的图像与所述目标蒙版子图像的对应区域的图像进行融合。Based on the vertex coordinates, the image of the target area of the reference face sub-image is fused with the image of the corresponding area of the target mask sub-image.
- 根据权利要求4所述的方法,其中,所述基于所述顶点坐标,将所述参考人脸子图像的目标区域的图像与所述目标蒙版子图像的对应区域的图像进行融合,包括:The method according to claim 4, wherein said merging the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image based on the vertex coordinates comprises:基于K个三角块的顶点坐标,对M个三角块的顶点坐标进行仿射变换;Based on the vertex coordinates of the K triangular blocks, affine transformation is performed on the vertex coordinates of the M triangular blocks;将经过仿射变换后的N个参考人脸子图像中第一参考人脸子图像的第一目标区域图像,与所述N个目标蒙版子图像中第一目标蒙版子图像的第二目标区域图像进行图像融合,得到N个处理后的目标蒙版子图像;The first target area image of the first reference face sub-image in the N reference face sub-images after the affine transformation, and the second target area of the first target mask sub-image in the N target mask sub-images Image fusion is carried out to obtain N processed target mask sub-images;对所述N个经过处理后的目标蒙版子图像进行重建,生成所述第二图像;Reconstructing the N processed target mask sub-images to generate the second image;其中,所述第一参考人脸子图像为:所述N个参考人脸子图像中的任一个;所述第一目标蒙版子图像为所述N个目标蒙版子图像中与所述第一参考人脸子图像对应的目标蒙版子图像;所述第一目标区域图像为所述第一参考人脸子图像的第一目标区域的图像、且所述第一目标区域与所述第一目标蒙版子图像的第二目标区域对应的图像区域。Wherein, the first reference face sub-image is: any one of the N reference face sub-images; the first target mask sub-image is one of the N target mask sub-images and the first A target mask sub-image corresponding to the reference face sub-image; the first target area image is an image of the first target area of the first reference face sub-image, and the first target area and the first target mask The image area corresponding to the second target area of the plate image.
- 根据权利要求3所述的方法,其中,所述基于所述目标蒙版图像构建第二图像金字塔之后,所述方法还包括:The method according to claim 3, wherein, after the second image pyramid is constructed based on the target mask image, the method further comprises:按照预设半径及浮点相对精度,对所述第二图像金字塔的每层图像进行导向滤波磨皮处理。According to the preset radius and the relative precision of the floating point, guide filter skinning treatment is performed on each layer image of the second image pyramid.
- 一种图像处理装置,所述装置包括:获取模块和图像处理模块;An image processing device, the device comprising: an acquisition module and an image processing module;所述获取模块,用于获取第一图像中的人脸区域的目标蒙版图像;The obtaining module is used to obtain the target mask image of the face area in the first image;所述获取模块,还用于基于所述目标蒙版图像的二值化人脸蒙版图像,获取参考人脸图像集合中与所述目标蒙版图像相匹配的参考人脸图像;The acquiring module is further configured to acquire, based on the binarized face mask image of the target mask image, a reference face image matching the target mask image in the set of reference face images;所述获取模块,还用于针对所述参考人脸图像以及所述目标蒙版图像进行图像处理,获得N个参考人脸子图像和N个目标蒙版子图像;The acquiring module is further configured to perform image processing on the reference face image and the target mask image to obtain N reference face sub-images and N target mask sub-images;所述图像处理模块,用于将所述获取模块获取的参考人脸子图像的目标区域的图像与所述获取模块获取的目标蒙版子图像的对应区域的图像进行融合,生成与所述第一图像对应的第二图像;The image processing module is used to fuse the image of the target area of the reference face sub-image acquired by the acquisition module with the image of the corresponding area of the target mask sub-image acquired by the acquisition module, and generate an image corresponding to the first mask sub-image. a second image corresponding to the image;其中,所述参考人脸图像集合包括多个经过肤质处理的人脸蒙版图像;所述目标区域的人脸肤质值高于所述目标蒙版图像中与所述目标区域对应区域的人脸肤质值。Wherein, the reference face image set includes a plurality of face mask images that have undergone skin texture processing; the face skin quality value of the target area is higher than that of the area corresponding to the target area in the target mask image. Human face skin quality value.
- 根据权利要求7所述的装置,其中,所述装置还包括:生成模块:The device according to claim 7, wherein the device further comprises: a generating module:所述获取模块,还用于获取N个经过肤质处理的人脸图像,N为正整数;The acquisition module is also used to acquire N face images processed through skin quality, where N is a positive integer;所述获取模块,还用于提取所述N个人脸图像中每个人脸图像的五官信息,并根据所述五官信息构建二值化蒙版图像;一个人脸图像对应一个二值化蒙版图像;The acquisition module is also used to extract facial features information of each facial image in the N facial images, and construct a binarized mask image according to the facial features information; one facial image corresponds to a binarized mask image ;所述生成模块,用于基于所述获取模块获取的N个经过肤质处理的人脸图像,以及与所述获取模块获取的每个人脸图像对应的二值化蒙版图像,生成所述参考人脸图像集合。The generating module is configured to generate the reference based on the N skin texture-processed face images obtained by the obtaining module, and a binary mask image corresponding to each face image obtained by the obtaining module. A collection of face images.
- 根据权利要求7所述的装置,其中,所述装置还包括:构建模块;The device according to claim 7, wherein the device further comprises: a building block;所述构建模块,用于基于所述参考人脸图像,构建第一图像金字塔,所述第一图像金字塔包括N个参考人脸子图像;The building module is configured to construct a first image pyramid based on the reference face image, and the first image pyramid includes N reference face sub-images;所述构建模块,还用于基于所述目标蒙版图像,构建第二图像金字塔,所述第二图像金字塔包括N个目标蒙版子图像。The construction module is further configured to construct a second image pyramid based on the target mask image, and the second image pyramid includes N target mask sub-images.
- 根据权利要求7所述的装置,其中,The apparatus according to claim 7, wherein,所述获取模块,还用于提取所述参考人脸图像的第一特征点和所述目标蒙版图像的第二特征点;The acquisition module is also used to extract the first feature point of the reference face image and the second feature point of the target mask image;所述获取模块,还用于基于所述第一特征点获取所述N个参考人脸子图像每张图像的二值化人脸蒙版图像,以及所述每张图像的二值化人脸蒙版图像所包含的M个三角块中每个三角块的顶点坐标;The acquisition module is also used to acquire the binarized face mask image of each of the N reference face sub-images based on the first feature point, and the binarized face mask image of each image. The vertex coordinates of each triangle block in the M triangle blocks contained in the plate image;所述获取模块,还用于基于所述第二特征点获取所述N个目标蒙版子图像每张图像的二值化人脸蒙版图像所包含的K个三角块中每个三角块的顶点坐标;The obtaining module is also used to obtain, based on the second feature point, the value of each of the K triangular blocks contained in the binarized face mask image of each of the N target mask sub-images vertex coordinates;所述图像处理模块,具体用于基于所述顶点坐标,将所述参考人脸子图像的目标区域的图像与所述目标蒙版子图像的对应区域的图像进行融合。The image processing module is specifically configured to fuse the image of the target area of the reference face sub-image with the image of the corresponding area of the target mask sub-image based on the vertex coordinates.
- 根据权利要求10所述的装置,其中,所述装置还包括:变换模块;The device according to claim 10, wherein the device further comprises: a transformation module;所述变换模块,用于基于所述获取模块获取的K个三角块的顶点坐标,对所述获 取模块获取的M个三角块的顶点坐标进行仿射变换;The transformation module is used to perform affine transformation on the vertex coordinates of the M triangle blocks obtained by the acquisition module based on the vertex coordinates of the K triangle blocks obtained by the acquisition module;所述图像处理模块,具体用于将经过仿射变换后的N个参考人脸子图像中第一参考人脸子图像的第一目标区域图像,与所述N个目标蒙版子图像中第一目标蒙版子图像的第二目标区域图像进行图像融合,得到N个处理后的目标蒙版子图像;The image processing module is specifically used to combine the first target area image of the first reference face sub-image in the N reference face sub-images after affine transformation with the first target in the N target mask sub-images Image fusion is performed on the second target area image of the mask sub-image to obtain N processed target mask sub-images;所述图像处理模块,具体还用于对所述N个经过处理后的目标蒙版子图像进行重建,生成所述第二图像;The image processing module is further configured to reconstruct the N processed target mask sub-images to generate the second image;其中,所述第一参考人脸子图像为:所述N个参考人脸子图像中的任一个;所述第一目标蒙版子图像为所述N个目标蒙版子图像中与所述第一参考人脸子图像对应的目标蒙版子图像;所述第一目标区域图像为所述第一参考人脸子图像的第一目标区域的图像、且所述第一目标区域与所述第一目标蒙版子图像的第二目标区域对应的图像区域。Wherein, the first reference face sub-image is: any one of the N reference face sub-images; the first target mask sub-image is one of the N target mask sub-images and the first A target mask sub-image corresponding to the reference face sub-image; the first target area image is an image of the first target area of the first reference face sub-image, and the first target area and the first target mask The image area corresponding to the second target area of the plate image.
- 根据权利要求9所述的装置,其中,The apparatus according to claim 9, wherein,所述图像处理模块,还用于按照预设半径及浮点相对精度,对所述构建模块构建的第二图像金字塔的每层图像进行导向滤波磨皮处理。The image processing module is further configured to perform guided filtering and microdermabrasion on each layer of the image of the second image pyramid constructed by the construction module according to the preset radius and the relative precision of the floating point.
- 一种电子设备,包括处理器、存储器及存储在所述存储器上并可在所述处理器上运行的程序或指令,所述程序或指令被所述处理器执行时实现如权利要求1至6中任一项所述的图像处理方法的步骤。An electronic device, comprising a processor, a memory, and a program or instruction stored on the memory and operable on the processor, when the program or instruction is executed by the processor, claims 1 to 6 are realized The steps of any one of the image processing methods.
- 一种可读存储介质,所述可读存储介质上存储程序或指令,所述程序或指令被处理器执行时实现如权利要求1至6中任一项所述的图像处理方法的步骤。A readable storage medium, storing programs or instructions on the readable storage medium, and implementing the steps of the image processing method according to any one of claims 1 to 6 when the programs or instructions are executed by a processor.
- 一种计算机程序产品,所述程序产品被至少一个处理器执行以实现如权利要求1至6中任一项所述的图像处理方法。A computer program product, the program product is executed by at least one processor to implement the image processing method according to any one of claims 1 to 6.
- 一种电子设备,其特征在于,所述电子设备用于执行如如权利要求1至6中任一项所述的图像处理方法。An electronic device, characterized in that the electronic device is configured to execute the image processing method according to any one of claims 1 to 6.
- 一种芯片,所述芯片包括处理器和通信接口,所述通信接口和所述处理器耦合,所述处理器用于运行程序或指令,实现如如权利要求1至6中任一项所述的图像处理方法。A chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, the processor is used to run a program or an instruction, and implement the method described in any one of claims 1 to 6 image processing method.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110653986.2A CN113469903A (en) | 2021-06-11 | 2021-06-11 | Image processing method and device, electronic equipment and readable storage medium |
CN202110653986.2 | 2021-06-11 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022258013A1 true WO2022258013A1 (en) | 2022-12-15 |
Family
ID=77869884
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/097859 WO2022258013A1 (en) | 2021-06-11 | 2022-06-09 | Image processing method and apparatus, electronic device and readable storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN113469903A (en) |
WO (1) | WO2022258013A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113469903A (en) * | 2021-06-11 | 2021-10-01 | 维沃移动通信有限公司 | Image processing method and device, electronic equipment and readable storage medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017026839A1 (en) * | 2015-08-12 | 2017-02-16 | 트라이큐빅스 인크. | 3d face model obtaining method and device using portable camera |
CN108509846A (en) * | 2018-02-09 | 2018-09-07 | 腾讯科技(深圳)有限公司 | Image processing method, device, computer equipment and storage medium |
CN109978754A (en) * | 2017-12-28 | 2019-07-05 | 广东欧珀移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
CN111445564A (en) * | 2020-03-26 | 2020-07-24 | 腾讯科技(深圳)有限公司 | Face texture image generation method and device, computer equipment and storage medium |
CN112215776A (en) * | 2020-10-20 | 2021-01-12 | 咪咕文化科技有限公司 | Portrait buffing method, electronic device and computer readable storage medium |
CN113469903A (en) * | 2021-06-11 | 2021-10-01 | 维沃移动通信有限公司 | Image processing method and device, electronic equipment and readable storage medium |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107146199B (en) * | 2017-05-02 | 2020-01-17 | 厦门美图之家科技有限公司 | Fusion method and device of face images and computing equipment |
CN108289141A (en) * | 2017-12-27 | 2018-07-17 | 努比亚技术有限公司 | A kind of the screen locking unlocking method and mobile terminal of mobile terminal |
CN109948526B (en) * | 2019-03-18 | 2021-10-29 | 北京市商汤科技开发有限公司 | Image processing method and device, detection equipment and storage medium |
CN111836058B (en) * | 2019-04-22 | 2023-02-24 | 腾讯科技(深圳)有限公司 | Method, device and equipment for playing real-time video and storage medium |
CN111583154B (en) * | 2020-05-12 | 2023-09-26 | Oppo广东移动通信有限公司 | Image processing method, skin beautifying model training method and related device |
-
2021
- 2021-06-11 CN CN202110653986.2A patent/CN113469903A/en active Pending
-
2022
- 2022-06-09 WO PCT/CN2022/097859 patent/WO2022258013A1/en unknown
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017026839A1 (en) * | 2015-08-12 | 2017-02-16 | 트라이큐빅스 인크. | 3d face model obtaining method and device using portable camera |
CN109978754A (en) * | 2017-12-28 | 2019-07-05 | 广东欧珀移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
CN108509846A (en) * | 2018-02-09 | 2018-09-07 | 腾讯科技(深圳)有限公司 | Image processing method, device, computer equipment and storage medium |
CN111445564A (en) * | 2020-03-26 | 2020-07-24 | 腾讯科技(深圳)有限公司 | Face texture image generation method and device, computer equipment and storage medium |
CN112215776A (en) * | 2020-10-20 | 2021-01-12 | 咪咕文化科技有限公司 | Portrait buffing method, electronic device and computer readable storage medium |
CN113469903A (en) * | 2021-06-11 | 2021-10-01 | 维沃移动通信有限公司 | Image processing method and device, electronic equipment and readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN113469903A (en) | 2021-10-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Jam et al. | A comprehensive review of past and present image inpainting methods | |
Wang et al. | An experiment-based review of low-light image enhancement methods | |
CN108765273B (en) | Virtual face-lifting method and device for face photographing | |
Guo et al. | LIME: Low-light image enhancement via illumination map estimation | |
Jiang et al. | Unsupervised decomposition and correction network for low-light image enhancement | |
CN110490828B (en) | Image processing method and system in video live broadcast | |
WO2019128508A1 (en) | Method and apparatus for processing image, storage medium, and electronic device | |
Guo et al. | Low-light image enhancement with regularized illumination optimization and deep noise suppression | |
Liu et al. | Knowledge-driven deep unrolling for robust image layer separation | |
EP3644599B1 (en) | Video processing method and apparatus, electronic device, and storage medium | |
KR101141643B1 (en) | Apparatus and Method for caricature function in mobile terminal using basis of detection feature-point | |
CN107749062B (en) | Image processing method and device | |
Shahamat et al. | Face recognition under large illumination variations using homomorphic filtering in spatial domain | |
CN111325657A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
CN111583154A (en) | Image processing method, skin beautifying model training method and related device | |
WO2020173024A1 (en) | Multi-gesture precise segmentation method for smart home scenario | |
Yang et al. | Joint-feature guided depth map super-resolution with face priors | |
CN112214773B (en) | Image processing method and device based on privacy protection and electronic equipment | |
CN111080670A (en) | Image extraction method, device, equipment and storage medium | |
WO2023066120A1 (en) | Image processing method and apparatus, electronic device, and storage medium | |
WO2022258013A1 (en) | Image processing method and apparatus, electronic device and readable storage medium | |
CN114187166A (en) | Image processing method, intelligent terminal and storage medium | |
Cui et al. | TPET: two-stage perceptual enhancement transformer network for low-light image enhancement | |
CN114862729A (en) | Image processing method, image processing device, computer equipment and storage medium | |
CN113379623B (en) | Image processing method, device, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22819610 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 23.04.2024) |