CN105913400A - Device for obtaining high-quality and real-time beautiful image - Google Patents

Abstract

The present invention discloses a device for obtaining a high-quality and real-time beautiful image. The device comprises an image obtaining module, an image integration beauty module, an image enhancement processing module, a skin area shading processing module and an image fusion module. The image obtaining module is configured to input video and extract the video according to the frame and take as a picture sequence; the output end of the image obtaining module is connected with the image integration beauty module, the image enhancement processing module, the skin area shading processing module and the image fusion module; and the output ends of the image enhancement processing module and the skin area shading processing module are connected with the image fusion module. Through adoption of GPU hardware acceleration characteristic, the device for obtaining a high-quality and real-time beautiful image is able to process a plurality of submodule of the system so as to solve the problem that the efficiency is low when a CPU is used; and moreover, each used submodule provided by the system can be applied to the GPU acceleration processing so as to obtain instant effect presentation while ensuring the real-time efficiency and apply to the special efficacy processing of a single image.

Description

A kind of device obtaining high-quality and real-time U.S. face

Technical field

The present invention relates to a kind of device obtaining high-quality and real-time U.S. face.

Background technology

U.S. face method is generally made up of multiple basic steps, the removal of noise, the people such as including image peripheral illumination and human body skin Body skin detection, Face datection, speckle dispelling, skin-whitening, image co-registration etc..

Image denoising as most basic be also a most important ring, follow-up algorithm process is had vital effect, at present The algorithm of denoising is more, generally includes Gaussian smoothing, bilinear filter is smooth, average filter smooth, three-dimensional based on Block-matching Denoising scheduling algorithm, the performance of the most each algorithm and effect all have difference in various degree, and each have limitation in various degree Property, this is for there being large effect on the algorithms selection of application scenarios.Such as, average filter smoothing efficiency is the highest, but often The more details such as face such as hair, eyelashes, eyebrow filtering out image has the region of obvious characteristic；Gaussian smoothing is in filtering half When footpath is less efficiency higher but time filter radius is bigger efficiency the lowest；Bilinear filter is smooth can be effectively maintained image edge details But mixed color phenomenon can be produced；Three-dimensional Denoising Algorithm based on Block-matching well can process white Gaussian noise but efficiency is the lowest. Therefore, selecting a kind of algorithm that can balance in efficiency and effect, the result to total algorithm is a kind of challenge, needs again simultaneously Well it is applicable to corresponding application scenarios.

Skin detection and Face datection, it is usually required mainly for process is skin and human face region, it is ensured that in skin and non-skin region Seam crossing is without obvious artificial trace, and this is determined by application scenarios.Face datection algorithm when for high-definition picture Performance is relatively low, and this carries out pyramid often caused by needs and successively detects human face region, is not particularly suited for other skin area simultaneously Such as arm, shoulder, neck etc., therefore select a kind of applicable skin and the detection algorithm of human face region, be provided simultaneously with higher Performance, particularly important.

Speckle removing, acne removing refers mainly to the regional area of skin area and processes, it is common practice to artificial hand selection dispels in region, uncomfortable Automatically processing for image.

Skin-whitening and image enhaucament can have multiple processing mode, including index mapping, logarithmic mapping, power function mapping, line Property intensification, Auto Laves etc., its purpose is that region dark in image carries out enhancement process, strengthens details and presents effect, with Time preferably retain the variations in detail of brighter areas, prevent whiting.

Generally speaking, just can complete owing to U.S. face method typically requires being mutually linked of several submethods, the most each submethod Faint change also can produce large effect to last effect, selects suitable submethod and is effectively combined in together, simultaneously Meet that efficiency is higher and effect presents immediately so that it is having more preferable application prospect, this is the problem that the present invention will solve.

Summary of the invention

It is an object of the invention to overcome the deficiencies in the prior art, it is provided that a kind of device obtaining high-quality and real-time U.S. face, borrow Help the hardware-accelerated characteristic of GPU, can solve use CPU time inefficiency problem, simultaneity factor propose and use each Individual submodule can well be applied to GPU acceleration and process, while guaranteed efficiency is in real time, can obtain instant effect in Existing.

It is an object of the invention to be achieved through the following technical solutions: a kind of device obtaining high-quality and real-time U.S. face, it Module, image enhancement processing module, skin area shade processing module and image is beautified including image collection module, image entirety Fusion Module；Described image collection module is used for input video, described video is pressed frame and extracts and as sequence of pictures, image The outfan of acquisition module beautifies module, image enhancement processing module, skin area shade processing module even respectively with image entirety Connecing, image entirety beautifies outfan and the image co-registration mould of module, image enhancement processing module, skin area shade processing module Block connects；

Described image entirety is beautified module and is included:

First image transform subblock: for the RGB color of input picture is transformed into YUV color space, simultaneously protect Stay UV passage；

Judge with filtering submodule: include judging unit, integrogram unit and box filter unit；Described judging unit judges Whether the size of sampling window is more than the threshold value preset: if it is send the image that image transform subblock is changed to integrogram Unit, otherwise sends to box filter unit；Integrogram unit is for generating the integrogram of luminance picture, described integrogram bag Include the integrogram of first order and quadratic term, the more all pixels in image are processed one by one, centered by each pixel In window, calculate average and the variance of all pixels in this window respectively；Described box filter unit carries out box filter to image Ripple；

Image denoising submodule: receive from judging and the output of filtering submodule, to each pixel, obtaining based on this picture After the average of the window centered by element and variance, carry out smothing filtering according to the average obtained and variance；

Image sharpening submodule: for image is sharpened process, the grain details of image is compensated lifting；

Image synthon module: the UV passage that before image and the denoising after sharpening, RGB is converted to is merged into YUV image；

Second image transform subblock: for the YUV image that image synthon module obtains is converted back RGB color；

Described image enhancement processing module is used for using nonlinear images to strengthen, and image carries out overall whitening and processes, by carrying The mode of luminance detail is kept to realize while rising the dark portion details of image, first by the range of image normalization to [0,1], Then the method using exponential function to map processes；

Described skin area shade processing module includes detection of skin regions unit and shade processing unit, described skin area Detector unit uses threshold process, is then divided into skin area when the pixel value of image is more than statistical value, is otherwise non-during detection Skin area, obtains the Preliminary detection of a skin area；Described statistical value is the skin to multiple images and non-skin region The class value carrying out statistical classification and obtain；Described shade processing unit is for after obtaining the shade of skin area, and employing refers to Further micronization processes made by shade by the Gaussian Blur determining window size；

Described image co-registration module is for beautifying module, image enhancement processing module, skin area shade processing in image entirety After module all completes to process, according to the shade of the skin area obtained respectively to the image after overall whitening and the figure after global de-noising As merging pixel-by-pixel；

Described image entirety beautifies module, image enhancement processing module, skin area shade processing module for using based on GPU Hardware-accelerated independently execute pixel-by-pixel three module.

The conversion formula of the first described image transform subblock is as follows:

$\left(\begin{array}{c}Y\\ U\\ V\end{array}\right)=\left(\begin{array}{ccc}0.299& 0.587& 0.114\\ -0.169& -0.331& 0.5\\ 0.5& -0.419& -0.081\end{array}\right)\left(\begin{array}{c}R\\ G\\ B\end{array}\right)$

The conversion formula of the second described image transform subblock is as follows:

$\left(\begin{array}{c}R\\ G\\ B\end{array}\right)=\left(\begin{array}{ccc}1& 0& 1.402\\ 1& -0.344& -0.714\\ 1& 1.772& 0\end{array}\right)\left(\begin{array}{c}Y\\ U\\ V\end{array}\right).$

Multiple in the class value that the described skin to multiple images and non-skin region carry out statistical classification and obtain are 1000 ?.

The invention has the beneficial effects as follows:

(1) present invention proposes a kind of should be readily appreciated that and the Image denoising algorithm process framework of excellent performance, utilizes based on image integration The rapid image denoising scheme of figure, it is ensured that calculate speed unrelated with the size of sampling window, the most well removes picture noise such as Details is kept while speckle.

(2) present invention proposes a kind of more efficient detection of skin regions process framework, and use the most slightly walks detection and carries out thin again The multi-step immediate processing method changed, can seam crossing implementation nothing in skin and non-skin region be stitched into very well.

(3) present invention uses nonlinear images Enhancement Method, and image carries out overall enhancing.

(4) present invention is by the hardware-accelerated characteristic of GPU, processes multiple submodules of native system, can solve to use Inefficiency problem during CPU, native system proposes and each submodule of using can well be applied at GPU acceleration simultaneously Reason, while guaranteed efficiency is in real time, can obtain instant effect and present, and effect when being applied to video capture presents immediately.

Accompanying drawing explanation

Fig. 1 is present configuration block diagram.

Detailed description of the invention

Technical scheme is described in further detail below in conjunction with the accompanying drawings:

As it is shown in figure 1, a kind of device obtaining high-quality and real-time U.S. face, it includes image collection module, image integrated beauty Change module, image enhancement processing module, skin area shade processing module and image co-registration module；Described image collection module For input video, described video being pressed frame and extracts and as sequence of pictures, the outfan of image collection module is whole with image respectively Body beautifies module, image enhancement processing module, the connection of skin area shade processing module, and image entirety beautifies module, image increases Strong processing module, the outfan of skin area shade processing module are connected with image co-registration module；

Described image entirety is beautified module and is included human body skin area smoothing processing, as face and the speckle of other skin area, The region erasing that other impact such as nevus is attractive in appearance, and the environment noise such as under-exposure etc. that global illumination is introduced, other later stage processes Other noise such as salt-pepper noise etc. of Shi Yinjin, damages place including the dropout in picture or video transmitting procedure, encoding and decoding Reason, blocking artifact etc., including:

First image transform subblock: for the RGB color of input picture is transformed into YUV color space, this is main It is to operate in luminance picture to improve efficiency, retains UV passage simultaneously；The conversion of the first described image transform subblock Formula is as follows:

$\left(\begin{array}{c}Y\\ U\\ V\end{array}\right)=\left(\begin{array}{ccc}0.299& 0.587& 0.114\\ -0.169& -0.331& 0.5\\ 0.5& -0.419& -0.081\end{array}\right)\left(\begin{array}{c}R\\ G\\ B\end{array}\right)$

Due to the calculating dependency of neighborhood territory pixel upper and lower before and after existing in the process of generation integrogram, this can relate to a up-downgoing Overhead time.In our scheme, a threshold value can be set, if the size of sampling window exceedes this threshold value, then use long-pending Component, otherwise uses box filtering.

Judge with filtering submodule: include judging unit, integrogram unit and box filter unit；Described judging unit judges Whether the size of sampling window is more than the threshold value preset: if it is send the image that image transform subblock is changed to integrogram Unit, otherwise sends to box filter unit；Integrogram unit is for generating the integrogram of luminance picture, described integrogram bag Including the integrogram of first order and quadratic term, wherein iterative formula is as follows

sum_i,j=sum_i,j-1+sum_i-1,j-sum_i-1,j-1+f_i,j；

${\mathrm{sumsq}}_{i,j}={\mathrm{sumsq}}_{i,j-1}+{\mathrm{sumsq}}_{i-1,j}-{\mathrm{sumsq}}_{i-1,j-1}+f_{i,j}^2;$

In formula, sum represent directly and, sumsq represents that quadratic sum, f represent brightness value, preserves two width integrograms obtained above Picture；Can be used for the quick filter based on window of next step image denoising, even if also do not interfere with calculating when sampling window is bigger Efficiency；

The most again all pixels in image are processed one by one, in the window centered by each pixel, calculate this window respectively In the average of all pixels and variance；

$E=\frac{{\mathrm{sum}}_{i+N,j+N}-{\mathrm{sum}}_{i+N,j-N-1}-{\mathrm{sum}}_{i-N-1,j+N}+{\mathrm{sum}}_{i-N-1,j-N-1}}{(2*N+1)*(2*N+1)}$

$Esq=\frac{{\mathrm{sumsq}}_{i+N,j+N}-{\mathrm{sumsq}}_{i+N,j-N-1}-{\mathrm{sumsq}}_{i-N-1,j+N}+{\mathrm{sumsq}}_{i-N-1,j-N-1}}{(2*N+1)*(2*N+1)}$

VAR=Esq-E²

In formula, E represents that average, VAR represent variance；I, j represent respectively relative to the image upper left corner vertically and horizontally Coordinate, N represents windows radius.

Described box filter unit carries out box filtering to image；The described computing formula using box to filter is as follows:

$E=\frac{\underset{m=-N}{\overset{N}{\Σ}}\underset{n=-N}{\overset{N}{\Σ}}{f}_{i+m,j+n}}{(2*N+1)*(2*N+1)}$

$\begin{array}{c}Esq=\frac{\underset{m=-N}{\overset{N}{\Σ}}\underset{n=-N}{\overset{N}{\Σ}}{f}_{i+m,j+n}^2}{(2*N+1)*(2*N+1)}\\ VAR=Esq-E^2\end{array}$

In formula, E represents that average, VAR represent variance；M, n represent vertically and horizontally relative with current pixel position respectively Distance.

Luminance picture is carried out denoising, owing to the eyes of people are more more sensitive than carrier chrominance signal to luminance signal, therefore believes in brightness Number noise can be more sensitive than chrominance signal noise, removing after luminance signal noise, the eyes of people may feel that significantly Change, does not interferes with total quality on the basis of improving computational efficiency simultaneously.

Image denoising submodule: receive from judging and the output of filtering submodule, to each pixel, obtaining based on this picture After the average of the window centered by element and variance, carry out smothing filtering according to the average obtained and variance；Its principle epigraph is the most flat Sliding, then the variance yields obtained is closer to 0, thus this pixel value is then closer to average E.The correcting mode of described smothing filtering For:

$k=\frac{VAR}{VAR+\mathrm{\β}+\mathrm{\ϵ}}$

f_i,j=E* (1-k)+f_i,j*k

In formula, β represents the parameter value of regulation, and its value is the biggest, and the degree representing smooth is the biggest, then the noise removed is the biggest；ε is One close to 0 decimal, its purpose is to prevent dividend is exception when 0；From the formula that pixel value is corrected In can draw, when regulation parameter value the biggest time, this pixel value is closer to E.

Image sharpening submodule: for image is sharpened process, the grain details of image is compensated lifting；Process public affairs Formula is:

$S_{i,j}=f_{i,j}+\frac{f_{i,j-1}+f_{i,j+1}+f_{i-1,j}+f_{i+1,j}-f_{i,j}*4}{4}*\mathrm{\α}$

In formula, S represents the image after sharpening, and α represents the degree of sharpening, the i.e. tribute to pixel value of the 4-neighborhood Laplce's gradient Offering percentage ratio, α value is the biggest, and sharpness is the biggest；

Image synthon module: the UV passage that before image and the denoising after sharpening, RGB is converted to is merged into YUV image；

Second image transform subblock: for the YUV image that image synthon module obtains is converted back RGB color；Institute The conversion formula of the second image transform subblock stated is as follows:

$\left(\begin{array}{c}R\\ G\\ B\end{array}\right)=\left(\begin{array}{ccc}1& 0& 1.402\\ 1& -0.344& -0.714\\ 1& 1.772& 0\end{array}\right)\left(\begin{array}{c}Y\\ U\\ V\end{array}\right)$

Described image enhancement processing module is used for using nonlinear images to strengthen, and image carries out overall whitening and processes, by carrying The mode of luminance detail is kept to realize while rising the dark portion details of image, first should be by the scope of image normalization to [0,1] In, the method then using exponential function to map processes:

$f_{i,j}=f_{i,j}^p$

In formula, p represents the degree of whitening；

Described skin area shade processing module includes detection of skin regions unit and shade processing unit:

Skin area, compared with non-skin region, is generally of the color being easier to distinguish, especially compared with dark black region, Application scenarios for U.S. face, it is usually required mainly for differentiation is skin and the hair of people, eyebrow, eyelashes, the subregion of eyes of people Process.Use threshold process, first mark off skin and noncutaneous region of roughly selecting, the RGB statistics of the skin area of usual people Value for [a, b, c], wherein a, b, c be the skin to 1000 images with non-skin region carry out that statistical classification obtains one Class value, is then divided into skin area when the pixel value of image is more than statistical value during detection, is otherwise non-skin region, obtains one The Preliminary detection of individual skin area；

Described shade processing unit, for after obtaining the shade of skin area, uses the Gaussian Blur of specified window size to screening Further micronization processes made by cover；After obtaining the shade of skin area, shade is made further micronization processes, and can not be direct Participate in image co-registration, otherwise have obvious artificial trace at the skin area of image and the seam crossing in non-skin region.Specifically, The Gaussian Blur using specified window size carries out shade processing, and the two-dimentional formula of Gaussian function is as follows:

$f(x,y)=\frac{1}{2{\mathrm{\π\σ}}^2}{e}^{-\frac{x^2+y^2}{2{\mathrm{\σ}}^2}};$

In formula, x, y represent respectively vertically and horizontally with the relative distance of current operation pixel, σ represents standard deviation.

Described image co-registration module is for beautifying module, image enhancement processing module, skin area shade processing in image entirety After module all completes to process, according to the shade of the skin area obtained respectively to the image after overall whitening and the figure after global de-noising As merging pixel-by-pixel；Integration formula is:

Final_i,j=B_i,j*(1-α_i.j)+F_i,j*α_i.j

In formula, B represents the image after global de-noising, and F represents the image after overall whitening, and α represents the screening of the skin area obtained Cover, Final represents image co-registration result.

In whole scheme, owing to the process step related to is more, but each step can independently execute pixel-by-pixel, therefore CPU can not in real time in the case of, use based on GPU hardware-accelerated, can process in real time, wherein mobile terminal use OpenGL ES accelerates.Described image entirety beautify module, image enhancement processing module, skin area shade processing module for use based on Hardware-accelerated independently execute pixel-by-pixel three module of GPU.

Claims (3)

1. the device obtaining high-quality and real-time U.S. face, it is characterised in that: it includes that image collection module, image are overall Beautify module, image enhancement processing module, skin area shade processing module and image co-registration module；Described Image Acquisition mould Block is used for input video, described video is pressed frame and extracts and as sequence of pictures；The outfan of image collection module respectively with image Entirety beautifies module, image enhancement processing module, the connection of skin area shade processing module, and image entirety beautifies module, image Enhancement process module, the outfan of skin area shade processing module are connected with image co-registration module；

Described image entirety is beautified module and is included:

First image transform subblock: for the RGB color of input picture is transformed into YUV color space, simultaneously protect Stay UV passage；

Judge with filtering submodule: include judging unit, integrogram unit and box filter unit；Described judging unit judges Whether the size of sampling window is more than the threshold value preset: if it is send the image that image transform subblock is changed to integrogram Unit, otherwise sends to box filter unit；Integrogram unit is for generating the integrogram of luminance picture, described integrogram bag Include the integrogram of first order and quadratic term, the more all pixels in image are processed one by one, centered by each pixel In window, calculate average and the variance of all pixels in this window respectively；Described box filter unit carries out box filter to image Ripple；

Image denoising submodule: receive from judging and the output of filtering submodule, to each pixel, obtaining based on this picture After the average of the window centered by element and variance, carry out smothing filtering according to the average obtained and variance；

Image sharpening submodule: for image is sharpened process, the grain details of image is compensated lifting；

Image synthon module: the UV passage that before image and the denoising after sharpening, RGB is converted to is merged into YUV image；

Second image transform subblock: for the YUV image that image synthon module obtains is converted back RGB color；

Described image enhancement processing module is used for using nonlinear images to strengthen, and image carries out overall whitening and processes, by carrying The mode of luminance detail is kept to realize while rising the dark portion details of image, first by the range of image normalization to [0,1], Then the method using exponential function to map processes；

Described skin area shade processing module includes detection of skin regions unit and shade processing unit, described skin area Detector unit uses threshold process, is then divided into skin area when the pixel value of image is more than statistical value, is otherwise non-during detection Skin area, obtains the Preliminary detection of a skin area；Described statistical value is the skin to multiple images and non-skin region The class value carrying out statistical classification and obtain；Described shade processing unit is for after obtaining the shade of skin area, and employing refers to Further micronization processes made by shade by the Gaussian Blur determining window size；

Described image co-registration module is for beautifying module, image enhancement processing module, skin area shade processing in image entirety After module all completes to process, according to the shade of the skin area obtained respectively to the image after overall whitening and the figure after global de-noising As merging pixel-by-pixel；

Described image entirety beautifies module, image enhancement processing module, skin area shade processing module for using based on GPU Hardware-accelerated independently execute pixel-by-pixel three module.

A kind of device obtaining high-quality and real-time U.S. face the most according to claim 1, it is characterised in that: described The conversion formula of one image transform subblock is as follows:

$\left(\begin{array}{c}Y\\ U\\ V\end{array}\right)=\left(\begin{array}{ccc}0.299& 0.587& 0.114\\ -0.169& -0.331& 0.5\\ 0.5& -0.419& -0.081\end{array}\right)\left(\begin{array}{c}R\\ G\\ B\end{array}\right)$

The conversion formula of the second described image transform subblock is as follows:

$\left(\begin{array}{c}R\\ G\\ B\end{array}\right)=\left(\begin{array}{ccc}1& 0& 1.402\\ 1& -0.344& -0.714\\ 1& 1.772& 0\end{array}\right)\left(\begin{array}{c}Y\\ U\\ V\end{array}\right).$

A kind of device obtaining high-quality and real-time U.S. face the most according to claim 1, it is characterised in that: described is right Multiple in the class value that the skin of multiple images and non-skin region carry out statistical classification and obtain are 1000.