WO2022135579A1

WO2022135579A1 - Skin color detection method and device, mobile terminal, and storage medium

Info

Publication number: WO2022135579A1
Application number: PCT/CN2021/141287
Authority: WO
Inventors: 杨敏
Original assignee: 百果园技术(新加坡)有限公司; 杨敏
Priority date: 2020-12-25
Filing date: 2021-12-24
Publication date: 2022-06-30
Also published as: CN112686965A

Abstract

Embodiments of the present application provide a skin color detection method and device, a mobile terminal, and a storage medium. The method comprises: detecting face data in image data, and with reference to the face data, separating head data matching a head contour; substituting a color value of the image data into a first mapping function to calculate a first target probability of the image data being skin data; generating a second mapping function with reference to a brightness value of the head data; substituting a brightness value of the image data into the second mapping function to calculate a second target probability of the image data being skin data; and detecting skin data in the image data by combining the first target probability and the second target probability.

Description

Skin color detection method, device, mobile terminal and storage medium

This application claims the priority of the Chinese patent application with application number 202011566266.4 filed with the China Patent Office on December 25, 2020, the entire contents of the above application are incorporated into this application by reference.

technical field

The embodiments of the present application relate to the technical field of computer vision, for example, to a skin color detection method, device, mobile terminal, and storage medium.

Background technique

With the rapid development of mobile Internet and mobile terminals, video data in mobile terminals has become a common information carrier in human activities, such as live broadcasts, video calls, etc. They contain a large amount of information about objects and become a way for people to obtain original information from the outside world. one.

For these video data, users usually choose beauty, and skin color detection is the basis of beauty, which determines that beauty can realize real-time processing of video data, and also affects the effect of beauty.

The mainstream skin color detection can be divided into skin color detection based on color space, skin color detection based on machine learning classification, and skin color detection based on deep learning image segmentation.

Among them, skin color detection based on color space has a wide range of applications in the field of real-time stream processing due to its fast and efficient characteristics, but the accuracy is low. High accuracy, but performance often becomes a bottleneck in real-time stream processing, and it is difficult to apply to limited devices such as mobile terminals.

SUMMARY OF THE INVENTION

In a first aspect, the embodiments of the present application provide a method for detecting skin color, including:

Detecting face data in the image data, and dividing the head data matching the head contour with reference to the face data;

Substituting the color value of the image data into the first mapping function to calculate the first target probability that the image data belongs to skin data;

generating a second mapping function with reference to the brightness value of the head data, the second mapping function being used to identify skin data based on the brightness value;

Substituting the luminance value of the image data into the second mapping function to calculate the second target probability that the image data belongs to skin data;

Skin data is detected in the image data in combination with the first target probability and the second target probability.

In a second aspect, the embodiments of the present application also provide a skin color detection device, including:

an image data detection module, configured to detect face data in the image data, and divide head data matching the head contour with reference to the face data;

a first target probability calculation module, configured to substitute the color value of the image data into a first mapping function to calculate the first target probability that the image data belongs to skin data;

a mapping function generation module, configured to generate a second mapping function with reference to the brightness value of the head data, where the second mapping function is used to identify skin data based on the brightness value;

A second target probability calculation module, configured to substitute the brightness value of the image data into the second mapping function to calculate the second target probability that the image data belongs to skin data;

A skin detection module configured to detect skin data in the image data in combination with the first target probability and the second target probability.

In a third aspect, an embodiment of the present application further provides a mobile terminal, where the mobile terminal includes:

at least one processor;

memory, arranged to store at least one program,

When the at least one program is executed by the at least one processor, the at least one processor implements the skin color detection method according to the first aspect.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the skin color detection according to the first party is implemented method.

Description of drawings

1 is a flowchart of a skin color detection method provided in Embodiment 1 of the present application;

FIG. 2 is an example diagram of a header data provided in Embodiment 1 of the present application;

3 is a schematic diagram of a first mapping function provided in Embodiment 1 of the present application;

4 is a schematic diagram of a second mapping function provided in Embodiment 1 of the present application;

5 is a grayscale distribution diagram corresponding to a probability distribution of skin color detection provided in Embodiment 1 of the present application;

6 is a schematic structural diagram of a skin color detection device provided in Embodiment 2 of the present application;

FIG. 7 is a schematic structural diagram of a mobile terminal according to Embodiment 3 of the present application.

Detailed ways

The present application will be described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all the structures related to the present application.

Example 1

1 is a flowchart of a skin color detection method provided in Embodiment 1 of the present application. This embodiment can perform skin color detection in combination with color values and brightness values. The method can be performed by a skin color detection device, which can be performed by software and/or hardware implementation, which can be configured in mobile terminals, such as mobile phones, tablet computers, smart wearable devices (such as smart watches, smart glasses, etc.), etc., including the following steps:

Step 101: Detect face data in the image data, and divide head data matching the head contour with reference to the face data.

In this embodiment, the video data waiting for skin color detection generally refers to video data generated, transmitted or played in a real-time service scenario.

Generally speaking, skin color detection can be performed on the video data in the mobile terminal that generates the video data. At this time, the camera of the mobile terminal can be turned on, and the camera can collect the video data.

In addition to performing skin color detection on the video data in the mobile terminal that generates the video data, skin color detection may also be performed on the video data on the mobile terminal that plays the video data, which is not limited in this embodiment.

For example, in a live broadcast business scenario, the video data waiting for skin color detection may refer to the video data used to carry the live broadcast content. The mobile terminal logged in by the host user generates the video data, and the video data is distributed to at least one The device logged in by the viewer user plays the video. At this time, the video data is usually detected by skin color on the mobile terminal logged in by the host user.

For another example, in the business scenario of a video call, the video data waiting for skin color detection may refer to the video data used to carry the content of the call, and the mobile terminal logged in by the user who initiates the call generates video data, and sends the video data to at least A device logged in by a user who is invited to a call is played. At this time, skin color detection is usually performed on the video data on the mobile terminal logged in by the user who initiated the call.

For another example, in a video conference business scenario, the video data waiting for skin color detection may refer to video data used to carry conference content, the mobile terminal logged in by the user who is speaking generates video data, and the video data is sent to at least A device logged in by a user participating in the conference plays the video. At this time, the mobile terminal logged in by the user who is usually speaking performs skin color detection on the video data.

The video data waiting for skin color detection may also refer to video data generated in a service scenario with low real-time requirements, such as a short video, which is not limited in this embodiment.

In different business scenarios, video data is usually collected from users. Therefore, the multi-frame image data of the video data usually contains the user's character image. The so-called character image can refer to the pixels in the image data used to represent the character.

The character image includes at least face data (pixels) for representing a human face, and may also include hair data (pixels) for representing hair and body data (pixels) for representing a body , limb data (pixel points) used to represent limbs, etc., which are not limited in this embodiment of the present application.

Wherein, for face data, body data, limb data, etc., skin data (pixel points) for representing skin may be included.

In this embodiment, face detection can be performed on multiple frames of image data respectively, and face data included in the image data can be identified. Using the face data as a high-quality anchor point, high-quality skin data can be captured.

In some cases, face data is represented by face key points, that is, given face data, key regions of the face are located, including eyebrows, eyes, nose, mouth, facial contours, and so on.

In other cases, a range of a specified shape can be generated based on the key points of the face, and the range is used to represent the face data, wherein the shape includes a rectangle, an ellipse, etc., and the range of the rectangle can also be called a face frame.

Face detection can be performed on multi-frame image data using the following methods:

1. Use artificial extraction features, such as haar features, use features to train classifiers, and use classifiers for face detection.

2. Inherit face detection from general object detection algorithms, for example, use Faster R-CNN to detect faces.

3. Convolutional neural network using cascade structure, for example, Cascade CNN (Cascade Convolutional Neural Network), MTCNN (Multi-task Cascaded Convolutional Networks, multi-task convolutional neural network).

These methods for implementing face detection can be integrated in the modules of the application, and the modules of the application are directly called to perform face detection on image data. These methods for implementing face detection can also be integrated in the SDK (Software Development Kit, software development kit). ), the SDK is used as the assembly data of the application, the application can request the SDK to perform face detection on multi-frame image data, the SDK detects the face data in the image data, and returns the face data to the application.

If the application is a system application supporting the camera, the system application usually provides API (Application Program Interface, Application Program Interface) for these methods of implementing face detection, as a face detection interface, for example, in the Android (Android) system , provides two face detection interfaces android.media.FaceDetector and android.media.FaceDetector.Face, in iOS system, provides two face detection interfaces AVCaptureMetadataOutput and CIDetector.

In response to this situation, you can call the face detection interface provided by the camera to request to detect the face data in the multi-frame image data collected by the camera, and receive one or more face frames returned by the face detection interface. The face frame is a rectangular frame , which can be used to frame the face data, that is, the face data is located in the face frame.

For the face detection interface provided by the system application, there is hardware support, and the face frame is calibrated based on a few key points of the face (2 key points of the face), the speed is fast, the performance consumption is very low, and the accuracy can meet the requirements.

Since the face data can reflect the spatial position information of the user's head to a certain extent, in this embodiment, the position, shape and other information of the face data can be referred to, and the image data can be divided into a matching head contour. Head data, thereby improving the accuracy of expressing parts such as heads and faces.

For the head contour, it is different when the face data faces the front and the side, and can present parts such as hair, face shape, ears, nose, mouth, etc. Correspondingly, the head data can reflect these parts in shape.

In one embodiment, in order to reduce the complexity of fitting the head data, increase the speed of calculation, and reduce the time-consuming of calculation, the shape of the head data can be simplified and fitted to a standard shape, for example, the Shape fitting is oval, circle, etc.

In the first embodiment of the present application, as shown in FIG. 2 , the head data includes face data 202 and hair data 203 , wherein the face data 202 is similar to the face contour in the head contour, mainly including the face , the hair data 203 is similar to the hair contour in the head contour, and mainly contains hair.

On the one hand, as shown in FIG. 2 , with reference to the shape, position and other information of the face data 201 , face data 202 matching the face contour in the head contour is divided into the image data.

In an example, for the face data recalled by a certain camera, if the face data 201 is framed by a face frame, the face data 202 meets the following conditions:

The shape of the face data is oval;

The focus of the face data is the midpoint of the frame edge corresponding to the width of the face frame;

The length of the short axis of the face data is equal to the width of the face frame;

The length of the long axis of the face data is equal to n (n is a positive number, such as 2) times the length of the target side. The target side is the hypotenuse corresponding to the first candidate side and the second candidate side as the right-angle side. The first candidate side is the width of the face frame with a specified first ratio, and the second candidate side is the height of the face frame with a specified second ratio.

In this example, let the width of the face frame be w, the height be h, and the four focal points are A(x ₁ , y ₁ ), B(x ₂ , y ₂ ), C(x ₃ , y ₃ ), D(x ₄ , y ₄ ), then the focus of the face data is F ₁ ((x ₁ +x ₂ )/2, (y ₁ +y ₂ )/2), F ₂ ((x ₃ +x _{4 )} )/2, (y ₃ +y ₄ )/2), the length of the short axis of the face data is 2b=w, and the length of the long axis of the face data is

Wherein, i is the first ratio, which belongs to a positive number less than or equal to 1, such as 0.5, and j is the second ratio, which belongs to a positive number less than or equal to 1, such as 0.5.

On the other hand, as shown in FIG. 2 , with reference to information such as the shape and position of the face data 201 , the image data is divided into hair data 203 matching the hair outline in the head outline.

In the head data, the hair data 203 surrounds the face data 202, and the two are independent of each other and do not overlap.

In an example, for the face data recalled by a certain camera, if the face data 201 is framed by a face frame, the hair data 203 meets the following conditions:

The shape of the hair data is an ellipse;

The focus of the hair data is the midpoint of the frame edge corresponding to the width of the face frame;

The length of the short axis of the hair data is equal to the width m in the face frame (m is a positive number, indicating that the hair data is magnified by m times the face data) times;

The length of the long axis of the hair data is equal to n×m times the length of the target side. The target side is the hypotenuse corresponding to the first candidate side and the second candidate side as the right-angle side, and the first candidate side is the face frame. The width takes the specified first ratio, and the second candidate edge is the height of the face frame and takes the specified second ratio.

In this example, let the width of the face frame be w, the height be h, and the four focal points are A(x ₁ , y ₁ ), B(x ₂ , y ₂ ), C(x ₃ , y ₃ ), D(x ₄ , y ₄ ), then the focus of the hair data is F ₁ ((x ₁ +x ₂ )/2, (y ₁ +y ₂ )/2), F ₂ ((x ₃ +x ₄ ) /2, (y ₃ +y ₄ )/2), the length of the short axis of the hair data is 2b=mw, and the length of the long axis of the hair data is

Among them, i is the first ratio, which belongs to a positive number less than or equal to 1, such as 0.5, and j is the first ratio, which belongs to a positive number less than or equal to 1, such as 0.5.

In an example, m can be a preset empirical value, which is a constant, and m can also be an adaptive variable, which is positively related to the area of the face data, that is, the larger the area of the face data, the larger the area of the hair data. Large, at this time, the value of m is larger, on the contrary, the area of the face data is smaller, and the area of the hair data is also smaller, at this time, the value of m is smaller.

The above-mentioned head data and its division method are only examples, and the face data recalled by different operating systems and different versions of the cameras are different. When implementing the embodiments of the present application, other head data and other head data may be set according to the actual situation. The division manner is not limited in this embodiment of the present application. In addition, in addition to the above header data and its division manner, those skilled in the art may also adopt other header data and its division manner according to actual needs, which are not limited in this embodiment of the present application.

In general, face data may include forehead, eyes, nose, mouth, cheeks, ears and other parts, and hair data may include hair and other parts. In the case of simplified face data, in addition to forehead, In addition to parts such as eyes, nose, mouth, cheeks, ears, etc., it may also include data such as part of hair, background, etc. In addition to parts such as hair, hair data may also include part of forehead, part of ears, part of two parts, neck, background data.

In addition, if there are multiple face data in the image data, the area of the multiple face data is counted respectively. If the face data is framed by a face frame, the width of the face frame is w and the height is h, then the face data The area S=w×h.

Compare the area of at least one face data, extract the k face data with the largest area (k is a positive integer, such as 3), and refer to the k face data with the largest area to divide the head data that matches the head contour, Due to the imaging characteristics of the camera and the method of recalling face data, the color description of face data with a larger area is more accurate. Selecting the k face data with the largest area to divide the head data that matches the head contour can reduce the In the case of the amount of calculation, the accuracy of the brightness value of the subsequent statistical skin data is guaranteed.

For example, the k face data with the largest area may be the face data sorted according to the area from large to small, and the k face data may be the top k face data in this sorting. However, it is not limited to such a selection method of selecting face data based on the size of the area.

In the case where there are multiple face data in the image data, head data matching the head contour may also be divided for all face data, which is not limited in this embodiment.

Step 102: Substitute the color value of the image data into the first mapping function to calculate the first target probability that the image data belongs to the skin data.

In this embodiment, a first mapping function can be set by counting color values of pixels in different samples such as skin data and non-skin data, and the first mapping function can be used to identify skin data based on color values, that is, the first mapping function can be used to identify skin data based on color values. The input of a mapping function is the color value, and the output is the probability that the current data belongs to skin data. At this time, the probability that the pixels in different samples are skin data under a certain color value can be counted, so that these color values and their probabilities can be calculated. fit to the first mapping function.

In order to maintain generality, the first mapping function can refer to the skin data of users of different races, different ages, and different skin tones, and can also refer to the skin data under different lighting conditions. Therefore, the first mapping function is relatively broad and relatively accurate. higher.

In general, the probability that the current data in the first mapping function belongs to skin data is a continuous value, such as [0-1], and in some cases, the probability that the current data in the first mapping function belongs to skin data is a discrete value, such as 0 1, which is not limited in this embodiment.

In an example, as shown in FIG. 3 , in the coordinate system where the first mapping function is located, the abscissa (horizontal axis) is the color value (X), and the ordinate (vertical axis) is the probability that the current data belongs to skin data ( P).

The first mapping function includes a first color mapping segment (the abscissa is [x ₀ , x ₁ ]), a second color mapping segment (the abscissa is [x ₁ , x ₂ ]), and a third color mapping segment connected in sequence (the abscissa is [x ₂ , x ₃ ]), wherein the probability of belonging to skin data in the first color mapping segment increases from 0 to 1, the probability of belonging to skin data in the second color mapping segment is 1, and the third color mapping segment The probability that a segment belongs to skin data drops from 1 to 0.

In an example, the second color map segment belongs to a line segment, and the first color map segment and the third color map segment belong to a curve, which can be fitted by using a polynomial function or the like.

The above-mentioned first mapping function is only an example. When implementing the embodiments of the present application, other first mapping functions may be set according to actual conditions. For example, the first mapping function includes a first color mapping segment and a second color mapping segment connected in sequence. , where the probability that the current data in the first color mapping segment belongs to skin data increases from 0 to 1, and the probability that the current data in the second color mapping segment belongs to skin data decreases from 1 to 0. At this time, the mapping function is a one-dimensional quadratic Function expressions such as equations, etc., are not limited in the embodiments of the present application. In addition, in addition to the above-mentioned first mapping function, those skilled in the art may also adopt other first mapping functions according to actual needs, which are not limited in this embodiment of the present application.

In different color spaces, color values have different chrominance components, and a corresponding mapping function can be set for each color component. ), blue chrominance component (B), then you can set the mapping function for the red chrominance component (R), set the mapping function for the green chrominance component (G), set the mapping function for the blue chrominance component (B), and for example , in the YCbCr color space, with blue chrominance components (Cb) and red chrominance components (Cr), you can set the mapping function for the blue chrominance component (Cb) and set the mapping function for the red chrominance component (Cr), and many more.

In this embodiment, the first mapping function may be applied to detect the skin color of the image data, so as to detect the degree to which each pixel in the image data represents the skin data in the dimension of color value.

In one example, the color value of each pixel of the image data can be marked on the horizontal axis in the coordinate system where the first mapping function is located. If the color value of the pixel in the image data is outside the first mapping function, the pixel The probability that the point belongs to the skin data is 0. If the color value of the pixel in the image data is within the first mapping function, the probability corresponding to the color value on the vertical axis can be found through the first mapping function, as the pixel belongs to the skin data. The first target probability of .

In some cases, the abscissa of the mapping function is the color value, and the ordinate is the probability that the pixel belongs to the skin data. In different color spaces, the color value has different chromaticity components, and each color component has a corresponding mapping function, For the same pixel, different probabilities can be calculated.

In view of this situation, the color value of each pixel in the image data can be queried, and the color value can be substituted into the corresponding first mapping function to map it as the candidate probability that the pixel belongs to the skin data under the color value. Based on these candidate probabilities , the first target probability that the pixel belongs to the skin data can be calculated by means of averaging, summing, product, linear fusion (that is, summing after configuring the weight), and these first target probabilities are reflected in the dimension of color value, ROI (region of interest, region of interest) area of skin color.

In one example, for the YCbCr color space, the image data has a blue chrominance component Cb, a red chrominance component Cr.

On the one hand, the color value of the blue chrominance component Cb is substituted into the first mapping function corresponding to the blue chrominance component Cb, so as to be mapped as the blue probability that the pixel belongs to the skin data under the blue chrominance component Cb, as the candidate probability

On the other hand, the color value of the red chrominance component Cr is substituted into the first mapping function corresponding to the red chrominance component Cr, so as to be mapped as the red probability that the pixel belongs to the skin data under the red chrominance component Cr, as the candidate probability

At this point, the blue probability can be calculated

with red probability

The product between , as the first target probability that the pixel belongs to the skin data

Step 103 , generating a second mapping function with reference to the luminance value of the header data.

If face data is detected in the image data, the face data can be corrected to obtain head data, and the brightness value is counted in the range of the head data. The brightness value of the brightness value has a high confidence in the brightness value of the skin data , which can be used to generate a second mapping function for the current user to improve the accuracy of the second mapping function for the current user, that is, in this embodiment, the second mapping function can be set by the brightness value of the header data, and the second mapping function The function can be used to identify skin data based on the brightness value, that is, the input of the mapping function is the brightness value, and the output is the probability that the current data belongs to skin data. At this time, the pixels of the head data can be counted as skin data under a certain brightness value. to fit these luminance values and their probabilities to the second mapping function.

In general, the probability that the current data in the second mapping function belongs to skin data is a continuous value, such as [0-1], and in some cases, the probability that the current data in the second mapping function belongs to skin data is a discrete value, such as 0 1, which is not limited in this embodiment.

In an example, as shown in FIG. 4 , in the coordinate system where the second mapping function is located, the abscissa (horizontal axis) is the luminance value (X), and the ordinate (vertical axis) is the probability that the current data belongs to skin data ( P).

The second mapping function includes a first luminance mapping segment (the abscissa is [x ₀ , x ₁ ]) and a second luminance mapping segment (the abscissa is [x ₁ , x ₂ ]) connected in sequence, wherein the first luminance The probability that the current data in the mapping segment belongs to skin data increases from 0 to 1, and the probability that the current data in the second luminance mapping segment belongs to skin data is 1.

In an example, the second luminance mapping segment belongs to a line segment, and the first luminance mapping segment belongs to a curve, which can be fitted by using a polynomial function or the like.

The above-mentioned second mapping function is only an example. When implementing the embodiments of the present application, other second mapping functions may be set according to actual conditions. For example, the second mapping function includes a first luminance mapping segment and a second luminance mapping segment connected in sequence. , the third brightness mapping segment, wherein, the first brightness mapping segment refers to the pixel point fitting in the hair data, and the probability of it belonging to the skin data rises from 0 to 1, and the second brightness mapping segment refers to the skin color range in the face data. Pixel point fitting, the probability that it belongs to skin data is 1, and the third brightness mapping segment refers to the pixel point fitting in the non-skin color range (such as glasses, teeth, eyebrows, etc.) in the face data, and its probability of belonging to skin data is from 1 drops to 0, etc., which are not limited in the embodiments of the present application. In addition, in addition to the above-mentioned second mapping function, those skilled in the art may also adopt other second mapping functions according to actual needs, which are not limited in this embodiment of the present application.

In a fitting method, if the head data includes face data matching the face contour in the head contour and hair data matching the hair contour in the head contour, in the face data, the pixel point The probability of belonging to skin data is high, and in hair data, the probability of pixels belonging to skin data is low. If the brightness value of the face data is greater than the brightness value of the hair data, then the skin color detection can be suppressed by using the brightness value.

On the one hand, the brightness value of the hair data is counted as the first reference brightness value. In the brightness value-probability coordinate system, the probability that the pixel corresponding to the first reference brightness value belongs to the skin data is 0.

On the other hand, the luminance values of the face data and hair data are counted as the second reference luminance value, and in the luminance value-probability coordinate system, the probability that the pixel corresponding to the second reference luminance value belongs to the skin data is 1.

At this time, in the first luminance mapping section, the first reference luminance value is used as the starting point and the second reference luminance value is used as the end point to fit the curve, and in the second luminance mapping section, the second reference luminance value is used as the starting point to fit the curve straight line, thereby generating the second mapping function.

In addition, for the case where the face data is an ellipse, it is more complicated to calculate the brightness value of the pixels in the ellipse, while the face data is mostly framed by a face frame, and the face frame is a rectangle. The operation of counting the brightness values is relatively simple, and the difference between the two is relatively small. Therefore, the face data can be replaced by the face data, while maintaining a certain accuracy, the simplicity of the operation is improved, the calculation speed is increased, and the calculation time is reduced. time consuming.

In an example, on the one hand, the brightness value of the hair data is counted as the first reference brightness value. In the brightness value-probability coordinate system, the probability that the pixel corresponding to the first reference brightness value belongs to the skin data is 0, and the other In one aspect, the brightness values of the face data and the hair data are counted as the second reference brightness value. In the brightness value-probability coordinate system, the probability that the pixel corresponding to the second reference brightness value belongs to the skin data is 1.

Step 104: Substitute the brightness value of the image data into the second mapping function to calculate the second target probability that the image data belongs to the skin data.

In this embodiment, the second mapping function can be applied to detect the skin color of the image data, so as to detect the degree to which each pixel point in the image data represents the skin data in the dimension of brightness value.

In an example, the brightness value of each pixel in the image data can be queried, and the brightness value can be substituted into the second mapping function to be mapped as the second target probability that the pixel belongs to the skin data, that is, in the second mapping function. The horizontal axis marks the brightness value of each pixel of the image data in the coordinate system at If the brightness value of the middle pixel is within the second mapping function, the probability corresponding to the brightness value on the vertical axis can be found through the second mapping function as the second target probability that the pixel belongs to the skin data.

Step 105, combining the first target probability and the second target probability to detect skin data in the image data.

For the same frame of image data, methods such as averaging, summing, multiplying, linear fusion (that is, summing after configuring weights) can be used to combine the first target probability in the dimension of color value and the probability in the dimension of luminance value. The second target probability is to perform skin color detection on the target data, so as to detect the degree to which the pixel points represent the skin data in the image data.

In an example, it can be considered that both the color value and the brightness value are important. At this time, for the same pixel in the image data, the product between the first target probability and the second target probability is calculated, as the pixel belongs to the skin data. The third target probability of , when the first target probability in the dimension of color value and the second target probability in the dimension of brightness value are both larger, the value of the third target probability multiplied by the two is larger, and in the color value When one of the first target probability in the dimension of value and the second target probability in the dimension of luminance value is larger and the other is smaller, the value of the third target probability multiplied by the two is smaller.

In addition, if the face data is not detected before, skin data can be detected in the image data based on the color space. The method based on the color space is simple to calculate, so the calculation speed is fast, the time-consuming is small, and the color of multiple skin data is counted. The value can maintain a high accuracy on the whole to meet the requirements of prior knowledge.

Since the color space of the image data is different, for different color spaces, you can use the method corresponding to the color space to detect the skin data in the face data, or convert the image data to the specified color space, use and specify The method corresponding to the color space of 1 detects skin data in the face data, which is not limited in this embodiment.

In an example, for the RGB color space, when the color value of a pixel meets the following conditions, it can be considered that the pixel is skin data (that is, the probability that the pixel belongs to skin data is 1), and the color value of the pixel does not meet the following conditions When the conditions are met, it can be considered that the pixel is not skin data (that is, the probability that the pixel belongs to skin data is 0):

R>95, G>40, B>20, R>G, R>B

(Max(R,G,B)-Min(R,G,B))>15

Abs(R-G)＞5

Among them, Max represents the maximum value, Min represents the minimum value, and Abs represents the absolute value.

Among them, R represents the red chrominance component, G represents the green chrominance component, and B represents the blue chrominance component.

In another example, for the YCbCr color space, when the color value of a pixel meets the following conditions, it can be considered that the pixel is skin data (that is, the probability that the pixel belongs to skin data is 1), and the color value of the pixel does not meet the following conditions. Under the following conditions, it can be considered that the pixel is not skin data (that is, the probability that the pixel belongs to skin data is 0):

c _x =109.38, c _y =152.02, θ = 2.53,

a=25.39, b=14.03

Among them, Cb represents the blue chrominance component, and Cr represents the red chrominance component.

In order for those skilled in the art to better understand the embodiments of the present application, the following examples are used to describe the skin color detection method in this embodiment.

As shown in Figure 5, for the same frame of image data, the left side is the grayscale distribution corresponding to the probability distribution obtained by using the color space-based skin color detection, and the right side is the probability distribution corresponding to the skin color detection obtained by applying this embodiment. Grayscale distribution, in which, the higher the grayscale of a pixel (more white), the higher the probability that the pixel belongs to skin data, and the lower the grayscale (more black), the lower the probability that the pixel belongs to skin data. .

Due to the combination of factors such as illumination, skin color, and hair color when collecting image data, the color value of the non-skin color area of the hair and face is relatively close to the skin color area. There is a big difference in the brightness value between the non-skin area and the skin area of the face.

Skin color detection based on color space considers hair, non-skin color areas of the face (such as eyebrows, eyes, nose, mouth, etc.), and background (such as electric lights, etc.) to belong to skin data, and this embodiment can be well excluded. Hair, non-skin color areas of the face (such as eyebrows, eyes, nose, mouth, etc.), background (such as electric lights, etc.), greatly improve the accuracy of skin color detection, and can be very good in subsequent beauty processing (such as microdermabrasion) Protect hair, non-skinned areas of the face (such as eyebrows, eyes, nose, mouth, etc.), background (such as electric lights, etc.).

In this embodiment, face data is detected in the image data, the head data matching the head contour is divided with reference to the face data, and the color value of the image data is substituted into the first mapping function to calculate that the image data belongs to the skin data The first target probability of The second target probability of the skin data, combined with the first target probability and the second target probability to detect skin data in the image data, the face data can reflect the spatial position information of the user's head to a certain extent, and the face data is used as the basic description Head data can ensure the accuracy of head data. Using head data as an anchor point, high-quality skin data and its close non-skin data can be captured, and more accurate skin data can be counted in current business scenarios. The brightness value is used to adaptively generate the second mapping function, which can ensure the accuracy of skin color detection by the second mapping function in the current business scene. The dimension of skin color detection improves the accuracy of skin color detection. In addition, operations such as face detection, drawing head data, statistics of color values and brightness values, generation and application of the first mapping function and the second mapping function are relatively simple. It is simple, requires less computation, is fast, and takes less time. When the performance of devices such as mobile terminals is limited, skin color detection can be performed on video data in real time.

In one embodiment, under the condition of improving the accuracy of skin color detection, some data on color values that are similar to skin data can be excluded, such as hair, non-skin color areas of the face, background, etc., so that in the subsequent beauty processing (such as Microdermabrasion) to protect data whose color is similar to skin data, slow down or avoid the loss of clarity of data similar to skin data due to beautification processing (such as microdermabrasion), so as to ensure the quality of beautification processing.

The method embodiments are described as a series of action combinations for the sake of simple description, but those skilled in the art should know that the embodiments of the present application are not limited by the described sequence of actions, because according to the embodiments of the present application , certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the actions involved in the embodiments described in the specification are not necessarily required by the embodiments of the present application.

Embodiment 2

6 is a structural block diagram of a skin color detection device provided in Embodiment 2 of the present application, which may include the following modules:

Image data detection module 601 is set to detect face data in image data, divide the head data that matches with head contour with reference to described face data;

The first target probability calculation module 602 is configured to substitute the color value of the image data into the first mapping function to calculate the first target probability that the image data belongs to skin data;

A mapping function generation module 603, configured to generate a second mapping function with reference to the brightness value of the head data, where the second mapping function is used to identify skin data based on the brightness value;

The second target probability calculation module 604 is configured to substitute the brightness value of the image data into the second mapping function to calculate the second target probability that the image data belongs to skin data;

The skin detection module 605 is configured to detect skin data in the image data by combining the first target probability and the second target probability.

In an embodiment of the present application, the image data detection module 601 includes:

a face detection interface calling module, configured to call the face detection interface provided by the camera to request to detect the face data in the image data collected by the camera;

The face frame receiving module is configured to receive at least one face frame returned by the face detection interface, where the face frame is used to frame face data.

In an embodiment of the present application, the head data includes face data and hair data; the image data detection module 601 includes:

a face data division module, configured to divide face data matching the face contour in the head contour with reference to the face data;

The hair data division module is configured to divide the hair data matching the hair contour in the head contour with reference to the face data, and the hair data surrounds the face data.

In an example of the embodiment of the present application, if the face data is framed by a face frame, the face data meets the following conditions:

The shape of the face data is an ellipse;

The length of the short axis of the face data is equal to the width in the face frame;

The length of the long axis of the face data is equal to n times the length of the target side. The first candidate edge is the width of the face frame with a specified first ratio, and the second candidate edge is the height of the face frame with a specified second ratio.

In an example of the embodiment of the present application, if the face data is framed by a face frame, the hair data meets the following conditions:

The shape of the hair data is an ellipse;

The length of the short axis of the hair data is equal to m times the width in the face frame, and the m is positively correlated with the area of the face data;

The length of the long axis of the hair data is equal to n×m times the length of the target side, and the target side is the hypotenuse when the first candidate side and the second candidate side are respectively used as right-angled sides of a right-angled triangle, so The first candidate edge is the width of the face frame with a specified first ratio, and the second candidate edge is the height of the face frame with a specified second ratio.

an area statistics module, configured to count the areas of a plurality of the face data respectively in response to the presence of a plurality of face data in the image data;

The area selection module is configured to divide the head data matching the head contour with reference to the k pieces of the face data with the largest area.

In an embodiment of the present application, the abscissa of the first mapping function is the color value, and the ordinate is the probability that the current data belongs to skin data;

The first mapping function includes a first color mapping section, a second color mapping section, and a third color mapping section that are connected in sequence, and the probability that the current data in the first color mapping section belongs to skin data increases from 0 to 1, so The probability that the current data in the second color mapping section belongs to skin data is 1, and the probability that the current data in the third color mapping section belongs to skin data decreases from 1 to 0.

In an embodiment of the present application, the first target probability calculation module 602 includes:

a color value query module, configured to query the color value of the pixel in the image data;

a candidate probability mapping module, configured to substitute the color value into the first mapping function to map the pixel point to the candidate probability of skin data under the color value;

The target probability combining module is configured to calculate the first target probability that the pixel point belongs to the skin data based on the candidate probability.

In an embodiment of the present application, the candidate probability mapping module includes:

The blue probability mapping module is configured to substitute the color value of the blue chrominance component into the first mapping function corresponding to the blue chrominance component, so as to map the pixel point to the skin under the blue chrominance component The blue probability of the data, as the candidate probability;

A red probability mapping module, configured to substitute the color value of the red chrominance component into the first mapping function corresponding to the red chrominance component, so as to map the pixel points to belong to skin data under the red chrominance component The red probability of , as the candidate probability;

The target probability combination module includes:

The first probability product calculation module is configured to calculate the product between the blue probability and the red probability as the first target probability that the pixel belongs to the skin data.

In an embodiment of the present application, the abscissa of the second mapping function is the brightness value, and the ordinate is the probability that the current data belongs to skin data;

The second mapping function includes a first luminance mapping segment and a second luminance mapping segment connected in sequence, the probability that the current data in the first luminance mapping segment belongs to skin data increases from 0 to 1, and the second luminance mapping segment The probability that the current data belongs to skin data is 1.

In an embodiment of the present application, the head data includes face data matched with the face contour in the head contour, hair data matched with the hair contour in the head contour,

The mapping function generation module 603 includes:

a first reference brightness value statistics module, configured to count the brightness value of the hair data as the first reference brightness value, and the probability that the first reference brightness value corresponds to the current data belonging to the skin data is 0;

The second reference brightness value statistics module is configured to count the brightness values of the face data and the hair data as the second reference brightness value, and the probability that the second reference brightness value corresponds to the current data belonging to the skin data is 1;

a first luminance mapping segment fitting module, configured to fit a curve with the first reference luminance value as a starting point and the second reference luminance value as an end point in the first luminance mapping segment;

The second luminance mapping segment fitting module is configured to fit a straight line with the second reference luminance value as a starting point in the second luminance mapping segment.

In an embodiment of the present application, the mapping function generation module 603 includes:

A face data replacement module, configured to replace the face data with the face data.

In an embodiment of the present application, the second target probability calculation module 604 includes:

a brightness value statistics module, configured to query the brightness values of pixels in the image data;

The luminance value mapping module is configured to substitute the luminance value into the second mapping function, so as to map the pixel to the second target probability that the pixel belongs to the skin data.

In an embodiment of the present application, the skin detection module 605 includes:

A second probability product calculation module, configured to calculate the product between the first target probability of the pixel and the second target probability of the pixel for the same pixel in the image data, As the third target probability that the pixel belongs to skin data.

The skin color detection apparatus provided by the embodiment of the present application can execute the skin color detection method provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

Embodiment 3

FIG. 7 is a schematic structural diagram of a mobile terminal according to Embodiment 3 of the present application. FIG. 7 shows a block diagram of an exemplary mobile terminal 12 suitable for use in implementing embodiments of the present application. The mobile terminal 12 shown in FIG. 7 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 7, the mobile terminal 12 takes the form of a general-purpose computing device. Components of the mobile terminal 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 connecting various system components including the system memory 28 and the processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA, Industry Standard Architecture) bus, Micro Channel Architecture (MCA, Micro Channel Architecture) bus, enhanced ISA bus, Video Electronics Standards Association (VESA, Video Electronics Standards Association) local bus and Peripheral Component Interconnect (PCI, Peripheral Component Interconnect) bus.

Mobile terminal 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the mobile terminal 12, including volatile and non-volatile media, removable and non-removable media.

System memory (also referred to as memory) 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . The mobile terminal 12 may include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive"). Although not shown in Figure 7, a disk drive for reading and writing to removable non-volatile magnetic disks (eg "floppy disks") and removable non-volatile optical disks (eg CD-ROM, DVD-ROM) may be provided or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The mobile terminal 12 may also communicate with one or more external devices 14 (eg, a keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with the mobile terminal 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the mobile terminal 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Moreover, the mobile terminal 12 can also communicate with one or more networks (such as a local area network (LAN, Local Area Network), a wide area network (WAN, Wide Area Network) and/or a public network, such as the Internet, through the network adapter 20. As shown, the network adapter 20 communicates with other modules of the mobile terminal 12 via the bus 18 . It should be understood that, although not shown in FIG. 7, other hardware and/or software modules may be used in conjunction with the mobile terminal 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID (Redundant Arrays) of Independent Disks, disk array) systems, tape drives, and data backup storage systems.

The processing unit 16 executes at least one functional application and data processing by running a program stored in the system memory 28, for example, to implement the skin color detection method provided by the embodiments of the present application.

Embodiment 4

The fourth embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, multiple processes of the above-mentioned skin color detection method can be realized, and the same effect can be achieved , in order to avoid repetition, it will not be repeated here.

Wherein, the computer-readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM, Read-Only Memory) ), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The storage medium may be a non-transitory storage medium.

The embodiments of the present application propose a skin color detection method, device, mobile terminal, and storage medium, so as to solve the problem of how to take into account the real-time performance and accuracy of skin color detection under the condition of limited performance.

Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described by the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present application is defined by The appended claims determine the scope.

Claims

A skin color detection method, comprising:

Detecting face data in the image data, and dividing the head data matching the head contour with reference to the face data;

Substituting the color value of the image data into the first mapping function to calculate the first target probability that the image data belongs to skin data;

generating a second mapping function with reference to the brightness value of the head data, the second mapping function being used to identify skin data based on the brightness value;

Substituting the luminance value of the image data into the second mapping function to calculate the second target probability that the image data belongs to skin data;

Skin data is detected in the image data in combination with the first target probability and the second target probability.
The method according to claim 1, wherein the detecting face data in the image data and dividing the head data matching the head contour with reference to the face data comprises:

calling the face detection interface provided by the camera to request to detect face data in the image data collected by the camera;

At least one face frame returned by the face detection interface is received, and the face frame is used to frame face data.
The method according to claim 1, wherein the head data includes face data and hair data; the detecting face data in the image data, and dividing the head matching the head contour with reference to the face data data, including:

dividing face data matching the face contour in the head contour with reference to the face data;

Hair data matching the hair contour in the head contour is divided with reference to the face data, the hair data surrounding the face data.
The method of claim 3, wherein, in response to the face data being framed by a face frame, the face data meets the following conditions:

The shape of the face data is an ellipse;

The focus of the face data is the midpoint of the frame edge corresponding to the width of the face frame;

The length of the short axis of the face data is equal to the width in the face frame;

The length of the long axis of the face data is equal to n times the length of the target side. The first candidate edge is the width of the face frame with a specified first ratio, and the second candidate edge is the height of the face frame with a specified second ratio.
The method of claim 3, wherein, in response to the face data being framed by a face frame, the hair data meets the following conditions:

The shape of the hair data is an ellipse;

The focus of the hair data is the midpoint of the frame edge corresponding to the width of the face frame;

The length of the short axis of the hair data is equal to m times the width in the face frame, and the m is positively correlated with the area of the face data;

The length of the long axis of the hair data is equal to n×m times the length of the target side, and the target side is the hypotenuse when the first candidate side and the second candidate side are respectively used as right-angled sides of a right-angled triangle, so The first candidate edge is the width of the face frame with a specified first ratio, and the second candidate edge is the height of the face frame with a specified second ratio.
The method according to claim 1, wherein the detecting face data in the image data and dividing the head data matching the head contour with reference to the face data comprises:

In response to the existence of a plurality of face data in the image data, count the areas of the plurality of face data respectively;

The head data matching the head contour is divided with reference to the k face data with the largest area.
The method according to claim 1, wherein the abscissa of the first mapping function is the color value, and the ordinate is the probability that the current data belongs to the skin data;

The first mapping function includes a first color mapping section, a second color mapping section, and a third color mapping section that are connected in sequence, and the probability that the current data in the first color mapping section belongs to skin data increases from 0 to 1, so The probability that the current data in the second color mapping segment belongs to skin data is 1, and the probability that the current data in the third color mapping segment belongs to skin data decreases from 1 to 0.
The method according to any one of claims 1-7, wherein said substituting the color value of the image data into a first mapping function to calculate the first target probability that the image data belongs to skin data, comprising: :

query the color value of the pixel in the image data;

Substituting the color value into the first mapping function to map the pixel as a candidate probability that the pixel belongs to skin data under the color value;

A first target probability that the pixel point belongs to skin data is calculated based on the candidate probability.
The method according to claim 8, wherein, substituting the color value into the first mapping function to map the pixel point to a candidate probability of skin data under the color value, comprising:

Substitute the color value of the blue chrominance component into the first mapping function corresponding to the blue chrominance component, so as to map the blue probability that the pixel belongs to the skin data under the blue chrominance component, as a candidate probability;

Substitute the color value of the red chrominance component into the first mapping function corresponding to the red chrominance component, so as to map to the red probability that the pixel belongs to the skin data under the red chrominance component, as a candidate probability ;

The calculating, based on the candidate probability, the first target probability that the pixel belongs to the skin data, including:

The product between the blue probability and the red probability is calculated as the first target probability that the pixel point belongs to the skin data.
The method according to claim 1, wherein the abscissa of the second mapping function is the brightness value, and the ordinate is the probability that the current data belongs to the skin data;

The second mapping function includes a first luminance mapping segment and a second luminance mapping segment connected in sequence, the probability that the current data in the first luminance mapping segment belongs to skin data increases from 0 to 1, and the second luminance mapping segment The probability that the current data belongs to skin data is 1.
11. The method of claim 10, wherein the head data includes face data matched to a face contour in the head contour, hair data matched to a hair contour in the head contour,

The generating a second mapping function with reference to the luminance value of the header data includes:

Counting the brightness value of the hair data as a first reference brightness value, the probability that the first reference brightness value corresponds to the current data belonging to skin data is 0;

Counting the brightness values of the face data and the hair data as a second reference brightness value, the probability that the second reference brightness value corresponds to the current data belonging to skin data is 1;

In the first luminance mapping segment, a curve is fitted with the first reference luminance value as a starting point and the second reference luminance value as an end point;

In the second luminance mapping section, a straight line is fitted with the second reference luminance value as a starting point.
The method according to claim 11, wherein the generating the second mapping function with reference to the luminance value of the header data, further comprises:

The face data is replaced with the face data.
The method according to any one of claims 1-7 and 9-12, wherein the luminance value of the image data is substituted into the second mapping function to calculate whether the image data belongs to the skin data The second target probability, including:

query the brightness value of the pixel point in the image data;

The luminance value is substituted into the second mapping function to map to the second target probability that the pixel belongs to skin data.
The method according to any one of claims 1-7 and 9-12, wherein the detecting skin data in the image data by combining the first target probability and the second target probability comprises:

For the same pixel in the image data, the product between the first target probability of the pixel and the second target probability of the pixel is calculated, as the pixel belonging to the skin data. The third target probability.
A skin color detection device, comprising:

an image data detection module, configured to detect face data in the image data, and to divide head data matching the head contour with reference to the face data;

a first target probability calculation module, configured to substitute the color value of the image data into a first mapping function to calculate the first target probability that the image data belongs to skin data;

a mapping function generation module, configured to generate a second mapping function with reference to the brightness value of the head data, where the second mapping function is used to identify skin data based on the brightness value;

A second target probability calculation module, configured to substitute the brightness value of the image data into the second mapping function to calculate the second target probability that the image data belongs to skin data;

A skin detection module configured to detect skin data in the image data in combination with the first target probability and the second target probability.
A mobile terminal, the mobile terminal comprising:

at least one processor;

memory, arranged to store at least one program,

When the at least one program is executed by the at least one processor, the at least one processor implements the skin color detection method according to any one of claims 1-14.
A computer-readable storage medium, storing a computer program on the computer-readable storage medium, and when the computer program is executed by a processor, implements the skin color detection method according to any one of claims 1-14.