WO2022135574A1 - Skin color detection method and apparatus, and mobile terminal and storage medium - Google Patents

Abstract

Provided in the embodiments of the present application are a skin color detection method and apparatus, and a mobile terminal and a storage medium. The method comprises: acquiring video data, wherein the video data comprises multiple frames of image data; respectively detecting facial data in the multiple frames of image data; in response to detecting facial data in first target image data, using a color value of the facial data as priori knowledge to correct a mapping function, wherein the mapping function is used for identifying skin data on the basis of the color value; and substituting a color value of second target image data into the mapping function, so as to detect skin data in the second target image data.

Description

Skin color detection method, device, mobile terminal and storage medium

This application claims the priority of the Chinese patent application with application number 202011563923.X filed with the China Patent Office on December 25, 2020, the entire contents of the above application are incorporated into this application by reference.

technical field

The embodiments of the present application relate to the technical field of computer vision, for example, to a skin color detection method, device, mobile terminal, and storage medium.

Background technique

With the rapid development of mobile Internet and mobile terminals, video data in mobile terminals has become a common information carrier in human activities, such as live broadcasts, video calls, etc. They contain a large amount of information about objects and become a way for people to obtain original information from the outside world. one.

For these video data, users usually choose beauty, and skin color detection is the basis of beauty, which determines that beauty can realize real-time processing of video data, and also affects the effect of beauty.

The mainstream skin color detection can be divided into skin color detection based on color space, skin color detection based on machine learning classification, and skin color detection based on deep learning image segmentation.

Among them, skin color detection based on color space has a wide range of applications in the field of real-time stream processing due to its fast and efficient characteristics, but the accuracy is low. Skin color detection based on machine learning classification and skin color detection based on deep learning image segmentation have higher However, in real-time stream processing, performance often becomes a bottleneck, which is difficult to apply to devices such as restricted mobile terminals.

SUMMARY OF THE INVENTION

In a first aspect, the embodiments of the present application provide a method for detecting skin color, including:

collecting video data, the video data includes multiple frames of image data;

respectively detecting face data in multiple frames of the image data;

In response to detecting human face data in the first target image data, using the color value of the human face data in the first target image data as a priori knowledge, modifying a mapping function for identifying skin based on the color value data, the first target image data is image data collected at a first time point;

Substitute the color value of the second target image data into the mapping function to detect skin data in the second target image data, where the second target image data is image data collected at a second time point.

In a second aspect, the embodiments of the present application also provide a skin color detection device, including:

a video data collection module, configured to collect video data, the video data including multiple frames of image data;

a face data detection module, configured to detect face data in multiple frames of the image data respectively;

The mapping function correction module is configured to modify the mapping function with the color value of the face data in the first target image data as a priori knowledge in response to detecting the face data in the first target image data, and the mapping function For identifying skin data based on color values, the first target image data is image data collected at a first time point;

A mapping detection module configured to substitute the color value of the second target image data into the mapping function to detect skin data in the second target image data, the second target image data being collected at a second time point image data.

In a third aspect, an embodiment of the present application further provides a mobile terminal, where the mobile terminal includes:

at least one processor;

memory, arranged to store at least one program,

When the at least one program is executed by the at least one processor, the at least one processor implements the skin color detection method according to the first aspect.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the skin color detection according to the first party is implemented method.

Description of drawings

1 is a flowchart of a skin color detection method provided in Embodiment 1 of the present application;

FIG. 2 is an example diagram of a time axis of video data according to Embodiment 1 of the present application;

FIG. 3 is an example diagram of a mapping function provided in Embodiment 1 of the present application;

FIG. 4 is an example diagram of a modified mapping function provided by Embodiment 1 of the present application;

5 is a grayscale distribution diagram corresponding to a probability distribution of skin color detection provided in Embodiment 1 of the present application;

6 is a schematic structural diagram of a skin color detection device provided in Embodiment 2 of the present application;

FIG. 7 is a schematic structural diagram of a mobile terminal according to Embodiment 3 of the present application.

Detailed ways

The present application will be described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, the drawings only show some, but not all, structures related to the present application.

Example 1

1 is a flowchart of a skin color detection method provided in Embodiment 1 of the present application. In this embodiment, color values of face data can be used as prior knowledge to perform skin color detection, and the method can be performed by a skin color detection device. The apparatus can be implemented by software and/or hardware, and can be configured in a mobile terminal, for example, a mobile phone, a tablet computer, a smart wearable device (such as a smart watch, smart glasses, etc.), etc., and includes the following steps:

Step 101: Collect video data.

In this embodiment, the video data waiting for skin color detection generally refers to video data generated, transmitted or played in a real-time service scenario.

Generally speaking, skin color detection can be performed on the video data in the mobile terminal that generates the video data. At this time, the camera of the mobile terminal can be turned on, and the camera can collect the video data.

In addition to performing skin color detection on the video data in the mobile terminal that generates the video data, skin color detection may also be performed on the video data on the mobile terminal that plays the video data, which is not limited in this embodiment.

For example, in a live broadcast business scenario, the video data waiting for skin color detection may refer to the video data used to carry the live broadcast content, the mobile terminal logged in by the anchor user generates the video data, and the video data is distributed to at least one The device logged in by the viewer user plays the video. At this time, the video data is usually detected by skin color on the mobile terminal logged in by the host user.

For another example, in the business scenario of a video call, the video data waiting for skin color detection may refer to the video data used to carry the content of the call, and the mobile terminal logged in by the user who initiates the call generates video data, and sends the video data to at least A device logged in by a user who is invited to a call is played. At this time, skin color detection is usually performed on the video data on the mobile terminal logged in by the user who initiated the call.

For another example, in a video conference business scenario, the video data waiting for skin color detection may refer to video data used to carry conference content, the mobile terminal logged in by the user who is speaking generates video data, and the video data is sent to at least A device logged in by a user participating in the conference plays the video. At this time, the mobile terminal logged in by the user who is usually speaking performs skin color detection on the video data.

The video data waiting for skin color detection may also refer to video data generated in a service scenario with low real-time requirements, such as a short video, which is not limited in this embodiment.

Step 102: Detect face data in multiple frames of image data respectively.

In different business scenarios, video data is usually collected from users. Therefore, the multi-frame image data of the video data usually contains the user's character image. The so-called character image can refer to the pixels in the image data used to represent the character.

The character image includes at least face data (pixels) for representing a human face, and may also include hair data (pixels) for representing hair and body data (pixels) for representing a body , limb data (pixel points) used to represent limbs, etc., which are not limited in this embodiment of the present application.

Wherein, for face data, body data, limb data, etc., skin data (pixel points) for representing skin may be included.

In this embodiment, face detection can be performed on multiple frames of image data respectively, and face data included in the image data can be identified. Using the face data as a high-quality anchor point, high-quality skin data can be captured.

In some cases, face data is represented by face key points, that is, given face data, key regions of the face are located, including eyebrows, eyes, nose, mouth, facial contours, and so on.

In other cases, a range of a specified shape can be generated based on the key points of the face, and the range is used to represent the face data, wherein the shape includes a rectangle, an ellipse, etc., and the range of the rectangle can also be called a face frame.

In a specific implementation, the following methods can be used to perform face detection on multi-frame image data:

1. Use artificial extraction features, such as haar features, use features to train classifiers, and use classifiers for face detection.

2. Inherit face detection from general object detection algorithms, for example, use Faster R-CNN to detect faces.

3. Convolutional neural network using cascade structure, for example, Cascade CNN (Cascade Convolutional Neural Network), MTCNN (Multi-task Cascaded Convolutional Networks, multi-task convolutional neural network).

These methods for implementing face detection can be integrated in the modules of the application, and the modules of the application are directly called to perform face detection on image data. These methods for implementing face detection can also be integrated in the SDK (Software Development Kit, software development kit). ), the SDK is used as the assembly data of the application, the application can request the SDK to perform face detection on multi-frame image data, the SDK detects the face data in the image data, and returns the face data to the application.

If the application is a system application supporting the camera, the system application usually provides API (Application Program Interface, Application Program Interface) for these methods of implementing face detection, as a face detection interface, for example, in the Android (Android) system , provides two face detection interfaces android.media.FaceDetector and android.media.FaceDetector.Face, in iOS system, provides two face detection interfaces AVCaptureMetadataOutput and CIDetector.

In response to this situation, you can call the face detection interface provided by the camera to request to detect the face data in the multi-frame image data collected by the camera, and receive one or more face frames returned by the face detection interface. The face frame is a rectangular frame , which can be used to frame the face data, that is, the face data is located in the face frame.

For the face detection interface provided by the system application, there is hardware support, and the face frame is calibrated based on a few key points of the face (2 key points of the face), the speed is fast, the performance consumption is very low, and the accuracy can meet the requirements of the first knowledge requirements.

The video data includes multiple frames of image data, which are denoted as P ₁ , P ₂ , ..., P _t-1 , P _t , P _t+1 , ... , P _n in the order of generation, where t and n are positive Integer, t+1<n, since the video data is generated in real time, n increases continuously with the generation of the video data until the generation of the video data is completed.

In this embodiment, each frame of image data in the video data is sequentially traversed to perform skin color detection. For the convenience of description, the image data collected at the first time point is called the first target image data, and the image data collected at the second time point will be called the first target image data. The image data is called the second target image data.

The second time point is located after the first time point. In general, the first time point is the closest time point to the second time point when face data is detected. On the time axis, the second time point may be the same as the second time point. The first time point is adjacent, that is, there is no other time point between the second time point and the first time point, and the second time point may not be adjacent to the first time point, that is, the second time point and the first time point other time points in between.

As shown in FIG. 2, T represents the time axis, and t ₀ to t ₁₁ represent time points.

For example, as shown in FIG. 2 , face data is detected in the image data corresponding to time points t ₂ and t ₇ respectively, then for time points t ₂ -t ₆ , time point t ₂ is the first time point, and the time point t ₃ -t ₆ are the second time points, for the time points t ₇ -t ₁₁ , the time point t ₇ is the first time point, and the time points t ₈ -t ₁₁ are the second time points.

Step 103 , in response to detecting the face data in the first target image data, correct the mapping function by using the color value of the face data as prior knowledge.

In this embodiment, a mapping function can be set by counting the color values of pixels in different samples such as skin data and non-skin data, and the mapping function can be used to identify skin data based on the color value, that is, the input of the mapping function is The color value is output as the probability that the current data belongs to skin data. At this time, the probability that the pixels in different samples are skin data under a certain color value can be counted, so that these color values and their probabilities can be fitted as a mapping function.

In order to maintain generality, the mapping function can refer to the skin data of users of different races, different ages, and different skin tones, and can also refer to the skin data under different lighting conditions. Therefore, the mapping function is relatively broad and the accuracy is relatively high, but , in the case of skin color detection for the current user, the accuracy is lacking.

In general, the probability that the current data in the mapping function belongs to skin data is a continuous value, such as [0-1]. In some cases, the probability that the current data in the mapping function belongs to skin data is a discrete value, such as 0, 1, this The embodiment does not limit this.

In an example, as shown in FIG. 3 , in the coordinate system where the mapping function is located, the abscissa (horizontal axis) is the color value (X), and the ordinate (vertical axis) is the probability (P) that the current data belongs to skin data .

The mapping function includes a first color map segment (the abscissa is [x ₀ , x ₁ ]), a second color map segment (the abscissa is [x ₁ , x ₂ ]), and a third color map segment (the abscissa is [x 1 , x 2 ]), which are connected in sequence The coordinates are [x ₂ , x ₃ ]), wherein the probability that the current data in the first color mapping segment belongs to skin data increases from 0 to 1, the probability that the current data in the second color mapping segment belongs to skin data is 1, and the third The probability that the current data in the colormap segment belongs to skin data decreases from 1 to 0.

In an example, the second color map segment belongs to a line segment, and the first color map segment and the third color map segment belong to a curve, which can be fitted by using a polynomial function or the like.

The above mapping function is only an example. When implementing the embodiments of the present application, other mapping functions may be set according to actual conditions. For example, the mapping function includes a first color mapping segment and a second color mapping segment connected in sequence, wherein the first color The probability that the current data in the mapping segment belongs to skin data increases from 0 to 1, and the probability that the current data in the second color mapping segment belongs to skin data decreases from 1 to 0. At this time, the mapping function is expressed by functions such as a quadratic equation, etc. etc., which are not limited in the embodiments of the present application. In addition, in addition to the above mapping function, those skilled in the art may also adopt other mapping functions according to actual needs, which are not limited in the embodiments of the present application.

In different color spaces, color values have different chrominance components, and corresponding mapping functions can be set for the color components. For example, in the RGB color space, there are red chrominance components (R), green chrominance components (G), For the blue chrominance component (B), the mapping function may be set for the red chrominance component (R), the mapping function may be set for the green chrominance component (G), and the mapping function may be set for the blue chrominance component (B), and for example, in In the YCbCr color space, with blue chrominance components (Cb) and red chrominance components (Cr), you can set the mapping function for the blue chrominance component (Cb), set the mapping function for the red chrominance component (Cr), etc. .

If face data is detected in the first target image data (that is, the first target image data is the image data collected at the first time point), the color value can be counted within the range of the face data, and the counted color value The confidence of the color value of the skin data is high, which can be used as a priori knowledge to correct the mapping function for the current user and improve the accuracy of the mapping function for the current user.

In an embodiment of the present application, step 103 may include the following steps:

Step 1031: Determine the mapping function.

In this embodiment, the abscissa of the mapping function is the color value, and the ordinate is the probability that the current data belongs to skin data, and the mapping function includes a first color mapping segment, a second color mapping segment, and a third color mapping segment connected in sequence, The probability that the current data in the first color map segment belongs to skin data increases from 0 to 1, the probability that the current data in the second color map segment belongs to skin data is 1, and the probability that the current data in the third color map segment belongs to skin data increases from 1 down to 0.

Step 1032: Detect skin data in the face data based on the color space.

In this embodiment, if there are multiple face data in the first target image data, the area of the multiple face data can be counted separately. If the face data is framed by a face frame, the width of the face frame is w and the height is w. is h, then the area of the face data

S=w×h.

Compare the areas of multiple personal face data, extract the face data with the largest area n (n is a positive integer, such as 3), the skin data to be detected, and the color value of the statistical skin data, due to the imaging characteristics of the camera, the callback face The data method can accurately describe the color of the face data with a larger area. Selecting the n face data with the largest area can ensure the accuracy of the color value of the subsequent statistical skin data under the condition of reducing the amount of calculation.

For example, the n face data with the largest area may be the face data sorted according to the area from large to small, and the n face data may be the top n face data in the sorting. However, it is not limited to such a selection method of selecting face data based on the size of the area.

In the case where there are multiple face data in the first target image data, skin data may also be detected in all the face data based on the color space, which is not limited in this embodiment.

For the detected face data, the color space-based method can be used to detect the skin color in the face data, and the pixels representing the skin data can be detected. The color space-based method is simple to calculate, so the calculation speed is fast and time-consuming. Counting the color values of multiple skin data can maintain a high accuracy as a whole, meeting the requirements of prior knowledge.

Since the color space of the image data is different, for different color spaces, you can use the method corresponding to the color space to detect the skin data in the face data, or convert the image data to the specified color space, use and specify The method corresponding to the color space of 1 detects skin data in the face data, which is not limited in this embodiment.

In an example, for the RGB color space, when the color value of a pixel meets the following conditions, it can be considered that the pixel is skin data (that is, the probability that the pixel belongs to skin data is 1), and the color value of the pixel does not meet the following conditions When the conditions are met, it can be considered that the pixel is not skin data (that is, the probability that the pixel belongs to skin data is 0):

R>95, G>40, B>20, R>G, R>B

(Max(R,G,B)-Min(R,G,B))>15

Abs(R-G)＞5

Among them, Max represents the maximum value, Min represents the minimum value, and Abs represents the absolute value.

Among them, R represents the red chrominance component, G represents the green chrominance component, and B represents the blue chrominance component.

In another example, for the YCbCr color space, when the color value of a pixel meets the following conditions, it can be considered that the pixel is skin data (that is, the probability that the pixel belongs to skin data is 1), and the color value of the pixel does not meet the following conditions. Under the following conditions, it can be considered that the pixel is not skin data (that is, the probability that the pixel belongs to skin data is 0):

Among them, Cb represents the blue chrominance component, and Cr represents the red chrominance component.

For the case of calling the face detection interface provided by the camera for face detection, the face data may include non-skin data such as hair data and background data. Reduce the area of face data on the edge, reduce the amount of non-skin data, and increase the proportion of skin data, so as to detect skin data in candidate area images based on color space, and improve the accuracy of skin data detection.

In a convergent example, if the face data is framed by a face frame, the center point P(x _{0 , y 0 ) of the face frame can be determined, and on the condition that the center point P(x 0 , y 0 ) is} maintained , reduce the width and height of the face frame as a candidate area image. Set the center point P(x ₀ , y 0 ) of the face frame as the center point P(x ₀ , y ₀ ) of the candidate region image, that is, keep the center point P(x ₀ , y _{0 )} unchanged, under this condition Next, reduce the width and height of the face frame as the width and height of the candidate area image.

Let the width of the face frame be w and the height be h, the candidate area image can be expressed as [x ₀ ±(a*w), y ₀ ±(b*h)[, where a is a coefficient less than 1, such as 0.4, b is a coefficient less than 1, such as 0.1.

Step 1033 , count the color values of the skin data.

For the pixel points belonging to the skin data, the average value of the color values among multiple pixel points can be counted as the color value of the entire skin data.

Step 1034: Mark the color value of the skin data on the horizontal axis of the coordinate system where the mapping function is located.

As shown in FIG. 4 , if the color value C of the skin data is counted, the color value C of the skin data can be marked on the horizontal axis of the coordinate system where the mapping function is located.

Due to the high precision of the color value C of the skin data, the color value C of the skin data is generally located in the second color mapping segment of the mapping function (the abscissa is [x ₁ , x ₂ ]).

Step 1035 , under the condition that the first color map segment and the third color map segment are maintained, reduce the second color map segment with reference to the color value of the skin data.

In a specific implementation, the mapping function can be converged to the color value C of the skin data as a whole, the range of the mapping function can be reduced, and some regions with low probability can be excluded.

In the process of converging the mapping function, the shape of the first color mapping segment and the third color mapping segment are kept unchanged as a precondition, and the color value of the user's skin data in the current scene is used as a reference to reduce the size of the second color mapping segment. range, and adjust the range of the first color map segment and the third color map segment accordingly.

In an embodiment of the present application, step 1035 includes the following steps:

Step 10351: Converging the second color map segment towards the color value of the skin data.

Taking the color value of the skin data as the specified position, such as the midpoint, the quantile point, etc., converge the second color map segment on the abscissa toward the location of the color value of the skin data, thereby narrowing the range of the second color map segment .

When converging the second color mapping segment, either end of the second color mapping segment may be converged, or both ends of the second color mapping segment may be converged at the same time, which is not limited in this embodiment.

As shown in FIG. 4 , the color value C of the skin data is marked on the horizontal axis, and the two ends of the second color mapping segment are converged toward the color value C of the skin data at the same time, and the abscissa of the second color mapping segment before convergence is [x ₁ , x ₂ ], the abscissa of the second color mapping segment after convergence is [x ₁ ', x ₂ '], where x ₁ '>x ₁ , x ₂ '<x ₂ .

In a convergent manner, the target length L _t can be determined. Assuming that the length of the second color map segment is L, the target length L _t is less than the length L of the second color map segment, that is, L _t <L. The color value C is used as the center point to reduce the second color map segment until the length L of the second color map segment is equal to the target length L _t , thereby reducing the range of the second color map segment.

As shown in FIG. 4 , the length of the second color map segment is L=x ₂ -x ₁ , the target length L _t =x ₂ ′-x ₁ ′, and the color value of the skin data is

In some cases, the target length L _t is a statistical value, which is a constant.

In other cases, the target length L _t is an adaptively adjusted value, which is a variable.

For example, the reference length T may be determined, and the reference length T is a statistical value, which is a constant, and is smaller than the length L of the second color map segment, that is, T<L.

Calculate the difference between the length L of the second color map segment and the reference length T as an adjustable length;

The time difference Δt between the second time point and the first time point is mapped to the correction coefficient w _t by the preset mapping method f(), and the correction coefficient w _t is positively correlated with the time difference, that is, the larger the time difference, the greater the correction coefficient w _t On the contrary, the smaller the time difference, the smaller the correction coefficient w _t .

Calculate the product between the adjustable length and the correction coefficient w _t as the correction length, and calculate the sum between the reference length T and the correction length as the target length L _t .

At this time, the calculation process of the target length L _t can be expressed as follows:

L _t =T+(LT)*w _t

w _t =f(Δt)

The closer the second time point is to the first time point, the lower the probability of changes in lighting and other conditions in the scene, and the higher the reference value of the color value of the skin data counted in the first time point. The second time point is closer to the first time point. The farther away, the higher the probability of changes in lighting and other conditions in the scene, and the lower the reference value of the color value of the skin data counted at the first time point. point) is farther and farther away from the current time (second time point), then gradually relax the target length L _t , so that the target length L _t is getting closer and closer to the length L of the second color mapping segment, so as to ensure that the second Accuracy of colormap segments.

Step 10352: Shift the first color mapping segment in a direction close to the second color mapping segment until the first color mapping segment is connected to the second color mapping segment.

As shown in FIG. 4 , if the convergence of the second color mapping segment is completed, the first color mapping segment can be shifted to the right on the horizontal axis, so that the first color mapping segment moves toward the second color mapping segment until the first color mapping segment is reached. The second color map segment is connected, that is, the first color map segment is connected to the second color map at the endpoint x ₁ ', and the abscissa of the first color map segment before translation is [x ₀ , x ₁ ], after translation The abscissa of the first color map segment is [x ₀ ', x ₁ '], where x ₀ '>x ₀ , x ₁ '>x ₁ , and x ₁ -x ₀ =x ₁ '-x ₀ ' .

Step 10353: Shift the third color mapping segment in a direction close to the second color mapping segment until the third color mapping segment is connected to the second color mapping segment.

As shown in FIG. 4 , if the convergence of the second color mapping segment is completed, the third color mapping segment can be shifted to the left on the horizontal axis, so that the third color mapping segment moves toward the second color mapping segment until the third color mapping segment is reached. The second color map segment is connected, that is, the third color map segment is connected to the second color map at the endpoint x ₂ '. Before the translation, the abscissa of the third color map segment is [x ₂ , x ₃ ]. After the translation The abscissa of the second color map segment is [x ₂ ', x ₃ '], where x ₂ '<x ₂ , x ₃ '<x ₃ , and x ₃ -x ₂ =x ₃ '-x ₂ ' .

Step 104: Substitute the color value of the second target image data into the mapping function to detect skin data in the second target image data.

The face data is detected at the first target image data corresponding to the first time point, and the color value of the face data in the face data is used as the prior knowledge to correct the mapping function. The mapping function can be located after the first time point. At the second time point, skin color detection is performed on the second target image data (that is, the second target image data is the image data collected at the second time point), so as to detect the degree to which the pixel points represent the skin data in the second target image data .

In one embodiment, the color value of each pixel of the second target image data can be marked on the horizontal axis in the coordinate system where the mapping function is located. If the color value of the pixel in the second target image data is outside the mapping function, Then the probability that the pixel belongs to the skin data is 0. If the color value of the pixel in the second target image data is within the mapping function, the probability corresponding to the color value on the vertical axis can be found through the mapping function, as the pixel belongs to. Probability of skin data.

In some cases, the abscissa of the mapping function is the color value, and the ordinate is the probability that the pixel belongs to the skin data. In different color spaces, the color value has different chromaticity components, and each color component has a corresponding mapping function, For the same pixel, different probabilities can be calculated.

In view of this situation, the color value of each pixel in the second target image data can be queried, and the color value can be substituted into the corresponding mapping function to map out the candidate probability that each pixel belongs to the skin data under the color value, based on These candidate probabilities can be calculated by averaging, summing, multiplying, linear fusion (that is, summing after configuring weights) to calculate the target probability that the pixel belongs to the skin data. These target probabilities reflect the ROI (region of interest, region of interest) area.

In one example, for the YCbCr color space, the second target image data has a blue chrominance component Cb, a red chrominance component Cr.

On the one hand, the color value of the blue chrominance component Cb is substituted into the mapping function corresponding to the blue chrominance component Cb, so as to map out the blue probability that the pixel belongs to the skin data under the blue chrominance component Cb, as the candidate probability

On the other hand, the color value of the red chrominance component Cr is substituted into the mapping function corresponding to the red chrominance component Cr, so as to map the red probability that the pixel belongs to the skin data under the red chrominance component Cr, as the candidate probability

At this point, the blue probability can be calculated

with red probability

The product between , as the target probability that the pixel belongs to the skin data

In addition, if the face data is not detected before, there is a lack of relatively high-quality anchor points, and it is difficult to capture high-quality skin data, the skin data can be detected in the image data based on the color space.

For example, as shown in FIG. 2 , if no face data is detected in the image data before time points t ₀ and t ₁ , skin data can be detected in the image data corresponding to t ₀ and t ₁ based on the color space method. .

After the skin color is detected, operations such as beautifying processing (such as skin resurfacing) can be performed on the video data according to the needs of the user.

In order for those skilled in the art to better understand the embodiments of the present application, the following specific examples are used to describe the skin color detection method in this embodiment.

As shown in Figure 5, for the same frame of image data, the left side is the grayscale distribution corresponding to the probability distribution obtained by using the color space-based skin color detection, and the right side is the probability distribution corresponding to the skin color detection obtained by applying this embodiment. Grayscale distribution, in which, the higher the grayscale of a pixel (more white), the higher the probability that the pixel belongs to skin data, and the lower the grayscale (more black), the lower the probability that the pixel belongs to skin data. .

In contrast, the skin color detection based on the color space considers clothes, hair, and backgrounds (such as electric lights, etc.) to belong to skin data, and this embodiment can well exclude clothes, hair, and backgrounds (such as electric lights, etc.). ), which greatly improves the accuracy of skin color detection, and can well protect clothes, hair, and backgrounds (such as lights, etc.) in subsequent beauty treatments (such as microdermabrasion).

In this embodiment, video data is collected, and the video data includes multiple frames of image data, and face data is detected in the multiple frames of image data respectively. If face data is detected in the first target image data, the face data The color value is used as a priori knowledge to correct the mapping function, the mapping function is used to identify skin data based on the color value, the first target image data is the image data collected at the first time point, and the color value of the second target image data is substituted into the mapping function , in order to detect skin data in the second target image data, the second target image data is the image data collected at the second time point, the video data has continuity, therefore, the content between multiple frames of image data has correlation, so that The previous image data can be used as the prior knowledge of the subsequent image data, and the face data can be used as the anchor to capture high-quality skin data. This adaptive modification of the mapping function can improve the accuracy of skin color detection performed by the mapping function in the current business scenario, and operations such as face detection, statistics of the color value of the skin data, correction of the mapping function and its application are relatively simple. , the calculation amount is small, the speed is fast, and the time-consuming is short. In the case of limited performance of mobile terminals and other equipment, the skin color detection of video data can be realized in real time.

In one embodiment, under the condition of improving the accuracy of skin color detection, some data similar to skin data, such as hair, clothes, background, etc., can be excluded in the dimension of color, so that in the subsequent beauty processing (such as microdermabrasion) Protect the data similar to the skin data in the processing), slow down or avoid the reduction of the clarity of the data similar to the skin data due to the beauty treatment (such as microdermabrasion), so as to ensure the quality of the beauty treatment.

The method embodiments are described as a series of action combinations for the sake of simple description, but those skilled in the art should know that the embodiments of the present application are not limited by the described sequence of actions, because according to the embodiments of the present application , certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the actions involved in the embodiments described in the specification are not necessarily required by the embodiments of the present application.

Embodiment 2

6 is a structural block diagram of a skin color detection device provided in Embodiment 2 of the present application, which may include the following modules:

The video data collection module 601 is configured to collect video data, and the video data includes multiple frames of image data;

A face data detection module 602, configured to detect face data in the multiple frames of the image data respectively;

The mapping function correction module 603 is configured to modify the mapping function with the color value of the face data in the first target image data as prior knowledge in response to detecting the face data in the first target image data, and the mapping function The function is used to identify skin data based on the color value, and the first target image data is the image data collected at the first time point;

The mapping detection module 604 is configured to substitute the color value of the second target image data into the mapping function, so as to detect skin data in the second target image data, the second target image data is at the second time point acquired image data.

In an embodiment of the present application, the face data detection module 602 includes:

a face detection interface calling module, configured to call the face detection interface provided by the camera, to request to detect the face data in the multiple frames of the image data collected by the camera;

The face frame receiving module is configured to receive at least one face frame returned by the face detection interface, where the face frame is used to frame face data.

In an embodiment of the present application, the mapping function correction module 603 includes:

The mapping function determination module is set to determine the mapping function, the abscissa of the mapping function is the color value, and the ordinate is the probability that the current data belongs to the skin data, and the mapping function includes the first color mapping segment and the second color that are connected in turn. mapping section and third color mapping section, the probability that the current data in the first color mapping section belongs to skin data increases from 0 to 1, the probability that the current data in the second color mapping section belongs to skin data is 1, and the The probability that the current data in the third color map segment belongs to skin data decreases from 1 to 0;

a face skin detection module, configured to detect skin data in the face data in the first target image data based on the color space;

a color value statistics module, configured to count color values for the skin data;

A color value marking module, configured to mark the color value based on the statistics of the skin data on the horizontal axis of the coordinate system where the mapping function is located;

The color map segment correction module is configured to reduce the second color map segment with reference to the color value based on the statistics of the skin data under the condition of maintaining the first color map segment and the third color map segment.

In an embodiment of the present application, the mapping function correction module 603 further includes:

an area statistics module, configured to count the areas of a plurality of the face data respectively in response to the existence of a plurality of face data in the first target image data;

A face extraction module, configured to extract the n pieces of face data with the largest area.

In an embodiment of the present application, the face skin detection module includes:

a face data convergence module, configured to converge the face data in the first target image data to obtain a candidate area image;

A candidate detection module, configured to detect skin data in the candidate region image based on the color space.

In an embodiment of the present application, the face data convergence module includes:

a center point determination module, configured to frame the face frame in response to the face data in the first target image data, and determine the center point of the face frame;

The face frame reduction module is configured to set the center point of the face frame as the center point of the candidate area image, and reduce the width and height of the face frame as the width and height of the candidate area image.

In an embodiment of the present application, the color map segment correction module includes:

A second color map segment convergence module configured to converge the second color map segment toward the color value based on the skin data statistics;

a first color map segment translation module, configured to translate the first color map segment in a direction close to the second color map segment, until the first color map segment is connected to the second color map segment;

A third color mapping segment translation module is configured to translate the third color mapping segment in a direction close to the second color mapping segment until the third color mapping segment is connected to the second color mapping segment.

In an embodiment of the present application, the second color map segment convergence module includes:

a target length determination module, configured to determine a target length, the target length being less than the length of the second color map segment;

A color map segment reduction module, configured to reduce the second color map segment with the color value based on the skin data statistics as a center point until the length of the second color map segment is equal to the target length.

In an embodiment of the present application, the target length determination module includes:

a reference length determination module, configured to determine a reference length, the reference length being less than the length of the second color mapping segment;

An adjustable length calculation module, configured to calculate the difference between the length of the second color map segment and the reference length as an adjustable length;

a correction coefficient calculation module, configured to map the time difference between the second time point and the first time point as a correction coefficient, and the correction coefficient is positively correlated with the time difference;

A correction length calculation module, configured to calculate the product between the adjustable length and the correction coefficient as the correction length;

The target length calculation module is configured to calculate the sum value between the reference length and the corrected length as the target length.

In an embodiment of the present application, the mapping detection module 604 includes:

a color value query module, configured to query the color value of a pixel in the second target image data;

A candidate probability mapping module, configured to substitute the color value of the pixel into the mapping function to map the pixel as a candidate probability that the pixel belongs to skin data under the color value;

The target probability calculation module is configured to calculate the target probability that the pixel point belongs to the skin data based on the candidate probability.

In an embodiment of the present application, the candidate probability mapping module includes:

The blue probability mapping module is configured to substitute the color value of the blue chrominance component into the mapping function corresponding to the blue chrominance component, so as to map the pixel points to the skin data under the blue chrominance component. The blue probability, as the candidate probability;

A red probability mapping module, configured to substitute the color value of the red chromaticity component into the mapping function corresponding to the red chromaticity component, so as to map the pixel to the red color of the skin data under the red chromaticity component probability, as a candidate probability;

The target probability calculation module includes:

The probability product calculation module is configured to calculate the product between the blue probability and the red probability as the target probability that the pixel belongs to the skin data.

In an embodiment of the present application, it also includes:

An image detection module configured to detect skin data in the image data based on the color space in response to the previously undetected face data.

The skin color detection apparatus provided by the embodiment of the present application can execute the skin color detection method provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

Embodiment 3

FIG. 7 is a schematic structural diagram of a mobile terminal according to Embodiment 3 of the present application. FIG. 7 shows a block diagram of an exemplary mobile terminal 12 suitable for use in implementing embodiments of the present application. The mobile terminal 12 shown in FIG. 7 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 7, the mobile terminal 12 takes the form of a general-purpose computing device. Components of the mobile terminal 12 may include, but are not limited to, one or more processors or processing units 16, a system memory (also referred to as memory) 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16). .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA, Industry Standard Architecture) bus, Micro Channel Architecture (MCA, Micro Channel Architecture) bus, enhanced ISA bus, Video Electronics Standards Association (VESA, Video Electronics Standards Association) local bus and Peripheral Component Interconnect (PCI, Peripheral Component Interconnect) bus.

Mobile terminal 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the mobile terminal 12, including volatile and non-volatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . The mobile terminal 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive"). Although not shown in Figure 7, a disk drive for reading and writing to removable non-volatile magnetic disks (eg "floppy disks") and removable non-volatile optical disks (eg CD-ROM, DVD-ROM) may be provided or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The mobile terminal 12 may also communicate with one or more external devices 14 (eg, a keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with the mobile terminal 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the mobile terminal 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Moreover, the mobile terminal 12 can also communicate with one or more networks (such as a local area network (LAN, Local Area Network), a wide area network (WAN, Wide Area Network) and/or a public network, such as the Internet, through the network adapter 20. As shown in FIG. 7 , the network adapter 20 communicates with other modules of the mobile terminal 12 via the bus 18 . It should be understood that, although not shown in FIG. 7, other hardware and/or software modules may be used in conjunction with the mobile terminal 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID (Redundant Arrays) of Independent Disks, disk array) systems, tape drives, and data backup storage systems.

The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the skin color detection method provided by the embodiments of the present application.

Embodiment 4

The fourth embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, each process of the above-mentioned skin color detection method can be achieved, and the same effect can be achieved, In order to avoid repetition, details are not repeated here.

Wherein, the computer-readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM, Read-Only Memory), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable of the above The combination. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The embodiments of the present application propose a skin color detection method, device, mobile terminal, and storage medium, so as to solve the problem of how to take into account the real-time performance and accuracy of skin color detection under the condition of limited performance.

In this embodiment, video data is collected, and the video data includes multiple frames of image data, and face data is detected in the multiple frames of image data respectively. If face data is detected in the first target image data, the face data The color value is used as a priori knowledge to correct the mapping function, the mapping function is used to identify skin data based on the color value, the first target image data is the image data collected at the first time point, and the color value of the second target image data is substituted into the mapping function , in order to detect skin data in the second target image data, the second target image data is the image data collected at the second time point, and the video data has continuity, therefore, the content between multiple frames of image data has correlation, so that The previous image data can be used as the prior knowledge of the subsequent image data, and the face data can be used as an anchor to capture high-quality skin data. This adaptive modification of the mapping function can improve the accuracy of skin color detection performed by the mapping function in the current business scenario, and operations such as face detection, statistics of the color value of the skin data, correction of the mapping function and its application are relatively simple. , the calculation amount is small, the speed is fast, and the time-consuming is short. In the case of limited performance of mobile terminals and other equipment, the skin color detection of video data can be realized in real time.

In one embodiment, under the condition of improving the accuracy of skin color detection, some data similar to skin data, such as hair, clothes, background, etc., can be excluded, so that in the subsequent beauty treatment (such as microdermabrasion), the The data similar to the skin data can slow down or avoid the situation that the data similar to the skin data will reduce the clarity caused by the beauty treatment (such as microdermabrasion), so as to ensure the quality of the beauty treatment.

Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described by the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present application is defined by The appended claims determine the scope.

Claims (15)

1. A skin color detection method, comprising:

   collecting video data, the video data includes multiple frames of image data;

   respectively detecting face data in multiple frames of the image data;

   In response to detecting human face data in the first target image data, using the color value of the human face data in the first target image data as a priori knowledge, modifying a mapping function for identifying skin based on the color value data, the first target image data is image data collected at a first time point;

   Substitute the color value of the second target image data into the mapping function to detect skin data in the second target image data, where the second target image data is image data collected at a second time point.
2. The method according to claim 1, wherein said detecting face data in multiple frames of said image data respectively comprises:

   calling the face detection interface provided by the camera to request to detect face data in the multiple frames of the image data collected by the camera;

   At least one face frame returned by the face detection interface is received, and the face frame is used to frame face data.
3. The method according to claim 1, wherein the modifying the mapping function using the color value of the face data in the first target image data as a priori knowledge, comprises:

   Determine a mapping function, the abscissa of the mapping function is the color value, and the ordinate is the probability that the current data belongs to the skin data, and the mapping function includes a first color map segment, a second color map segment, and a third color map connected in sequence segment, the probability that the current data in the first color mapping segment belongs to skin data increases from 0 to 1, the probability that the current data in the second color mapping segment belongs to skin data is 1, and the current data in the third color mapping segment The probability that the data belongs to skin data decreased from 1 to 0;

   Detecting skin data in the face data in the first target image data based on the color space;

   Counting color values on the skin data;

   Marking the color value based on the skin data statistics on the horizontal axis of the coordinate system where the mapping function is located;

   On the condition that the first color map segment and the third color map segment are maintained, the second color map segment is reduced with reference to the color value based on the statistics of the skin data.
4. The method according to claim 3, wherein the modifying the mapping function using the color value of the face data in the first target image data as a priori knowledge, further comprises:

   In response to the existence of a plurality of face data in the first target image data, count the areas of the plurality of face data respectively;

   Extracting the n pieces of the face data with the largest area.
5. The method according to claim 3, wherein the detecting skin data in the face data in the first target image data based on the color space comprises:

   Converging the face data in the first target image data to obtain a candidate area image;

   Skin data is detected in the candidate region image based on a color space.
6. The method according to claim 5, wherein the converging the face data in the first target image data to obtain the candidate region image comprises:

   In response to the face data in the first target image data being framed by a face frame, determining the center point of the face frame;

   The center point of the face frame is set as the center point of the candidate area image, and the width and height of the face frame are reduced as the width and height of the candidate area image.
7. The method of claim 3 , wherein the first color map segment and the third color map segment are maintained, and the first color map segment is reduced with reference to the color value based on the skin data statistics. Two colormap segments, including:

   Converging the second color map segment toward the color value based on the skin data statistics;

   Translating the first color map segment in a direction close to the second color map segment until the first color map segment joins the second color map segment;

   Translating the third color map segment in a direction close to the second color map segment until the third color map segment joins the second color map segment.
8. 8. The method of claim 7, wherein the converging the second color map segment toward the color values based on the skin data statistics comprises:

   determining a target length, the target length being less than the length of the second color map segment;

   The second color map segment is reduced with the color value based on the skin data statistics as a center point until the length of the second color map segment is equal to the target length.
9. The method of claim 8, wherein the determining the target length comprises:

   determining a reference length, the reference length being less than the length of the second color map segment;

   Calculate the difference between the length of the second color map segment and the reference length as an adjustable length;

   mapping the time difference between the second time point and the first time point as a correction coefficient, and the correction coefficient is positively correlated with the time difference;

   Calculate the product between the adjustable length and the correction coefficient as the correction length;

   The sum of the reference length and the correction length is calculated as the target length.
10. The method according to any one of claims 1-9, wherein the substituting the color value of the second target image data into the mapping function to detect skin data in the second target image data comprises:

    query the color value of the pixel in the second target image data;

    Substitute the color value of the pixel into the mapping function to map out the candidate probability that the pixel belongs to skin data under the color value;

    The target probability that the pixel point belongs to skin data is calculated based on the candidate probability.
11. The method according to claim 10, wherein said substituting the color value of the pixel point into the mapping function to map out the candidate probability of the pixel point belonging to skin data under the color value, comprising:

    Substitute the color value of the blue chromaticity component into the mapping function corresponding to the blue chromaticity component, so as to map the blue probability that the pixel belongs to the skin data under the blue chromaticity component, as a candidate probability;

    Substitute the color value of the red chrominance component into the mapping function corresponding to the red chrominance component, so as to map to the red probability that the pixel point belongs to the skin data under the red chrominance component, as a candidate probability;

    The calculating, based on the candidate probability, the target probability that the pixel belongs to the skin data, including:

    The product between the blue probability and the red probability is calculated as the target probability that the pixel belongs to the skin data.
12. The method according to any one of claims 1-9, further comprising:

    Skin data is detected in the image data based on the color space in response to no prior detection of face data.
13. A skin color detection device, comprising:

    a video data acquisition module, configured to collect video data, the video data including multiple frames of image data;

    a face data detection module, configured to detect face data in multiple frames of the image data respectively;

    The mapping function correction module is configured to modify the mapping function with the color value of the face data in the first target image data as a priori knowledge in response to detecting the face data in the first target image data, and the mapping function For identifying skin data based on color values, the first target image data is image data collected at a first time point;

    A mapping detection module, configured to substitute the color value of the second target image data into the mapping function to detect skin data in the second target image data, the second target image data being collected at a second time point image data.
14. A mobile terminal, the mobile terminal includes:

    at least one processor;

    memory, arranged to store at least one program,

    When the at least one program is executed by the at least one processor, the at least one processor implements the skin color detection method according to any one of claims 1-12.
15. A computer-readable storage medium, storing a computer program on the computer-readable storage medium, and when the computer program is executed by a processor, implements the skin color detection method according to any one of claims 1-12.

Images