CN110929239B - Terminal unlocking method based on lip language instruction - Google Patents

Abstract

The invention relates to a terminal unlocking method based on lip language instructions. During the acquisition process, several frames of images are taken to acquire the face, and some key feature points are extracted. In the verification process, the key feature points that need to be recognized for the face are extracted in the same way, and the Euclidean distance of the face features is calculated by the facenet network, and the threshold is compared and judged. When collecting, users can design command actions by themselves, and just make the same actions when identifying, so that action commands are not easily stolen by others, and the security of authentication is improved. At the same time, the lip language instruction unlocking method does not require large-scale operations on the terminal, which greatly reduces the hardware performance requirements and improves the recognition speed. The invention can avoid the problem of excessive gradient caused by accumulation of a certain quadrant in the space, improve the efficiency of network learning and training, play the effect of active learning and training model, and solve the problem of easy exposure of traditional fixed command actions.

Description

A terminal unlocking method based on lip language instruction

technical field

The invention relates to a terminal unlocking method based on lip language instructions, and belongs to the technical field of image information processing.

Background technique

At present, the terminal unlocking methods mainly include: face, fingerprint, and iris. However, these information can be easily forged, and the static identification method is easy to be cracked, and the security is poor, which can easily lead to the leakage of private information. The invention adopts the method of unlocking by lip language instruction, realizes dynamic unlocking, and can improve the security of authentication.

The existing lip-language unlocking technology relies heavily on deep learning. It needs to train a specific single command model on the PC side, and then deploy it on the terminal. Users need to match fixed command actions. The effect of this method is poor, it does not adapt to the user's data, it can only adapt to fixed command actions, and the commands are easily exposed.

SUMMARY OF THE INVENTION

Purpose of the invention: Aiming at the deficiencies of the existing unlocking technology, a terminal unlocking method based on lip language instruction is provided.

Technical solution: a terminal unlocking method based on lip language instruction, comprising the following steps:

Step 1, the terminal camera collects the lip language instruction video frame unlocked by the user, the terminal performs face detection and extracts face features, and simultaneously extracts the lip region video frame;

Step 2, extract feature points from the lip video frame data set, and match the feature points of adjacent frames, and mark the position coordinates;

Step 3. Use the frame difference method to extract the change feature of the feature point position, that is, the algebraic feature of the lip movement;

Step 4. Match faces in the database;

Step 5. If the matching is successful, it is necessary to recognize that the person makes the same lip language command action to the terminal camera, and the terminal also extracts the lip feature points, and calculates the algebraic features of the lip movement to determine whether the match is an unlock command;

Step 6. When matching the face or the matching instruction is unsuccessful, it indicates that the matching fails, and skips to Step 4.

In a further embodiment, the step 1 is further:

Step 1-1. Calculate the color histogram of the RGB space for each frame of the video clip, each channel is divided into 32 intervals according to the pixel value, and normalized to obtain 96-dimensional features; the features of each frame are The vectors form a matrix, and the dimensionality reduction process is performed on the modified matrix to calculate the initial cluster center:

In the formula, C _n represents the cluster center of the nth segment, _fn represents the feature vector of the nth frame, and fn ₊₁ represents the n+1th feature vector;

Calculate the similarity of each new frame to the current cluster center, and specify a threshold σ. When the similarity is greater than the threshold, it is judged that f _n belongs to the cluster center C _n , and f _{n is} added to C _n at this time, Update to get a new cluster center C _n′ :

In the formula, _fn represents the feature vector of the nth frame, _Cn represents the cluster center of the nth segment, and _Cn′ represents the new cluster center obtained by updating;

When the similarity is less than the threshold, it is judged that f _n belongs to a new cluster center, and at this time, f _n is used to initialize the new cluster center C _n′ :

C _n′ =f _n

Step 1-2, firstly identify the contour of the face, remove the background, cut the lips of the face in the video frame, and locate the position of the contour points of the facial features in the face, including the coordinates of the tip of the nose and the leftmost lip. The coordinates of the side, the coordinates of the far right side of the lip, and the coordinates of the center point of the mouth. According to these coordinates, the image containing the details of the lip is cropped, and the crop size is calculated by this formula:

In the formula, L _MN represents the distance between the coordinates of the nose tip and the coordinates of the center point of the mouth, x _right represents the abscissa of the rightmost feature point of the lip, y _right represents the ordinate of the rightmost feature point of the lip, x _left represents the abscissa of the leftmost feature point of the lip, and y _left represents the ordinate of the leftmost feature point of the lip;

Steps 1-3, perform deviation correction on the cropped lip image, train the lip image based on the bipartite model of the convolutional neural network, and judge whether the extracted lip image is a valid image:

In the formula, l represents the number of convolution layers, k represents the convolution kernel, b represents the convolution bias, M _j represents the input local receptive value, β represents the output parameter, and down() represents the pooling function.

In a further embodiment, the step 2 is further:

Step 2-1. For the cropped image extracted in step 1, build a D3D model to accelerate network convergence, and introduce a loss function to correct the model:

表示的是交叉熵损失，{y_i＝k}为指示函数，logit(pre)表示网络输出概率，σ是比例系数；In the formula,

represents the cross-entropy loss, {y _i =k} is the indicator function, logit(pre) represents the network output probability, and σ is the proportional coefficient;

Among them, P({Z|X})=∑ _k=1 P(π||X), that is, the sum of the probabilities formed after all paths are combined;

Step 2-2, extract feature points from the images of two adjacent frames and obtain two sets of feature point sets:

p={p ₁ , p ₂ , p ₃ . . . p _n }

p'={p ₁ ', p ₂ ', p ₃ '...p _n '}

According to the center of the adjacent two groups of feature points, the pixel value of the window W of its neighborhood is used as the descriptor of the feature point, and the pixel interpolation of the neighborhood of the two groups of feature points is calculated separately:

In the formula, S represents the pixel interpolation of the two groups of feature points, x represents the abscissa of the pixel, y represents the ordinate of the pixel, W represents the field window, which is used as a descriptor in this formula, p represents the previous frame image, p' represents the next frame of image;

Step 2-3, according to the pixel interpolation obtained in step 2-2, to find the matching point according to the matching coefficient between the feature point and the neighborhood window:

In the formula, G represents the gray value of the image of the previous frame, G′ represents the gray value of the image of the next frame, C represents the matching coefficient, and other symbols have the same meanings as above.

In a further embodiment, the step 3 is further:

Step 3-1. Record the images of three adjacent individual frames, denoted as f(n+1), f(n), and f(n-1), respectively, and the gray values corresponding to the three frames of images are respectively denoted as G( n+1) ^x,y , G(n) ^x,y , G(n-1) ^x,y , using the frame difference method to get the image P′:

P′=|G(n+1) ^x,y -G(n) ^x,y |∩|G(n) ^x,y -G(n-1) ^x,y |

Compare the image P' with the preset threshold T to analyze the circulation, and extract the moving target. The comparison conditions are:

In the formula, N represents the total number of pixels in the area to be detected, τ represents the suppression coefficient of illumination, A represents the image of the complete frame, and T is the threshold.

In a further embodiment, the step 4 is further:

Step 4-1. On multi-user terminals, such as safes and door locks, face recognition needs to be performed to match whether the user's face exists in the database; on single-user private terminals, such as mobile phones and tablets, no need to perform face recognition. Face recognition, face verification is enough, the facenet network is used to calculate the Euclidean distance of the face features, and the comparison threshold is judged:

表示正样本对，

表示负样本对，

表示平样本对，α表示正样本对与负样本对之间的约束范围，Φ表示三元组的集合；In the formula,

represents a positive sample pair,

represents the negative sample pair,

represents the flat sample pair, α represents the constraint range between the positive sample pair and the negative sample pair, and Φ represents the set of triples;

Introduce the neuron model:

h _W,b (x)=f(W ^T x)

In the formula, W represents the weight vector of the neuron, W ^T x represents the nonlinear transformation of the input vector x, and f(W ^T x) represents the activation function transformation of the weight vector;

Assign the input vector x to x _i , bringing in W ^T x :

In the formula, n represents the series of the neural network, and b represents the bias.

In a further embodiment, the step 5 is further: in the acquisition process, the center of the lips is used as the coordinate origin to establish a coordinate axis, and the inner lip area in the lip grayscale image is fitted into a combination of two semi-ellipses, and the upper inner lip corresponds to The upper ellipse, the lower inner lip corresponds to the lower ellipse, and the frame difference method is used to extract the change feature of the corresponding feature point position, that is, the algebraic feature of the lip movement between frames:

Record the images of two adjacent separate frames, denoted as f(n+1), f(n), respectively, and the gray values corresponding to the two frames of images are denoted as G(n+1) ^x,y , G(n) ^{x, y} , and the image P' is obtained by the frame difference method:

P′=|G(n+1) ^x,y -G(n) ^x,y |

Compare the image P' with the preset threshold T to analyze the circulation, and extract the moving target. The comparison conditions are:

In the formula, N represents the total number of pixels in the area to be detected, τ represents the suppression coefficient of illumination, A represents the image of the complete frame, and T is the threshold.

Beneficial effects: The present invention relates to a terminal unlocking method based on lip language commands. When collecting, users can design command actions by themselves, and they can make the same actions when recognizing, so that action commands are not easily stolen by others, and the security of authentication is improved. sex. At the same time, the lip language instruction unlocking method does not require large-scale operations on the terminal, which greatly reduces the hardware performance requirements and improves the recognition speed. The invention can avoid the problem of excessive gradient caused by the accumulation of a certain quadrant in the space by reducing the matrix, extracting feature points, initializing the cluster center, and using the facenet network to calculate the Euclidean distance of the face features, and improving the network learning and The training efficiency has the effect of actively learning the training model, which solves the problem that the traditional fixed command actions are easy to be exposed.

Description of drawings

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a schematic diagram of establishing a coordinate system for lips according to the present invention.

FIG. 3 is a cropped image including lip details in the lip language unlocking instruction of the present invention.

FIG. 4 is a schematic diagram of introducing a neuron model according to the present invention.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without one or more of these details. In other instances, some technical features known in the art have not been described in order to avoid obscuring the present invention.

The applicant believes that in the field of lip language unlocking, the existing technology relies heavily on deep learning, and a specific single command model needs to be trained on the PC side, and then deployed on the terminal for use, and the user needs to match fixed command actions. The effect of this method is poor, and it does not adapt to the user's data, but can only adapt to fixed command actions, and the commands are easily exposed. Therefore, how to build a lip language model and continuously improve the active learning of the machine Sex is crucial.

In order to solve the above-mentioned problems existing in the prior art, the present invention proposes a terminal unlocking method based on lip language instructions. When collecting, the user can design an instruction action by himself, and the same action can be made when recognizing, so that the action instruction is not easy. It is stolen by others, which improves the security of authentication.

The technical solutions of the present invention will be further described below through examples and in conjunction with the corresponding drawings.

First, the terminal camera collects the lip language instruction video frame of the user unlocking, the terminal detects the face and extracts the face features, and extracts the video frame of the lip area at the same time; Each channel is divided into 32 intervals according to the pixel value, and normalized to obtain 96-dimensional features; the eigenvectors of each frame are formed into a matrix, and the dimensionality reduction process is performed on the modified matrix to calculate the initialization cluster center:

In the formula, C _n represents the cluster center of the nth segment, _fn represents the feature vector of the nth frame, and fn ₊₁ represents the n+1th feature vector;

Calculate the similarity of each new frame to the current cluster center, and specify a threshold σ. When the similarity is greater than the threshold, it is judged that f _n belongs to the cluster center C _n , and f _{n is} added to C _n at this time, Update to get a new cluster center C _n′ :

In the formula, _fn represents the feature vector of the nth frame, _Cn represents the cluster center of the nth segment, and _Cn′ represents the new cluster center obtained by updating;

When the similarity is less than the threshold, it is judged that f _n belongs to a new cluster center, and at this time, f _n is used to initialize the new cluster center C _n′ :

C _n′ =f _n

Identify the contour of the face, remove the background, cut the lips of the face in the video frame, and locate the contour points of the facial features in the face, including the coordinates of the tip of the nose, the coordinates of the leftmost lip, and the lip The rightmost coordinates, the coordinates of the center point of the mouth, according to these coordinates, the image containing the lip details is cropped, and the crop size is calculated by this formula:

In the formula, L _MN represents the distance between the coordinates of the nose tip and the coordinates of the center point of the mouth, x _right represents the abscissa of the rightmost feature point of the lip, y _right represents the ordinate of the rightmost feature point of the lip, x _left represents the abscissa of the leftmost feature point of the lip, and y _left represents the ordinate of the leftmost feature point of the lip;

Correct the deviation of the cropped lip image, train the bipartite model based on the convolutional neural network on the lip image, and judge whether the extracted lip image is a valid image:

In the formula, l represents the number of convolution layers, k represents the convolution kernel, b represents the convolution bias, M _j represents the input local receptive value, β represents the output parameter, and down() represents the pooling function.

Next, extract feature points to the lip video frame data set, and match the feature points of adjacent frames, and mark the position coordinates;

For the extracted cropped image, build a D3D model to accelerate network convergence, and introduce a loss function to correct the model:

表示的是交叉熵损失，{y_i＝k}为指示函数，logit(pre)表示网络输出概率，σ是比例系数；In the formula,

represents the cross-entropy loss, {y _i =k} is the indicator function, logit(pre) represents the network output probability, and σ is the proportional coefficient;

Among them, P({Z|X})=∑ _k=1 P(π||X), that is, the sum of the probabilities formed after all paths are combined;

Extract feature points from the images of two adjacent frames and obtain two sets of feature points:

p={p ₁ , p ₂ , p ₃ … p _n }

p′={p ₁ ′, p ₂ ′, p ₃ ′ … p _n ′}

According to the center of the adjacent two groups of feature points, the pixel value of the window W of its neighborhood is used as the descriptor of the feature point, and the pixel interpolation of the neighborhood of the two groups of feature points is calculated separately:

In the formula, S represents the pixel interpolation of the two groups of feature points, x represents the abscissa of the pixel, y represents the ordinate of the pixel, W represents the field window, which is used as a descriptor in this formula, p represents the previous frame image, p' represents the next frame of image;

According to the pixel interpolation obtained above, to find matching points according to the matching coefficient between the feature points and the neighborhood window:

In the formula, G represents the gray value of the image of the previous frame, G′ represents the gray value of the image of the next frame, C represents the matching coefficient, and other symbols have the same meanings as above.

Next, use the frame difference method to extract the change feature of the feature point position, that is, the algebraic feature of the lip movement; record the images of three adjacent individual frames, denoted as f(n+1), f(n), f(n- 1), the gray values corresponding to the three frames of images are respectively recorded as G(n+1) ^x,y , G(n) ^x,y , G(n-1) ^x,y , and the image P′ is obtained by the frame difference method :

P′=|G(n+1) ^x,y -G(n) ^x,y |∩|G(n) ^x,y -G(n-1) ^x,y |

Compare the image P' with the preset threshold T to analyze the circulation, and extract the moving target. The comparison conditions are:

In the formula, N represents the total number of pixels in the area to be detected, τ represents the suppression coefficient of illumination, A represents the image of the complete frame, and T is the threshold.

Step 4. Match faces in the database: On multi-user terminals, such as safes and door locks, face recognition needs to be performed to match whether the user's face exists in the database; on single-user private terminals, such as mobile phones, The tablet does not need face recognition, just face verification. The facenet network is used to calculate the Euclidean distance of the face features, and the comparison threshold is judged:

表示正样本对，

表示负样本对，

表示平样本对，α表示正样本对与负样本对之间的约束范围，Φ表示三元组的集合；In the formula,

represents a positive sample pair,

represents the negative sample pair,

represents the flat sample pair, α represents the constraint range between the positive sample pair and the negative sample pair, and Φ represents the set of triples;

Introduce the neuron model:

h _W,b (x)=f(W ^T x)

In the formula, W represents the weight vector of the neuron, W ^T x represents the nonlinear transformation of the input vector x, and f(W ^T x) represents the activation function transformation of the weight vector;

Assign the input vector x to x _i , bringing in W ^T x :

In the formula, n represents the series of the neural network, and b represents the bias.

Step 5. If the matching is successful, it is necessary to recognize that the person makes the same lip language command action to the terminal camera, and the terminal also extracts the lip feature points, and calculates the algebraic features of the lip movement to determine whether the match is an unlock command; during the acquisition process Use the center of the lips as the coordinate origin to establish the coordinate axis, and fit the inner lip area in the gray image of the lips into a combination of two semi-ellipses. The upper inner lip corresponds to the upper ellipse, and the lower inner lip corresponds to the lower ellipse. The frame difference method is used to extract the corresponding feature points. The change feature of the position is the algebraic feature of the lip movement between frames:

Record the images of two adjacent separate frames, denoted as f(n+1), f(n), respectively, and the gray values corresponding to the two frames of images are denoted as G(n+1) ^x,y , G(n) ^{x, y} , and the image P' is obtained by the frame difference method:

P′=|G(n+1) ^x,y -G(n) ^x,y |

Compare the image P' with the preset threshold T to analyze the circulation, and extract the moving target. The comparison conditions are:

In the formula, N represents the total number of pixels in the area to be detected, τ represents the suppression coefficient of illumination, A represents the image of the complete frame, and T is the threshold.

When matching the face or the matching instruction is unsuccessful, it will prompt that the matching fails, continue to match the face in the database, repeat the above steps, and temporarily lock the terminal device if the matching fails more than three times.

To sum up, in view of the deficiencies of the prior art, the present invention proposes a terminal unlocking method based on lip language instructions. During the acquisition process, several frames of images are taken to acquire the face, and some key feature points are extracted. In the verification process, the key feature points that need to be recognized for the face are extracted in the same way, and the Euclidean distance of the face features is calculated by the facenet network, and the threshold is compared and judged. In the acquisition process, the center of the lips is used as the coordinate origin to establish the coordinate axis, and the inner lip area in the lip grayscale image is fitted into a combination of two semi-ellipses (the upper inner lip corresponds to the upper ellipse, and the lower inner lip corresponds to the lower ellipse), and the frame difference is used. The method extracts the change feature of the corresponding feature point position, that is, the algebraic feature of the lip motion between frames, and calculates the judgment threshold. In the verification process, the lip motion features are extracted in the same way for comparison and judgment. By reducing the matrix dimension, extracting feature points, initializing cluster centers, and using facenet network to calculate the Euclidean distance of face features, the problem of excessive gradients caused by accumulation of a certain quadrant in the space can be avoided, and the efficiency of network learning and training can be improved. , has the effect of active learning and training model, and solves the problem that the traditional fixed command action is easy to be exposed.

As mentioned above, although the present invention has been shown and described with reference to specific preferred embodiments, this should not be construed as limiting the invention itself. Various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims (1)

1. a terminal unlocking method based on lip language instruction is characterized in that comprising the following steps: Step 1, the terminal camera collects the lip language instruction video frame unlocked by the user, the terminal performs face detection and extracts face features, and simultaneously extracts the lip region video frame; Step 1-1. Calculate the color histogram of the RGB space for each frame of the video clip, each channel is divided into 32 intervals according to the pixel value, and normalized to obtain 96-dimensional features; the features of each frame are The vectors form a matrix, and the dimensionality reduction process is performed on the modified matrix to calculate the initial cluster center:

In the formula,

represents the cluster center of the nth fragment,

represents the feature vector of the nth frame,

Represents the n+1th eigenvector; Calculate the similarity of each new frame to the current cluster center and specify a threshold

, when the similarity is greater than the threshold, judge

belong to the cluster center

, then the

join in

, update the new cluster center

:

In the formula,

represents the feature vector of the nth frame,

represents the cluster center of the nth fragment,

Indicates that the update obtains a new cluster center; When the similarity is less than the threshold, judge

belongs to the new cluster center, at this time use

Initialize new cluster centers

:

Step 1-2, firstly identify the contour of the face, remove the background, cut the lips of the face in the video frame, and locate the position of the contour points of the facial features in the face, including the coordinates of the tip of the nose and the leftmost lip. The coordinates of the side, the coordinates of the far right side of the lip, and the coordinates of the center point of the mouth. According to these coordinates, the image containing the details of the lip is cropped, and the crop size is calculated by this formula:

In the formula,

represents the distance between the coordinates of the tip of the nose and the coordinates of the center of the mouth,

represents the abscissa of the rightmost feature point of the lip,

represents the ordinate of the rightmost feature point of the lip,

represents the abscissa of the leftmost feature point of the lip,

Represents the ordinate of the leftmost feature point of the lip; Steps 1-3, perform deviation correction on the cropped lip image, train the lip image based on the bipartite model of the convolutional neural network, and judge whether the extracted lip image is a valid image:

where l represents the number of convolution layers, k represents the convolution kernel, b represents the convolution bias,

represents the local receptive value of the input,

represents the output parameter,

represents the pooling function; Step 2, extract feature points from the lip video frame data set, and match the feature points of adjacent frames, and mark the position coordinates; Step 2-1. For the cropped image extracted in step 1, build a D3D model to accelerate network convergence, and introduce a loss function to correct the model:

In the formula,

represents the cross-entropy loss,

is the indicator function,

represents the network output probability,

is the scale factor; in,

, that is, the sum of the probabilities formed after all paths are merged; Step 2-2, extract feature points from the images of two adjacent frames and obtain two sets of feature point sets:

According to the center of the adjacent two groups of feature points, the pixel value of the window W of its neighborhood is used as the descriptor of the feature point, and the pixel interpolation of the neighborhood of the two groups of feature points is calculated separately:

In the formula, S represents the pixel interpolation of the two groups of feature points, x represents the abscissa of the pixel, y represents the ordinate of the pixel, W represents the field window, which is used as a descriptor in this formula, p represents the previous frame image,

Represents the next frame of image; Step 2-3, according to the pixel interpolation obtained in step 2-2, to find the matching point according to the matching coefficient between the feature point and the neighborhood window:

In the formula,

represents the gray value of the previous frame image,

Represents the gray value of the next frame of image, C represents the matching coefficient, and the rest of the symbols have the same meaning as above; Step 3. Use the frame difference method to extract the change feature of the feature point position, that is, the algebraic feature of the lip movement; Step 3-1. Record the images of three adjacent individual frames, respectively denoted as

,

,

, the corresponding grayscale values of the three frames of images are respectively recorded as

,

,

, using the frame difference method to obtain the image

:

the image

Compare and analyze the liquidity with the preset threshold T , and extract the moving target. The comparison conditions are:

In the formula,

represents the total number of pixels in the area to be detected,

represents the suppression coefficient of illumination,

represents the image of the complete frame, T is the threshold; Step 4. Match faces in the database; Step 4-1. On multi-user terminals, such as safes and door locks, face recognition needs to be performed to match whether the user's face exists in the database; on single-user private terminals, such as mobile phones and tablets, no need to perform face recognition. Face recognition, face verification is enough, the facenet network is used to calculate the Euclidean distance of the face features, and the comparison threshold is judged:

In the formula,

represents a positive sample pair,

represents the negative sample pair,

represents a flat sample pair,

represents the constraint range between the positive sample pair and the negative sample pair,

represents a set of triples; Introduce the neuron model:

In the formula,

represents the weight vector of the neuron,

represents the nonlinear transformation of the input vector x ,

Represents the activation function transformation of the weight vector; Assign the input vector x as

, bring in

:

In the formula, n represents the series of the neural network, and b represents the bias; Step 5. If the matching is successful, it is necessary to recognize that the person makes the same lip language command action to the terminal camera, and the terminal also extracts the lip feature points, and calculates the algebraic features of the lip movement to determine whether the match is an unlock command; In the acquisition process, the center of the lip is used as the coordinate origin to establish the coordinate axis, and the inner lip area in the gray image of the lips is fitted into a combination of two semi-ellipses. The upper inner lip corresponds to the upper ellipse, and the lower inner lip corresponds to the lower ellipse. The frame difference method is used. Extract the change feature of the corresponding feature point position, that is, the algebraic feature of the lip movement between frames: Record the images of two adjacent separate frames, denoted as

,

, the corresponding gray values of the two frames of images are respectively recorded as

,

, using the frame difference method to obtain the image

:

the image

Compare and analyze the liquidity with the preset threshold T , and extract the moving target. The comparison conditions are:

In the formula,

represents the total number of pixels in the area to be detected,

represents the suppression coefficient of illumination,

represents the image of the complete frame, T is the threshold,

represents the gray value corresponding to the nth frame image,

Indicates the gray value corresponding to the n+1th frame image; Step 6. When matching the face or the matching instruction is unsuccessful, it indicates that the matching fails, and skips to Step 4.

Images