TWI722705B - Method for automatically labelling muscle feature points on face - Google Patents

Abstract

A method for automatically labelling muscle feature points on face implemented by a face image analysis apparatus is disclosed and includes following steps: obtaining an identified image showing a face of a user; performing a face recognition procedure to the identified image for obtaining multiple strong reference points on the face; performing a fuzzy comparison procedure on the face of the identified image based on a pre-trained training model for generating a comparison result; automatically labelling multiple muscle feature points on the face according to the comparison result, wherein the multiple muscle feature points respectively locate at multiple weak reference points of the face; and, displaying the multiple muscle feature points and the identified image on a display unit.

Description

Automatic marking method of facial muscle feature points

The invention relates to a method for marking a face, in particular to a method for marking facial muscle feature points.

Since human muscles (especially facial muscles) will gradually become loose and sagging with age, some users will choose to use skin care products to maintain muscles and skin, and use cosmetics to cover up Relaxation of muscles, or through exercise, medical beauty and other means to slow down the occurrence of the above-mentioned relaxation and sagging phenomenon.

Generally speaking, users will sit in front of a mirror to perform maintenance and makeup, or use a smartphone, a tablet computer, or a special makeup assisting device to assist the maintenance and makeup actions in order to increase the speed.

However, the above-mentioned device can only assist the user in maintenance and makeup, but cannot actively analyze the user's current muscle state, and cannot let the user know whether the maintenance and makeup actions they perform are really effective.

In view of this, there is a need in this technical field to provide a novel method that can effectively analyze the user’s facial image and automatically mark the user’s current muscle state. The multiple muscle feature points, which allows the user to easily understand the current muscle state and judge whether the maintenance method performed is effective.

The main purpose of the present invention is to provide a method for automatically marking facial muscle feature points, which can automatically mark a plurality of muscle feature points representing the state of facial muscles on a user's facial image.

In order to achieve the above objective, the method for automatically marking facial muscle feature points of the present invention is mainly applied to a facial image analysis device, and includes the following steps: obtaining a to-be-recognized image, wherein the to-be-recognized image includes at least the user’s face; Perform a face recognition procedure on the image to be recognized to obtain multiple strong reference points on the face; perform blur comparison on the image to be recognized based on a training model completed in advance and generate a recognition result; according to the recognition As a result, a plurality of muscle feature points are marked on the image to be recognized, wherein the plurality of muscle feature points respectively fall on the plural weak reference points on the face; and, the image to be recognized and the plurality of muscle feature points are overlapped and displayed by a display unit .

Compared with the related technology, the present invention can automatically mark at least four muscle feature points by recognizing the user's facial image, and the muscle feature points marked by the present invention can be used to represent the muscle state of the face. Therefore, the user can quickly understand his current facial muscle state through the marked muscle feature points, and confirm whether the maintenance measures taken are effective.

1: Facial image analysis device

10: processor

11: Display unit

12: Image capture unit

13: Input unit

14: wireless transmission unit

15: storage unit

151: Artificial Intelligence Training Algorithm

152: Training Materials

153: training model

1531: The first regression tree

1532: The second regression tree

1533: the nth regression tree

1534: Judgment endpoint

1535: weight

2: Image to be recognized

3, 4: muscle feature points

31, 41: the first muscle feature point

32, 42: Second muscle feature point

33, 43: third muscle feature point

34, 44: Fourth muscle feature point

5: Face positioning frame

61: First tear groove tangent

62: Second tear groove tangent

63: The first law pattern tangent

64: The second law pattern tangent

65: The first puppet pattern tangent

66: The second puppet pattern tangent

67: Vertical line at the first corner of the eye

68: Vertical line at the second corner of the eye

69: First jawbone branch tangent

70: Second jawbone branch tangent

80: predict feature points

81: The first adjusted forecast point

82: Second adjusted forecast point

83: adjusted prediction point n-1

84: Final prediction point

S10~S20: Establishment steps

S30~S36: Identification steps

S320~S330: Identification steps

FIG. 1 is a first specific embodiment of a schematic diagram of a facial image analysis device of the present invention.

Fig. 2 is a first embodiment of the block diagram of the facial image analysis device of the present invention.

Fig. 3 is a first specific embodiment of the schematic diagram of muscle feature points of the present invention.

Fig. 4 is a first specific embodiment of the flow chart of establishing a training model of the present invention.

Fig. 5 is a first specific embodiment of the schematic diagram of the marking position of the muscle feature points of the present invention.

Fig. 6 is a first specific embodiment of the training data schematic diagram of the present invention.

FIG. 7 is a first specific embodiment of the identification flowchart of the present invention.

FIG. 8 is a second specific embodiment of the identification flowchart of the present invention.

Fig. 9 is a first specific embodiment of a schematic diagram of a training model of the present invention.

Fig. 10 is a first specific embodiment of the feature point adjustment schematic diagram of the present invention.

With regard to a preferred embodiment of the present invention, the detailed description is given below in conjunction with the drawings.

Please refer to FIGS. 1 and 2. FIG. 1 is a first specific embodiment of a schematic diagram of a facial image analysis device of the present invention, and FIG. 2 is a first specific embodiment of a block diagram of the facial image analysis device of the present invention.

The present invention discloses a method for automatically marking facial muscle feature points (hereinafter referred to as the marking method), which is mainly applied to the facial image analysis device 1 as shown in FIG. 1 and FIG. 2 (hereinafter referred to as the analysis device 1 for short). ). Specifically, the present invention mainly obtains the facial image of the user through the analysis device 1, and recognizes the facial image based on the pre-trained artificial intelligence model, and then marks the facial image based on the recognition result. Multiple muscle feature points that can be used to represent the user's current facial muscle state. Through the multiple muscle feature points automatically marked by the analysis device 1, the user can quickly and effectively understand his current facial muscle state, and then evaluate his current use of cosmetics, skin care products and various maintenance methods (such as (Exercise or medical beauty) is effective.

The analysis device 1 shown in Fig. 1 and Fig. 2 is mainly used to assist the user in applying makeup. Specifically, the analysis device 1 can provide makeup suggestions before the user applies makeup, provide makeup assistance when applying makeup, and analyze and evaluate makeup after the makeup is completed. It is worth mentioning that other electronic devices (such as smart mobile devices, tablet computers, etc.) have hardware components similar to the analysis device 1, and the corresponding application software is installed to perform the various controls of the marking method of the present invention. In the case of steps, the marking method of the present invention is not limited to being applied to the analysis device 1 shown in FIG. 1 and FIG. 2, and can be implemented by various electronic devices.

As shown in FIGS. 1 and 2, the analysis device 1 mainly includes a processor 10, a display unit 11, an image capture unit 12, an input unit 13, a wireless transmission unit 14 and a storage unit 15. The processor 10 is electrically connected to the display unit 11, the image capture unit 12, the input unit 13, the wireless transmission unit 14 and the storage unit 15 through a bus to integrate and control these components.

The analysis device 1 can instantly capture a user's photo (mainly a photo containing the user's facial image) through the image capturing unit 12, and extract the user's facial image from the photo and display it on the display unit 11. On display. The main technical feature of the present invention is that the analysis device 1 can recognize the captured facial image, and automatically mark the facial image with multiple muscle feature points that can be used to represent the user's current facial muscle state , And the display unit 11 simultaneously displays the facial image and multiple muscle feature points. In this way, the user can quickly confirm his current muscle state by analyzing the information displayed on the device 1.

The input unit 13 is arranged on one side of the analysis device 1 and can be a physical input unit or a touch input unit. Through the input unit 13, the user can interact with the analysis device 1 to operate the analysis device 1 and send instructions to the analysis device 1. For example, users can pass The input unit 13 selects different functions of the analysis device 1 (such as makeup assist function, facial muscle analysis function, etc.), or switches the makeup steps or makeup suggestions (such as next page, previous page) provided by the analysis device 1.

In one embodiment, the display unit 11 may be a touch screen capable of accepting user operations. In this embodiment, the input unit 13 and the display unit 11 are integrated as a whole, and do not exist separately.

The wireless transmission unit 14 is used to connect to a network, and the analysis device 1 can be connected to a remote electronic device or server through the network. In the present invention, the analysis device 1 can mainly use one or more algorithms to perform artificial intelligence model training procedures, facial image recognition procedures, and muscle feature points marking procedures. The smart model can be stored in the analysis device 1 or in a remote electronic device or server, without limitation. Furthermore, the user can also operate a user terminal (not shown in the figure) to connect to the analysis device 1 via a network, so as to directly maintain and update the firmware of the analysis device 1 from a remote location.

In one embodiment, the analysis device 1 captures the user's facial image in real time through the image capturing unit 12, and recognizes the facial image to analyze the user's current facial muscle state. In another embodiment, the analysis device 1 can also use the wireless transmission unit 14 to download photos taken in advance by the user from a remote electronic device or server, and recognize the facial images in the photos to Evaluate the state of the facial muscles of the user when the photo was taken. In another embodiment, the analysis device 1 can also read the photos taken by the user in advance from the storage unit 15 and recognize the facial images in the photos to evaluate the user’s performance when taking the photos. The state of facial muscles.

The storage unit 15 stores the algorithms and models required by the analysis device 1 to execute the marking method of the present invention. Specifically, at least a training model 153 used for fuzzy comparison is stored and used for training the training Model 153's artificial intelligence training algorithm 151 and a lot of training resources 152, but not limited. As mentioned above, the artificial intelligence training algorithm 151, training data 152, and training model 153 can also be stored in a remote electronic device or server, and the analysis device 1 can be connected to a remote electronic device or server via a network. , And remotely access the artificial intelligence training algorithm 151, training data 152, and training model 153.

In another embodiment, the artificial intelligence training algorithm 151 can be embedded in the processor 10 as a part of the firmware of the processor 10, but it is not limited.

In the present invention, the manufacturer of the analysis device 1 can import a large amount of training data 152 into the storage unit 15 in advance. These training data 152 are mainly facial images of unspecified people, and manually mark these facial images. Multiple muscle feature points defined by experts (such as physicians, beauticians, etc.) are generated. In this way, the analysis device 1 can perform the training procedure of the training model 153 on the training data 152 through the artificial intelligence training algorithm 151. In the present invention, the positions of the multiple muscle feature points can be used to represent the muscle state of the face.

The main technical feature of the present invention is that when the analysis device 1 obtains a new photo (for example, the user’s face image is captured in real time by the image capture unit 12, or a pre-photographed image is read from the storage unit 15). Photo), based on the training model 153, the photo can be blurred and compared to automatically mark the positions of multiple muscle feature points on the facial image in the photo. In this way, the user can determine the current muscle state of his face based on the position of the muscle feature points automatically marked by the analysis device 1.

Please also refer to FIG. 3, which is a first specific embodiment of the schematic diagram of the muscle feature points of the present invention. As shown in FIG. 3, after the analysis device 1 obtains a to-be-recognized image 2 containing the user's facial image through the image capturing unit 12, the wireless transmission unit 14, or the storage unit 15, it can be based on the pre-training completed The training model 153 performs a fuzzy comparison on the image 2 to be identified to find out the image 2 The position of the complex muscle feature point 3 on the facial image of, and the multiple muscle feature 3 is marked on the image 2 to be recognized and then displayed on the display unit 11.

In the embodiment of FIG. 3, the analysis device 1 can automatically mark at least four muscle feature points 3 in the image 2 to be recognized based on the training model 153, including the first muscle located on the upper left side of the face (corresponding to the position of the laughing muscle on the left) Feature point 31, the second muscle feature point 32 located on the lower left side of the face (corresponding to the position of the left masseter muscle), the third muscle feature point 33 located on the upper right side of the face (corresponding to the position of the right laughing muscle), and the third muscle feature point 33 located on the lower right side of the face ( The fourth muscle feature point 34 corresponding to the position of the right masseter muscle. In the embodiment of FIG. 3, four muscle feature points 3 are taken as an example, but in other embodiments, the number of muscle feature points 3 is not limited to four.

It is worth mentioning that within a certain range, if the positions of the first muscle feature point 31 and the third muscle feature point 33 are closer to the inside of the face (close to the nose) and the upper side of the face (close to the eyes), it means using The better the muscles are (for example, the younger, the tighter the muscles). Similarly, within a certain range, if the positions of the second muscle feature point 32 and the fourth muscle feature point 34 are closer to the inside of the face (near the mouth) and the upper side of the face (near the nose), it indicates the muscles of the user The better the state.

As mentioned above, the marking method of the present invention is based on the pre-trained training model 153 to recognize the image 2 to be recognized, and automatically mark at least four muscle feature points 3 on the image 2 to be recognized after the recognition is completed. Effectively achieve the technical effect of allowing users to judge their current facial muscle state by the positions of at least four muscle feature points 3.

Compared with multiple strong reference points on the face (such as eyes, nose, mouth and other organs, or parts with obvious characteristics), the muscle feature point 3 referred to in the present invention mainly corresponds to the weak reference points on the face point. Therefore, if the analysis device 1 is used to implement the marking method of the present invention to automatically input for the user The imported image 2 to be recognized marks the muscle feature points 3, and the analysis device 1 needs to establish a training model 153 for performing blur comparison in advance.

Please continue to refer to FIG. 4, which is a first specific embodiment of the flow chart for establishing the training model of the present invention. Fig. 4 is used to specifically illustrate the procedure for establishing the training model 153 of the present invention.

As shown in FIG. 4, first, the user obtains a large amount of training data 152, which are facial images of the same or different people. Then, the user marks the training data 152 with at least four muscle feature points based on the feature point setting rules defined by experts (such as doctors or beauticians) based on the actual facial muscle states shown in these facial images (step S10 ). As mentioned above, the present invention uses at least four muscle feature points to represent the muscle state of the face. In other words, during the training phase, the user can also manually update the training data 152 according to the muscle state of the facial image. Mark the muscle feature points as the training basis of the training model 153.

Please also refer to FIG. 5, which is a first specific embodiment of the schematic diagram of the marking position of the muscle feature points of the present invention. In the present invention, the muscle feature points 4 mainly include a first muscle feature point 41 and a second muscle feature point 42 located on the left side of the face, and a third muscle feature point 43 and a fourth muscle feature point 44 located on the right side of the face. . Moreover, the at least four muscle feature points 4 are respectively located on four weak reference points of the face. Wherein, the number of weak reference points corresponds to the number of muscle feature points 4. Specifically, if the number of muscle feature points 4 marked on the training data 152 by the user is greater than four, each muscle feature point 4 needs to be located on a weak reference point of the face.

More specifically, as shown in FIG. 5, the user can analyze and judge the facial image in the training data 152 with the naked eye or an algorithm, so as to obtain the first tear on the left side of the face based on the actual muscle state of the face. The groove tangent line 61, the first law pattern tangent line 63, the first marionette line tangent line 65, the first corner vertical line 67, and the first mandible ramus tangent line 67 constitute auxiliary lines and obtain the face The second tear groove tangent 62, the second law pattern tangent 64, the second marionette tangent 66, the second corner vertical line 68, and the second jawbone branch tangent 70 on the right constitute auxiliary lines. In the present invention, the user can determine the specific position of each muscle feature point 4 by forming the auxiliary line of the above-mentioned area, and directly mark at least four muscle feature points 4 on the training data 152.

When the user is marking the muscle feature point 4, it is mainly to make the first muscle feature point 41 fall on the first tear groove tangent line 61, the first law pattern tangent line 63, the first eye corner vertical line 67 and the left side of the face. Within the area formed by the first jawbone branch tangent line 69; make the second muscle feature point 42 fall on the first decree pattern tangent line 61, the first marionette pattern tangent line 65, the first eye corner vertical line 67 and the first jawbone on the left side of the face Within the area formed by the branch tangent line 69; the third muscle feature point 43 is made to fall on the second tear groove tangent line 62, the second nasolabial tangent line 64, the second eye vertical line 68 and the second jawbone branch tangent line 70 on the right side of the face Within the area constituted; and make the fourth muscle feature point 44 fall on the right side of the face by the second law pattern tangent 64, the second marionette pattern tangent 66, the second eye corner vertical line 68 and the second jawbone branch tangent 70 Within the area. However, the above is only one of the specific implementation examples of the present invention, but it is not limited thereto.

In addition, the analysis device 1 can also separately determine whether the muscle feature points 4 fall within the area constituted by the auxiliary line of the corresponding area, as an aid for the user to pre-determine whether there is an error in marking the muscle feature points 4 , In order to filter out the wrong marking situation or raise a warning for the user to confirm again. Similarly, such an auxiliary method can also be selectively used as an auxiliary method for checking whether the result of the automatic marking of the training model 153 is incorrect after the training model 153 described later is established, especially when the training data 152 is relatively insufficient. Can play a certain auxiliary function.

Return to Figure 4. The user also uses the analysis device 1 or other electronic devices to perform a preliminary face recognition procedure on the training data 152, thereby locating facial features from the training data 152 that can be used as a basis for training. In one embodiment, the analysis device 1 is mainly executed by the processor 10 The Histogram of Oriented Gradient (HOG) algorithm of the Dlib Face Landmark system executes the face recognition program on the training data 152, and generates the face that indicates the effective facial features on the training data 152 Position the frame (step S12). In one embodiment, at least part of the facial features are strong reference points in the face.

Please refer to FIG. 6 at the same time, which is a first specific embodiment of the training data schematic diagram of the present invention. As shown in Figure 6, the HOG algorithm mainly converts a photo into a complex vector, and determines the position of the face image in the photo according to the combination of the size, direction, and shape of these vectors, and generates a picture that can cover the entire A face positioning frame 5 of the face image. In this embodiment, the four muscle feature points 4 all fall within the face positioning frame 5, and the connection of the four muscle feature points 4 can form a rectangular frame or a trapezoidal frame.

Return to Figure 4. After step S12, the analysis device 1 can execute the artificial intelligence training algorithm 151 according to the training data 152 (step S14) to build the training model 153. More specifically, the artificial intelligence training algorithm 151 is based on the training data 152, the marked muscle feature points 4 on the training data 152, the face positioning frame 5 on the training data 152, and the plural faces in the face positioning frame 5. Feature points to perform the training program.

It is worth mentioning that the user can selectively mark the muscle feature points 4 first or generate the face positioning frame 5 first according to the actual situation. In other words, the steps S10 and S12 do not have an inevitable sequence relationship in execution.

In the present invention, the artificial intelligence training algorithm 151 mainly analyzes the training data 152 during the training process, and records at least the correspondence between the multiple feature points in the face positioning frame 5, the The corresponding relationship between the plural muscle feature points 4 (such as the rectangular frame or ladder The size, shape, angle, etc. of the shaped frame, and the correspondence between each muscle feature point 4 and one or more feature points (especially strong reference points) in the face positioning frame 5 (step S16).

It is worth mentioning that in step S16, the artificial intelligence training algorithm 151 can count the probability of each muscle feature point 4 appearing at each position while executing the training program (for example, where these muscle feature points 4 have never appeared) Position), and the basic positioning rules of each muscle feature point 4 (for example, the first muscle feature point 41 must be higher than the second muscle feature point 42, and the third muscle feature point 43 must be located on the right side of the first muscle feature point 41, etc.) , Is used as the prediction reference value of the analysis device 1 when performing blur comparison for the newly inputted image 2 to be recognized (detailed later).

After step S16, the artificial intelligence training algorithm 151 can generate a plurality of muscle feature point positioning rules based on the analyzed correspondence relationship (step S18), and establish a training model at least according to the plurality of muscle feature point positioning rules, judgment depth and regression times 153 (Step S20). After the training model 153 is established, the analysis device 1 can perform a fuzzy comparison on the newly input image 2 to be recognized, so as to automatically mark the plural muscle features on the facial image in the image 2 to be recognized based on the training model 153 Point 3.

In the present invention, the training model 153 generated by the artificial intelligence training algorithm 151 is mainly a regressor, which includes multiple Cascade regression trees with the same content. Each regression tree has a plurality of judgment endpoints, and the content of at least a part of the judgment endpoints corresponds to the above-mentioned multiple muscle feature point positioning rules (for example, as shown in FIG. 9 and detailed later).

It is worth mentioning that before executing the artificial intelligence training algorithm 151, the user can set the actual factors such as the hardware performance of the analysis device 1, the required recognition accuracy, and the acceptable execution time. The number of regression times and the number of endpoints. In this embodiment, the number of complex regression trees is the same as the number of regressions, and the number of complex judgment endpoints is the same as the judgment depth. Without considering the hard In the case of physical performance, identification accuracy, and execution time, the greater the number of regressions and the depth of judgment, the more accurate the identification results will be.

Refer to FIG. 7, which is a first specific embodiment of the identification flowchart of the present invention. FIG. 7 discloses each execution step of the marking method of the present invention, and the marking method is mainly applied to the analysis device 1 shown in FIG. 1 and FIG. 2.

When the user wants to recognize the muscle state of the face through the analysis device 1, the image to be recognized 2 (as shown in FIG. 3) containing the user's facial image can be input to the analysis device 1, so that the analysis device 1 Obtain the user's to-be-recognized image 2 (step S30). In one embodiment, the analysis device 1 can capture the image 2 to be recognized in real time through the image capture unit 12 thereon. In another embodiment, the analysis device 1 can read from the storage unit 15 the to-be-identified image 2 pre-stored by the user. In another embodiment, the analysis device 1 can receive the image 2 to be recognized from the outside via the wireless transmission unit 14.

After obtaining the image 2 to be recognized, the analysis device 1 can execute a face recognition program on the image 2 to be recognized through the processor 10 to generate a face positioning frame 5 as shown in FIG. 6 (step S32). The face positioning frame 5 marks the face image in the image to be recognized 2, and the face positioning frame 5 covers multiple strong reference points in the face image (such as eyes, nose, mouth and other organs, or Obvious feature points).

In one embodiment, the processor 10 mainly uses the Histogram of Oriented Gradient (HOG) algorithm of the Dlib Face Landmark system to execute the face recognition program on the image 2 to be recognized, and perform the face recognition program on the image 2 to be recognized. The face positioning frame 5 is generated. In the present invention, the processor 10 only performs training based on the multiple feature points in the face positioning frame 5 of the training data 152. Therefore, when performing the recognition process, the processor 10 also only performs training based on the face positioning frame of the image 2 to be recognized. Multiple feature points within 5 for identification.

After step S32, the processor 10 performs a blur comparison on the image 2 to be recognized based on the training model 153, and generates a recognition result (step S34). Thereby, the processor 10 can directly mark the positions of the four muscle feature points 3 on the image 2 to be recognized according to the recognition result (step S36), wherein the four muscle feature points 3 will respectively fall on the image to be recognized. The face positioning frame 5 on the image corresponds to at least four weak reference points of the face, respectively.

After step S36, the processor 10 may control the display unit 11 to display the image 2 to be recognized and the four muscle feature points 3 (step S38), and make the marks of the four muscle feature points 3 overlapped and displayed on the face in the image 2 to be recognized. The corresponding position of the image.

In the marking method of the present invention, the processor 10 performs blur comparison on the image 2 to be recognized based on the training model 153. Therefore, the position of the four muscle feature points 3 automatically marked by the processor 10 on the image 2 to be recognized will be Corresponds to the positioning rules analyzed based on a large amount of training data 152.

Specifically, the number of muscle feature points 3 automatically marked on the image 2 to be recognized by the processor 10 through blur comparison corresponds to the number of muscle feature points 4 marked in each piece of training data 152. In other words, the number of muscle feature points 3 marked by the processor 10 in step S36 and the number of muscle feature points 3 displayed on the image 2 to be recognized in step S38 correspond to the storage unit 15 The number of muscle feature points 4 marked on each training data 152 (ie, plural), but not limited to the above four.

In one embodiment, the four muscle feature points 3 automatically marked by the processor 10 will include the first muscle feature point 31 and the second muscle feature point 32 located on the left side of the face image in the image 2 to be recognized, as well as those located on the face. The third muscle feature point 33 and the fourth muscle feature point 34 on the right side of the partial image, and the connection of these four muscle feature points 3 form a rectangular frame or trapezoidal frame.

In another embodiment, the position of the first muscle feature point 31 automatically marked by the processor 10 will fall on the first tear groove tangent line 61, the first legal pattern tangent line 63, and the first tear groove tangent line 63 on the left side of the facial image as described previously. The vertical line 67 of the corner of the eye and the tangent line 69 of the first jawbone branch constitute the auxiliary line; the position of the second muscle feature point will fall on the tangent line 63 of the first law pattern and the first puppet pattern on the left side of the face image The tangent line 65, the vertical line 67 of the first corner of the eye, and the tangent line 69 of the first jawbone branch constitute the auxiliary line; the third muscle feature point will fall on the second tear groove tangent line 62 and the second tear groove on the right side of the face image. The tangent line 64 of the law pattern, the vertical line 68 of the second corner of the eye and the tangent line 70 of the second jawbone constitute the auxiliary line; the fourth muscle feature point will fall on the tangent line 64 of the second law pattern on the right side of the face image The second marionette line tangent 66, the second corner vertical line 68, and the second jawbone branch tangent 70 constitute the area formed by the auxiliary line.

As described above, with the four muscle feature points 3 automatically marked by the analysis device 1, the user can quickly and effectively judge his current facial muscle state. For example, if the positions of the first muscle feature point 31 and the third muscle feature point 33 are close to the eyes, it means that the user's muscles are quite tight. For another example, if the second muscle feature point 32 and the fourth muscle feature point 34 are far away from the mouth and close to the chin, it means that the user's muscles are quite relaxed and maintenance is necessary.

Please continue to refer to FIG. 8, which is a second specific embodiment of the identification flowchart of the present invention. FIG. 8 is used to further explain the specific content of step S34 in FIG. 7.

As mentioned in the foregoing, the training model 153 in the present invention is mainly a regressor containing multiple Cascade regression trees, and the artificial intelligence training algorithm 151 will build the training model 153 based on a large amount of training data. The analysis of 152 records the probability of each muscle feature point appearing in each position and the basic positioning rules. When the processor 10 performs a blur comparison on the image 2 to be identified based on the training model 153, it will first perform the blur comparison on the image 2 to be identified according to the basic positioning rules and probability. A plurality of predicted feature points are randomly generated (step S340). In this embodiment, the plurality of predicted feature points are randomly generated (initial estimate), but locations that are impossible to appear (for example, above the eyebrows, inside the mouth, etc.) will be excluded based on the probability, and will conform to the basic positioning Rules (for example, a plurality of predicted feature points fall within the face positioning frame 5, and the connection of a plurality of predicted feature points can form a rectangular frame or a trapezoidal frame, etc.).

It is worth mentioning that the number of the plurality of predicted feature points corresponds to the number of the marked muscle feature points 4 in each piece of training data 152. In other words, the number of predicted feature points randomly generated by the processor 10 in step S340 is the same as the number of marked muscle feature points 4 on each piece of training data 152 used to train the training model 153 (ie, Plural), but this number is not limited to four.

Next, the processor 10 merges the image 2 to be recognized and a plurality of predicted feature points into one of the complex regression trees of the training model 153 (step S342), and obtains a complex analysis result of the regression tree (step S344). After step S344, the processor 10 adjusts the plurality of predicted feature points respectively according to the obtained analysis results of the plurality of pens, and generates a plurality of adjusted feature points (step S346).

In the present invention, the analysis results mainly record the weight of the combination of the image 2 to be recognized and the plurality of predicted feature points compared with at least a part of the plurality of training data 152 respectively. More specifically, the weight can be used to indicate the correspondence between one or more strong reference points in the image 2 to be recognized and each predicted feature point, and one or more strong reference points and each muscle feature in the training data 152. The similarity of the correspondence between points 4. In other words, the analysis result can be used to indicate the combination of the image to be recognized 2 and the current predicted feature points, and the similarity of each training data 152 (or the summarized/classified data group) and the combination of the muscle feature points 4 .

In one embodiment, the higher the similarity, the higher the weight, but it is not limited. In addition, in the above step S346, the processor 10 is mainly based on different weights (ie, plural analysis Result) to adjust the coordinate values of each predicted feature point on the image 2 to be recognized, thereby generating a plurality of adjusted feature points. In addition, the number of the plurality of adjusted feature points is the same as the number of predicted feature points (for example, four).

As mentioned above, the training model 153 includes complex regression trees, and the number of regression trees corresponds to the number of regressions preset by the user. After step S346, the processor 10 will determine whether all the complex regression trees in the training model 153 have been executed (step S348), that is, determine whether the execution times of the aforementioned steps S342, S344, and S346 are equal to the regression times. .

In the present invention, the processor 10 will replace the aforementioned plurality of predicted feature points with the plurality of adjusted feature points generated in step S346 before the execution of the complex regression tree is completed, and based on the to-be-identified image 2 and the plurality of adjusted features Point, perform step S342, step S344, and step S346 again in the next regression tree to continuously adjust the plurality of feature points. Moreover, when the complex regression trees are all executed, the processor 10 will use the plurality of adjusted feature points generated after the last execution of step S346 as the finally determined plurality of muscle feature points 3, and output the plurality of muscle features Point 3 to end the fuzzy comparison procedure (step S350). In the present invention, the analysis device 1 automatically marks the plurality of muscle feature points 3 on the image 2 to be recognized, that is, the finally determined plurality of muscle feature points 3 output by the processor 10 after step S350.

Please refer to FIG. 9 and FIG. 10 at the same time. FIG. 9 is the first specific embodiment of the schematic diagram of the training model of the present invention, and FIG. 10 is the first specific embodiment of the feature point adjustment schematic diagram of the present invention. For ease of understanding, four predicted feature points will be used as an example below in conjunction with FIG. 9 and FIG. 10 (that is, the number of marked muscle feature points 4 on each training data 152 is four).

As shown in the figure, before performing the fuzzy comparison, the processor 10 randomly generates four predicted feature points 80 on the image to be recognized 2 according to the basic positioning rule and probability. It should be noted that these four The predicted feature points 80 only comply with the basic positioning rules and statistical probabilities summarized by the artificial intelligence training algorithm 151 in the training procedure, but cannot represent the actual muscle state of the face image in the image 2 to be recognized.

Then, the processor 10 merges the image 2 to be recognized and the four predicted feature points 80 into the first regression tree 1531 in the complex regression tree of the training model 153. The first regression tree 1531 has multiple judgment endpoints 1534, and each judgment endpoint 1534 represents a rule, and these rules are the artificial intelligence training algorithm 151 based on the plural training data 152 when training the training model 153 Obtained.

In the present invention, at least part of the content of the judgment endpoint 1534 of the first regression tree 1531 corresponds to the multiple muscle feature point positioning rule generated by the artificial intelligence training algorithm 151 in step S18 of FIG. 4, in other words, the processor 10 will determine at the judgment endpoint 1534 whether the correspondence between the four prediction feature points 80 on the image 2 to be recognized (for example, the size and shape of the rectangular frame or trapezoid frame) is the same as that indicated by the judgment endpoint 1534 The rule is similar, or it is determined whether the corresponding relationship between each predicted feature point 80 on the image to be recognized 2 and one or more strong reference points in the face positioning frame 5 of the image 2 to be recognized is consistent with this judgment endpoint 1534 The indicated rules are similar, that is, the judgment is YES or NO.

It is worth mentioning that when judging yes or no at each judging endpoint 1534, the processor 10 mainly judges the probability that the content of the judging endpoint 1534 is similar (ie, yes) and dissimilar (ie, no). The probability of being 100% similar (ie, 0% dissimilar) or 100% dissimilar (ie, 0% similar) will not occur. After each judgment endpoint 1534 in the regression tree 1531 has been judged, the processor 10 can obtain multiple analysis results, and each analysis result corresponds to a weight 1535.

In the present invention, the weight 1535 is used to indicate the correspondence between one or more strong reference points on the face image in the image 2 to be recognized and each predicted feature point 80 and one or more training data The similarity of the correspondence between one or more strong reference points in 152 (or a group of similar multiple training data 152) and each muscle feature point 4, or the similarity on the face image in the image 2 to be recognized The similarity of the corresponding relationship between each predicted feature point 80 and the corresponding relationship between each muscle feature point 4 in one or more training data 152 (or the group), etc.

For example, if the face image in the image 2 to be recognized is very similar to the first type of training data 152 (for example, the face shape with a big nose is pointed out), the weight 1535 will be higher; if the image in the image 2 to be recognized If the facial image is not similar to the second type of training data 152 (for example, both indicate the face shape with small eyes), the weight 1535 will be lower.

After all the judgment endpoints 1534 in the first regression tree 1531 are judged and multiple weights 1535 are obtained, the processor 10 further adjusts the coordinate value of each predicted feature point 80 on the image 2 to be identified according to the weights, and The first adjusted prediction point 81 is generated. For example, if the face image in the image to be recognized 2 is similar to the training data 152 of the first type (for example, the face shape with a big nose is pointed out), the coordinate value of the first adjusted prediction point 81 will be toward the first type The position of the muscle feature point 4 on the training data 152 moves, and the adjustment range will correspond to the obtained weight 1535 (the higher the weight 1535, the larger the adjustment range).

As shown in FIG. 10, compared to the predicted feature point 80, the first adjusted predicted point 81 will be closer to the actual position of the muscle feature point 3 on the face image in the image 2 to be recognized. Among them, the number of the plurality of first adjusted prediction points 81 is the same as the number of the aforementioned prediction feature points 80.

Next, as shown in FIG. 9, after obtaining the four first adjusted prediction points 81, the processor 10 further integrates the to-be-identified image 2 and the four first adjusted prediction points 81 into the second regression of the training model 153 Tree 1532. In the present invention, the second regression tree 1532 and the aforementioned first regression tree 1531 have the same number of complex judgment endpoints 1534 with related content, but because the input parameters have been changed (from the four predictions) The measured feature points 80 become four first adjusted prediction points 81), so the final analysis results obtained will be different (that is, the weights 1535 obtained will be different).

Similarly, after all the judgment endpoints 1534 in the second regression tree 1532 are judged and multiple weights 1535 are obtained, the processor 10 further adjusts each first adjusted prediction point 81 to be identified according to the weights 1535 obtained. The coordinate values on the image 2 are generated, and the second adjusted predicted point 82 is generated.

In the present invention, the processor 10 will continue to perform the above actions. When the n-1th regression tree (not shown in the figure) is executed, the processor 10 will generate four n-1th adjusted prediction points 83. Next, the processor 10 merges the four n-1 adjusted predicted points 83 into the last regression tree of the training model 153 (ie, the nth regression tree 1533). After the execution of the nth regression tree 1533 is completed, the processor 10 will also obtain multiple analysis results (ie, multiple weights 1535). At this time, the processor 10 will adjust the four n-th regression trees according to the multiple weights 1535. 1 The adjusted prediction point 83, and four final prediction points 84 are generated. In the present invention, the processor 10 will output these four final prediction points 84 as the four muscle feature points 3 generated after the analysis device 1 automatically analyzes them, and automatically mark the four muscle feature points 3 in the to-be-identified Image 2 on.

It is worth mentioning that the number of the regression trees depends on the hardware performance of the analysis device 1, the identification accuracy required by the user, and the processing time acceptable to the user. Generally speaking, the greater the number of regression trees, the closer the position of the final prediction point 84 will be to the position where the actual muscle feature point 3 on the face image in the image 2 to be recognized should exist.

Through the marking method of the present invention, the analysis device 1 can automatically mark four muscle feature points on the user's facial image based on artificial intelligence comparison, thereby allowing the user to clearly and quickly understand their current facial muscle state , And then confirm whether the maintenance measures taken are effective, which is quite convenient for users and practical.

The above are only preferred specific examples of the present invention, and are not limited to the scope of the patent of the present invention. Therefore, all equivalent changes made by using the content of the present invention are included in the scope of the present invention in the same way. Bright.

S30~S38: Identification steps

Claims (9)

A method for automatically marking facial muscle feature points is applied to a facial image analysis device having a processor, a display unit, and a storage unit, wherein the storage unit stores a training model completed in advance, and the method includes : A) Obtain an image to be recognized of a user; b) Perform a face recognition program on the image to be recognized by the processor to generate a face positioning frame, wherein the face positioning frame marks the image to be recognized A face in the image and covers a number of strong reference points on the face; c) the processor performs a fuzzy comparison on the image to be recognized based on the training model and generates a recognition result; d) based on the recognition result A plurality of muscle feature points are marked on the image to be recognized, wherein the plurality of muscle feature points respectively fall in the face positioning frame and correspond to a plurality of weak reference points on the face, and the plurality of muscle feature points include A first muscle feature point and a second muscle feature point on the left side of the face, a third muscle feature point and a fourth muscle feature point on the right side of the face, and the connection of the four muscle feature points is formed A rectangular frame or a trapezoidal frame; and e) displaying the image to be recognized and the plurality of muscle feature points overlapped by the display unit, wherein the first muscle feature point falls on a first tear groove tangent line on the left side of the face , In the area constituted by a first law pattern tangent line, a first eye corner vertical line and a first jawbone branch tangent line, the third muscle feature point falls on a second tear groove tangent line on the right side of the face, a first Within the area formed by the tangent line of the two statute lines, the vertical line of the second corner of the eye, and the tangent line of the second jawbone branch. The method for automatically marking facial muscle feature points according to claim 1, wherein the second muscle feature point falls on the tangent line of the first law pattern on the left side of the face, the first vertical line of the corner of the eye, and the first In the area formed by the tangent line of the jawbone branch and the tangent line of the first marionette pattern, the fourth muscle characteristic point falls on the tangent line of the second law pattern on the right side of the face, the vertical line of the second eye corner, and the tangent line of the second jawbone branch And a second puppet pattern tangent to the area constituted. The method for automatically marking facial muscle feature points according to claim 1, wherein the step a) is to capture the image to be recognized in real time by an image capture unit of the facial image analysis device, and the storage unit reads the pre-recognized image. The stored image to be recognized or the image to be recognized is received from the outside through a wireless transmission unit of the facial image analysis device, wherein the image to be recognized includes at least the facial image of the user. The method for automatically marking facial muscle feature points according to claim 1, wherein the step b) is to execute the face on the image to be recognized by the Histogram of Oriented Gradient (HOG) algorithm of the Dlib Face Landmark system Face recognition program and generate the face positioning frame. The method for automatically marking facial muscle feature points according to claim 1, wherein step a) further includes the following steps: a01) respectively marking the plurality of muscle feature points on the plural training data, wherein the plural training data is the face A02) Perform the face recognition program on each training data through the Histogram of Oriented Gradient (HOG) algorithm of the Dlib Face Landmark system to generate the face location on each training data. Box; a03) Execute an artificial intelligence training algorithm based on the plural training data to analyze and record the correspondence between the plural muscle feature points in each training data, as well as the position of each muscle feature point and the face Correspondence between one or more strong reference points in the frame; a04) generate a plurality of muscle feature point positioning rules based on the corresponding relations obtained by analysis; and a05) Establish the training model at least according to the multiple muscle feature point positioning rules, a judgment depth and a regression number. The method for automatically marking facial muscle feature points according to claim 5, wherein the training model is a regressor, and the regressor includes a plurality of Cascade regression trees with the same content, and each regression tree Each has a plurality of judgment endpoints, at least a part of the plurality of judgment endpoints corresponds to the plurality of muscle feature point positioning rules, wherein the number of the complex regression tree is the same as the number of regressions, and the number of the complex judgment endpoints is the same as the judgment depth . The method for automatically marking facial muscle feature points according to claim 6, wherein the step c) includes the following steps: c01) randomly generating a correspondence on the image to be recognized according to a basic positioning rule and a probability indicated by the training model A plurality of predicted feature points of a plurality of muscle feature points; c02) the image to be identified and the plurality of predicted feature points are imported into one of the complex regression trees of the training model; c03) a complex analysis of the regression tree is obtained Result; c04) adjust the plurality of predicted feature points according to the complex analysis result and generate a plurality of adjusted predicted feature points; c05) execute the plurality of adjusted predicted feature points again before the complex regression tree is executed Step c02) to step c04); and c06) after the complex regression tree is executed, the plurality of adjusted predicted feature points are used as the plurality of muscle feature points of the image to be recognized. The method for automatically marking facial muscle feature points according to claim 7, wherein the complex analysis result records the image to be recognized, the plurality of predicted feature points, and at least the plurality of training data. A weight of a part of the comparison, the weight being the correspondence between one or more strong reference points in the image to be recognized and each of the predicted feature points, and one or more strong reference points in the training data and each of the muscles A similarity of the corresponding relationship between the feature points, wherein the step c04) adjusts a standard value of each predicted feature point on the image to be recognized according to the plurality of weights. The method for automatically marking facial muscle feature points according to claim 1, wherein the processor recognizes in the face positioning frame that a plurality of regions on the face constitute an auxiliary line, and the plurality of regions constitute an auxiliary line including the left side of the face A first tear groove tangent, a first law pattern tangent, a first vertical eye corner, a first jawbone branch tangent and a first marionette pattern tangent, and a second tear groove tangent on the right side of the face, a first Two tangent lines of the law pattern, a second vertical line of the corner of the eye, a second tangent line of the jawbone branch, and a second tangent line of the puppet pattern, wherein the processor determines whether the plurality of muscle feature points fall on the auxiliary line formed by the plurality of regions. It is determined that the recognition result is incorrect when any of the muscle feature points does not fall within the corresponding area.

Images