KR102497694B1 - System and Method for correcting facial asymmetry based expression training using correction mirror - Google Patents

Abstract

The present invention determines the user's facial asymmetry through analysis of user's facial images of various facial expressions, performs facial expression training targeting the facial expression of the balance model most similar to the user's facial musculoskeletal system, and performs training for the user through affiliated companies. It is about the technology that enables the provision of coaching services for
According to the present invention, by expressing training information for the asymmetric position of the user's face and facial asymmetry correction through a mirror surface in conjunction with a correction mirror through a user terminal, a state in which the user naturally recognizes the asymmetric state of his or her face while looking in the mirror. Facial asymmetry correction training can be performed.
In addition, accurate facial expression training can be performed by providing a coaching service for the user's training performance through an affiliated company.

Description

System and method for correcting facial asymmetry based expression training using correction mirror}

The present invention determines the user's facial asymmetry through analysis of user's facial images of various facial expressions, performs facial expression training targeting the facial expression of the balance model most similar to the user's facial musculoskeletal system, and performs training for the user through affiliated companies. It is about the technology that enables the provision of coaching services for

The movement of the face interacts with the numerous muscles that make up the face by being connected to 22 bone pieces. Facial asymmetry is a phenomenon that appears as a result of these complex causes of the musculoskeletal system, and it appears as facial asymmetry.

The main cause of such facial asymmetry is very closely related to an individual's lifestyle (masticating, sleeping posture, resting the chin with the hand, postural imbalance of the upper and lower body), and these habits affect the musculoskeletal system of the face for a long time, and thus facial muscles It is manifested in the contraction relaxation and tension of the muscle, whether it is used or not, and the changes in the skeletal structure of the face and head caused by the muscle.

Also, the consequences of facial asymmetry can have a very profound effect on the balance and movement of the temporomandibular joint.

The temporomandibular joint is a part that regulates the meridian system and brain activity, and the temporomandibular joint is also the root of brain activity treatment. The reason is that numerous nerves and blood vessels pass around the temporomandibular joint, so when facial asymmetry occurs, it becomes the balance of the nervous system. This is because breaking can cause many problems.

If left unattended, it can cause chronic headaches, edema, color changes, shoulder hangs, dizziness, vomiting, difficulty in chewing, and disorders of the nervous and muscular system, such as abnormal secretion of hormones. It is necessary to prevent its progress.

In addition, facial asymmetry can not only give a bad impression on interpersonal relationships, but also negatively affect self-efficacy in social relationships because it causes appearance stress and complexes.

This is a factor that lowers self-confidence, leading to psychological passivity in accurately conveying one's intentions to the other person or persuading the other person, affecting the area of leading emotional expression and self-reliance, and limiting or reducing one's social influence to subordinate to the environment. The easier it is to go in that direction.

Due to these results, facial asymmetry is not limited to disease problems, but is also very closely related to psychological issues in life related to individual sociability and self-efficacy, so it is necessary to solve it.

To this end, facial asymmetry-related procedures or surgical methods have been used, but currently used facial asymmetry-related procedures or surgical methods use specialized equipment in hospitals, and thus have the disadvantage of requiring a lot of cost.

Accordingly, there may be a method for performing facial correction training that is currently openly provided online, but this is general-purpose and has a disadvantage in that customized training based on one's own facial imbalance is impossible. In addition, this service has a disadvantage in that the user cannot check whether he or she is currently performing training accurately.

1. Japanese Patent Laid-open Publication No. 2019-524397 (Title: Method for realizing an orthodontic appliance for treating facial asymmetry) 2. Korean Patent Publication No. 10-2015-0059629 (Name: Mirror that can grasp the balance of the human body)

Therefore, the present invention was created in view of the above circumstances, and by expressing training information for the asymmetric position of the user's face and facial asymmetry correction through a mirror surface in conjunction with a correction mirror through a user terminal, so that the user looks in the mirror Facial asymmetry correction based on facial expression training using a corrective mirror that enables not only to perform facial asymmetry correction training while naturally recognizing one's own facial asymmetry, but also to provide coaching services for the user's training performance through affiliated companies. The technical purpose is to provide a system and its service method.

여기서, X 와 Y 는 보정된 각 픽셀의 위치좌표, X0 와 Y0 는 제1 교정영상의 우측 첫 위치 좌표, X1 과 Y1 은 제2 교정영상의 우측 첫 위치 좌표, σ는 카메라의 고유 왜곡보정율, θ 는 제1 교정영상에서 사용자 얼굴의 왼쪽과 오른쪽 눈썹의 상단을 연결한 직선과 수평선 대비 기울기이다.According to one aspect of the present invention for achieving the above object, the grid pattern and the RGB LEDs are based on a mirror surface on which RGB LEDs are disposed at each intersection of the grid pattern and a control signal received from the user terminal through communication with the user terminal. A correction mirror configured to include an electronic module for controlling output and a cradle for mounting a user terminal on one side of the mirror surface, and facial asymmetry training through the mirror surface by communicating with the correction mirror according to the execution of the face correction application. Information is provided, but a reference line face image including the reference line expressed on the correction mirror and various expression images including neutral expressions are taken and provided to the service server, and the training information is provided by communicating with the coaching service terminal matched by the service server. In addition to expressing through a correction mirror, training feedback information including a corresponding user face image is transmitted to the coaching service terminal, and the user terminal outputs the coaching information received from the coaching service terminal, and the reference line received from the user terminal. Based on the face image, a plurality of points of each part including eyes, ears, and mouth are extracted, and a gravity line of gravity reference corresponding to the reference line is created based on the distance between the gravity line and each part point on both sides and between the part points The asymmetric attribute value is calculated, and the asymmetric type is determined based on the slope and direction of the slope between each gravity line and the reference line, and the facial structure most similar to the user's face in the pre-registered standard model based on the facial expression image A service server and a service server that determine a standard model of , generate a training schedule targeting the facial expression of the standard model based on the analyzed facial motion based on the facial expression image, and provide the training schedule to the coaching service terminal matched with the corresponding user terminal. Applying the user's facial expression training schedule received from the service server to the coaching application installed and provided from, and transmitting the training information to the user terminal matched by the service server, as well as corresponding to the training feedback information received from the user terminal Coaching service providing coaching information It is configured to include a terminal, wherein the user terminal captures a first correction image of a face state looking at the mirror surface and a second correction image of a face state watching the user terminal while being mounted on a cradle of a correction mirror, A face asymmetry correction system based on facial expression training using a correction mirror is stored, characterized in that a reference line face image, which is a face image displayed as a mirror surface, is generated using the following equation.

Here, X and Y are the position coordinates of each corrected pixel, X0 and Y0 are the coordinates of the first right position of the first calibration image, X1 and Y1 are the coordinates of the first right position of the second calibration image, and σ is the camera's unique distortion correction rate. , θ is the slope of the line connecting the tops of the left and right eyebrows of the user's face in the first correction image and the horizontal line.

여기서, X 와 Y 는 보정된 각 픽셀의 위치좌표, X0 와 Y0 는 제1 교정영상의 우측 첫 위치 좌표, X1 과 Y1 은 제2 교정영상의 우측 첫 위치 좌표, σ는 사용자 단말 카메라의 고유 왜곡보정율, θ 는 제1 교정영상에서 사용자 얼굴의 왼쪽과 오른쪽 눈썹의 상단을 연결한 직선과 수평선 대비 기울기이다.In addition, according to another aspect of the present invention for achieving the above object, facial asymmetric training information is provided using a user terminal communicating with a calibration mirror equipped with RGB LEDs at the intersection of the grid pattern and the grid pattern, and coaching In the facial asymmetry correction service method based on facial expression training using a correction mirror that provides coaching information on a training state performed through a user terminal through a service terminal, the user terminal takes a picture of the face on a grid pattern represented by a correction mirror. A first step of photographing and transmitting a face image including the reference line of the correction mirror and various expression images including expressionless expressions to the service server, and analyzing the reference line face image in the service server. The second step of determining the asymmetric type based on whether or not and the direction of the inclination between the gravity line and the reference line by extracting the facial asymmetry attribute and setting the gravity line of the gravity reference in the reference line face image at a position corresponding to the reference line , The third step of analyzing the facial expression image in the service server and determining the standard model of the facial structure most similar to the user's facial characteristics from the previously registered standard models, and the fourth step of analyzing the user's facial movement based on the facial expression image in the service server , A fifth step of generating a training schedule targeting the facial expression of the standard model based on the asymmetric type and facial movement in the service server, and matching the coaching service terminal for providing coaching information to the user terminal in the service server, and matching The training schedule of the corresponding user is transmitted to the coaching service terminal, and facial correction training is performed through communication between the matched user terminal and the coaching service terminal. It is configured to include a sixth step of driving an RGB LED at a position corresponding to the training information, and the first step is to obtain a first correction image of a face looking at a correction mirror through a camera of the user terminal in a service server. The step of doing, and the user through the camera of the user terminal Acquiring a second correction image of a face state watching the terminal, and generating a reference line face image in the form of a face image displayed as a mirror surface by applying the first correction image to the following equation in the user terminal There is provided a facial asymmetry correction service method based on expression training using a correction mirror, characterized in that configured.

Here, X and Y are the position coordinates of each corrected pixel, X0 and Y0 are the coordinates of the first right position of the first calibration image, X1 and Y1 are the coordinates of the first right position of the second calibration image, and σ is the inherent distortion of the user terminal camera The correction rate, θ, is the slope of a line connecting the tops of the left and right eyebrows of the user's face and the horizontal line in the first correction image.

According to the present invention, by expressing training information for the asymmetric position of the user's face and facial asymmetry correction through a mirror surface in conjunction with a correction mirror through a user terminal, a state in which the user naturally recognizes the asymmetric state of his or her face while looking in the mirror. Facial asymmetry correction training can be performed.

In addition, accurate facial expression training can be performed by providing a coaching service for the user's training performance through an affiliated company.

1 is a diagram showing a schematic configuration of a facial asymmetry correction system based on expression training using a correction mirror according to a first embodiment of the present invention.
Fig. 2 is a view for explaining the configuration of the correcting mirror 100 shown in Fig. 1;
FIG. 3 is a functionally separated view showing the configuration of the user terminal 200 shown in FIG. 1;
4 is a block diagram showing the internal configuration of the service server 400 shown in FIG. 1 by functionally separating it;
5 is a view for explaining the operation of the facial asymmetry correction system based on facial expression training using a correction mirror shown in FIG. 1;
6 is a diagram illustrating an expression state output through the mirror surface 110 of the correction mirror 100 in the operation of FIG. 5;
FIG. 7 is a diagram for explaining in detail a reference line face image generation process (ST100) shown in FIG. 5;
8 is a diagram illustrating a process of generating a synchronization image in the operation of FIG. 7;
FIG. 9 is a diagram for explaining in detail a facial asymmetry type determination process (ST200) shown in FIG. 5;
10 is a diagram for explaining a method of extracting an asymmetric attribute value in FIG. 9;
11 is a diagram for explaining an asymmetric type determination method (ST300) in FIG. 9;

The embodiments described in the present invention and the configurations shown in the drawings are only preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so the scope of the present invention is limited to the embodiments and drawings described in the text. should not be construed as being limited by That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as being consistent with meanings in the context of related art, and cannot be interpreted as having ideal or excessively formal meanings that are not clearly defined in the present invention.

1 is a diagram showing a schematic configuration of a facial asymmetry correction system based on facial expression training using a correction mirror according to a first embodiment of the present invention.

Referring to FIG. 1, the facial correction system based on expression training using a correction mirror according to the present invention includes a correction mirror 100 and a user who communicates with the correction mirror 100 to control the operation of the correction mirror 100. It includes a terminal 200, a coaching service terminal 300 providing a coaching service for facial correction training, and a service server 400.

Here, the correction mirror 100 and the user terminal 200 are user-side terminals for performing facial correction training, and the user terminal 200 can communicate with the correction mirror 100 and perform personal communication such as a smart phone having a camera function. can be a terminal.

The coaching service terminal 300 is an operator's terminal that communicates with the user terminal 200 to provide a coaching service for the user to perform accurate training, and can be a terminal of various service companies related to facial correction. may be a plastic surgery clinic, a physical therapy company, a manual therapy company, an esthetician, a dermatologist, an oriental medicine clinic, or an interview or presentation coaching company that provides facial expression management services. At this time, the service provider may activate its own service more by combining facial correction training coaching with its own service.

In addition, the coaching service terminal 300 transmits training information to the user terminal 200 matched by the service server 400, and in response thereto, collects training feedback information input from the user terminal 200 to provide training coaching information. is provided to the corresponding user terminal 200 .

The service server 400 supplies a facial correction application to the user terminal 200 of the user who has registered as a member, analyzes the face image applied from the user terminal 200 to determine the user's facial asymmetry type, and determines the user's facial asymmetry. A training schedule corresponding to the type is created and provided to the matched coaching service terminal 300 .

That is, the user terminal 200 matched by the service server 400 communicates with the coaching service terminal 300 to provide a facial correction training service to the user.

At this time, the service server 400 manages user information receiving facial correction coaching service for each coaching service terminal 300, and calculates the cost of the coaching service provider based on the number of users providing the coaching service to the coaching service provider. You may receive payment from For example, the coaching service provider may pay a monthly basic amount according to the use of a facial correction coaching service program to the service server, but the coaching service cost per user may be deducted from the next month's basic amount.

2 is a diagram for explaining the configuration of the correction mirror 100 shown in FIG. 1, and a smart phone is shown as the user terminal 200.

Referring to FIG. 2 (A), the calibration mirror 100 is integrally provided with a smartphone holder 120 capable of holding the smartphone 200 on one side of the mirror surface 110. At this time, the smartphone holder 120 may be configured to automatically communicate with the mirror surface 110 when the smartphone 200 is mounted, and may be configured to be connected by physical connection between communication terminals or by a wireless communication method such as Bluetooth. there is. In addition, the smartphone holder 120 is provided with a sensor 121 for detecting whether the smartphone 200 is mounted.

In addition, a mirror cover 130 having a general mirror structure for covering the mirror surface 110 is provided on the upper side of the mirror surface 110 . The mirror cover 130 has a size that covers the mirror surface 110 and is coupled to the upper side of the mirror surface 110 in a folder shape to set an open/closed state. That is, when the mirror cover 130 is covered on the mirror surface 110, the correction mirror 100 is used as a general mirror, and when the mirror cover 130 is opened from the mirror surface 110, according to the present invention Used as a correction mirror. At this time, a sensor for detecting the open/closed state of the mirror cover 130 may be additionally provided on one side of the mirror surface 110, and the sensor transmits open/closed state information of the mirror cover 130 to the electronic module 140. can

In addition, around the mirror surface 110, for example, on the lower side of the rear surface or inside the pedestal, an electronic module 140 and a power supply unit 150 for providing a facial asymmetry correction service through the mirror surface 110 in conjunction with the smartphone 200 ) is provided.

On the other hand, as shown in FIG. 2 (B), the mirror surface 110 has a half mirror structure having a certain area and a groove formed in a lattice shape while being disposed on the rear surface of the mirror plate 111. A grid plate 112, an LED lighting unit 113 arranged at regular intervals on the edge of the grid plate 112, and RGB LEDs disposed at each intersection of the grid plate 112 while being disposed on the rear surface of the grid plate 112 LED matrix 114.

The mirror plate 111 is formed by coating an 80% silver nitrate aqueous solution on the rear surface of a plate-shaped glass or plastic substrate, and the grid plate 112 disposed on the rear surface is transmitted through the mirror plate 111 without shading treatment. It consists of a half-mirror structure displayed on the top.

The grating plate 112 is formed by forming grid pattern grooves of horizontal and vertical lines at regular intervals and thicknesses on a substrate of the same material and size as the mirror plate 111 .

The LED lighting unit 113 is disposed on the edge of the grid plate 112 and scatters light to the grid pattern groove lines formed on the grid plate 112, thereby making the grid plate 112 shine in a grid shape.

The LED matrix 114 is configured by arranging RGB LEDs (D) at each intersection of the grid plate 112 on a substrate having the same size as the grid plate 112 .

Here, the LED lighting unit 113 and the LED matrix 114 are electrically connected to the electronic module 140 and operated according to a control signal of the electronic module 140 .

The electronic module 140 communicates with the smartphone 200 mounted on the holder 120 based on the smartphone mounting detection signal applied from the sensor 121, and according to the control signal provided from the smartphone 200 Each RGB LED (D) of the LED lighting unit 113 and the LED matrix 114 is individually controlled on/off.

In particular, the electronic module 140 controls the LED matrix 114 to display the asymmetrical position of the user's face reflected on the mirror surface 110, and provides training information for correcting the asymmetrical part as training direction information at the training position. Control to display output.

In addition, the electronic module 140 receives open/close state information of the mirror cover 130 from the calibration mirror 100 from a sensor (not shown), and when the mirror cover 130 is opened, the LED lighting unit 113 is used as the calibration mirror. It may be configured to transmit a control signal for driving. That is, when the mirror cover 130 is opened, the lattice pattern is automatically expressed through the mirror surface 110 .

The power supply unit 150 supplies driving power to the electronic module 140 and the detection sensor 121, and is connected to the LED lighting unit 113 and the LED matrix 114 according to a control signal applied from the electronic module 150. Power to each RGB LED (D) of the LED lighting unit 113 and the LED matrix 114 is supplied or cut off.

3 is a functionally separated view of the configuration of the user terminal 200 shown in FIG. ) to provide facial training services.

Referring to FIG. 3 , the user terminal 200 includes a communication connection module 210, an image acquisition module 220, an image synchronization module 230, and a calibration service module 240.

The communication connection module 210 communicates with the calibration mirror 100, the service server 400, and the service terminal 300, and may communicate with different devices in different ways.

In particular, in the present invention, the communication connection module 210 automatically executes a face correction application based on a pairing signal received from the correction mirror 100 in a communication-connected state with the correction mirror 100.

The image acquisition module 220 transmits a control signal to the calibration mirror 100 in a state where the calibration application is automatically executed to drive the LED lighting unit 113 and, through the LED matrix 114, the image corresponding to the reference line. The RGB LED (D) is operated, and an image when a preset motion such as a user blinks twice is recognized is acquired as a calibration image.

At this time, the image acquisition device 220 acquires a first correction image of a face state of looking at the mirror surface 110 of the correction mirror 100 and a second correction image of a face state of gaze of the camera of the user terminal 200. do.

The image synchronization module 230 analyzes the first correction image and the second correction image using the OpenCV algorithm, and calculates the position and size of each part of the face of the second correction image necessary for generating the synchronization image and training schedule.

In addition, the image synchronization module 230 synchronizes the image reflected on the mirror surface 110 of the correction mirror 100 with the image captured by the user terminal 200 so that the first correction image is captured by the correction mirror ( 100) is converted into a synchronization image in the same state as the face image displayed on the mirror plate 110, that is, a reference line face image.

In this case, the image synchronization module 230 may also generate a synchronization image by tracking a positional change of a predetermined feature point in the first calibration image. This is in consideration of the change in the position of the user's face reflected in the correction mirror 100 . Accordingly, as the position of the user's face reflected on the mirror surface 110 of the correction mirror 100 changes, the asymmetric position and training information are also changed and expressed.

The calibration service module 240 transmits the reference line face image to the service server 400, and based on the asymmetric position of the user's face provided from the service terminal 300, the reference line face image and the RGB LEDs of the LED matrix 114 (D) A control signal for driving the RGB LED (D) corresponding to the asymmetrical position in the first color by matching the position is generated, and the control signal is transmitted to the calibration mirror 100.

In addition, the calibration service module 240 generates the reference line face image and the LED matrix 114 based on the training information including the training position, direction, and number of times (time) corresponding to the asymmetrical position provided from the service terminal 300. By matching the positions of the RGB LEDs (D), a control signal for driving the training position and the RGB LEDs (D) corresponding to the training direction around the training position with the second color is generated and transmitted to the calibration mirror 100.

In addition, the calibration service module 240 may output training information as audio through a speaker or display it through a display window. In particular, the calibration service module 240 may be configured to display and output training status information performed through the calibration mirror 100 on a display window, and to use the display window as a dashboard for changing a training schedule by a user.

FIG. 4 is a block diagram showing the functional separation of the internal configuration of the service server 400 shown in FIG. 1 .

Referring to FIG. 4 , the service server 400 includes a communication connector 410 , a data storage 420 , and a service control unit 430 .

The communication connection unit 410 communicates with the user terminal 200 and the coaching service terminal 300 .

The data storage 420 stores a facial correction application to be provided to the user terminal 200, a coaching application to be provided to the service terminal 300, reference information for determining the facial asymmetry state, and training contents (training information) for each facial asymmetry type. Various types of information for providing a facial correction training service according to the present invention are stored, including training reference information to generate, information on subscribed user terminals and correction mirrors, subscribed service terminal information, and service matching information for these information.

In particular, the data storage 420 includes an Open CV algorithm for analyzing facial conditions using face images, a CNN model and an RNN model in which each part of the face is learned, a standard model of balance for each facial type, and reference information for each asymmetric type. Information including is stored. Here, the CNN model is an AI algorithm for measuring movement of facial parts, and the RNN model is an AI algorithm for measuring interaction, vertical relaxation information and direction of facial muscles moving in conjunction with the CNN model. In addition, standard model information is stored together with a plurality of different facial expression images for each standard model.

The service control unit 430 provides a facial correction application to the user terminal 200 subscribed as a member, and provides a coaching application to the service terminal 300 subscribed as a member, and provides matching information for them to the data storage 420. save and manage

In addition, the service control unit 430 receives the reference line face image from the user terminal 200, determines an asymmetric state, determines an unbalanced type corresponding to the asymmetric state, and generates training information corresponding to the unbalanced type. . And, the service control unit 430 provides the training information to the coaching service terminal 300 matched with the corresponding user terminal 200 .

The service control unit 430 includes a service management module 431, an information collection module 432, an asymmetric analysis module 433, and a service matching module 434.

The service management module 431 performs user sign-up processing based on member sign-up information including personal information and user terminal information about the user who wants to receive facial correction service, as well as service companies who want to provide coaching services. Coaching member sign-up processing is performed based on member sign-up information including personal information about and service terminal information.

In addition, the service management module 431 provides a facial correction application to the user terminal 200 registered as a member, and provides a coaching application to the service terminal 300 registered as a member.

The information collection module 432 acquires a variety of different facial expression images including a neutral expression face image including a reference line transmitted from the user terminal 200, that is, a reference line face image and an expressionless image.

The asymmetry analysis module 433 analyzes the face image of the reference line collected by the information acquisition unit 432 to extract an asymmetry attribute value, determines the asymmetric type of the user's face based on this, and determines the asymmetric type of the corresponding user's face in the model big data. The standard model most similar to the facial musculoskeletal system is determined, and the facial expression image of the user is applied to the CNN (Convolutional Neural Network) model and the RNN (Recurrent Neural Network) model in which each part of the face has been trained. Analyze facial movements.

In addition, the asymmetry analysis module 433 compares the user's asymmetric attribute value and the balanced facial numerical information of the standard model, analyzes the difference, and creates a facial expression training schedule consisting of training information to have a facial expression image of the standard model, , and create a coaching application applying it.

The service matching module 434 matches and manages the user terminal 200 and the service terminal 300 in response to the unbalanced type, and sets the training schedule according to the matching information to the service terminal 300 matched with the corresponding user terminal 200. send to

Subsequently, a facial correction service method based on facial expression training using a correction mirror configured as described above will be described with reference to a flowchart shown in FIG. 5 .

First, the user accesses the service server 400 through the user terminal 200, performs a series of membership registration processes, and installs a facial correction application provided from the service server 400 in the user terminal 200. And, the user has a calibration mirror 100 that works with his or her user terminal 200 .

In addition, the service company accesses the service server 400 through the coaching service terminal 300, performs a series of membership registration processes, and installs a coaching application provided from the service server 400 in the coaching service terminal 300. . At this time, the service company may add its own company advertisement information and work-related information to the coaching application.

In the above state, the user sets the user terminal 200 and the correction mirror 100 to a communication connection state, and the user terminal 200 executes a face correction application. From then on, the user terminal 200 operates according to the face correction application.

First, the user terminal 200 transmits various expression images and reference line face images generated in conjunction with the correction mirror 100 to the service server 400 in a state in which the face correction program is executed (ST100). At this time, the user terminal 200 responds to this by outputting various induction sentences for inducing facial expressions having 7 preset facial expressions of laughter, joy, surprise, fear, sadness, anger, and disgust including expressionless expressions. It is possible to shoot images of different facial expressions.

In addition, in the ST100 step, the user terminal 200 displays the grid pattern through the calibration mirror 100 as shown in FIG. 6 (A), and also displays the reference line as shown in FIG. 6 (B). (L1, L2, L3) are displayed and output, and a face image is captured in a state in which the user positions his or her face according to the reference lines (L1, L2, L3) of the correction mirror 100. Then, a reference line face image is generated using the face image and then transmitted to the service server 200 .

In addition, the service server 400 determines the facial asymmetry type based on the reference line face image received from the user terminal 200 (ST200). The process of generating the reference line face image and determining the facial asymmetry type will be described in detail below.

Subsequently, the service server 400 analyzes the facial expression image received from the user terminal 200 and determines a standard model having characteristics similar to the facial values of the corresponding user among previously registered standard symmetric facial models (ST300).

In addition, the service server 400 analyzes the user's facial movement based on the facial expression image received from the user terminal 200 (ST400).

At this time, the service server 400 uses GANs (Generative Adversarial Networks, deep convolutional generative adversarial networks) learned from the big data facial image data set for seven different facial expression images to add six facial expressions. create with

Six new facial expression images are generated through the combination of facial expression images in the manner shown in Table 2 below.

default expression
synthetic expression
Facial expression generated by GAN
2. Joy
1. Expressionless
8. Satisfaction
3. Surprise
9. Boundary
4. Fear
10. Worry
5. Sorrow
11. Discouragement
6. Anger
12. Hardened
7. Dislike
13. Contempt

Referring to Table 1, the basic facial expression and synthesized facial expression are facial expression images obtained from the user terminal 100 in step ST100, and facial expressions generated through GAN are additionally generated facial expression images. By sequentially combining images, fear images, sadness images, anger images, and disgust images, a satisfaction image, a caution image, a worry image, a discouragement image, a hardened image, and a contempt image are sequentially created. For example, in Table 1, a “satisfied” facial expression image is generated by combining a joy expression image and an expressionless expression image, and a “border” facial expression image is generated by combining a “surprise” facial expression image and an expressionless expression image. That is, the service server 400 analyzes the user's facial movement by applying a total of 13 facial expression images to the CNN model and the RNN model in which each part of the face has been learned. At this time, the service server 400 presents a preset short sentence to measure the movement of the user's facial muscles during a conversation, acquires a face image reading the sentence from the user terminal 200, and displays the preset conversation content. A face image in a state of outputting and listening to a corresponding conversation may be obtained from the user terminal 200 and additionally reflected in facial motion analysis.

Next, the service server 400 creates a facial expression training schedule consisting of training information for each session based on the facial asymmetry type determined in step ST200 and the facial motion analysis result determined in step ST400 (ST500).

At this time, the service server 200 creates a facial expression training schedule targeting the facial expression image of the standard model for each facial expression shown in Table 1, that is, the facial expression of the standard image. That is, the service server 200 compares the user's facial expression image with the standard image and creates a facial expression training schedule for displaying the corresponding facial expression.

For example, the service server 200 may select a certain number of facial expression images in the order of increasing facial movements related to the asymmetric attribute, and generate training information to make the corresponding facial expressions of the standard model.

In addition, the service server 200 compares each facial expression image of the corresponding user with the facial expression image of the standard model to calculate the facial movement difference for each facial expression image, selects a certain number of facial expression images in the order of the largest facial movement difference, and selects the corresponding facial expression. Training information to be the facial expression of this standard image can be generated.

This facial expression training schedule includes basic facial expression training consisting of muscle relaxation for each expression, facial movement, point acupressure, and massage, and practice of moving two or more facial parts simultaneously after practicing the movements of the facial parts individually, Step-by-step training consisting of making facial expressions step by step and then making facial expressions at once, deriving improvement points from expressionless state, generating and presenting images of soft and likeable facial expressions, and training on areas in need of improvement It may include liking training in which elements are derived and training is performed. Here, the standard model for liking training can be directly selected by the user through the user terminal 100 from among a plurality of models similar to the user's facial musculoskeletal system.

Thereafter, the service server 400 transmits facial asymmetry information and coaching service information for each service type of the service provider to the user terminal 300, and based on the affiliate information selected from the user terminal 300, the coaching service terminal of the corresponding service provider ( 300), the training schedule generated in step ST200 is transmitted to match the user terminal 200 and the coaching service terminal 300 (ST600). At this time, the selection of the service provider may be arbitrarily matched by the service server 400 according to the user's facial asymmetry. For example, in the case of the first asymmetric type, it can be matched with aesthetics, dermatology, oriental medicine, in the case of the second asymmetric type, physical therapy, manual therapy, and in the case of the third asymmetric type, plastic surgery.

In addition, the user terminal 300 expresses the asymmetric facial position by driving the RGB LEDs (D) disposed at the asymmetric position of the user's face reflected in the correction mirror 100 based on the facial asymmetry information provided from the service server 400. let it do

That is, the user terminal 300 compares the asymmetric position and the LED matrix 114 to determine the position of the RGB LED (D) corresponding to the asymmetric position information found in the LED matrix 114, and determines the position of the RGB LED (D) at the corresponding position. ) is generated as a first color and transmitted to the calibration mirror 100 . Accordingly, as shown in FIG. 6(C), information on the asymmetric position LP1 is displayed on the user's face looking at the correction mirror 100 through the mirror surface 110 of the correction mirror 100. Accordingly, the user can recognize his/her facial asymmetry while looking at the correction mirror 100 .

Thereafter, the user terminal 200 matched by the service server 400 and the coaching service terminal 300 form a direct communication path, and perform facial correction training processing by transmitting and receiving mutual training information (ST700).

That is, the coaching service terminal 300 transmits training information to the corresponding user terminal 200 according to the operation of the coaching application, and the user terminal 200 outputs the training information on the screen or through audio so that the user performs training. In addition to requesting to do so, it controls to display and output training information to the calibration mirror 100 communicatively connected to the corresponding user terminal 200 . In the present invention, the display window of the user terminal 200 can be used as a dashboard for training services. For example, through the display window of the user terminal 200 , training image and training dashboard information including the current training status may be displayed and output.

At this time, the user terminal 200 controls training for outputting training position and training direction information in order to express training information including the training position, direction, and number of times (time) corresponding to the asymmetrical position on the LED matrix 114. The signal is transmitted to the calibration mirror 100 so that training information is displayed and output through the calibration mirror 100 (ST800). At this time, as shown in FIG. 6(D), the training information includes training position LP2 and training direction LP3 information, and is expressed through the mirror surface 110 of the correction mirror 100. Here, the training direction information is expressed around the training position, and by setting the RGB LEDs (D) corresponding to the corresponding direction at the position to be sequentially accumulated for a certain length, it can be displayed so that the directionality appears, The stronger the training intensity, the longer the length representing the training direction can be displayed.

Thereafter, the user terminal 200 transmits training feedback information including the face image of the user training to the coaching service terminal 300, and the coaching service terminal 300 provides coaching information for accurate training based on the training feedback information. It is provided to the user terminal 200 . In this case, the training face image is an image representing the user's training state and may be a first calibration image captured by a camera.

Next, with reference to FIG. 7, the reference line face image generation process (ST100) shown in FIG. 5 will be described in more detail.

Referring to FIG. 7, in a state in which the user terminal 200 is connected to the calibration mirror 100, the user terminal 200 transmits a lighting control signal to the calibration mirror 100 to turn on the LED lighting unit 113 ( ON) state, as shown in FIG. 6(A), the grid pattern is controlled to be displayed through the mirror surface 110 of the correction mirror 100 (ST110).

In addition, the user terminal 200 transmits an RGB control signal to the correction mirror 100 to drive RGB LEDs (D) corresponding to reference lines for face photographing in the LED matrix 130 . The user terminal 200 sets the RGB LEDs (D) arranged on the horizontal lines corresponding to the vertical 1/3 and 2/3 points of the LED matrix 130 to the on state with the calibration mirror 100 . That is, the first to third reference lines L1, through the mirror surface 110 of the calibration mirror 100 as shown in FIG. Reference lines of L2 and L3) are expressed (ST120). Here, the first to third reference lines L1, L2, and L3 are reference lines for locating the user's face area, and the eyebrows and chin of the user's face are the first reference line L1 and the second reference line L2. ), and is a reference line for positioning the nose on the third reference line (L3).

In the above state, the user terminal 200 monitors a user image in a state of looking at the correction mirror 100 through a camera, and when a preset confirmation motion is recognized in the user image, the face looking at the correction mirror 100. An image is obtained as a first calibration image (ST130).

At this time, the user terminal 200 may output a guide voice requesting the location of a predetermined face part in the inner region of the first to third reference lines L3 displayed and output by the correction mirror 100 . In addition, the user terminal 200 analyzes the user image received through the camera, and when a preset confirmation motion, for example, blinking one eye twice in succession appears, recognizes that all face parts are located in the reference line area and corresponds to the corresponding By capturing a face image of a viewpoint, a first calibration image is acquired.

That is, the first correction image is the cradle of the correction mirror 100 in a state where the user's face is located in the inner region of the first to third reference lines L3 expressed on the mirror surface 110 of the correction mirror 100 ( 120) is a facial image looking at the correction mirror 100 photographed by the smart phone 200 in a state of being mounted.

Subsequently, the user terminal 200 displays and outputs a calibration background screen composed of the same grid pattern represented by the calibration mirror 100 and the first to third reference lines, and displays the first to third reference lines on the calibration background screen. A second calibration image in a state of looking at the user terminal is obtained by taking a face image in a state of gazing at a camera with the user's face positioned in the same way as in the calibration mirror 100 as a reference (ST140). At this time, the user terminal 200 may acquire a second correction image by determining that the user blinks one eye twice in succession and capturing a face image at that time.

In addition, the user terminal 200 analyzes the first calibration image obtained in the step ST400 and the second calibration image obtained in the step ST500 using the OpenCV algorithm, and uses a preset equation to generate the first calibration image. A synchronization image for the face, that is, a reference line face image is generated (ST150).

That is, the user terminal 200 transmits a trapezoid-shaped first correction image tilted to one side to synchronize an image reflected on the mirror surface 110 of the correction mirror 100 with an image captured by the user terminal 20. Convert to the same rectangle as the image expressed in (110). For example, when the user terminal 200 is located on the right side of the correction mirror 100 as shown in FIG. 2, a trapezoidal shape inclined to the left is converted into a rectangle.

In the present invention, a synchronization image is generated using the correction function of Equation 1 based on the first position coordinates on the right of the first and second correction images.

Here, X and Y are the position coordinates of each corrected pixel, X0 and Y0 are the coordinates of the first right position of the first calibration image, X1 and Y1 are the coordinates of the first right position of the second calibration image, and σ is the user terminal (e.g., smart The distortion correction rate, θ, of the camera provided in the phone) is the slope of the horizontal line and the straight line connecting the tops of the left and right eyebrows of the user's face in the first correction image. σ may be obtained from a user terminal camera manufacturer or by calculating a perspective-related distortion correction rate of a camera lens through a known equation.

That is, as shown in FIG. 8 , the face displayed as the mirror surface 110 of the correction mirror 100 when the first correction image taken by the user terminal 200 while looking at the correction mirror 100 is captured. It is converted into a synchronized image in the same state as the image, that is, a reference line face image.

Next, with reference to FIG. 9, the facial asymmetry type determination process (ST200) shown in FIG. 5 will be described in detail.

Referring to FIG. 9, the service server 400 creates a measurement line for determining facial asymmetry by connecting preset points P1 to P8 and facial reference lines in the reference line face image (ST210). . At this time, the site points include the lacrimal glands (P1, P2) on each side, the tips of the whites of the outer corners of the eyes (P3, P4), the tails of the lips (P5, P6), the tips of the ears (P7, P8), and the upper lip groove (P9). do. In addition, the measurement line is a line connecting both ear tip points P7 and P8 perpendicularly to the first reference line L1, and connecting both tear glands P1 and P2 and the white tips of the eyes P3 and P4. A line connecting the points of the tails of the lips P5 and P6, and a line connecting the points of the tails of the lips P5 and P6 and the grooves of the lips P9 are included.

Then, as shown in FIG. 10 , the service server 200 establishes a first gravity line GL1 and a second gravity line GL2 perpendicular to the point where the third reference line L3 and the first reference line L1 meet. ) is generated, and a gravity line consisting of a third gravity line GL3 perpendicular to the first gravity line GL1 is generated based on the upper lip tail point among the lip tails (ST220).

In addition, the service server 200 measures the distance between each part point on both sides, the distance between different part points, the distance between measurement lines, the angle, and the area of the surface formed by the measurement line connecting the points based on the gravity line. An asymmetric attribute value is calculated based on the value (ST230). At this time, the service server 200 sets a comparison line by rotating all measurement lines of the face parallel to the first and second gravity lines GL1 and GL2, and based on the difference between the comparison line and the user's face measurement line. Calculate the asymmetric attribute value. In the present invention, the asymmetry attribute is an item for determining asymmetry, the height of both ear tips, the size of both eyes, the horizontal height of both eyes, the horizontal height of the lip tip, the left and right balance of the nose, the left and right balance of the mouth, the left and right eyebrows and the corners of the mouth length, and the left and right areas of the face.

At this time, the service server 400 transmits the attribute value calculated in step ST230 to the user terminal 200, and the user terminal 200 can display and output it through the calibration mirror 100.

Next, the service server 200 compares the reference line shown in FIG. 10 with the gravity line, determines whether or not the reference line is inclined and the direction of the reference line based on the gravity line, and determines the asymmetric type according to the inclination state of the reference line. (ST240). As shown in FIG. 11, the asymmetric type can be classified into three types according to the inclination and direction of the inclination of the first and second reference lines L1 and L2, and the first asymmetric type (A) is the first When only the reference line L1 is inclined, the second asymmetric type is when the second reference line L2 and the first reference line L1 are inclined in different directions, and the third asymmetric type is when the second reference line L2 and the first reference line L1 are inclined in different directions. This is a case where the line L2 and the first reference line L1 are inclined in the same direction.

The first asymmetry type (A) is the facial asymmetry state at the earliest stage, and only the second reference line (L2) is set to an inclined state. This is a state in which the lower jaw is distorted to the left or the right, and the asymmetry is not yet evident in areas other than the lower jaw.

The second asymmetric type (B) is a condition in which the distortion of the lower jaw becomes more severe in the first asymmetry type (A) state, and is the most common facial asymmetry type. In the second asymmetric type (B), both the first and second reference lines L1 and L2 are inclined, and are set to be inclined in different directions. This is analyzed as a type in which the externally rotated upper jaw (maxilla) is twisted in the same direction as the lower jaw as the temporal bone is pushed in the direction the lower jaw is distorted, and the eyes and cheekbones sink in the direction the lower jaw is distorted, causing the face to look distorted. The nose is often crooked. In this case, as aging progresses, wrinkles around the eyes and nasolabial folds are induced more deeply in a distorted direction.

The third asymmetry type (C) is the most severe type of facial asymmetry, in which the first and second reference lines L1 and L2 are both inclined, and are set to be tilted in the same direction. This is a state in which the asymmetry has progressed further in the second asymmetric type (B) and the displacement of the skull, such as the temporal bone and the sphenoid bone, has progressed. This is the upward state, and the displacement of the facial bones and skull is analyzed as the most advanced type.

100: correction mirror, 200: user terminal,
300: coaching service terminal, 400: service server.

Claims (13)

A mirror surface on which RGB LEDs are disposed at each intersection of the lattice pattern, an electronic module communicating with a user terminal and controlling the output of the lattice pattern and RGB LEDs based on a control signal received from the user terminal, and one side of the mirror surface A calibration mirror configured to include a cradle for mounting a user terminal on the;
According to the execution of the facial correction application, it communicates with the correction mirror to provide facial asymmetry training information through the mirror surface, but the service server captures various facial images including a reference line face image and expressionless expression expressed in the correction mirror. In addition to expressing the training information through the correction mirror by communicating with the coaching service terminal matched by the service server, the corresponding training feedback information including the user's face image is transmitted to the coaching service terminal, and the coaching service A user terminal outputting coaching information received from the terminal;
Based on the reference line face image received from the user terminal, a plurality of points of each part including the eyes, ears, and mouth are extracted, and a gravity line of gravity reference corresponding to the reference line is created, and the distance between the gravity line and each part point on both sides Calculate the asymmetric attribute value based on the distance between the and part points, determine the asymmetric type based on whether or not and the direction of the slope between each gravity line and the reference line, and in the standard model registered in advance based on the facial expression image Determines the standard model of the facial structure most similar to the user's face, creates a training schedule targeting the facial expression of the standard model based on the facial motion analyzed based on the facial expression image, and provides it to the coaching service terminal matched with the user terminal a service server and,
The user's facial expression training schedule received from the service server is applied to the coaching application provided and installed from the service server, and training information is transmitted to the user terminal matched by the service server, as well as training feedback information received from the user terminal. It is configured to include a coaching service terminal providing corresponding coaching information,
The user terminal captures a first correction image of a face state of looking at the mirror while being mounted on a cradle of a correction mirror, and a second correction image of a face state of gaze of the user terminal, and obtains a correction image using the following equation. 1 A facial asymmetry correction system based on expression training using a correction mirror, characterized in that it generates a reference line face image, which is a face image displayed as a mirror surface of the correction image.

Here, X and Y are the position coordinates of each corrected pixel, X0 and Y0 are the coordinates of the first right position of the first calibration image, X1 and Y1 are the coordinates of the first right position of the second calibration image, and σ is the camera's unique distortion correction rate. , θ is the slope of the contrast between the straight line and the horizontal line connecting the tops of the left and right eyebrows of the user's face in the first correction image.
According to claim 1,
The mirror surface of the calibration mirror is a mirror plate having a half mirror structure, a grid plate having grid-shaped grooves disposed on the rear surface of the mirror plate, an LED lighting unit disposed at regular intervals on the edge of the grid plate, and a grid plate disposed on the rear surface of the grid plate. It is configured to include an LED matrix in which RGB LEDs are disposed at each intersection of
The electronic module is configured to include an electronic module characterized in that for controlling the on / off of the LED lighting unit and each RGB LED of the LED matrix according to a control signal provided from the user terminal Facial expression using a correction mirror, characterized in that Training-based facial asymmetry correction system.
According to claim 2,
The user terminal drives the LED lighting unit of the calibration mirror to scatter the LED light into the grooves formed on the grid plate so that the grid pattern is displayed on the mirror surface and drives the RGB LEDs at the preset line positions of the LED matrix to form the reference line. In the state where the display is output, the user's face is photographed in a state where the face is arranged based on the reference line to obtain a reference line face image, and the RGB LEDs of the LED matrix corresponding to the asymmetric position are operated with a first color, A facial asymmetry correction system based on facial expression training using a correction mirror, characterized in that configured to control the RGB LEDs of the LED matrix corresponding to the training position and training direction corresponding to the asymmetric position to operate in a second color.
delete According to claim 1,
The correction mirror is configured by additionally coupling a mirror cover of a general mirror structure for covering the mirror surface to the upper side of the mirror surface to open and close in a folder shape,
The user terminal is provided with open/closed state information of the mirror cover from the correcting mirror and controls the lattice pattern to be displayed through the correcting mirror when the mirror cover is open.
According to claim 1,
The service server includes a first asymmetric type in which only the second horizontal reference line is inclined, a second asymmetric type in which the second horizontal reference line and the first horizontal reference line are inclined in different directions, and the second horizontal reference line and the first horizontal reference line. 1 A facial asymmetry correction system based on expression training using a correction mirror, characterized in that the asymmetric type is determined by dividing the asymmetric type into a third asymmetric type in which horizontal lines are inclined in the same direction.
According to claim 1,
The service server generates an additional facial expression image by combining the facial expression images received from the user terminal, analyzes the facial movement using the user facial expression image including the additionally generated facial expression image, and standardizes the additionally generated facial expression image. Create training information to have a model's expression,
The additionally generated facial expression image is a satisfaction image, a warning image, and a worry image by sequentially combining the expressionless image received from the user terminal with the joy image, surprise image, fear image, sadness image, anger image, and disgust image received from the user terminal. , A face asymmetry correction system based on facial expression training using a correction mirror, characterized in that it sequentially generates a discouragement image, a stiffening image, and a contempt image.
Provides facial asymmetry training information using a user terminal that communicates with a correction mirror equipped with RGB LEDs at the intersection of the lattice pattern and the lattice pattern, and provides coaching information on the training status performed through the user terminal through the coaching service terminal. In the facial expression training-based facial asymmetry correction service method using a correction mirror to
In addition to controlling the display of reference lines for face capture on the grid pattern represented by the correction mirror in the user terminal, a face image including the reference line of the correction mirror and various expression images including expressionless expressions are photographed and transmitted to the service server. The first step of doing,
The service server analyzes the face image of the reference line, extracts the facial asymmetry attribute, sets the gravity line of the gravity standard at the position corresponding to the reference line in the face image of the reference line, and determines whether or not the gradient between the gravity line and the reference line and the direction of the inclination are determined. A second step of determining the asymmetric type with
A third step of analyzing facial expression images in the service server and determining a standard model of a facial structure most similar to user facial characteristics from previously registered standard models;
A fourth step of analyzing the user's facial movement based on the facial expression image in the service server;
A fifth step of generating a training schedule targeting the facial expression of the standard model based on the asymmetric type and facial movement in the service server;
The service server matches the coaching service terminal for providing coaching information to the user terminal, transmits the user's training schedule to the matched coaching service terminal, and performs facial correction training through communication between the matched user terminal and the coaching service terminal. However, the user terminal is configured to include a sixth step of driving an RGB LED at a position corresponding to the asymmetric attribute and training information on the user's face reflected on the mirror surface through the correction mirror,
The first step includes obtaining a first correction image of a face state of looking at a correction mirror through a camera of the user terminal in the service server, and second correction of a face state of looking at the user terminal through a camera of the user terminal. A correction mirror, characterized in that it comprises the steps of acquiring an image and generating a reference line face image in the form of a face image displayed as a mirror surface by applying the first correction image to the following equation at the user terminal. Facial expression training-based facial asymmetry correction service method.

Here, X and Y are the position coordinates of each corrected pixel, X0 and Y0 are the coordinates of the first right position of the first calibration image, X1 and Y1 are the coordinates of the first right position of the second calibration image, and σ is the inherent distortion of the user terminal camera The correction rate, θ, is the slope of the straight line connecting the tops of the left and right eyebrows of the user's face in the first correction image and the horizontal line.
delete According to claim 8,
In the first step, the user terminal outputs various induction sentences for inducing expressions of expressionless expression, joy, surprise, fear, sadness, anger, and disgust, and obtains a reaction expression for each induction sentence as a user's facial expression image to provide service. A facial asymmetry correction service method based on facial expression training using a correction mirror, characterized in that for transmission to the server.
According to claim 8,
In the fourth step, the service server additionally generates a user facial expression image by combining the user facial expression images acquired in the first step, analyzes facial movements using the user facial expression image including the additionally generated facial expression image,
In the fifth step, the service server generates training information to have a standard model expression including an additionally generated expression image,
The additionally generated facial expression image is a satisfaction image, a caution image, a worry image, a discouragement image, a hardening image, and a contempt image by sequentially combining a joy image, a surprise image, a fear image, a sadness image, an anger image, and a disgust image in an expressionless image. A facial asymmetry correction service method based on facial expression training using a correction mirror, characterized in that sequentially additionally generated.
According to claim 8,
In the fifth step, the service server selects a certain number of facial expression images in order of increasing facial movements related to the asymmetry attribute and generates training information targeting the selected facial expression. Asymmetry correction service method.
According to claim 8,
In the fifth step, the service server compares the user's facial expression image with the facial expression image of the standard model, calculates the facial movement difference for each facial expression image, selects a certain number of facial expression images in the order of the largest facial movement difference, and aims for the selected facial expression. A facial asymmetry correction service method based on facial expression training using a correction mirror, characterized in that for generating training information to do.

Images