KR102009844B1 - Facial expression recognition device and management service server for dementia patient using the same - Google Patents

Abstract

A server for providing a management service for a dementia patient according to an embodiment of the present invention may include: a communication part which receives terminal state information and the facial image of a user from a user terminal, and transmits a login result performed based on the received facial image to the user terminal; a storage part which stores a result of processing the information received from the user terminal, an image analysis program for analyzing the face image of the user, and user identification information; and a control part which performs user the log-in determination of the user and emotion determination based on the received facial image of the user, and determines at least one of an emergency situation and a necessity of function limitation based on the received terminal state information, and controls the specific function of the user terminal according to the determination result. The availability of a mobile device can be improved by providing contents which the dementia patient prefers.

Description

Facial expression recognition device and dementia patient management service server using the same {{FACIAL EXPRESSION RECOGNITION DEVICE AND MANAGEMENT SERVICE SERVER FOR DEMENTIA PATIENT USING THE SAME}

The present invention relates to a facial recognition apparatus and a management service server for dementia patients using the same. More specifically, the present invention relates to a system for determining the feelings of dementia patients based on facial imaging information received from a terminal used by a dementia patient, managing a risk situation, and helping rehabilitation of dementia patients.

Recent advances in medical technology have prolonged the life span of humankind and rapidly increasing the number of older people. The elderly population is expected to increase more rapidly in the coming future, and it is urgently prepared for the increase in the elderly.

As the proportion of the elderly population rises rapidly, chronic diseases in the elderly also increase. In particular, dementia is a disease accompanied by a decrease in cognitive ability, which is a major obstacle to daily life and social life, and has emerged as an important social problem.

Early dementia patients who can live at home can have some daily life, but sometimes they forget to call their family outside to complain. In this way, devices that help people with dementia basic daily life can be a great help. In addition, as interest in non-pharmaceutical treatment techniques such as music therapy, recall therapy, and cognitive training against dementia patients is increasing, there is an increasing number of individual non-drug treatments in homes or nursing homes. However, there is still a lack of facilities and manpower to care for people with dementia as well as a very low budget. In addition, there is also a lack of a mobile support system that provides content to support non-drug treatments and help people with dementia daily lives.

Meanwhile, the prior art of the data processing system related to dementia is disclosed in Korean Patent Laid-Open Publication No. 10-2016-0001164 (a data processing apparatus for preventing dementia and a data processing method using the same).

The present invention is to identify the dementia patient's preferences by recognizing the facial expression of the dementia patient, to increase the usability of the mobile device by providing the content preferred by the dementia patient, and at the same time to support the non-drug treatment of dementia patients Designed for

The server for providing a dementia patient management service according to an embodiment of the present invention receives terminal state information and a facial image of a user from a user terminal, and transmits a login result performed based on the received facial image to a user terminal. The user based on the communication unit, a result of processing the information received from the user terminal, an image analysis program for analyzing the user's face photographed image, a storage unit for storing user identification information, and a face photographed image of the user. The controller may perform a log-in decision and an emotional determination of the terminal, determine at least one of an emergency situation or a necessity of function limitation based on the received terminal state information, and control a specific function of the user terminal according to the determination result. have.

Various embodiments of the present disclosure may help dementia patients to use mobile devices more easily. In particular, the present invention can support a dementia patient to log in just by photographing a facial image even without inputting an ID and password directly on the mobile device.

In addition, various embodiments of the present invention allows dementia patients to perform cognitive training using a quiz based on their family photo on a mobile device.

In addition, various embodiments of the present invention analyze the face image of the dementia patient to automatically analyze and determine preference information for each multimedia content viewed by the dementia patient, so that the dementia patient can play the content prefered.

1 is a view showing a schematic configuration of a system for providing a dementia patient management service according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a server 100 that provides a dementia patient management service according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating a configuration of a user terminal according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating a login process using a face of a user in a user terminal according to an exemplary embodiment of the present invention.
5 is a flowchart illustrating a process of calculating preference information based on user facial information according to an exemplary embodiment of the present invention.
6 is a diagram illustrating an example of training content according to an embodiment of the present invention.
7 to 8 are diagrams illustrating a signal flow system in an emergency according to an embodiment of the present invention.
9 is a diagram illustrating an example of a notification service according to an embodiment of the present invention.
10 is a diagram illustrating a sequence of preprocessing steps for facial image analysis according to an exemplary embodiment of the present invention.
11 is a view for explaining the structure of the AlexNet model proposed in the image analysis process of the user image analysis unit according to an embodiment of the present invention.
12 is a diagram illustrating an example of a Facial Action Coding System according to an embodiment of the present invention.
13 illustrates a process of extracting a fixed feature using a pre-trained model according to an embodiment of the present invention.
14 illustrates AU recognition accuracy for each feature according to an embodiment of the present invention.
FIG. 15 illustrates transfer learning in a model that is pre-trained using imageNet Data according to an embodiment of the present invention.
16 is a diagram illustrating a structure of a Bvlc alexnet model according to an embodiment of the present invention.
17 is a diagram illustrating the development status of the Android tensorflow inference library according to an embodiment of the present invention.
18 is a diagram illustrating the structure of an Android tensorflow inference library according to an embodiment of the present invention.
19 is a diagram illustrating a process of applying a facial expression recognition model developed in Python to Android according to an embodiment of the present invention.
20 is a diagram illustrating an example of an operation of recognizing an expression through specific user image information according to an exemplary embodiment of the present invention.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the drawings, similar reference numerals are used for similar elements.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may exist in the middle. Should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Also, in the present specification, the device may be a general apparatus (or thing) connected to a gateway and applied to the Internet of Things (IoT). For example, the device may be a wireless pager, smartphone, tablet PC, computer, temperature sensor, humidity sensor, acoustic sensor, motion sensor, proximity sensor, gas sensor, heat sensor, refrigerator, CCTV, TV, washing machine, dehumidifier , Lights, fire alarms, and the like. However, this is not limitative.

In addition, a device may be used interchangeably with a device or a device in the present specification, and the device, the device, and the device may be described in the same expression.

Also, in the present specification, a service may include various services that can be performed by a device. The service may include a service based on communication with a server or another device, and a service operable in the device. The service applied to the present disclosure is preferably understood as a broad concept including various services that can be performed in the device in addition to the services described as examples in the present disclosure.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. Hereinafter, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a view showing a schematic configuration of a system for providing a dementia patient management service according to an embodiment of the present invention.

 The server 100 supporting the dementia patient management service application may communicate with the user terminal 200 using the application and receive necessary state information from the user terminal 200. In addition, the server 100 may provide some information on the operation result to the user terminal 200 to control some functions. The user terminal may communicate with the guardian terminal 400 and the peripheral device 301 capable of the IoT in the living space 300. When it is determined that the server 100 is in an emergency situation, the server 100 may perform a function of calling the medical institution 501.

2 is a block diagram illustrating a configuration of a server 100 that provides a dementia patient management service according to an exemplary embodiment of the present invention.

The server 100 may include a communication unit 110, a storage unit 120, and a controller 130. The storage unit 120 may store basic content 121, content preference information 122, rehabilitation content 123, institution contact 124, image analysis program, and data 125. The controller 130 may include a voice recognition processor 131, a user image analyzer 132, a situation information analyzer 133, a customized content information provider 134, an emergency determiner 135, and a rehabilitation trainer 136.

First, the communication unit 110 may use a network for data transmission and reception between a user device and a server, and the type of the network is not particularly limited. The network may be, for example, an IP (Internet Protocol) network providing a transmission / reception service of a large amount of data through an Internet protocol (IP), or an All IP network integrating different IP networks. The network may include a wired network, a wireless broadband network, a mobile communication network including WCDMA, a high speed downlink packet access (HSDPA) network, and a long term evolution (LTE) network, LTE advanced (LTE-A). ), Or one of a mobile communication network including 5G (Five Generation), a satellite communication network, and a Wi-Fi network, or a combination of at least one of them.

The communication unit 110 according to an embodiment of the present invention may receive user information transmitted from a user terminal and state information of the user terminal. In addition, the communication unit 110 may receive a face-photographed image of the user to be used in place of the ID and password in order to determine the member information of the user according to an embodiment of the present invention. In addition, the communication unit 110 may receive the state information of each device (eg, whether there is open, usage, operating time, etc.) from the peripheral device supporting the IoT function installed in the user's living place according to an embodiment of the present invention. It is also possible to send signals for direct control of the device. In addition, the communication unit 110 may perform a communication function for calling a medical institution or a guardian under the judgment of the controller that the user (dementia patient) is exposed to an emergency situation. In addition, the communication unit 110 may receive necessary materials (multimedia content, rehabilitation content, medical institution contact information, geographic information of a medical institution, etc.) from a web server through the Internet.

The storage unit 120 may include, for example, an internal memory or an external memory. The internal memory may be, for example, volatile memory (for example, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (for example, OTPROM (one). time programmable ROM (PROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash), hard drives, Or it may include at least one of a solid state drive (SSD).

The external memory may be a flash drive such as compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (XD), It may further include a multi-media card (MMC) or a memory stick. The external memory may be functionally and / or physically connected to the electronic device through various interfaces.

According to an embodiment of the present disclosure, the storage unit 120 may store basic content 121, content preference information 122, rehabilitation content 123, institution contact 124, image analysis program, and data 125. First, the basic content 121 may refer to multimedia content basically provided to an application for providing a management service for dementia patients. The basic content 121 may be reproduced for the purpose of focusing attention of a user (dementia patient), for example. As a result, caregivers caring for dementia patients will be able to temporarily focus their attention on viewing content and perform separate tasks. The basic content 121 may be composed of a TV program, a music video, a movie, animation, music, and the like. However, according to various embodiments of the present disclosure, the multimedia content provided to the user is not limited to the basic content 121, and the multimedia content previously stored in the user terminal 200 may be selected and executed on the application.

 The content preference information 122 refers to information on a result of evaluating the emotional state of the user with respect to the content being played based on the user's facial image received from the user terminal. Accordingly, the content preference information 122 may include information about a preference of each user, a genre of the content, and a preference of each topic. The content preference information 122 may be used to sort the contents according to the preference order of the user when providing a list of executable contents to the user terminal.

The rehabilitation content 123 may refer to content prepared for rehabilitation of a user (dementia patient). The rehabilitation content 123 may include, for example, learning materials and games (eg, memory games) for training the cognitive ability of the user. In addition, the rehabilitation content 123 may include a quiz material consisting of a picture of the surrounding people. Nearby portraits may be uploaded from the user terminal, for example, by a guardian.

The institution contact 124 may be referred to when calling the medical institution because it is determined that the user's condition is an emergency. The institution contact 124 may include the address of the medical institution as well as the contact information of the medical institution. In addition, the institution contact 124 may include contact information for various organizations such as a police station and a fire station, as well as a medical institution. Accordingly, when an emergency situation occurs, information on the emergency situation and location information and movement path information of the user terminal 200 carried by the dementia patient may be automatically provided to the relevant institution. According to various embodiments of the present disclosure, when a call to an emergency situation is made to a police station contact of the agency contact 124 (eg, requesting help of a police station based on a user's disappearance determination), the police station is reminded of CCTV location information or CCTV photographing information for shooting a missing person. It may receive through the communication unit 110.

The image analysis program and the data 125 according to an embodiment of the present invention may identify the user (determining login information) and facial expression and emotion of the user based on artificial intelligence based on artificial intelligence. The image analysis program may determine, for example, the expression of the user using Google's TensorFlow, Convolutional Neural Network (CNN), and extended Cohn-Kanade (CK +) dataset. The CK + dataset is classified into four data groups: 'Image, Emotion, labels, Facial Action Coding System (FACS) labels, Landmarks', and the facial recognition function according to an embodiment of the present invention includes image data including a face image. Emotion labels containing the image's emotional labels, and FACS labels data encoding all anatomically possible facial expression changes can be used. Image data is image data that is taken and provided in units of frames by capturing 593 facial expression images from 123 people. Each facial expression data may be included from a normal natural expression to a moment of making a specific facial expression.

Emotion labels data may be data that classifies what expressions are made on 327 facial expression images out of a total of 593 facial expression images. Based on the FACS investigators Guide confirmed by the visual inspection of the emotion researcher, 327 images that have been validated can be selected and emotion can be classified from the last peak frame of the image data corresponding to each image. A number between 0 and 7 (ie 0 = nuetral, 1 = anger, 2 = contempt, 3 = disgust, 4 = fear, 5 = happy, 6 = sadness, 7 = surprise) corresponding to an emotion classified in the file Can be taken. However, the classification result is not limited to the above seven and may be classified into more various items.

The operation of recognizing a facial expression of a user will be described later in more detail with reference to the user image analyzer 132.

The controller 130 may also be referred to as a processor, a controller, a microcontroller, a microprocessor, a microcomputer, or the like. On the other hand, the control unit may be implemented by hardware (hardware) or firmware (firmware), software, or a combination thereof.

In the case of implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code may be stored in a memory and driven by the controller. The memory may be located inside or outside the user terminal and the server, and may exchange data with the controller by various known means.

According to an embodiment of the present disclosure, the controller 130 may include a voice recognition processor 131, a user image analyzer 132, a context information analyzer 133, a customized content information provider 134, an emergency determiner 135, and a rehabilitation trainer 136.

First, the voice recognition processor 131 may recognize a user voice input while an application is executed and process a related function. For example, when a user inputs a request to play a specific multimedia content on a dementia patient management service application by voice, the voice recognition processor 131 may recognize a corresponding voice command and extract a content name in the voice command. Thereafter, the controller 130 may determine a function to be provided to the user based on the command translated by the voice recognition processor 131 and control to provide the corresponding function.

According to various embodiments of the present disclosure, the controller 130 may designate and manage a voice command frequently requested by a user more than a predetermined number of times based on a result processed by the voice recognition processor 131. The controller 130 may store information on a content or a specific function (eg, a weather forecast request, DMB playback, etc.) frequently requested by a user, and update the content preference information 122 by adding points to frequently executed content. . In addition, the controller 130 may provide a service for automatically asking whether to execute a specific function that is frequently executed. For example, for a specific function executed more than a preset number of times in the last week, the controller 130 may classify the function as a user interest function. The controller 130 analyzes time information on which the user's interest function is mainly executed, and if there is a history of playing the corresponding function at a predetermined time period, the controller 130 provides voice guidance to the user, such as 'will execute the OO function' by voice in the same time zone. Can be provided.

The user image analyzer 132 may use the image analysis program 125 to obtain information about a user's facial expression and emotion state from the user's facial image by using the image analysis program 125. Although the user image analyzer 132 mainly describes evaluating collected user face image information, the user image analyzer 132 may also perform a learning process necessary to extract desired information from the user face image information.

According to an embodiment of the present disclosure, the user image analyzer 132 may be configured to view information (genre, subject, title, etc.) of the multimedia content being executed and the corresponding content when the user's face photographed image is a video captured while playing the multimedia content. It is possible to determine the emotional state of the user of the user can be calculated based on the preference information for the specific content of the user. The content preference information calculated in this manner may be stored in the storage 120 and updated as the user emotional state is further analyzed.

The operation of recognizing the facial expression of the user in the user image analyzer 132 may be performed as follows. The user image analyzer 132 may perform a preprocessing operation of transforming a received face image of a user according to a TensorFlow and a CNN model to recognize a facial expression of a user. The data preprocessing operation may be performed in the order shown in FIG. 10.

10 is a diagram illustrating a sequence of preprocessing steps for facial image analysis according to an exemplary embodiment of the present invention.

The user image analysis unit 132 selects 327 image data with an emotion label for easy management and processing of data, and neutralizes the first frame before the expression changes, and expresses the last three frames where the expression is clearly expressed according to the emotion label. After dividing into, the image data may be divided by facial expressions. The user image analyzer 132 then converts the RGB image to a gray image to remove useless portions of the data and improves the recognition rate, and finds a human face using an haarcascade face detection of OpenCV to a suitable size (255x255). You can freeze and save the video. In this case, the user image analyzer 132 may invert the image left and right to increase the amount of data and increase the data and randomly mix the images. This prevents overfitting early in the training process. In addition, the user image analyzer 132 separates and stores training data and evaluation data separately, so that the recognition performance can be objectively evaluated.

The data format of the saved image is modified to meet the requirements of the high level api 'tensorflow.contrib.learn' provided by TensorFlow, and the format can be configured as the following table.

Data type Data format training data float32 eval data float32 training labels int32 eval labels int32

The user image analyzer 132 loads the stored data before training and evaluating the model and performs a final preprocessing process. At this time, for example, the data is normalized from 0 to 1, and the size of the data is 255x255 to further increase the amount of data. A selective random technique may be applied to randomly extract a 224x224 size image from a non-overlapping image. Thereafter, the user image analyzer 132 may unfold the data into a one-dimensional array, input the data into a model for designing, and start training and evaluation.

The user image analyzer 132 may extract features of the input data after a preprocessing process and modeling (eg, based on AlexNet model structure).

11 is a view for explaining the structure of the AlexNet model proposed in the image analysis process of the user image analysis unit according to an embodiment of the present invention.

Referring to FIG. 11, the calculation process of the AlexNet model can be divided into two types. The first is to extract 256 features from the input data through five convolution operations, and the second is to classify the data through the full connect operation using the extracted features.

If a 224x224 sized image is pre-processed into a model and the image is expanded into a one-dimensional array, 96 kernels of size 11x11 and stride of 4 can be combined with the composite product. After passing the first convolutional layer, the 55x55x96 image is output, which is then combined with 256 kernels of 5x5 size and 1 stride. The user image analyzer 132 may repeat this operation to obtain 256 features of 6x6 size after the operation of passing the last fifth composite product layer and the max-pooling layer.

The user image analyzer 132 may classify the extracted data through three dense layers and two dropout layers. In the case of training the model, the error can be corrected through softmax cross entropy, and in the case of evaluation, the classified (predicted) values can be checked through the softmax and argmax functions.

According to an embodiment of the present invention, three kinds of models may be used in the model design process. The first is a model trained directly using the CK + dataset, the second is the model using the Action Unit Selective technique, and the last is a pre-trained model pre-trained with a large amount of ImageNet data.

First, the model trained directly using the CK + dataset recognizes eight facial expressions (neutral, angry, contempt, disgust, fear, happy, sadness, surprise) and six faces to verify the performance of the model. It can be divided into cases of recognizing facial expressions (angry, disgust, fear, happy, sadness, surprise). According to an exemplary embodiment of the present invention, the user image analyzer 132 may perform an additional experimental operation while changing a training method such as changing a data set or a patch setting in order to improve the performance of the model. The result of checking the evaluation accuracy according to the change of the training method for the model for recognizing the eight facial expressions may be calculated as shown in Table 2 below.

input
size reverse random
opt batch
size steps total
step stopping
step accuracy loss overfit 224 x 224 x normal 20 58 x 150 58 x 150 x 16 25.00k 0.7537 1.32 10.00k 224 x 224 x normal 20 58 x 150 58 x 150 x 16 35.00k 0.7537 1.23 9.00k 224 x 224 x normal 20 58 x 150 58 x 150 x 16 22.00k 0.7611 1.07 10.00k 224 x 224 o normal 20 117 x 50 117 x 50 x 16 24.00k 0.7126 1.18 10.00k 224 x 224 o normal 20 117 117 x 16 x 10 18.72k 0.7798 0.69 16.00k 224 x 224 o normal 20 117 117 x 36 x 56 235.87k 0.8208 1.56 10.00k 224 x 224 o full 20 117 117 x 160 18.72k 0.7761 0.51 x 224 x 224 o selective 20 117 117 x 160 18.72k 0.7947 0.44 x 224 x 224 o selective 20 117 117 x 300 35.10k 0.8134 0.83 16.00k 224 x 224 o selective 20 117 117 x 200 23.40k 0.8022 0.65 16.00k 224 x 224 o selective 20 117 117 x 276 32.29k 0.8134 0.75 20.00k 224 x 224 o selective 20 117 117 x 200 23.40k 0.8097 0.51 x 224 x 224 o selective 20 117 117 x 200 23.40k 0.8059 0.45 x 224 x 224 o selective 20 117 117 x 200 23.40k 0.7985 0.52 x

The user image analyzer 132 according to an embodiment of the present invention may increase the accuracy by adding reverse data to the input data and applying a selective random technique to increase the amount of data by several times.

By adding reverse data to the input data and evaluating the models that recognize the eight facial expressions and the six facial expressions when the selective random technique is applied, the results are written in a confusion table and the accuracy of each facial expression is verified. The result is as follows.

Neutral Angry Contempt Disgust Fear Happy Sadness Surprise Neutral 0.9242 0 0 0 0.0151 0.0606 0 0 Angry 0.3571 0.4285 0 0 0 0 0.2142 0 Contempt 0.3333 0 0.1666 0 0 0.1666 0 0.3333 Disgust 0.1388 0 0 0.8611 0 0 0 0 Fear 0 0 0.25 0 0.75 0 0 0 Happy 0 0 0 0 0 One 0 0 Sadness 0.3333 0.3333 0.1111 0 0 0 0.2222 0 Surprise 0 0 0 0 0 0 0 One

Table 3 shows a confusion table of eight facial expression recognition models. (Input image size 224x224)

Angry Disgust Fear Happy Sadness Surprise Angry 0.7857 0 0 0 0.2142 0 Disgust 0 One 0 0 0 0 Fear 0 0 One 0 0 0 Happy 0 0 0 One 0 0 Sadness 0.7222 0 0 0 0.2777 0 Surprise 0 0 0 0 0 One

Table 4 shows a confusion table of six facial expression recognition models. (Input image size 224x224)

Neutral Angry Contempt Disgust Fear Happy Sadness Surprise Neutral 0.9242 0 0 0.0303 0.0454 0 0 0 Angry 0.6428 0.2857 0 0 0 0 0.0714 0 Contempt 0.3333 0 0.5 0 0.1666 0 0 0 Disgust 0.3888 0 0 0.6111 0 0 0 0 Fear 0 0 0.25 0 0.75 0 0 0 Happy 0 0 0 0 0 One 0 0 Sadness 0.5 0 0.0555 0 0 0 0.4444 0 Surprise 0 0 0 0 0 0 0 One

Table 5 shows a confusion table of eight facial expression recognition models. (Input image size 112x112)

Angry Disgust Fear Happy Sadness Surprise Angry 0.6071 0.0714 0.1071 0 0.2142 0 Disgust 0 One 0 0 0 0 Fear 0 0 One 0 0 0 Happy 0 0 0 One 0 0 Sadness 0.2777 0 0 0 0.7222 0 Surprise 0 0 0 0 0 One

Table 6 shows a confusion table of six facial expression recognition models. (Input image size 112x112)

According to the presented confusion table, the user image analyzer 132 according to an embodiment of the present invention may calculate about 80% accuracy for eight facial expressions and about 90% accuracy for six facial expressions.

Second, the user image analyzer 132 according to an embodiment of the present invention may utilize a model to which an action unit selective (AUS) technique is applied.

Action unit (AU) is a concept in the Facial Action Coding System (FACS) developed by Paul Ekman and Wallace V. Friesen. FACS is a system that encodes and classifies all anatomically possible facial muscle movements into AUs, and is based on numerous studies that read emotions from facial expressions.

12 is a diagram illustrating an example of a Facial Action Coding System according to an embodiment of the present invention.

The development sequence of the AUS technique is as follows. First, images data including emotion labels and FACS labels data are extracted from the CK + dataset, and all FACS labels data is checked.

Angry Contempt Disgust Fear Happy Sadness Surprise Total AU0 0 0 0 0 0 0 0 0 AU1 0 One One 22 0 26 81 131 AU2 0 One 0 10 0 7 81 99 AU3 0 0 0 0 0 0 0 0 AU4 40 One 36 21 0 23 One 122 AU5 6 0 0 16 0 0 70 92 AU6 8 0 18 3 66 0 0 95 AU7 32 0 33 6 7 One 0 79 AU8 0 0 0 0 0 0 0 0 AU9 3 0 58 0 0 0 0 61 AU10 2 0 7 4 0 0 2 15 AU11 0 0 0 2 0 0 0 2 AU12 One 5 2 2 67 0 3 80 AU13 0 0 0 0 2 0 0 2 AU14 7 17 2 5 0 0 One 32 AU15 3 4 2 0 0 23 One 33 AU16 One 0 0 4 4 0 3 12 AU17 39 5 40 3 0 27 0 114 AU18 4 0 0 0 0 2 0 6 AU19 0 0 0 0 0 0 0 0 AU20 0 0 0 25 One 0 One 27 AU21 0 0 0 One 0 0 0 One AU22 0 0 0 0 0 0 0 0 AU23 36 One 2 0 0 3 One 43 AU24 33 2 7 0 0 One 0 43 AU25 0 0 9 23 67 0 82 181 ...

Table 7 shows nine AU screening tables and shows the number of appearances of AU for each facial expression.

After the AU screening is finished, the user image analyzer 132 may classify the data into images including AUs and images not including the AUs for each of the selected AUs, and configure the AU binary classifier using a pre-trained model. The pre-trained model is a pre-trained model that uses a large amount of data to compensate for the lack of data and extract fixed features, which is suitable for facial features and AUS method using AU.

13 illustrates a process of extracting a fixed feature using a pre-trained model according to an embodiment of the present invention.

The user image analyzer 132 according to an embodiment of the present invention alternately substitutes 256 features extracted by using a pre-trained model into the configured AU binary classifier and checks the accuracy of the AU binary classifier for each feature. When performing the AUS technique, the user image analyzer 132 utilizes changes in facial muscles, that is, spatial information, and may utilize 256 feature maps of 6x6 size before the fully-connected layer as a feature. Table 8 below shows the process after pseudo AU screening with pseudo code.

# Initialize
Images, AU_labels = CK_plus.load () # images and image's action unit labels array
AUs = [AU1, AU2, AU4, AU5, AU6, AU7, AU12, AU17, AU25]
result = [] [] # 2-D array

# each action unit
for AU in AUs:
# separate AU images and non AU images
labels = []
for i in range (length (AU_labels)):
if AU in AU_labels [i]:
labels.append (1) # true
else:
labels.append (0) # false

# extract 256-features
features = pre-trained_AlexNet_model (Images)

# train model
original_model.train (features, labels)

# evaluate each feature
for feature in features:
eval_score = original_model.evaluate (feature)
index = AUs.getIndex (AU) # 0, 1, 2, ...
result [index] .append (eval_score)

Thereafter, the user image analyzer 132 may select one of the AUs best found for each feature, compare the accuracy, and store the index in order of the feature having the highest accuracy. This is to construct a model using the feature that finds AU well regardless of any AU. Feature-specific AU recognition accuracy may be calculated as shown in FIG. 15.

14 illustrates AU recognition accuracy for each feature according to an embodiment of the present invention.

According to an embodiment of the present invention, the user image analyzer 132 uses only 240 features in the facial expression recognition model to which the AUS technique is applied. Reflection).

The AUS method is applied to the model as follows. The 256 feature data extracted using the pre-trained model are selected and rearranged to 240 feature data, and the rearranged data is connected to the last fully-connected layer to reconstruct the model. The following iterative training and evaluation process completes the facial expression recognition model using the AUS technique.

The facial expression recognition model to which the AUS technique is applied also confirmed the performance of all eight facial expression recognition models and six facial expression recognition models, and the accuracy of each facial expression can be calculated as follows.

Neutral Angry Contempt Disgust Fear Happy Sadness Surprise Neutral 0.8034 0.2119 0.2356 0.1972 0 0 0.083 0.0231 Angry 0.1157 0.7874 0 0.0498 0 0 0.01 0 Contempt 0.0036 0 0.7144 0 0 0 0.1193 0 Disgust 0.0015 0 0.0006 0.7524 0.0029 0.0017 0 0 Fear 0.036 0 0.0056 0.0002 0.9971 0.0048 0.0267 0.0037 Happy 0.024 0 0.0067 0 0 0.993 0 0 Sadness 0.0137 0.0007 0.0022 0.0004 0 0 0.76 0 Surprise 0.002 0 0.035 0 0 0.0005 0.0011 0.9732

Table 9 shows confusion tables of eight facial expression recognition models.

Angry Disgust Fear Happy Sadness Surprise Angry 0.9967 0.0696 0 0.0002 0.1081 0 Disgust 0 0.9254 0.0013 0.0005 0.0007 0 Fear 0 0 0.9988 0.0016 0.0452 0.0296 Happy 0 0 0 0.9978 0 0 Sadness 0.0033 0.005 0 0 0.8456 0 Surprise 0 0 0 0 0.0004 0.9704

Table 10 shows a confusion table of six facial expression recognition models.

Third, the user image analyzer 132 may use a pre-trained model through ImageNet data.

In deep learning, there are not many people who can experiment with enough data sets, so it is rare to train the CNN model from beginning to end. Accordingly, the concept of transfer learning emerged. Refer to FIG. 15 for the transfer learning.

FIG. 15 illustrates transfer learning in a model that is pre-trained using imageNet Data according to an embodiment of the present invention.

Transfer learning is a concept used to solve other related problems by using information obtained while solving a problem. The pre-trained model can be used to transfer specific information to the model to be used in the experiment using this transfer learning. When training a CNN model, we use a proven, pre-trained, pre-trained CNN model, and training the model using a very large data set such as ImageNet can provide specific knowledge (eg, general feature extraction methods for image recognition; model's weights).

The user image analyzer 132 according to an embodiment of the present invention may use bvlc_alexnet as a pre-trained AlexNet model. The bvlc_alexnet is a model converted to use a model trained in Caffe, a C ++ based deep learning framework, in TensorFlow.

The Bvlc alexnet model contains the structure information of all hidden layers of AlexNet, and the weight and bias values of each node. Conv1 has weight value and bias value for 11 by 11 size 3-channel convolution operation, conv3 has weight value and bias value for 3 by 3 size 256-channel convolution operation, and each output has 96-channel and 384-chanel. Conv2, conv4, and conv5 have only weight and bias values that correspond to half of the previous layer, so that the convolution operation is executed twice through separate processing. Similarly, fc6, fc7, and fc8 have weight and bias values for the last full connect operation.

The structure of the Bvlc alexnet model is as shown in FIG.

The user can import the bvlc alexnet model and obtain the desired output value from the input image through manual operation. According to an exemplary embodiment of the present invention, a weight value and a bias value of conv1, conv2, conv3, conv4, and conv5 are called to extract a 6-by-sized 256-channel feature from an input image, and have a great effect on facial expression recognition performance. Unless this was done, we trained a separate fully-connected layer to reduce the size of the model and speed up computation.

The accuracy of each facial expression of the eight facial expression recognition models and the six facial expression recognition models constructed using the pre-trained AlexNet model can be calculated as shown in the following table.

Neutral Angry Contempt Disgust Fear Happy Sadness Surprise Neutral 0.8715 0.2143 0.0667 0.1717 0.0025 0.0086 0.03 0.0648 Angry 0.0873 0.7793 0 0.0089 0 0.0005 0.0033 0 Contempt 0.0039 0 0.8533 0 0 0.0033 0.0011 0 Disgust 0.0018 0 0.0017 0.8178 0 0.0629 0 0 Fear 0.0048 0 0.0067 0.0011 0.995 0.0014 0.0011 0.0004 Happy 0.0197 0 0.0383 0.0006 0.0013 0.9233 0 0 Sadness 0.0097 0.0064 0 0 0.0013 0 0.9644 0 Surprise 0.0012 0 0.0333 0 0 0 0 0.9348

Table 11 shows a confusion table of eight facial expression recognition models.

Angry Disgust Fear Happy Sadness Surprise Angry 0.9957 0.0239 0 0.0033 0.0044 0 Disgust 0 0.975 0.0088 0.0043 0 0 Fear 0 0 0.9788 0.0005 0.0011 0.0012 Happy 0 0 0.0013 0.9919 0 0 Sadness 0.0043 0.0011 0.0025 0 0.9944 0 Surprise 0 0 0.0088 0 0 0.9988

Table 12 is a confusion table of six facial expression recognition models.

17 is a view showing the development status of the Android tensorflow inference library according to an embodiment of the present invention, Figure 18 is a view showing the structure thereof.

The server according to an embodiment of the present invention may use the android inference library shown in FIGS. 17 and 18 to apply tensorFlow to a mobile. The Android inference library acts as an interface that connects Android apps written in the Java language with TensorFlow written in the C ++ language. It can provide functions to load graphs, set inputs, and calculate specific output using models.

19 is a diagram illustrating a process of applying a facial expression recognition model developed in Python to Android according to an embodiment of the present invention.

Referring to FIG. 19, a process of applying a facial expression recognition model developed in Python will be described. The process of applying the model developed using TensorFlow to android app can be done as follows. First, we freeze the developed model to integrate various files composing the model into one file and manage each layer information in one graph. Next, optimize the model to use only the functions required for facial expression recognition (input and output) and move the optimized model file to the android app. And after setting the model file path, input and output node, you can get the output value when you input data into the model by using the function provided by android inference library.

20 is a diagram illustrating an example of an operation of recognizing an expression through specific user image information according to an exemplary embodiment of the present invention.

Accordingly, the user image analyzer 132 according to an exemplary embodiment of the present invention may perform a function of recognizing an expression based on user image information as shown in FIG. 20.

In other words, the user image analyzer 132 may perform an operation of preprocessing and modeling corresponding information in order to evaluate a learning process and a user's facial image. The user image analyzer 132 may classify the facial expression and the emotional state of the user into predetermined items (eg, fear, joy, surprise, etc.) based on the received facial information of the user.

The context information analyzer 133 may determine whether the user wanders based on the state information of the user terminal received from the user terminal. The state information of the user terminal may refer to information about a terminal position measured by a communication unit 110 (eg, GPS) of the user terminal, and a moving speed of the user terminal detected by a sensor such as an acceleration sensor (not shown). have. In addition, the state information of the user terminal may further include battery remaining information and information on a function that is being executed for a predetermined time period in the user terminal. The situation information analysis unit 133 determines whether the user roams, and if it is determined that the user (dementia patient) roams outdoors, the user terminal 200 immediately displays information on the screen of the user's location from the current user's location to the user's residence. Can be displayed. According to various embodiments of the present disclosure, the contextual information analyzing unit 133 may collect user information (including residence information) from the user terminal 200 to perform a guide to return to a residence. Alternatively, the information about the residence or the guardian may be stored and managed as member information (not shown) in the server 100 side storage unit 120.

The context information analyzer 133 may analyze the context information based on the user information and the schedule information as well as the state information of the user terminal. The server 100 may be linked with an SNS including a scheduler and an instant messenger of a user input to the user terminal 200. Accordingly, the contextual information analysis unit 133 may grasp the schedule of the user. According to various embodiments of the present disclosure, information on a schedule of a user (including a medication schedule) may be received and managed in the dementia patient management application. In this case, the situation information analyzer 133 may store and manage the schedule information of the user in the storage unit 120 on the server 100 and provide the schedule information as a notification to the user terminal 200. Notifications may include push notifications or text message notifications with sound. Depending on the importance of the schedule you have previously checked (if the user has previously checked for an important event), The guardian may be further informed of the schedule.

The personalized content information providing unit 134 may determine whether there is content preference information of the corresponding user when a list of multimedia contents held in the server is requested from the user terminal. Thereafter, if the content preference information exists, the personalized content information providing unit 134 may provide a content list based on the content preference information. At this time, the content list may be sorted in order of preference of the user. In addition, it is possible to provide information reflecting the user's preference by various methods. For example, the personalized content information providing unit 134 may add the content based on the user's content preference information generated by the user image analyzer 133, the execution cycle for each content, the number of voice inputs for requesting a specific content, and the cumulative playback count for each content. The preference information 122 may be generated.

In addition, the personalized content information providing unit 134 may automatically execute the content most preferred by the user based on the content preference information 122 or change the content provided based on the real-time response information of the user when the content is played. For example, if the user's request is related to 'playing the drama', the user may play a new episode of the drama most preferred by the user among a plurality of multimedia contents corresponding to the drama.

The emergency determination unit 135 may determine a risk of an accident of a user based on state information received from the user terminal 200 and peripheral devices supporting the IoT function. In this case, the accident risk may be classified by item, and if the determined accident risk is greater than or equal to a preset criterion, the controller may perform at least one of a peripheral device control supporting a medical institution call, a guardian call, and an IoT function according to the item.

The items for evaluating the risk of an accident may include, for example, the use of a specific heating device, the use of gas, the seizure, the use of a front door, the lack of remaining battery power, and no response.

The emergency determination unit 135 receives the location information of the user when calling the medical institution based on the accident risk, calls the medical institution of the nearest location from the corresponding location, and when the medical institution call is performed, a notification of the medical institution call (eg, Text message) can be delivered to the guardian terminal.

In addition, the emergency determination unit 135 may control the communication function of the user terminal to perform a call connection between the terminal of the user and the guardian terminal when the guardian calls based on the risk of an accident. Accordingly, the emergency determination unit 135 may support the guardian to directly check the status of the user (dementia patient) by performing an automatic telephone connection from the user (dementia patient) terminal to the guardian terminal in an emergency.

In addition, the emergency determination unit 135 uses a specific heater when a user (dementia patient) uses a gas inside the house where the user lives, and goes out to the outside. It can be determined as a risk factor in the case of a home or a smart outlet). For example, when it is determined that the power is continuously turned on for a predetermined time after the user starts using the iron, the emergency determination unit 135 may record the voice data of the user that is photographed or recorded through the duration and other user terminals, or Risk can be calculated based on other peripherals (eg indoor CCTV). To this end, when the first risk is detected, the camera and the recording function of the user terminal may be activated. Determination of the second risk based on the information collected using the first risk detection and the use time of the specific heater, the camera and recording function of the user terminal activated in response to the detection of the first risk, the second risk is determined It can be controlled to automatically turn off the power of the heater. In this case, the emergency determination unit 135 may send a signal to control the peripheral device registered in the user terminal 200 through the peripheral device control unit 275 of the user terminal 200. Alternatively, according to various embodiments of the present disclosure, since the server 100 may grasp information about an IoT peripheral device held for each user, the emergency determination unit 135 may directly limit the use of a specific peripheral device in question.

  The rehabilitation training unit 136 may perform cognitive training of the user based on the picture of the person around the user. At this time, the rehabilitation training unit 136 may generate a quiz based on a picture of a nearby person and provide the user with a memory to perform memory learning.

In addition, the rehabilitation training unit 136 may execute rehabilitation content 123 provided by the server 100. The rehabilitation content 123 may include various types of content for training memory, such as a game, an image.

3 is a block diagram illustrating a configuration of a user terminal according to an exemplary embodiment of the present invention.

The user terminal 200 may include a communication unit 210, a storage unit 220, an input unit 230, a display unit 240, a camera unit 250, an audio module 260, and a controller 270.

The storage unit 220 may store the user information 221, the schedule information 222, the training record 223, and the photo information 224. The controller 270 may include a voice recognition processor 271, a situation information provider 272, an emergency contact unit 273, a content execution unit 274, and a peripheral device controller 275.

The communication unit 210 may include, for example, a cellular module, a WiFi module, a Bluetooth module, a GPS module (eg, a GPS module, a Glonass module, a Beidou module, or a Galileo module), an NFC module, and a radio frequency (RF) module.

The cellular module may provide, for example, a voice call, a video call, a text service, or an internet service through a communication network. According to an embodiment of the present disclosure, the cellular module may perform identification and authentication of an electronic device in a communication network using a subscriber identification module (eg, a SIM card). According to an embodiment of the present disclosure, the cellular module may perform at least some of the functions that the processor may provide. According to an embodiment, the cellular module may include a communication processor (CP).

Each of the WiFi module, the Bluetooth module, the GPS module, or the NFC module may include, for example, a processor for processing data transmitted and received through the corresponding module. According to some embodiments, at least some (eg, two or more) of the cellular module, WiFi module, Bluetooth module, GPS module, or NFC module may be included in one integrated chip (IC) or IC package.

The RF module may transmit / receive, for example, a communication signal (eg, an RF signal). The RF module may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), an antenna, or the like. According to another embodiment, at least one of the cellular module, the WiFi module, the Bluetooth module, the GPS module, or the NFC module may transmit and receive an RF signal through a separate RF module.

The communication unit 210 according to an embodiment of the present invention may be configured to include a short range communication module 211, and the short range communication module 211 may be used to determine whether the user terminal is indoors by performing a beacon transmission, for example. In addition, according to various embodiments of the present disclosure, the short range communication module 211 may support communication between a peripheral device of the user terminal and the user terminal.

In addition, the communication unit 210 according to an embodiment of the present invention may receive data and commands transmitted from the server 100 and transmit user information, schedule information, and photo information, which are information stored in the user terminal, to the server 100 as necessary. . In addition, the communication unit 210 may transmit state information (eg, location information, moving distance information, etc.) of the user terminal to the server 100.

The storage unit 220 according to an embodiment of the present invention may store the user information 221, the schedule information 222, the training record 223, and the photo information 224. The user information 221 may collectively refer to all information about a user's personality. For example, the user information 221 may include identification information of a user's peripheral device, information of a medical institution treated by the user, age of the user, disease information (dementia, or complications) of the user, a user's residence address, and a user's guardian's contact information. It may include.

The schedule information 222 is schedule information including a medication schedule of the user. The schedule information 222 may be retrieved and updated from the linked SNS including the instant messenger of the user. In addition, the schedule information 222 may be checked differently by importance for each schedule item by the user. Accordingly, the server 100 may provide a separate additional notification service for the important schedule of the user.

The training record 223 may include a history of the user executing the rehabilitation content 123 using the application. In addition, the training record 223 may include training results of the user (eg, quiz scores, change of grades, etc.).

The picture information 224 may include a picture taken by a user, picture information downloaded or input. The photo information 224 may be used to produce contents for training for recognition of nearby persons during rehabilitation training.

The input unit 230 may include a key input unit, a touch panel, a stylus pen, or the like. The input unit 230 may convert a user input into a signal and transmit the signal to the controller 270.

The input unit 230 according to an embodiment of the present invention may receive a user's touch input or key input for schedule input, user information input, and the like.

The display unit 240 may include a panel, a hologram device, or a projector. The panel may, for example, be implemented to be flexible, transparent, or wearable. The panel may be composed of a touch panel and one module. The hologram device may show a stereoscopic image in the air by using interference of light. The projector may display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device. According to an embodiment, the display unit 240 may further include a control circuit for controlling a panel, a hologram device, or a projector.

The display unit 240 according to an embodiment of the present invention may display the overall operation of the dementia patient management service application on the screen.

The camera unit 250 is, for example, a device capable of capturing still images and moving images. According to an embodiment, the camera unit 250 may include one or more image sensors (eg, a front sensor or a rear sensor), a lens, an image signal processor (ISP), or Flash (eg, LED or xenon lamp, etc.).

The camera unit 250 according to an embodiment of the present invention may be activated in a user authentication step for logging in. The camera unit 250 may photograph the user's face in front of the user terminal in the user authentication step for logging in. The captured user facial image (or image) may be provided to the server 100 to log in to the application. According to various embodiments of the present disclosure, the camera unit 250 may be activated to unlock the user terminal, and the activated camera unit 250 may provide the controller 270 with data about the captured front face image of the user.

In addition, the camera unit 250 may be used to determine the facial expression of the user to investigate the user's preference for each content. The camera unit 250 may capture a face of a user who is activated and watched while the multimedia content is executed. At this time, the captured user's face photographed image may be provided to the server 100 to determine an expression.

For example, the audio module 260 may bidirectionally convert a sound and an electric signal. At least some components of the audio module may be included in the input / output interface. The audio module may process sound information input or output through, for example, a speaker, a receiver, an earphone, a microphone, or the like.

The audio module 260 according to an embodiment of the present invention may receive various analog sound data including a voice of a user and convert it to a digital signal. In addition, the audio module 260 may output a notification sound in various notification situations such as a schedule related notification, a road guide notification, a danger notification, and the like. In addition, the audio module 260 may output audio according to content execution and audio data required for overall driving of the dementia patient management service application.

According to an embodiment of the present disclosure, the controller 270 may include a voice recognition processor 271, a context information providing unit 272, an emergency contact execution unit 273, a content execution unit 274, and a peripheral device controller 275.

The voice recognition processor 271 may receive and recognize a voice command of a user for executing the dementia patient management service application according to an exemplary embodiment of the present invention. According to various embodiments of the present disclosure, the application may be executed when named under a specific name. In addition, when the application does not perform server communication, the voice recognition processor 271 may process a voice command for a function that can be performed offline.

The contextual information providing unit 272 may collect state information such as GPS information of the user terminal under the control of the contextual information analysis unit 133 of the server 100 and transmit the state information to the server 100. When the user is roaming, the situation information providing unit 272 may display, on the screen, information for guiding a road from the location of the current user terminal to the user's house under the control of the server 100. In addition, the situation information provider 272 may receive a signal from the server 100 and display a push notification for schedule information, training reservation information, and the like.

The emergency contact execution unit 273 may perform a related operation as it receives a signal for an emergency situation transmitted from the server 100. When the emergency contact execution unit 273 receives a command related to a guardian call from a server, the emergency contact execution unit 273 may automatically connect to a predetermined guardian contact. As the emergency contact performing unit 273 receives a signal from the emergency determining unit 135 in an emergency, the emergency contact performing unit 273 may perform not only a telephone but also various types of communication functions including a text message. For example, the text message sent to the called medical institution may include information about the dementia level of the user, complication information, and information about an existing treatment medical institution (if the called medical institution and the existing treatment medical institution are different).

The content execution unit 274 may execute the training content and the multimedia content provided by the server 100. The attribute information for each content may be information included at the time of download or when stored in the server 100. Otherwise, the content execution unit 274 may extract attribute information (eg, thumbnails, playback time, major characters, etc.) for the content information to be reproduced during playback.

When it is determined that the server 100 is in an emergency situation, the peripheral controller 275 may control the peripheral device registered in the user terminal 200 under the control of the server. The peripheral device controller 275 may perform an operation of controlling ON / OFF operations and other functions of peripheral devices supporting the Internet of Things, including disconnecting the local area communication module 211.

4 is a flowchart illustrating a login process using a face of a user in a user terminal according to an exemplary embodiment of the present invention.

If the user terminal 200 performs an operation 410 of executing an application, the user terminal 200 may perform an operation 415 of capturing a user's face image. The photographed facial image may be transmitted to the server 100 supporting the corresponding application in operation 420 to determine whether to log in. Thereafter, the server 100 may perform operation 425 of analyzing the received image. The received image may be analyzed based on user identification information (feature value of face information of a member) previously held in the server 100, thereby allowing a user of the received image to log in based on the analysis information. The server 100 may perform user identification based on additional information on various other information (eg, identification information of the user terminal).

The server 100 may perform operation 430 to determine whether the identity is confirmed through the above process. When the server 100 determines that the identification is completed, the server 100 may perform operation 435 of providing login information (screen information of the application displayed in the login state) to the user terminal 200. Thereafter, the user terminal 200 may perform operation 440 of displaying a login screen.

In this manner, the present invention can log in the application only by face recognition without directly inputting ID and password according to the characteristics of the dementia patient.

5 is a flowchart illustrating a process of calculating preference information based on user facial information according to an exemplary embodiment of the present invention.

The user terminal 200 may perform operation 510 of selecting a multimedia content execution function according to a user's request in operation 505 in which the user terminal 200 is pre-logged in to the application and a screen of a login state is displayed. After operation 510, the user terminal 200 may perform operation 515 for requesting a content list from the server 100.

Accordingly, the server 100 may perform operation 520 of extracting a content list based on content preference information. Although not shown in the drawing, if the content preference information does not exist, the content list may be extracted and provided to the user terminal in a preset manner.

After operation 520, the server 100 may perform operation 525 of providing a content list based on content preference information. The content list may be arranged based on the content preference information, the content in the order preferred by the user.

Thereafter, when operation 530 in which specific content is selected by a user input is performed, the user terminal 200 executes the corresponding content, and simultaneously performs operation 535 of capturing the user's face. Thereafter, the user terminal 200 may perform operation 540 of providing a face-photographed image to the server 100.

When the server 100 receives the face photographed image, the server 100 may analyze the received face photographed image and perform operation 545 for determining a preference for the corresponding content. Thereafter, the server 100 updates the content preference information based on the preference for the specific content calculated in operation 545.

6 is a diagram illustrating an example of training content according to an embodiment of the present invention.

The server 100 may generate rehabilitation content such as 610 based on a photo uploaded by the user and other information (name, relationship). This may be simple learning content (for viewing and memorizing purposes), or may be content that provides a quiz after screening specific information (eg, a name). Each item 611 may be configured to include a name and a relationship between the picture and the user (a dementia patient).

7 to 8 are diagrams illustrating a signal flow system in an emergency according to an embodiment of the present invention.

7 illustrates an operation for calling a medical institution in an emergency situation. When an emergency occurs, the user terminal 200 may provide terminal status information to the server 100. Accordingly, the server 100 may determine the risk based on the state information of the terminal (which may include information transmitted from other peripheral devices besides the user terminal). If the server 100 determines that the risk is a dangerous and urgent situation (eg, when the location of the user terminal is detected around an outdoor road and the voice information of the user input to the terminal is a crying sound), the location of the user is determined. Call to the nearest medical institution 501 (or guardian terminal).

8 illustrates an operation of controlling a peripheral device of a dementia patient in an emergency situation. After the server 100 receives the state information (including information transmitted from the peripheral device) from the user terminal 200 existing in the indoor 300, the server 100 may determine the necessity of the function limitation based on the terminal state information. For example, when the user determines that the rate of decrease of the battery level of the terminal is extremely rapid or the battery level is not enough due to continuous viewing of the content, the server 100 establishes a communication connection with the user terminal and a peripheral device connected to the near field communication. It may be released or controlled to perform a power saving mode of the user terminal. Alternatively, if it is determined that the symptoms of dementia of the user are manifested based on the image information and the audio information of the user, the user terminal is requested to temporarily limit the use of a specific peripheral device (eg, a heating device). The function of the surrounding IoT apparatus may be controlled (for example, the state of the heating apparatus may be changed to OFF).

9 is a diagram illustrating an example of a notification service according to an embodiment of the present invention.

The dementia patient management service application according to an embodiment of the present invention may perform integrated management on SNS information, a scheduler, and medication information directly input by a user (or guardian) previously stored in the user terminal 200. For example, the application may provide a medication schedule notification 911 and other general schedule notification 912 together on the lock screen of the user terminal displayed as 910. According to various embodiments of the present disclosure, the application may perform a notification for suggesting the execution of the training content in a time zone in which the user mainly executes the training content. In this way, even if the user does not directly specify the behavior pattern of the user, or the frequency of the voice command can be recorded later to propose the action. For example, if the training content is executed at 3:00 pm for one week, the application provides a notification message such as 'Do you want to run the training content' on the screen of the user terminal before a predetermined time before 3:00 pm, If the user responds, the training content can be executed immediately.

Although the present invention has been described in detail with reference to the above examples, those skilled in the art can make modifications, changes, and variations to the examples without departing from the scope of the invention. In short, in order to achieve the intended effect of the present invention, it is not necessary to separately include all the functional blocks shown in the drawings or to follow all the orders shown in the drawings in the order shown; Note that it may fall within the scope.

100: server
110: communication unit
120: storage unit
130: control unit
131: speech recognition processing unit
132: user image analysis unit
133: situation information analysis unit
134: custom content information provider
135: emergency judgment
136: Rehabilitation Training Department
200: user terminal
210: communication unit of the user terminal
220: storage unit of the user terminal
230: input unit
240: display unit
250: camera unit
260: audio module
270: control unit of the user terminal
271: speech recognition processing unit
272: situation information provider
273: emergency contact unit
274: content execution unit
275: peripheral control unit

Claims (13)

In the server providing dementia patient management services,
Receives terminal status information and a facial image of the user from the user terminal, and transmits the login result performed based on the received facial image to the user terminal, the IoT function installed in the user terminal and the user's place of residence A communication unit for receiving status information from supporting peripheral devices;
Stores the result of processing the information received from the user terminal, an image analysis program for analyzing the user's face image, content identification information that is information about user identification and user preference by content, content genre, and content subject part; And
Based on the received facial image of the user, the user determines the user's log-in and determines an emotion, and determines at least one of an emergency situation or a necessity of function limitation based on the received terminal state information, and according to the determination result. Controls a specific function of the user terminal, manages a keyword of a voice command of a user requested over a predetermined criterion, classifies a specific function executed over a predetermined criterion as a user interest function, and the user interest function is equal to or greater than a reference frequency. Control unit for supporting to automatically execute the function of interest in the time of execution, including;
The control unit
A user image analyzer configured to analyze an emotional state of a user according to user identification and reproduction of a specific content from a face photographed image received from the user terminal;
When the list of multimedia contents held in the server is requested from the user terminal, it is determined whether the content preference information of the corresponding user exists, and if the content preference information of the user exists, it is displayed on the list based on the content preference information. Personalized content information providing unit for changing the content in the order preferred by the user;
A situation information analyzer configured to perform a road guidance from the user location to the user's residence if it is determined that the user roams outdoors based on the remaining battery information of the user terminal, the location of the terminal, and the moving speed of the terminal;
Determining an accident risk for each item of the user based on state information received from the user terminal and the peripheral devices supporting the IoT function, and if the determined accident risk is more than a predetermined criterion, calling a medical institution according to the item, An emergency determination unit configured to perform at least one of automatic telephone connection to a guardian terminal and control of a peripheral device supporting the IoT function; Including;
The content preference information is
Characterized in that the information is generated based on the emotional state of the user for each content, the number of voice inputs requesting specific content, and the cumulative number of playbacks for each content determined based on the user's facial image photographed while playing the multimedia content. Dementia Patient Management Services server. delete The method of claim 1,
The user image analysis unit
When the face photographed image of the user is received in the log-in step, the received face photographed image is compared with the pre-stored user identification information to determine whether there is a match.
After login, if a face photographed image of a user photographed in a state where multimedia content is played from the user terminal is received, the dementia patient characterized by analyzing the facial expression change of the user and a corresponding emotional state of the user from the face photographed image of the user Management service server. The method of claim 1,
The user image analysis unit
In order to perform facial expression recognition from a facial image of a user, a composite product neural network (CNN), CK + dataset classified into four data groups, Images, Emotion lebels, Facial Action Coding System laves, and Landmarks,
A preprocessing operation and a modeling operation of input data for analyzing a face photographed image of a user are performed.
Images data with Emotion label are inputted, the first frame before the expression is changed with a negative expression, the last three frames where the expression is clearly expressed are divided with the expression with the Emotion label, and the RGB data is grayed out. And converting the data into data, fixing the human face to a predetermined size, and inverting image data left and right to double the amount of data, and then randomly mixing the dementia patient management service server. The method of claim 4, wherein
The user image analysis unit
When performing the preprocessing operation, the data is normalized between 0 and 1, and selective random extraction is performed to randomly extract a 224 * 224 size image from a 255 * 255 size image in order to increase the amount of insufficient data. Performing a technique, and then spreading the data in a one-dimensional array and inputting the data into a model for designing the dementia patient management service. The method of claim 4
The user image analysis unit
The modeling operation is performed to input the data in which the preprocessing operation is completed to the model.
The modeling operation
A first process of extracting 256 features through five convolution operations based on the AlexNet model, and a second process of classifying data through a full connect operation using the extracted features Patient care services server. The method of claim 6,
The user image analysis unit
In performing the modeling operation, if a 224 * 224 sized image, which is unfolded in a one-dimensional array according to a preprocessing operation, is input as data, 96 kernels having a size of 11 * 11 and a stride of 4 are combined and computed. When the 55 * 55 * 96 image is output, it repeats the multiplication operation with 256 kernels with 5 * 5 size and 1 stride. A dementia patient care server comprising 256 features of size 6 * 6 past the pooling layer. The method of claim 4, wherein
The user image analysis unit
Facial expression recognition is performed using a model trained directly using CK + dataset, a model applied with AUS (Action Unit Selective) technique, and a pre-trained model using ImageNet data.
The model trained directly using the CK + dataset is trained by inverting and segmenting the input data to increase the amount of data,
The model applying the AUS technique selects 256 feature data extracted using a pre-trained model, rearranges the data to 240 features, and uses the rearranged data. Connect to the last Fully-connected layer and reconfigure
The model pre-trained through the ImageNet data uses the concept of transfer learning, and loads the pre-trained model, applies weight and bias as it is, and then changes the setting of input data and output label. Dementia patient management service server, characterized in that trained as. delete The method of claim 1,
The communication unit receives the status information, user information and schedule information of the user terminal from the user terminal,
The situation information analysis unit
If it is determined that the user is roaming based on the received status information, and the user is determined to be roaming, in real time, the user guides the route from the current location of the user terminal to the user's residence address included in the user information. Determining a schedule for each user time based on the scheduled information, the dementia patient management service server, characterized in that for controlling to automatically execute a notification according to the schedule in the user terminal. delete The method of claim 1,
The control unit
Including rehabilitation training department,
The rehabilitation training unit
Dementia patient management service server, characterized in that to perform the user's cognitive training based on the picture of the surrounding people, to generate a quiz based on the picture of the surrounding people to provide the user with memory learning. delete

Images