KR102118585B1 - Smart Mirror Chatbot System and Method for Senior Care - Google Patents

Abstract

The present invention relates to a smart mirror chatbot system for senior care and a method thereof. More specifically, the present invention relates to a smart mirror chatbot system for senior care and a method thereof. The smart mirror chatbot system for senior care performs a conversation with a user through a character displayed on a smart mirror, analyzes a mild cognitive impairment degree and emotion of the user based on contents of the performed conversation and image information of the user collected through a camera of the smart mirror, and transmits state information of the user in accordance with the analyzed emotion to a guardian.

Description

Smart Mirror Chatbot System and Method for Senior Care}

The present invention relates to a smart mirror chatbot system and method for senior care, and more specifically, to perform a conversation with a user through a character displayed on the smart mirror, the content of the conversation performed, and collected through the camera of the smart mirror The present invention relates to a smart mirror chatbot system and method for senior care that can analyze a user's degree of mild cognitive impairment and emotion based on the user's image information and transmit the user's status information according to the analyzed emotion to a guardian.

Senior healthcare technology is being actively researched in line with the increase in the senior population and their interest in health according to the aging age. Currently, the population of people over 65 years old in Korea is more than 6.6 million, exceeding 12% of the total population. Aging society is rapidly spreading not only in Korea but also in developed countries such as France, Sweden, the United States, Germany, and Japan, and in the 21st century, all countries must respond to the social phenomenon of aging.

Meanwhile, research on smart mirrors has been actively conducted recently, and people's interest in smart mirrors is also increasing. Smart mirror is a mirror that reflects the shape of an object by reflecting light, as well as performing a function that can connect to the Internet and output various information such as weather, calendar, and major news. It is a mirror. However, the smart mirrors used so far are only unilaterally providing general information such as the weather of the day and main news, and are not able to provide customized information suitable for the user's emotions in consideration of the user's emotions. .

In addition, as the aging society accelerates, the need for management of the elderly is greatly increased, and especially in the case of an elderly person living alone, when an emergency occurs, it can be in great danger if no special measures can be taken. As is the big problem. In order to solve this problem, a smart mirror that can be used more easily than a general smart device by an elderly person provides a conversation considering the user's emotion, collects the user's video or voice, and analyzes and manages the user's condition. There is a need for an existing technology, but there is no such conventional technology.

The present invention conducts a conversation with a user through a character displayed on a smart mirror, analyzes the user's degree of mild cognitive impairment and emotion based on the content of the conversation performed and the user's video information collected through the camera of the smart mirror, It is an object of the present invention to provide a smart mirror chatbot system and method for senior care, which can transmit the user's status information according to the analyzed emotion to the guardian.

In order to solve the above problems, as a smart mirror chatbot system for senior care implemented by a service server that can be accessed by a guardian's terminal and a smart mirror device, the smart mirror device is made of a transparent material that illuminates the user's appearance. mirror; A display unit that displays an image or information and is installed on the rear side of the mirror; A speaker unit for outputting sound; A microphone unit for collecting voice information of the user; A wireless communication unit wirelessly communicating with the service server and the guardian's terminal; Camera unit for collecting the user's video information; And a control unit controlling an operation of the smart mirror device through communication with the service server, wherein the control unit identifies the user based on voice information and video information of a user who wants to operate the smart mirror device. A user identification unit for operating the smart mirror device; A chatbot interface unit for performing a conversation with a user through a character displayed on the display unit; And a call mode providing unit for performing a video call through communication with a terminal of the guardian, wherein the service server is a mild cognitive impairment that derives mild cognitive impairment information based on the user's voice information and the video information. Information elicitation unit; And an emotion information deriving unit that derives emotion information based on the image information of the user. And a user status notification unit that transmits the user's status information derived based on the mild cognitive impairment information and the emotion information to a guardian's terminal.

In one embodiment of the present invention, the user identification unit, face identification unit; And a voice identification unit, wherein the face identification unit extracts a user's face image based on the image information of the user; Determining a user's face by comparing the pre-stored user's face image with the face image extracted from the image information; And if it is determined as the user's face, operating the smart mirror device; and the voice identification unit determines the user's voice by comparing the received user's voice information with the pre-stored user's voice information. step; And when it is determined by the user's voice, operating the smart mirror device.

In one embodiment of the present invention, the chatbot interfacing unit comprises a chatbot that displays smart mirror providing information including one or more of weather, schedule, and news requested by the user in the form of a character according to the user's voice information for the information request. Provided through voice, and displays the corresponding information on the display screen of the display unit, and the character can be displayed in an area of the display unit that does not overlap the area of the mirror on which the user's appearance is projected based on the image information of the user. have.

In one embodiment of the present invention, the chatbot interface unit, performing one or more questions related to the diagnosis of mild cognitive impairment through a character displayed on the display unit; Receiving voice information of a user in response to the question through the microphone unit; Receiving feedback voice information corresponding to the corresponding response from the service server based on the user's voice information for the response; And providing the feedback voice information through the character; and the character may be displayed on an area of the display unit that does not overlap the area of the mirror on which the user's appearance is projected based on the image information of the user. have.

In one embodiment of the present invention, the call mode providing unit transmits the user's voice information and video information to the guardian's terminal, and transmits the guardian voice information and the guardian's video information received from the guardian's terminal to the smart mirror device. To make a video call.

In an embodiment of the present invention, the user state diagnosis unit further includes a feature information extractor; the feature information extractor derives voice feature information according to the amplitude, length and pattern of the user's voice information, A face image may be extracted based on the user's image information, and face feature information may be derived based on the extracted face image.

In one embodiment of the present invention, the hardness cognitive impairment information extracting unit is a first artificial person analyzing the degree of hardness cognitive impairment based on voice characteristic information including amplitude, length and pattern derived based on the user's voice information. Deriving a first analysis result by the neural network; Deriving a second analysis result by a second artificial neural network analyzing the degree of mild cognitive impairment based on the facial feature information derived from the image information of the user; And deriving longitudinal cognitive impairment information based on the first analysis result and the second analysis result.

In one embodiment of the present invention, the emotion information deriving unit may derive the emotion information by a third artificial neural network analyzing the user's emotion based on the facial feature information derived from the image information of the user.

In one embodiment of the present invention, the user status notification unit derives the user's status information according to a preset criterion based on the user's longitude cognitive impairment information and the emotion information, and the derived user's status information is preset. If the criteria are met, the user's status information may be transmitted to the guardian's terminal.

In order to solve the above problems, in one embodiment of the present invention, as a method of providing a smart mirror chatbot service for senior care performed by a service server to which a terminal of a guardian and a smart mirror device can access, the smart The mirror device includes a mirror made of a transparent material that illuminates a user's appearance; A display unit that displays an image or information and is installed on the rear side of the mirror; A speaker unit for outputting sound; A microphone unit for collecting voice information of the user; A wireless communication unit wirelessly communicating with the service server and the guardian's terminal; Camera unit for collecting the user's video information; And a control unit for controlling the operation of the smart mirror device through communication with the service server, wherein the service server comprises: a user identification unit; Chatbot interface unit; A call mode providing unit; Mild Cognitive Disability Information Derivation Unit; And an emotional information deriving unit; And a user status notifying unit, comprising: a user identification step of operating the smart mirror device by identifying the user based on voice information and video information of a user to operate the smart mirror device by the user identification unit; A chatbot interfacing step of performing a conversation with a user through a character displayed on the display unit by the chatbot interfacing unit; A call mode providing step of providing a voice call or a video call through communication with a terminal of the guardian by the call mode providing unit; A long-term cognitive disability information deriving step of deriving long-term cognitive disability information based on the feature information on the user's voice information and the feature information on the image information; And an emotion information deriving step of deriving emotion information based on feature information on the user's image information by the emotion information deriving unit. And a user status notification step of transmitting, by the user status notification unit, the user's status information derived based on the mild cognitive impairment information and the emotion information to a guardian's terminal. How to do it.

According to an embodiment of the present invention, it is possible to provide an interactive partner to a user through a character displayed on a smart mirror device, thereby exerting an effect of satisfying a lack of emotion.

According to an embodiment of the present invention, information is provided interactively through a smart mirror device, and a result of analyzing a user's state based on the collected voice and video information is delivered to a guardian, and the user's state It can exert the effect of taking effective measures by the guardian.

According to an embodiment of the present invention, by generating a situation such as having a real conversation with a character, the user's voice information is analyzed from the user's facial expression, voice pattern, keywords of words used, and the current user's emotion and health status It can exert an effect that can be analyzed for.

According to an embodiment of the present invention, it is possible to exert the effect of recognizing the emotion contained in the user's image information using an artificial neural network model learned based on the user's image information.

According to an embodiment of the present invention, it is possible to provide convenience and conversational counterparts to the elderly living alone to measure and analyze the feeling of satisfaction and health in real time and deliver it to the guardian.

According to an embodiment of the present invention, the derived user's status information is determined according to a predetermined criterion, and the result of analyzing the user's status is transmitted to the guardian, even if the guardian of the user of the senior class who is living alone does not live together By receiving the user's status, it is possible to exert the effect of taking appropriate measures according to the user's status.

1 schematically illustrates a form of a smart mirror chatbot system for senior care according to an embodiment of the present invention.
2 schematically shows a configuration of a smart mirror device according to an embodiment of the present invention.
Figure 3 schematically shows the internal configuration of the control unit according to an embodiment of the present invention.
4 schematically shows an internal configuration of a service server according to an embodiment of the present invention.
5 schematically illustrates a screen of a smart mirror device provided by a user identification unit according to an embodiment of the present invention.
6 schematically illustrates a screen of a smart mirror device provided by a chatbot interface unit according to an embodiment of the present invention.
7 schematically illustrates a step of performing a chatbot interface unit according to an embodiment of the present invention.
8 schematically shows a screen of a smart mirror device provided by a chatbot interface unit according to an embodiment of the present invention.
9 schematically illustrates a screen of a smart mirror device provided by a call mode providing unit according to an embodiment of the present invention.
10 schematically illustrates the operation of the feature information extracting unit according to an embodiment of the present invention.
11 schematically illustrates the operation of the feature information extractor according to an embodiment of the present invention.
12 schematically illustrates the operation of the longitudinal cognitive impairment information extracting unit according to an embodiment of the present invention.
13 schematically illustrates the operation of the emotion information extracting unit according to an embodiment of the present invention.
14 exemplarily illustrates a computing device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . Also, when a part “includes” a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise specified.

Further, terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and/or includes a combination of a plurality of related described items or any one of a plurality of related described items.

In the present specification, the term “unit” includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be realized by using two or more hardware, and two or more units may be realized by one hardware. Meanwhile,'~unit' is not limited to software or hardware, and'~unit' may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example,'~ unit' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, and procedures. , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables. The functions provided within components and'~units' may be combined into a smaller number of components and'~units' or further separated into additional components and'~units'. In addition, the components and'~ unit' may be implemented to play one or more CPUs in the device or secure multimedia card.

The "user terminal" mentioned below may be implemented as a computer or a portable terminal capable of accessing a server or another terminal through a network. Here, the computer includes, for example, a laptop equipped with a web browser (WEB Browser), a desktop (desktop), a laptop (laptop), and the like, and the portable terminal is, for example, a wireless communication device that guarantees portability and mobility. , PCS(Personal Communication System), GSM(Global System for Mobile communications), PDC(Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code It may include all kinds of handheld-based wireless communication devices such as Division Multiple Access (2000), W-Code Division Multiple Access (W-CDMA), and Wireless Broadband Internet (Wibro) terminals. In addition, the "network" is a wired network or a mobile radio communication network or satellite, such as a local area network (LAN), a wide area network (WAN), or a value added network (VAN). It can be implemented with any kind of wireless network, such as a communication network.

1 schematically illustrates a form of a smart mirror chatbot system for senior care according to an embodiment of the present invention.

The terminal of the guardian of FIG. 1 may correspond to the aforementioned user terminal, and the service server 2000 corresponds to a computing device including one or more processors and one or more memories.

The smart mirror 100 device 1000 may communicate with the service server 2000, and the control unit 500 of the smart mirror 100 device 1000 may communicate with the service server 2000 through the smart mirror ( 100) It is possible to control the operation of the device 1000. The control unit 500 corresponds to a computing device including one or more processors and one or more memories.

The smart mirror device 1000 includes a mirror 100 that illuminates the user's appearance, and the mirror 100 is made of a transparent material as well as illuminating the user's appearance, or an object on the back through the mirror 100 or The environment can be seen visually through the mirror 100. In the smart mirror device 1000 of the present invention, a display unit 600 for displaying an image or information is installed on the rear surface of the mirror 100 as described above. In addition, the smart mirror device 1000, a speaker unit 400 for outputting sound; A microphone unit 300 for collecting voice information of the user; A wireless communication unit wirelessly communicating with the service server 2000 and the guardian's terminal; A camera unit 200 for collecting user's image information may be further included.

The smart mirror device 1000 may identify the user, operate the smart mirror device 1000, and perform a conversation with the user through a character displayed on the display unit 600. In addition, it is possible to execute a video call through communication with the guardian's terminal.

Hereinafter, the configuration of the smart mirror device 1000 will be described in more detail.

2 schematically shows a configuration of a smart mirror device 1000 according to an embodiment of the present invention.

Specifically, Figure 2 (a) shows a form viewed from the front of the smart mirror device 1000 of the present invention. The smart mirror device 1000 may include a mirror 100 of a transparent material, illuminate a user and surrounding environment, and a camera unit 200 that collects image information of a user at the top of the smart mirror device 1000; And a microphone unit 300 for collecting voice information of the user.

Figure 2 (b) shows a form viewed from the side of the smart mirror device 1000 of the present invention. As shown in (b) of FIG. 2, the front side is provided with a mirror 100 of a transmissive material that illuminates the user's figure, and at the top of the mirror 100 is a camera unit 200 that collects user image information. It is mounted. A display unit 600 for displaying information or images is installed on the rear surface of the mirror 100. In addition, a speaker unit 400 for outputting sound and a control unit 500 for controlling the operation of the smart mirror device 1000 are provided on the rear surface of the display unit 600. The speaker unit 400 may be provided in another area of the smart mirror device 1000 even if it is not necessarily the rear surface of the display unit 600.

Hereinafter, the configuration of the control unit 500 of the smart mirror device 1000 of the present invention will be described in more detail.

3 schematically shows an internal configuration of the control unit 500 according to an embodiment of the present invention.

As illustrated in FIG. 3, the control unit 500 of the smart mirror device 1000 identifies the user based on voice information and video information of a user who wants to operate the smart mirror device 1000 to identify the smart A user identification unit 510 for operating the mirror device 1000; A chatbot interface unit 520 that performs a conversation with a user through a character displayed on the display unit 600; And a call mode providing unit 530 for performing a video call through communication with the terminal of the guardian.

Specifically, the user identification unit 510 includes a face identification unit 511 and a voice identification unit 512, and the face identification unit 511 is a camera unit 200 of the smart mirror device 1000. The user's face may be determined based on the collected user's image information, and when it is determined as the user's face, the smart mirror device 1000 may be operated. The voice identification unit 512 determines the user's voice based on the user's voice information collected by the microphone unit 300 of the smart mirror device 1000, and the collected voice information is determined as the user's voice In this case, the smart mirror device 1000 may be operated.

The chatbot interfacing unit 520 performs a conversation with a user through a character displayed on the display unit 600 of the smart mirror device 1000. Receives the user's voice information for the information request, and according to the received user's voice information, the smart mirror providing information including one or more of the weather, schedule, and news requested by the user is voiced through a chatbot displayed in the form of a character. Provided, and displays the information corresponding to the display screen of the display unit 600. In addition, the chatbot interfacing unit 520 performs one or more questions related to the diagnosis of mild cognitive impairment through the character in order to diagnose the degree of mild cognitive impairment of the user, and the microphone unit 300 of the smart mirror device 1000. Through this, feedback voice information for the response may be provided according to the user's voice information for the response corresponding to the question.

The call mode providing unit 530 transmits the user's voice information and video information to the guardian's terminal, and transmits the guardian voice information and the guardian video information received from the guardian's terminal to the smart mirror device 1000 to make a video call. Run Through the operation of the call mode providing unit 530, the user can make a video call with the guardian through the smart mirror device 1000.

The control unit 500 illustrated in FIG. 3 may further include elements other than the illustrated components, but for convenience, only components related to the smart mirror chatbot system for senior care according to embodiments of the present invention are displayed.

4 schematically shows the internal configuration of a service server 2000 according to an embodiment of the present invention.

As shown in FIG. 4, the service server 2000 includes: a longitudinal cognitive impairment information deriving unit 2120 for deriving longitudinal cognitive impairment information based on the user's voice information and the video information; And an emotion information deriving unit 2130 for deriving emotion information based on the image information of the user; And a user status notification unit 2200 that transmits the user's status information derived based on the mild cognitive impairment information and the emotion information to a guardian's terminal. And a smart mirror device setting unit 2300 for setting a smart mirror device based on the hardness cognitive impairment information and the emotion information.

Specifically, the user status diagnosis unit 2100 may derive the user's status information by deriving the mild cognitive impairment information and emotion information based on the user's voice information and video information. The user state diagnosis unit 2100 includes a feature information extraction unit 2110, a long-term cognitive disability information extraction unit 2120, and an emotion information extraction unit 2130.

The feature information deriving unit 2110, face feature information and voice based on the user's video information collected from the camera unit 200 of the smart mirror device 1000 and the user's voice information collected from the microphone unit 300 Feature information can be derived. Specifically, voice characteristic information is derived according to the amplitude, length, and pattern of the voice information, face image is extracted based on image information, and face feature information is derived based on the extracted face image.

In one embodiment of the present invention, the feature information deriving unit 2110 basically receives the user's image information and voice information, and thus face feature information and voice feature information through a model learned through machine learning techniques such as deep learning. Can be derived.

The module for deriving the feature information included in the feature information deriving unit 2110 separately extracts the face image included in the image information and recognizes the hardness using artificial neural network technology including temporal concepts such as RNN, LSTM, and GRU. You can analyze the degree of disability and derive a mild cognitive disability score. In addition, in addition to this, the feature information extracting unit 2110 may analyze an image according to a preset rule rather than machine learning.

The mild cognitive disability information deriving unit 2120 derives mild cognitive disability information based on the user's voice information and video information. Specifically, the longitudinal cognitive impairment information extraction unit 2120 includes a first artificial neural network 2121, a second artificial neural network 2122, and a comprehensive diagnosis module 2123. The first artificial neural network 2121 derives a first analysis result analyzing the degree of mild cognitive impairment based on the voice feature information, and the second artificial neural network 2122 determines the degree of hardness cognitive impairment based on the facial feature information. The second analysis result is analyzed. Thereafter, the comprehensive diagnosis module 2123 may derive longitudinal cognitive impairment information based on the first analysis result and the second analysis result. Preferably, the mild cognitive impairment information may be derived in the form of a level or score according to the user's degree of mild cognitive impairment.

The emotion information deriving unit 2130 derives emotion information based on the user's image information. Specifically, the emotion information deriving unit 2130 includes a third artificial neural network 2131, and the third artificial neural network 2131 analyzes emotion information based on face feature information derived from the user's image information. . Based on the facial feature information derived from the user's image information, emotion information is derived by a third artificial neural network 2131 that analyzes the user's emotion. Preferably, the emotion information may be data in vector form according to emotions such as user's joy, surprise, sadness, anger, fear, interest, stress, fear, boredom, comfort, discomfort, anxiety, and neutrality. .

The user status notification unit 2200 transmits the user's status information derived based on the hardness cognitive impairment information and the emotion information to the terminal of the guardian. Specifically, the user status notification unit 2200 derives the user's status information according to a preset criterion based on the mild cognitive disability information and emotion information, and when the derived user's status information corresponds to the preset criterion, The user's status information is transmitted to the guardian's terminal.

The smart mirror device setting unit 2300 sets the smart mirror device 1000 based on the hardness cognitive impairment information and the emotion information. As mentioned in the description of FIG. 3, the smart mirror device 1000 communicates with a user through a character displayed on the display unit 600 of the smart mirror device 1000, and the smart mirror device setting unit 2300 The feedback voice information for the response may be derived according to the voice information of the user received from the smart mirror device 1000, and the feedback voice information derived to the smart mirror device 1000 may be transmitted. In addition, the smart mirror device setting unit 2300 sets and displays the chatbot interfacing unit 520 of the smart mirror device 1000 based on the hardness cognitive disorder information and the emotion information derived from the user state diagnosis unit 2100. It is also possible to set the voice of the character displayed on the unit 600 and the speed of the voice to be different.

On the other hand, the DB (2400) of the service server 2000 is transmitted to the control unit 500 of the smart mirror device 1000, one or more question information related to the diagnosis of mild cognitive impairment, collected by the smart mirror device 1000 User's voice information and video information, user's long-term cognitive impairment information derived by user status diagnosis unit 2100, diagnostic information including emotion information and status information, user's guardian's terminal information, address, email and Guardian information including a phone number may be stored.

The service server 2000 illustrated in FIG. 4 may further include elements other than the illustrated components, but for convenience, only components related to the smart mirror chatbot system for senior care according to embodiments of the present invention are displayed. .

5 schematically illustrates a screen of the smart mirror device 1000 provided by the user identification unit 510 according to an embodiment of the present invention.

The control unit 500 of the smart mirror device 1000 includes a user identification unit 510, and the user identification unit 510 accesses the smart mirror device 1000 based on the user's video information and audio information. Alternatively, a user who wants to operate the smart mirror device 1000 may be identified.

Preferably, the user identification unit 510 includes a face identification unit 511 and a voice identification unit 512, and the face identification unit 511 is based on the user's video information. Extracting an image (S100); Determining a user's face by comparing the user's previously stored face image and the face image extracted from the image information (S110); And if it is determined as the user's face, operating the smart mirror device 1000 (S120); performing, the voice identification unit 512, the received voice information of the user and the pre-stored user's voice Comparing the information to determine the user's voice (S200); And when it is determined by the user's voice, operating the smart mirror device 1000 (S210).

In step S100, the face identification unit 511 extracts the user's face image based on the user's image information.

In step S110, the user's face is determined by comparing the face image extracted in step S110 and the previously stored face image. The smart mirror device 1000 may pre-store the user's face image to determine the user and set it for user recognition. The face identification unit 511 compares the previously stored face image of the user and the face image extracted from the image information, and if it is a face image of the same person, it is determined as the user's face, and when it is determined that it is not the same face image It is determined that the person accessing the smart mirror device 1000 is not a preset user.

In step S120, when it is determined as the user's face in step S110, the smart mirror device 1000 is operated. The smart mirror device 1000 in which the user has been identified and operated is displayed with smart mirror providing information including the corresponding date, time, and weather, as shown in FIG. 5(b), and the smart mirror device 1000 is operated. Can be operated.

Preferably, the user identification unit 510 further includes a proximity sensor, when a user approaches the smart mirror device 1000, transmits a user access signal to the camera unit 200 to operate the camera unit 200 In addition, it may be possible to collect the user's video information.

Meanwhile, in step S200, the voice identification unit 512 determines the user's voice by comparing the received user's voice information with the pre-stored user's voice information. In order to discriminate the user, the smart mirror device 1000 may pre-store the user's voice information and set it for user recognition. The voice identification unit 512 compares the previously stored user's voice information with the received user's voice information, and if each voice information is the same person's voice information, it is determined as the user's voice and is not the same person's voice information. In the case, it is determined that the person who intends to operate the smart mirror device 1000 is not a set user.

In step S210, when it is determined by the user's voice in step S200, the smart mirror device 1000 is operated. The smart mirror device 1000 in which the user has been identified and operated may be operated in the form of providing smart mirror providing information desired by the user according to the user's voice, executing a video call, or talking with a character.

5A shows a state in which the smart mirror device 1000 does not operate. The smart mirror device 1000 in a form not accessed by the user can function only as a mirror that illuminates a person or an object in front of the mirror like a general mirror.

Meanwhile, FIG. 5B schematically illustrates a state in which the smart mirror device 1000 is operated by a user approaching. When the user approaches the smart mirror device 1000, the user's appearance is reflected, and the camera unit 200 of the smart mirror device 1000 collects the user's video information. In addition, the mirror 100 included in the smart mirror device 1000 of the present invention is made of a transparent material, and the display screen of the display 600 installed on the rear surface of the mirror 100 can be transmitted. FIG. 5B shows that information such as the date, time, and weather is displayed on the display screen of the display unit 600. In another embodiment of the present invention, the smart mirror device 1000 may include a proximity sensor, and when the proximity sensor recognizes a user, the smart mirror device 1000 may be operated. Information such as the date, time, and weather of the day is displayed on the display screen of the display unit 600, and the user operating the smart mirror device 1000 can check the information provided by the smart mirror device 1000.

In this way, the user identification unit 510 identifies the user and operates the smart mirror device 1000 to prevent unnecessary waste of power when the smart mirror device 1000 is not used. It is possible to exert the effect of allowing only the user to use the smart mirror device 1000.

6 schematically shows a screen of the smart mirror device 1000 provided by the chatbot interface unit 520 according to an embodiment of the present invention.

The control unit 500 of the smart mirror device 1000 includes a chatbot interface unit 520, and the chatbot interface unit 520 is a user through a character displayed on the display unit 600 of the smart mirror device 1000. And conduct conversations and provide information.

Preferably, the chatbot interfacing unit 520 is a chatbot that displays smart mirror provision information including one or more of weather, schedule, and news requested by the user in the form of a character according to the user's voice information for the information request. Provided through voice, and displays the corresponding information on the display screen of the display unit 600, and the character does not overlap the area of the mirror 100 on which the user's appearance is reflected based on the user's image information. It is displayed in the area of the display unit 600.

FIG. 6 schematically illustrates a state in which the smart mirror device 1000 receives voice information for a request for information about a user's weather from a user and provides information about the weather requested by the user. According to what is shown in FIG. 6, the user “tell me the weather today!” That is, the smart mirror device 1000 requests information, and the smart mirror device 1000 receives the voice information of the user and displays the smart mirror providing information on the weather requested by the user in the display unit of the smart mirror device 1000. It is displayed on the display screen of (600). In addition, the voice of the weather is provided through a chatbot displayed in the form of a character such as “Today's weather is clear and 3° lower than yesterday~~.” The present invention is a smart mirror chatbot system for the care of seniors. As an invention relating to, the user of the smart mirror device 1000 can mainly dominate the senior layer, relatively, seniors who are difficult to operate a smartphone and a computer can obtain themselves through voice, as shown in FIG. The desired information may be requested from the smart mirror device 1000, and the smart mirror device 1000 may provide information requested by the user through the operation of the chatbot interface unit 520.

On the other hand, the character capable of providing the smart mirror providing information by voice is displayed on the area of the display unit 600 that does not overlap the area of the mirror 100 on which the user's appearance is reflected based on the image information of the user. . The smart mirror device 1000 including the mirror 100 of the transparent material and the display unit 600 installed on the rear surface of the mirror 100 illuminates a user accessing the smart mirror device 1000, and the display unit 600 ) To display the character. The smart mirror device 1000 of the present invention displays a character in an area of the display unit 600 that does not overlap with the user's appearance reflected on the mirror 100. According to FIG. 6, when the user views the area L1 projected on the mirror 100 and the character display area L2 of the display unit 600, the user's appearance and characters are displayed so as not to overlap. do. As such, the chatbot interfacing unit 520 displays the mirrored user's appearance and the displayed character so as not to overlap. Specifically, the chatbot interfacing unit 520, based on the image information, deriving an area in which the user is projected on the mirror 100 (S300); Deriving a character display area of the display unit 600 that does not overlap the area where the user is illuminated (S310); And displaying the character in the character display area (S320).

In step S300, based on the image information, the user derives an area L1 that is illuminated on the mirror 100.

In step S310, the display unit 600 installed on the rear surface of the mirror 100 and the display unit 600 in which the user is illuminated do not overlap based on the region L1 where the user derived in step S300 is projected on the mirror 100. ), the character display area L2 is derived.

In step S320, the character is displayed in the character display area L2 derived in step S310.

In this way, by displaying the character and providing the information requested by the user, the user of the senior class who can live alone and feel loneliness can actually exert the effect of feeling emotional satisfaction by receiving the feeling of actually interacting with the character. have.

Figure 7 schematically shows the execution steps of the chatbot interface unit 520 according to an embodiment of the present invention, Figure 8 is a smart mirror device provided by the chatbot interface unit 520 according to an embodiment of the present invention The screen of 1000 is schematically illustrated.

As illustrated in FIG. 7, the chatbot interfacing unit 520 includes: performing one or more questions related to diagnosis of mild cognitive impairment through a character displayed on the display unit 600 (S1000); Receiving the user's voice information on a response corresponding to the question through the microphone 300 (S1100); Receiving feedback voice information corresponding to the corresponding response from the service server 2000 based on the user's voice information for the response (S1200); And providing the feedback voice information through the character (S1300).

Specifically, in step S1000, the chatbot interfacing unit 520 performs one or more questions related to the diagnosis of mild cognitive impairment through characters displayed on the display unit 600. The question above is, for example, as shown in FIG. 8, “How was your day today?” Or “What was the result of ooo you did yesterday?” As such, it is a question that asks about daily life, but may be a question that can diagnose a state of a user's memory and a state of a current emotion according to a user's response to the corresponding question through the corresponding question. The question information may be stored in the smart mirror device 1000 or may be updated by receiving it from the service server 2000.

In step S1100, the user's voice information is received in response to the question performed in step S1000. The received voice information is transmitted to the service server 2000.

In step S1200, feedback voice information corresponding to the corresponding response is received from the service server 2000 based on the user's voice information for the response. Preferably, the service server 2000 is derived by the first artificial neural network 2121 that determines the user's emotion according to the amplitude, length, and pattern of the keyword or the user's voice information included in the user's response to the question. Feedback voice information to be provided to the user may be determined according to a predetermined criterion based on the user's emotion information. The feedback voice information may be included and stored in the question information of the service server 2000, and the feedback for the corresponding response among the feedback voice information stored according to a preset criterion based on the voice information of the user who answered the question Audio information may be determined.

In step S1300, the chatbot interfacing unit 520 provides feedback voice information received from the service server 2000 in step S1200 through the character. The character is displayed on the display 600 by adjusting the expression and mouth shape as if they are actually speaking according to the feedback voice information, and outputs feedback voice information received from the service server 2000 through the speaker.

On the other hand, the service server 2000 of the present invention derives the longitudinal cognitive impairment information and emotion information based on the user's voice information and video information, and transmits the derived mild cognitive impairment information and emotion information to the smart mirror device 1000. can do. Accordingly, preferably, the chatbot interfacing unit 520 receives the user's longitude and cognitive disability information and the emotion information from a service server, and based on the received mild cognitive disability information and the emotion information, a preset criterion According to this, the voice of the character and the speed of the voice providing the feedback voice information may be converted.

In one embodiment of the present invention, when the user's emotion information is determined to be sad, the chatbot interface unit 520 receiving the emotion information may convert to a more sweet and comfortable voice mode. The voice mode may have a plurality of voice modes according to emotion information, and voice may be converted according to a preset criterion. In addition, when it is determined that the user does not understand the feedback voice information well based on the mild cognitive impairment information and the emotion information, the voice speed of the provided voice may be converted to provide the feedback voice information at a slightly slower rate. In addition, the character providing the feedback voice information may be changed according to a predetermined criterion based on the received mild cognitive impairment information and the emotion information.

According to FIG. 8, a character is displayed on the display unit 600 of the smart mirror device 1000 by the chatbot interface unit 520 of the smart mirror device 1000, and the user is asked a question through the character. Perform. The character asked, "How was your day today?", and the user replied to the question, "There was a day today, so I'm not feeling well." As described above, the chatbot interfacing unit 520 receives the user's response to the question and transmits it to the service server 2000, and receives feedback voice information from the service server 2000 to provide feedback voice information to the user. Can be. Thereafter, the voice information transmitted to the service server 2000 may then derive the user's status information by the operation of the service server 2000 described later.

In this way, the user can feel the deficiency of emotion, such as loneliness, by receiving the feeling of actually communicating with the character displayed on the smart mirror device 1000, and the smart mirror device 1000 collects from the user By transmitting the user's voice information to the service server 2000, it is possible to exert the effect of analyzing the user's mild cognitive impairment information and emotion information.

9 schematically illustrates a screen of the smart mirror device 1000 provided by the call mode providing unit 530 according to an embodiment of the present invention.

According to an embodiment of the present invention, the control unit 500 of the smart mirror device 1000 includes a call mode providing unit 530, and the call mode providing unit 530 provides user voice information and video information. It transmits to the terminal of the guardian and transmits the guardian voice information and the guardian image information received from the terminal of the guardian to the smart mirror device 1000 to perform a video call. In addition, the video call can be performed in the same manner as the other terminal as well as the terminal of the guardian. When a user makes a request for a video call or when a video call is received from a terminal of a guardian, as shown in FIG. 9, the user shines on the mirror 100 and the area L1 that the user shines on the mirror 100 The appearance of an object making a video call is displayed on the area L3 of the display unit 600 that does not overlap with the area L1, and the smart mirror device 1000 may be executed in a video call mode. In this way, the user can make a video call through the smart mirror device 1000, and can exert the effect of communicating with the family through the video call.

10 schematically illustrates the operation of the feature information extractor 2110 according to an embodiment of the present invention.

The service server 2000 that communicates with the smart mirror device 1000 of the present invention includes a user state diagnosis unit 2100, and the user state diagnosis unit 2100 includes a feature information extraction unit 2110 and a hardness recognition disorder. It includes an information extraction unit 2120. As illustrated in FIG. 10, the user's voice information received from the microphone unit 300 of the smart mirror device 1000 includes the feature information extractor 2110 including the amplitude, length, and pattern of the user's voice information. Derives voice feature information. The voice feature information including the derived amplitude, length, and pattern is transmitted to the longitude cognitive impairment information extracting unit 2120, and the longitude cognitive impairment information extracting unit 2120 provides the longitude cognitive impairment information based on the transmitted voice feature information. Can be derived. The derived hardness cognitive impairment information may be transmitted to the guardian's terminal by the user's hardness cognitive impairment information according to a preset criterion by the user state notification unit 2200 described later.

11 schematically illustrates the operation of the feature information extractor 2110 according to an embodiment of the present invention.

The service server 2000 that communicates with the smart mirror device 1000 of the present invention includes a user state diagnosis unit 2100, and the user state diagnosis unit 2100 includes a feature information extraction unit 2110, a hardness recognition disorder. It includes an information extracting unit 2120 and an emotion information extracting unit 2130. As illustrated in FIG. 11, the user's image information received from the camera unit 200 of the smart mirror device 1000 extracts a face image based on the user's image information, and features a face based on the extracted face image. Derive information. The image information may be a dynamic image for a user including a facial expression change capable of recognizing a user's emotion or a plurality of static images for a user recorded continuously at a constant cycle. Image information should be interpreted in the broadest sense, including having two or more image frames. Accordingly, the feature information deriving unit 2110 classifies the received image information into a plurality of frames, and extracts a face image from each frame. Then, face feature information is derived based on the extracted face image. The facial feature information may mean feature information about the unique characteristics capable of recognizing the facial expression, such as the inclination of the eye, the inclination of the eyebrow, the inclination of the mouth, and the inclination of the mouth. It is transmitted to the disability information extracting unit 2120 and the emotion information extracting unit 2130, and the long-term cognitive impairment information extracting unit 2120 derives long-term cognitive impairment information based on the transmitted facial feature information, and the emotion information extracting unit ( 2130) derives emotion information based on the transmitted facial feature information. The derived long-term cognitive impairment information and emotion information may then be transmitted to the guardian's terminal according to a preset criterion by the user status notification unit 2200.

12 schematically illustrates the operation of the longitudinal cognitive impairment information extracting unit 2120 according to an embodiment of the present invention.

The service server 2000 of the present invention includes a longitudinal cognitive disability information exposing unit 2120 included in the user state diagnosis unit 2100, and the longitudinal cognitive disability information exposing unit 2120 is as shown in FIG. 12. Likewise, the first artificial neural network 2121, the second artificial neural network 2122, and the comprehensive diagnosis module 2123 are included, and the longitudinal cognitive impairment information extracting unit 2120 is an amplitude derived based on the user's voice information. Deriving a first analysis result by the first artificial neural network 2121 that analyzes the degree of mild cognitive impairment based on voice feature information including length and pattern (S400); Deriving a second analysis result by a second artificial neural network (2122) that analyzes the degree of mild cognitive impairment based on the facial feature information derived from the user's image information (S410); And deriving longitudinal cognitive impairment information based on the first analysis result and the second analysis result (S420).

In step S400, the first analysis result is derived by the first artificial neural network 2121 that analyzes the degree of long-term cognitive impairment based on the voice characteristic information including the amplitude, length, and pattern derived from the feature information deriving unit 2110. do. The first analysis result may be derived by analyzing a user's speech based on the amplitude, length, and pattern, and analyzing a user's language proficiency based on keywords used by the user.

In step S410, the second analysis result is derived by the second artificial neural network 2122 that analyzes the degree of mild cognitive impairment based on the facial feature information derived from the feature information deriving unit 2110. The facial feature information derived from the face image extracted based on the user's image information refers to characteristic information about a unique characteristic capable of recognizing a facial expression such as an eye tilt, an eyebrow tilt, a mouth shape, and a mouth tilt. It is possible to derive the second analysis result analyzing the degree of mild cognitive impairment according to the user's facial expression based on the feature information.

In step S420, the comprehensive diagnosis module 2123 analyzes the degree of hardness cognitive impairment based on voice feature information by the first artificial neural network 2121 in step S400 and the second artificial in step S410. Based on the facial feature information by the neural network 2122, mild cognitive impairment information is derived based on the second analysis result analyzing the degree of mild cognitive impairment. The mild cognitive impairment information may be derived by a score or level, for example, when the user's degree of mild cognitive impairment is the lowest and 1 when the user's degree of mild cognitive impairment is the highest. . Preferably, the comprehensive diagnosis module 2123 may be configured to automatically control to what extent weight or obstacle information is derived for the video information and the audio information, and learn about the weight. You can construct the model to do so. That is, in this configuration, if the user is not in a state of speaking through facial feature information such as a mouth shape from the image information received from the smart mirror device 1000, the image information is automatically processed more importantly and the voice information is less. It can be important. In this way, the cognitive impairment information guiding unit 2120 may derive the cognitive impairment information based on the user's voice feature information and face feature information.

13 schematically illustrates the operation of the emotion information extracting unit 2130 according to an embodiment of the present invention.

The service server 2000 of the present invention includes an emotion information extracting unit 2130 including a user state diagnosis unit 2100, and the emotion information extracting unit 2130 is as shown in FIG. 13(a). Likewise, it includes a third artificial neural network (2131). The emotion information deriving unit 2130 may derive emotion information by a third artificial neural network 2131 that analyzes a user's emotion based on face feature information derived from the user's image information. The third artificial neural network 2131 includes one or more of a user's joy, surprise, sadness, anger, fear, interest, stress, fear, boredom, comfort, discomfort, anxiety, and neutrality based on facial feature information Derives emotion information of the user. The emotion information is a vector form according to emotions such as joy, surprise, sadness, anger, fear, interest, stress, fear, boredom, comfort, discomfort, anxiety, and neutrality as shown in FIG. 13B. It can be data. The emotion information shown in (b) of FIG. 13 is anger 0.01, discomfort 0.00, fear 0.00, happiness 72.00, neutral 24.09, sadness 0.00, surprise 3.9, and emotion information is derived. It can be judged that you are feeling happy in the video information of time.

On the other hand, the service server 2000 includes a user status notification unit 2200, and the user status notification unit 2200, the hardness cognitive impairment information and the emotion derived from the cognitive impairment information estimator 2120 Deriving the user's status information according to a predetermined criterion based on the emotion information derived by the information deriving unit 2130 (S500); Determining whether the derived user's status information corresponds to a predetermined criterion (S510); And if the user's status information corresponds to a predetermined criterion, transmitting the user's status information to the terminal of the guardian (S520).

In step S500, the user's status information is derived according to a predetermined criterion based on the mild cognitive impairment information and emotion information. In one embodiment of the present invention, the user's longitudinal cognitive impairment information was derived as 7, and the preset criterion is when the longitudinal cognitive impairment information is 5 or more or when there is a difference of 3 or more than the past derived cognitive impairment information. , Assuming that it is judged as a mild cognitive disability risk condition, the user is judged to be a mild cognitive disability risk condition. In addition, among the emotion details including one or more of joy, surprise, sadness, anger, fear, interest, stress, fear, boredom, comfort, discomfort, anxiety, and neutrality included in the emotion information, the corresponding vector value is the most When it is assumed that a large emotional detail item is derived into the corresponding emotional state, or when the corresponding vector value is 0.5 or more, assuming that the corresponding emotional detail item is derived into an emotional state, the user from whom the emotional information as shown in FIG. 13(b) is derived It can be judged that you are feeling happiness. In this way, the user's status information is derived.

In step S510, it is determined whether the user's status information corresponds to a predetermined criterion based on the user's status information derived in step S500. In one embodiment of the present invention, the preset criterion is that when the user's status information is a mild or disability risk condition and an emotional state such as depression or sadness, the status information is transmitted to the guardian's terminal, and the user's status information Is a mild cognitive disability risk condition, but in the case of an emotional state such as happiness or joy, it is determined whether the user's status information corresponds to the predetermined criteria, assuming that the status information is not transmitted to the terminal of the guardian. .

In step S520, in step S510, when the user's status information corresponds to a preset criterion, the user's status information is transmitted to the terminal of the user's guardian. In one embodiment of the present invention, when the preset criteria described in step S510 is applied, the user's status information is sent to the terminal of the guardian when the user's status information is a mild cognitive disorder safety state and an emotional state corresponding to sadness. It may not be transmitted.

In this way, the derived user's status information is judged according to a predetermined criterion, and the result of analyzing the user's status is transmitted to the guardian, so that the guardian of the senior senior user who lives alone does not live with the user's status. By receiving the notification, it is possible to exert the effect of taking appropriate action according to the user's condition.

14 exemplarily shows an internal configuration of a computing device according to an embodiment of the present invention.

As shown in FIG. 14, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, and an input/output subsystem ( I/Osubsystem) 11400, a power circuit 11500, and a communication circuit 11600. At this time, the computing device 11000 may correspond to the service server 2000 connected to the smart mirror device 1000 or the aforementioned user terminal.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. have. The memory 11200 may include software modules required for the operation of the computing device 11000, a set of instructions, or other various data included in the learned embedding model.

At this time, access to the memory 11200 from other components such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100.

The peripheral interface 11300 may couple input and/or output peripherals of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute various functions for the computing device 11000 and process data by executing a software module or a set of instructions stored in the memory 11200.

The input/output subsystem 11400 can couple various input/output peripherals to the peripheral interface 11300. For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor or keyboard, mouse, printer or a touch screen or sensor to the peripheral interface 11300 as required. According to another aspect, the input/output peripherals may be coupled to the peripheral interface 11300 without going through the input/output subsystem 11400.

The power circuit 11500 may supply power to all or part of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or power. It can include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with other computing devices using at least one external port.

Or, as described above, if necessary, the communication circuit 11600 may transmit and receive an RF signal, also known as an electromagnetic signal, including an RF circuit, to enable communication with other computing devices.

The embodiment of FIG. 14 is only an example of the computing device 11000, and the computing device 11000 may omit some components illustrated in FIG. 14 or further include additional components not illustrated in FIG. 14, or 2 It may have a configuration or arrangement that combines more than one component. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, etc. in addition to the components illustrated in FIG. 14, and various communication methods (WiFi, 3G, LTE) in the communication circuit 1160. , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software including integrated circuits specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be recorded in a computer readable medium by being implemented in the form of program instructions that can be performed through various computing devices. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

The device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configures the processing device to operate as desired, or processes independently or collectively You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computing devices and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if substituted or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

As a smart mirror chatbot system for senior care implemented by a service server that can be accessed by a guardian's terminal and a smart mirror device,
The smart mirror device,
A mirror made of a transparent material that illuminates the user's appearance; A display unit that displays an image or information and is installed on the rear side of the mirror; A speaker unit for outputting sound; A microphone unit for collecting voice information of the user; A wireless communication unit wirelessly communicating with the service server and the guardian's terminal; Camera unit for collecting the user's video information; And a control unit for controlling the operation of the smart mirror device through communication with the service server.
The control unit,
A user identification unit for operating the smart mirror device by identifying the user based on voice information and video information of a user who intends to operate the smart mirror device;
A chatbot interface unit for performing a conversation with a user through a character displayed on the display unit; And
Includes a call mode providing unit for executing a video call through communication with the guardian's terminal;
The service server,
A longitudinal cognitive disability information deriving unit for deriving longitudinal cognitive disability information based on the user's voice information and the image information; And an emotion information deriving unit that derives emotion information based on the image information of the user. And
It includes; a user status notification unit for transmitting the user's status information derived based on the hardness cognitive impairment information and the emotion information to the terminal of the guardian
The chatbot interface unit,
Deriving an area where a user is projected on the mirror based on the image information; Deriving a character display area of the display unit that does not overlap the area where the user is projected; And displaying the character in the character display area.
The chatbot interface unit,
Performing one or more questions related to diagnosis of mild cognitive impairment through a character displayed on the display unit;
Receiving voice information of a user in response to the question through the microphone unit;
Receiving feedback voice information corresponding to the corresponding response from the service server based on the user's voice information for the response; And
And providing the feedback voice information through the character.
The longitudinal cognitive disability information extraction unit,
Deriving a first analysis result by a first artificial neural network analyzing a degree of mild cognitive impairment based on voice feature information including amplitude, length and pattern derived based on the user's voice information;
Deriving a second analysis result by a second artificial neural network analyzing the degree of mild cognitive impairment based on the facial feature information derived from the image information of the user; And
And deriving longitudinal cognitive impairment information based on the first analysis result and the second analysis result.
The emotion information deriving unit,
Emotion information is derived by a third artificial neural network that analyzes the user's emotion based on the facial feature information derived from the user's image information,
The chatbot interface unit,
Voice and voice of a character that receives the user's mild cognitive impairment information and the emotion information from the service server and provides the feedback voice information according to a preset criterion based on the received mild cognitive impairment information and the emotional information Smart mirror chatbot system to convert the speed of the.
The method according to claim 1,
The user identification unit, a face identification unit; And a voice identification unit;
The face identification unit,
Extracting a user's face image based on the user's image information;
Determining a user's face by comparing the user's previously stored face image with the face image extracted from the image information; And
If it is determined as the user's face, operating the smart mirror device;
The voice identification unit,
Determining a user's voice by comparing the received user's voice information with a pre-stored user's voice information; And
If it is determined by the user's voice, operating the smart mirror device; performing, smart mirror chatbot system.
The method according to claim 1,
The chatbot interface unit,
According to the user's voice information for the information request, smart mirror providing information including one or more of the weather, schedule, and news requested by the user is provided by voice through a chatbot displayed in the form of a character, and corresponds to the display screen of the display unit A smart mirror chatbot system that displays information.
delete The method according to claim 1,
The call mode providing unit,
A smart mirror chatbot system that transmits a user's voice information and video information to the guardian's terminal, and transmits the guardian voice information and the guardian's video information received from the guardian's terminal to the smart mirror device to perform a video call.
The method according to claim 1,
The user state diagnosis unit further includes a feature information deriving unit;
The feature information introduction unit,
Deriving voice feature information according to the amplitude, length and pattern of the user's voice information,
A smart mirror chatbot system that extracts a face image based on the user's video information and derives facial feature information based on the extracted face image.
delete delete The method according to claim 1,
The user status notification unit,
Based on the user's long-term cognitive impairment information and the emotion information, the user's status information is derived according to a preset criterion, and when the derived user's status information corresponds to a preset criterion, the user's status is transferred to the guardian's terminal A smart mirror chatbot system that transmits information.
As a method of providing a smart mirror chatbot service for senior care performed by a service server that can be accessed by the guardian's terminal and the smart mirror device,
The smart mirror device,
A mirror made of a transparent material that illuminates the user's appearance; A display unit that displays an image or information and is installed on the rear side of the mirror; A speaker unit for outputting sound; A microphone unit for collecting voice information of the user; A wireless communication unit wirelessly communicating with the service server and the guardian's terminal; Camera unit for collecting the user's video information; And a control unit for controlling the operation of the smart mirror device through communication with the service server.
The service server includes a user identification unit; Chatbot interface unit; A call mode providing unit; Mild Cognitive Disability Information Derivation Unit And an emotion information deriving unit; And a user status notification unit;
A user identification step of identifying the user based on voice information and video information of a user who wants to operate the smart mirror device by the user identification unit and operating the smart mirror device;
A chatbot interfacing step of performing a conversation with the user through a character displayed on the display unit by the chatbot interfacing unit;
A call mode providing step of providing a voice call or a video call through communication with a terminal of the guardian by the call mode providing unit;
A long-term cognitive disability information deriving step of deriving long-term cognitive disability information based on feature information on the user's voice information and feature information on the video information by the long-term cognitive disability information deriving unit; And an emotion information deriving step of deriving emotion information based on the feature information on the user's image information by the emotion information deriving unit. And
Including, by the user status notification unit, the user status notification step of transmitting the user's status information derived on the basis of the hardness cognitive impairment information and the emotion information to the terminal of the guardian,
The chatbot interfacing step,
Deriving an area in which the user is projected on the mirror based on the image information; Deriving a character display area of the display unit that does not overlap the area where the user is projected; And displaying the character in the character display area.
The chatbot interfacing step,
Performing one or more questions related to diagnosis of mild cognitive impairment through a character displayed on the display unit;
Receiving voice information of a user in response to the question through the microphone unit;
Receiving feedback voice information corresponding to the corresponding response from the service server based on the user's voice information for the response; And
And providing the feedback voice information through the character.
The longitudinal cognitive disability information extraction step,
Deriving a first analysis result by a first artificial neural network analyzing a degree of mild cognitive impairment based on voice feature information including amplitude, length and pattern derived based on the user's voice information;
Deriving a second analysis result by a second artificial neural network analyzing the degree of mild cognitive impairment based on the facial feature information derived from the image information of the user; And
And deriving longitudinal cognitive impairment information based on the first analysis result and the second analysis result.
The emotion information deriving step,
Emotion information is derived by a third artificial neural network that analyzes the user's emotion based on the facial feature information derived from the user's image information,
The chatbot interfacing step,
Voice and voice of a character that receives the user's mild cognitive impairment information and the emotion information from the service server and provides the feedback voice information according to a preset criterion based on the received mild cognitive impairment information and the emotional information How to provide a smart mirror chatbot service, to transform the speed of the.

Images