WO2020209414A1

WO2020209414A1 - Face recognition-based smart dispenser using image sensor

Info

Publication number: WO2020209414A1
Application number: PCT/KR2019/004330
Authority: WO
Inventors: 천재두; 이민정
Original assignee: 주식회사 에버정보기술
Priority date: 2019-04-11
Filing date: 2019-04-11
Publication date: 2020-10-15
Also published as: KR20200120778A; KR102185492B1

Abstract

A face recognition-based smart dispenser using an image sensor according to an embodiment of the present invention is provided with the image sensor, which periodically captures and acquires images of a medicine taker, wherein authentication of the medicine taker is performed through the image sensor, a medicine-taking schedule of the medicine taker is stored in advance for the authenticated medicine taker, a command to take medicine is issued according to the stored medicine taking schedule, and the medicine taker is authenticated by using the image of the medicine taker.

Description

Smart dispenser based on facial recognition using image sensor

The present invention relates to a smart dispenser, and more particularly, to a smart dispenser based on facial recognition using an image sensor capable of authenticating a taker using an image sensor and performing a dosage command of the taker.

In modern society, interest in health is increasing day by day. In addition to the interests of the times, as technology advances rapidly, such as the advancement of data analysis methods and tools by collecting real-time data, it has become possible to monitor health conditions and receive personalized health care services.

In addition, convenience and customization of health services and related systems are being strengthened by diversification of customer needs and improvement of expectations according to changes in consumer's consciousness, and lifestyle-related diseases are prevented based on accumulated personal health data. Personalized healthcare businesses such as weight management are growing rapidly.

Recently, according to the National Statistical Office, Korea has experienced a'population reversal phenomenon' in which the proportion of the elderly population exceeds the proportion of the youth population from 2017, and the age of aging is in full swing after the working age population turns to a decline.

In addition, since the elderly take several drugs at the same time, it is very likely that side effects may occur due to interactions between drugs. Therefore, if you discontinue the drugs or take additional drugs to relieve pain, consult your doctor or pharmacist in advance. You should consult.

Drugs must be taken in a fixed dose at a fixed time and in a timely manner in order to achieve optimal effects. However, in the case of elderly patients, due to complications, the number of drugs to be taken is large, and the number of drugs to be taken is also large, so the correct drug must be taken at the specified time.

Therefore, in order to solve the above-described problem, the face using an image sensor capable of performing image sensor authentication so that the drug can be administered according to a prescribed dosing schedule and that other people are not authorized to receive the drug. Research on recognition-based smart dispensers is needed.

[Prior patent literature]

Korean Patent Publication No. 2013-024065 (published on March 08, 2013)

An object of the present invention is to use an image sensor to trigger a notification when the dose time is reached, to recognize and open the biometric information through an image sensor connected to the device, to allow the patient to take the drug, and at the same time to take the dose by the image sensor. It provides a smart dispenser based on facial recognition using an image sensor that can recognize and automatically check whether or not to take it.

The smart dispenser based on facial recognition using an image sensor according to an embodiment of the present invention has an image sensor that periodically photographs and acquires an image of a taker, performs authentication of the taker through the image sensor, and It is characterized in that the dose schedule of the taker is stored in advance, and the medication command is executed according to the stored dose schedule, and authentication for the corresponding taker is performed using the image of the taker.

In the above, the dispenser comprises: a control unit configured to receive an image acquired by periodically photographing an image of a taker through the image sensor, and to recognize a taker and authenticate the taker; A display unit that serves to visually provide various notifications to a user; A key input unit for providing a key input such as a screen display, power on/off, etc.; An output unit provided to output voice guidance to the outside; Recognizes the facial expression features of the taker from at least one of the taker's image, can store a plurality of facial expression recognition results, detects a feature related to the temporal change of the taker's expression, and based on the detected feature It includes a situation judgment unit that can estimate mental health status.

In the above, the dispenser predicts the state of the taker based on an artificial intelligence learning algorithm by analyzing the image photographed through the image sensor, and displays whether to take it through the display unit under the control of the controller according to the prediction result It further includes a prediction unit for providing a prediction result to the control unit to provide a voice guidance on the dose through the output unit.

In the above, the control unit is characterized in that to store and manage at least one of a dose, a type of drug to be taken, a dose time, a state of the taker, and a taking history in order to perform a dosage command of the taker.

In the above, the authentication of the taker is characterized by extracting the taker's feature points from the facial image based on pixel information from the taker's face image acquired by photographing by the image sensor.

In the above, the situation determination unit is characterized in that it uses a skin care algorithm to check the health of the user according to the skin change.

In the above, the authentication of the taker is characterized in that the situation determination unit checks the heart rate by programming a health management algorithm in order to check the state of the taker.

The smart dispenser based on facial recognition using the image sensor of the present invention can perform dose to the authenticated taker through image sensor authentication, thus enhancing security and preventing medical accidents that may be taken by others in advance. can do.

In addition, it is possible to provide health and healthcare services through the administration of medications for the elderly living alone or for one or two family members.

In addition, additional additional services are provided by acquiring non-contact biometric information for the object's reluctance, discomfort, and ease of use, and by processing and managing AI (deep learning) using the acquired big data of the subject's unique biometric information ( Health, skin, taking medicines, etc.)

In addition, by executing the medication order according to the prescribed dosage schedule, drug abuse is prevented, and even if the user is not aware of the dosage schedule, the dosage, the type of medicine being taken, the dosage time, the condition of the user, the dosage history, etc. are checked and administered, and medication management There is an advantage in that it can be performed smartly, and furthermore, it is possible to reduce unnecessary loss of health care and social medical expenses.

1 is a front view of a face recognition-based smart dispenser using an image sensor according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a detailed configuration of a smart dispenser based on facial recognition using the image sensor of FIG. 1.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention can add, change, or delete other elements within the scope of the same idea. Other embodiments included within the scope of the inventive concept may be easily proposed, but it will be said that this is also included within the scope of the inventive concept. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described with the same reference numerals.

1 is a front view of a face recognition-based smart dispenser using an image sensor according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a detailed configuration of a face recognition-based smart dispenser using the image sensor of FIG. 1.

The dispenser 100 of the present invention basically uses the image sensor 170 to sound a notification when the dose time is reached, and recognizes the biometric information through the image sensor 170 connected to the device and locks the drug inlet 101 with a locking device. ) Is opened and the user is allowed to take the medicine, and at the same time, the user's taking action is recognized through the image sensor 170 to automatically check and record whether or not to take the medicine.

Referring to FIG. 1, the dispenser 100 of the present invention shows a display unit 120 for displaying various guides on the front side, a key input unit 140 for receiving external inputs, and a user's appearance for use in biometric authentication. It includes an image sensor 170 for photographing.

As shown in FIG. 2, the dispenser 100 of the present invention includes a control unit 110, a display unit 120, a prediction unit 130, a key input unit 140, an output unit 150, and a biometric authentication unit 160. , An image sensor 170, and a situation determination unit 180.

The image sensor 170 is located in front of the dispenser 100 using the image sensor 170 authentication, and if the taker is within the recognition distance, the image can be periodically photographed and acquired, and the acquired image is transferred to the controller 110. Is transmitted. At this time, the recognition distance is preferably within 1m.

In addition, the image sensor 170 is an image sensor used to image the entire face of the taker, and an auxiliary image capable of imaging different angles of the taker (for example, the position of the taker's hand or the side of the taker's face). A plurality of sensors can also be used in combination. Through this method, it is possible to compensate for the limitation of situation recognition that may occur using the single image sensor 170 (for example, a backlight situation or an error such as a hardware failure of the image sensor itself). In addition, it is desirable to additionally provide an illuminance sensor to obtain a face image of the patient in order to utilize it in a non-bright place to facilitate facial recognition.

Furthermore, a camera capable of performing the function of the image sensor 170 may be provided to acquire a plurality of images and transmit them to the control unit 110 to be used for user authentication.

The controller 110 may store the drug taking schedule of the taker for the taker authenticated and the recognized taker through the image sensor 170, and execute a dosage command according to the stored taking schedule.

At this time, the control unit 110 through a central control server (not shown) that can provide such information about additional disease status through personal ID information (for example, resident number) of the user or personal ID information of the user. You can learn. That is, after the user's authentication is completed, additional information (personal ID information, disease status) of the user can be used.

The control unit 110 may use a feature point extraction method to perform recognition of a taker and authentication of a corresponding taker. For example, a feature point of the taker may be extracted from the facial image based on pixel information from the face image of the taker acquired by photographing by the image sensor 170. It may be a feature point constituting the face (face), for example, the center of the eyebrow, the edges of the eyes, the corner of the mouth, the center of the lips, and the like. In this case, facial contours and contours such as eyebrows, eyes, nose, mouth, and ears may be extracted using a boundary line extraction method, and feature points may be extracted by checking a pixel value of each pixel within each contour.

Furthermore, at least one complexion area may be extracted using the detected feature points. Here, the complexion area is an area that is divided into an area that can be distinguished from the face, and may be an area constituting the face.

The control unit 110 may have a built-in memory to store a dose, a type of drug to be taken, a dose time, a state of a taker, a taking history, etc., and may be a programmable microcontroller (MCU) for this purpose.

In addition, the control unit 110 receives a prediction result for the state of the taker based on an artificial intelligence learning algorithm from the prediction unit 130 and displays whether to take it through the display unit 120 or the output unit 150. It can be controlled to provide voice guidance for

The display unit 120 serves to visually provide various notifications to the user, and may display, for example, a dose, a type of drug to be taken, a dosage time, a state of the user, and a dosage history.

The prediction unit 130 analyzes the image captured through the image sensor 170, predicts the state of the taker based on an artificial intelligence learning algorithm, and displays the prediction result through the display unit 120 or the output unit 150. Enables voice guidance to be performed. Here, the artificial intelligence learning algorithm may be, for example, an artificial neural network, a radial basis function (RBF) neural network, and a support vector machine algorithm.

Specifically, it is more preferable to use a support vector machine algorithm capable of estimating a case that deviates from the pattern by extracting and classifying facial feature points.

More specifically, the predictor 130 analyzes the taker's image and predicts the taker's state, and most of the taker's images are used to recognize the taker's face and hands. In more detail, the prediction unit 130 is used to predict the state of the taker by extracting feature points (eg, eyes, nose, mouth, etc.) of the face and hands from the imaged images of the face and hands of the taker.

However, when such a user wears an additional facial artificial structure on the face, there is a problem that it is difficult for the predictor 130 to accurately extract the feature points from the image captured through the image sensor 170.

For example, these additional facial structures may be glasses, sunglasses, hats, masks, artificial beards, etc. worn by the user.

Like this, if the user's face cannot be recognized due to the artificial structure, "Please remove the artificial structure from the face." Or "Please check it as it is currently impossible to recognize due to artificial structures." A voice guide such as, etc. may be output or displayed on the display unit 120.

In addition, furthermore, the prediction unit 130 includes the artificial structures (for example, sunglasses, masks) described above as a factor that analyzes and learns the image captured through the image sensor 170, You can make them learn. As a result, through such learning, the predictor 130 may better estimate the wearer's wearing of the artificial structure and request the user to better classify the removal of the artificial structure through the display unit 120 or the output unit 150.

That is, various voice guidance or visual indications related to the dose are made according to the state of the taker, and the prediction results and the taking history of the taker are stored in a separate database, so that the self-learning and evolution can be achieved by converting into big data.

The situation determination unit 180 may recognize the facial expression characteristics of the taker from each of the plurality of image sensor 170 images captured and acquired in time series, and store the plurality of facial expression recognition results as time series data. Features related to temporal changes in facial expressions may be detected, and the mental health status of the user may be estimated based on the detected features.

For example, if a face with a smiling expression is continuously recognized, the drug administration time is delayed, or if a face with a depressed expression is continuously recognized, the medicine administration time may be accelerated so that it can be provided to the user. It can also be used to determine a health state according to facial expression characteristics.

In addition, the situation determination unit 180 additionally recognizes not only the facial features of the taker in time series, but also the hands that actually take the medicine to the mouth of the taker, and features points of the taker's face (for example, the position of the mouth). Based on the relationship between the characteristics of the hand and the hand, it is possible to better check whether the user is taking medication. That is, by additionally recognizing and authenticating the change in the result of facial expression recognition and the change between the feature points of the face and the hand, it is possible to estimate the health status of the taker and increase the estimation or prediction accuracy of whether the taker takes medicine.

Furthermore, the situation determination unit 180 predicts what problems will occur in health (breathing status, activity level, heart rate status) according to skin changes (pores, wrinkles, sebum, etc.) using a skin care algorithm to check the status of the user. May be. For example, in the case of stomach cancer, the image provided through the image sensor 170, which causes rapid changes such as skin temperature or flushing, can be analyzed and checked, and through this, medical accidents can be prevented in advance.

In addition, the situation determination unit 180 can check the heart rate by programming a health management algorithm to check the state of the taker. For example, anger, contempt, disgust, fear, happiness, and expressionlessness in the face image obtained from the image sensor 170 You can also identify a series of emotions such as, sadness, and surprise.

More specifically, the situation determination unit 180 analyzes the user's time series data and images obtained through the image sensor 170 to estimate the change in blood flow on the user's face, thereby estimating the heart rate. It also becomes possible. In addition, a method such as deep learning as described above may be additionally used for this estimation. Furthermore, it is possible to perform authentication for the user by checking the heart rate using such a health management algorithm. For example, user authentication may be performed according to whether the heart rate measurement value matches the previously registered heart rate of the corresponding user.

The key input unit 140 may provide a user to directly input a key input to give a command to take medicine or to input a key such as a screen display or power on/off.

The output unit 150 is provided in the form of a speaker so as to output voice guidance to the outside.

The voice guidance may be, for example, a dosage instruction guide, a dosage time warning alarm, an authentication status alarm, and a dosage voice guidance.

The biometric authentication unit 160 is a separate authentication means other than the image sensor 170 image authentication, and may be an alternative authentication means in situations where authentication of the image sensor 170 continuously fails or is unavailable (dark places, etc.), For example, a biometric authentication method using fingerprint, iris, voice, and heart rate may be adopted. In addition, for biometric authentication, it is desirable to receive and store a fingerprint, iris, voice, and heart rate as the biometric authentication information of the user in advance.

In addition, in addition to the image sensor 170 and the biometric authentication unit 160 described above, it may further include auxiliary sensing means. More specifically, the auxiliary detection means may be a distance detection sensor, such as a radar or LiDar, that can measure a distance to a user. When such an auxiliary sensing means is a sensor capable of sensing the distance, for example, when a user enters the dispenser 100 within a certain distance, it recognizes such a situation and starts so that the image sensor 170 can be imaged, and the user moves a certain distance. If it deviates, it can be utilized to stop capturing the image sensor 170. Through this utilization, it is possible to solve a problem in that the image sensor 170 operates to capture an object other than the patient and analyze the captured image through the prediction unit 130 or the control unit 110.

In addition, when the dispenser is operated using only the image sensor 170 and the auxiliary sensing means, it is possible to effectively care for the health of the users through the presence or absence of the taker's medicine and estimation of the health status of the taker in a non-contact manner with the user. This is particularly helpful for patients who are difficult to authenticate through physical contact or for the elderly (for example, dementia patients) to manage their medications smartly and further effectively manage their health. Can give.

In addition, as described above, the control unit 110 may utilize additional information (personal ID information, disease status) of the user through the central control server. For example, in the case of disease status, the control unit 110 may use the individual's injury and illness code.

More specifically, when the injured code of the person who can be acquired through the control unit 110 is'dementia', the dispenser 100 simultaneously utilizes the image sensor 170 and the auxiliary sensing means as described above, and the biometric authentication unit In the case where the patient's injury code is'schizophrenia', as another example, when the patient's injury code is'schizophrenia', the imaging method of the image sensor 170 is expected to be in a state where the movement of the object is large. By setting, it is possible to provide a customized method according to the condition of the disease such as the patient's injury and disease code.

That is, by actively using the patient's injury and illness code in the operation of the control unit 110, the prediction unit 130, and the situation determination unit 180, it is possible to change and utilize various recognition methods in response to the disease the patient is suffering from.

Furthermore, the dispenser of the present invention may additionally include a wireless communication unit (not shown) capable of wireless communication in connection with an external management server. The dispenser of the present invention can be used to perform user authentication in connection with a management server through a wireless communication unit, or to establish a DB for dosage management, skin care, and health management (heart rate check), and to check and manage history. It is also possible to analyze and predict the patient's condition more accurately by using the dosage data related to the taker (dose, type of drug taken, taking time, taker status and taking history, etc.) converted into big data based on artificial intelligence algorithm.

[Explanation of code]

100: dispenser

110: control unit 120: display unit

130: prediction unit 140: key input unit

150: output unit 160: biometric authentication unit

170: image sensor 180: situation determination unit

Claims

Equipped with an image sensor that periodically photographs and acquires an image of a taker

Performing the taker authentication through the image sensor, pre-stores the taker's taking schedule for the authenticated taker, and executes a dosage command according to the stored taking schedule,

Facial recognition-based smart dispenser using an image sensor, characterized in that performing authentication for the corresponding taker using the image of the taker.
The method of claim 1,

The dispenser

A control unit configured to receive an image acquired by periodically photographing an image of a taker through the image sensor, and to recognize a taker and authenticate the taker;

A display unit that serves to visually provide various notifications to a user;

A key input unit for providing a key input such as a screen display, power on/off, etc.;

An output unit provided to output voice guidance to the outside; And

Recognizes the facial expression features of the taker from at least one of the taker's image, can store a plurality of facial expression recognition results, detects a feature related to the temporal change of the taker's expression, and based on the detected feature A smart dispenser based on facial recognition using an image sensor including a situation determination unit that can estimate mental health.
The method of claim 2,

The dispenser

By analyzing the image captured through the image sensor, the state of the taker is predicted based on an artificial intelligence learning algorithm, and according to the prediction result, the taking status is displayed through the display unit under the control of the control unit, or a voice for taking the dose is displayed through the output unit. Facial recognition-based smart dispenser using an image sensor further comprising a prediction unit for providing a prediction result to the control unit to provide guidance.
The method of claim 2,

The control unit

A smart dispenser based on facial recognition using an image sensor, characterized in that storing and managing at least one of a dose, a type of drug to be taken, a dose time, a state of a taker, and a taking history in order to perform a dose command of a taker.
The method of claim 1,

Certification for the above taker,

A face recognition-based smart dispenser using an image sensor, characterized in that the feature points of the taker are extracted from the face image based on pixel information from the face image of the taker acquired by photographing by the image sensor.
The method of claim 2,

The situation determination unit uses a skin care algorithm to check the health of a user according to a change in skin. A smart dispenser based on facial recognition using an image sensor.
The method of claim 2,

Certification for the above taker,

The smart dispenser based on facial recognition using an image sensor, characterized in that the situation determination unit is made by programming a health management algorithm to check the state of the patient and checking the heart rate.