KR102293433B1 - System for providing mental health care service using smartphone sensors - Google Patents

Abstract

Provided is a system for providing a mental health measurement service using a smartphone sensor, which comprises: a user terminal which uploads at least one sensor data measured by at least one sensor in real time or periodically, and receives a report on a mental health state measured based on the at least one sensor data; and a mental health measurement service providing server. The mental health measurement service providing server includes: a storage unit for mapping and storing at least one reference sensor data and at least one mental health measurement data; a comparison unit for receiving the at least one sensor data received from the user terminal and comparing the received data with the at least one reference sensor data; an extraction unit for extracting the mental health measurement data stored after being previously mapped to the at least one reference sensor data including the at least one sensor data of the user terminal; and a transmission unit for transmitting the extracted mental health measurement data to the user terminal. The mental health state can be analyzed and identified using only the sensor of the smartphone.

Description

A system for providing mental health measurement service using a smartphone sensor {SYSTEM FOR PROVIDING MENTAL HEALTH CARE SERVICE USING SMARTPHONE SENSORS}

The present invention relates to a system for providing a mental health measurement service using a smartphone sensor, and provides a platform that can measure mental health using the correlation between the data measured by the sensor built into the smartphone and the diagnostic criteria of DSM-5. to provide.

The world is facing a catastrophic situation due to a new infectious disease called Corona (COVID-19). These infectious disease disasters bring trauma to the entire nation at the national level, and at the same time have a negative impact on the mental health of not only confirmed patients and their bereaved families, but also the entire nation. In particular, since the socioeconomic loss and subsequent economic crisis due to this corona may afflict society due to an increase in suicide rate after the end of the corona situation, it is necessary to prepare for a mentaldemic, in which mental trauma is prevalent like an epidemic. The government is taking preemptive measures to prevent the damage to mental health from Corona. The National Trauma Center has been installed and operated at the National Mental Health Center in accordance with the Act.

At this time, a method of using biometric information signals to manage mental health has been researched and developed. Korean Patent Publication No. 2020-0103397 (published on September 02, 2020) and Korean Patent No. 10-1913850 (No. Notice on the 01th of the month), generates stress standard information by correcting the biosignal received from the biosignal sensor, creates a stress guiding screen, and measures and measures the user's biometric data through the generated stress guiding screen A configuration for calculating the user's stress measurement information by comparing the biometric data with at least one of the stress standard information and the biosignal, extracting features from the biometric data, and predicting the user's stress index based on the extracted features; In order to determine the user's psychological state by analyzing the user's psychological state data obtained through the state determination kit, the user's psychological state is determined through interaction with the content, and the user's biological signal measured through the sensor A method of obtaining data about a user's saliva acquired through data and a saliva analyzer, and determining or supporting a user's psychological state using each data is disclosed, respectively.

However, the above-mentioned configurations are thought to have a low barrier to entry because they do not need to see a psychiatrist, but in fact, each HMD device or content is purchased and equipment that can analyze saliva is required, so another barrier to infrastructure construction exists. . The Ministry of Health and Welfare has released apps such as the mental health self-examination app, smart touch me, mind more, mind program, and demonic pat-to-dak, which can perform self-examination, but this does not reach the level where users simply respond to questionnaires and see the results. no, the effect is insignificant. Accordingly, research and development of a platform that allows users to analyze and understand mental health status using only the sensor of a smartphone is required, even if the user does not build an infrastructure of hardware or software and does not respond to a questionnaire.

An embodiment of the present invention uses a GPS, a touch sensor, a gyro sensor, an accelerometer, an illuminance sensor, and a microphone that are basically installed in a smartphone to determine the user's activity amount, exercise amount, mobile phone usage, data usage, sleep time and Based on the experiment in which the correlation between at least one sensor data and the DSM-5 diagnostic standard (5th edition of Diagnostic and Statistical Manual of Mental Disorder) was significant, the sensor data and By using the correlation between DSM-5, scores for such measures as social activity, emotional factor, exercise amount, social interest, sleep time and stable surroundings are given and displayed in a visualized graph. It is possible to provide a method of providing a mental health measurement service using a smart phone sensor, which allows individuals or countries to manage their mental health status by notifying individuals or countries. However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention uploads at least one sensor data measured by at least one sensor in real time or periodically, and is measured based on the at least one sensor data. A user terminal for receiving a report on the mental health state, a storage unit for mapping and storing at least one reference sensor data and at least one mental health measurement data, and at least one sensor data received from the user terminal to receive at least one A comparison unit for comparing with reference sensor data, an extraction unit for extracting mental health measurement data previously mapped to at least one reference sensor data including at least one sensor data of the user terminal, and the extracted mental health measurement data to the user terminal It includes a mental health measurement service providing server including a delivery unit for transmitting to.

According to any one of the above-described problem solving means of the present invention, each individual's mental health state is automatically detected using GPS, touch sensor, gyro sensor, acceleration sensor, illuminance sensor, microphone, etc. which are basically installed in a smartphone. There is no need to fill out a separate questionnaire, interview with a psychiatrist, or purchase additional hardware or software. It can provide the basis for the establishment of prevention strategies and the introduction of psychological vaccine programs, and can lay the foundation for the provision of customized services for each class and target.

1 is a view for explaining a system for providing a mental health measurement service using a smart phone sensor according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a mental health measurement service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a mental health measurement service using a smart phone sensor according to an embodiment of the present invention is implemented.
5 is an operation flowchart illustrating a method of providing a mental health measurement service using a smart phone sensor according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

The terms "about", "substantially", etc. to the extent used throughout the specification are used in a sense at or close to the numerical value when the manufacturing and material tolerances inherent in the stated meaning are presented, and serve to enhance the understanding of the present invention. To help, precise or absolute figures are used to prevent unfair use by unscrupulous infringers of the stated disclosure. As used throughout the specification of the present invention, the term “step for (to)” or “step for” does not mean “step for”.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' refers to components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

In this specification, some of the operations or functions described as being performed by the terminal, apparatus, or device may be performed instead of in a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a system for providing a mental health measurement service using a smart phone sensor according to an embodiment of the present invention. Referring to FIG. 1 , the mental health measurement service providing system 1 using a smart phone sensor may include at least one user terminal 100 and a mental health measurement service providing server 300 . However, since the mental health measurement service providing system 1 using the smart phone sensor of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to the mental health measurement service providing server 300 through the network 200 . In addition, the mental health measurement service providing server 300 may be connected to at least one user terminal 100 through the network 200 .

Here, the network refers to a connection structure in which information exchange is possible between each node, such as a plurality of terminals and servers, and an example of such a network includes a local area network (LAN), a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi (Wi-Fi) , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( Near-Field Communication) networks, satellite broadcast networks, analog broadcast networks, Digital Multimedia Broadcasting (DMB) networks, and the like are included, but are not limited thereto.

In the following, the term at least one is defined as a term including the singular and the plural, and even if at least one term does not exist, each element may exist in the singular or plural, and may mean the singular or plural. it will be self-evident In addition, that each component is provided in singular or plural may be changed according to embodiments.

The at least one user terminal 100 transmits the sensor data measured by at least one sensor using a mental health measurement service related web page, an app page, a program or an application using a smart phone sensor to the mental health measurement service providing server 300 ) may be a terminal that transmits In addition, the user terminal 100 may be a terminal that receives and outputs a report or data on a mental health state from the mental health measurement service providing server 300 .

Here, the at least one user terminal 100 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100, for example, as a wireless communication device that guarantees portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smart phone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

The mental health measurement service providing server 300 may be a server that provides a mental health measurement service web page, an app page, a program, or an application using a smart phone sensor. In addition, the mental health measurement service providing server 300 may be a server that maps and stores at least one reference sensor data and mental health measurement data. In addition, the mental health measurement service providing server 300 collects sensor data from at least one user terminal 100 and compares it with reference sensor data to determine whether it is within a normal range, and mental health measurement data therefor It may be a server that outputs and delivers to the user terminal 100 .

Here, the mental health measurement service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop.

2 is a block diagram for explaining a mental health measurement service providing server included in the system of FIG. 1, and FIG. 3 is an embodiment in which a mental health measurement service using a smart phone sensor according to an embodiment of the present invention is implemented. It is a figure for demonstrating an example.

Referring to FIG. 2 , the mental health measurement service providing server 300 may include a storage unit 310 , a comparison unit 320 , an extraction unit 330 , a transmission unit 340 , and a modeling unit 350 . .

Mental health measurement service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 is a mental health measurement service application, program using a smart phone sensor. , when transmitting an app page, a web page, etc., at least one user terminal 100 may install or open a mental health measurement service application, program, app page, web page, etc. using a smart phone sensor. In addition, a service program may be driven in at least one user terminal 100 using a script executed in a web browser. Here, the web browser is a program that enables the use of a web (WWW: World Wide Web) service, and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, and the like. In addition, the application means an application on the terminal, for example, includes an app (App) executed in a mobile terminal (smartphone).

Referring to FIG. 2 , the storage unit 310 may map and store at least one reference sensor data and at least one mental health measurement data. At this time, the at least one sensor includes a GPS sensor for measuring the moving distance of the user terminal 100 , a touch sensor for measuring the number of times the screen of the user terminal 100 is turned on, and the number of steps of the user having the user terminal 100 . A gyro sensor and an accelerometer sensor for measuring 100) may include a microphone for measuring the noise of the located environment.

The at least one reference sensor data is 10 km to 50 km when the total moving distance is expressed in Km for 24 hours by the GPS sensor, 50 to 90 times when the total number of turns on in the touch sensor is counted for 24 hours, and a gyro sensor and If the accelerometer counts the total number of pedometers for 24 hours, it shows 3300, the data usage measured by the data usage meter for 24 hours is 307 to 587 Mbyte, and the touch sensor measured by the touch sensor and the light sensor The time when the measured illuminance is less than the preset illuminance may be 7 to 8 hours for 24 hours, and the number of decibels (dB) of noise input into the microphone may be 6 to 12 times for 24 hours less than 30 dB.

At least one mental health measurement data mapped with at least one reference sensor data and stored is a community corresponding to a social activity based on a movement distance measured through the GPS of the user terminal 100, the user terminal 100 The effect corresponding to the emotional factor based on the number of times the screen is turned on, measured using the touch sensor, the amount of exercise corresponding to the number of steps measured by the gyro sensor and the acceleration sensor of the user terminal 100 (Excercise), the user terminal Mood including social interest and interest based on the data usage used in 100, sleep based on the user's sleep time measured by the touch sensor and illuminance sensor of the user terminal 100, and the user terminal ( 100) may be data including at least one scale including a stable corresponding to a stable environment measured by a micrometer.

At least one scale constituting at least one mental health measurement data is based on the DSM-5 diagnostic criteria (5th edition of Diagnostic and Statistical Manual of Mental Disorder), if two or more scales out of a total of six scales do not exist in the normal range. In some cases, it may be configured to be marked as a mental health disorder. For example, when there are two or more DSM-5 depending on the type of mental illness, that is, depression, anxiety, smartphone addiction, schizophrenia, bipolar disorder, etc., it may be possible to predict the diagnosis of various diseases. For example, in the case of depression, it can be predicted as a high-risk group for depression when there are 5 or more in a total of 6 sensor scales.

At this time, DSM-5 announced the diagnostic criteria for major depressive disorder as shown in Table 1 below. In an embodiment of the present invention, it is experimentally proven that there is a significant correlation between at least one sensor data capable of measuring symptoms indicative of major depressive disorder and symptoms of major depressive disorder, and, accordingly, at least one sensor data It is used to make a diagnosis according to the diagnostic criteria for major depressive disorder.

DSM-5 Criteria for Major Depressive Disorder depressed mood Most of the day, nearly every day; may be subjective (e.g. feels sad, empty, hopeless) or observed by others (e.g. appears tearful); in children and adolescents, can be irritable mood Loss of interest/pleasure Markedly diminished interest/pleasure in all (or almost all) activities most of the day, nearly every day; may be subjective or observed by others Weight loss or gain Significant weight loss (without dieting) or gain (change of >5% body weight in a month), or decrease or increase in appetite nearly every day; in children, may be failure to gain weight as expected Insomnia or hypersomnia Nearly every day Psychomotor agitation or retardation Nearly every day and observable by others (not merely subjectively restless or slow) Fatigue Or loss of energy nearly every day Decreased concentration Nearly every day; may be indecisiveness; may be subjective or observed by others

In one embodiment of the present invention, the terms of the reference sensor data to be databased and the sensor data measured and uploaded by the user terminal 100 are distinguished. In the former case, it is a set reference value for measuring the user's state, and the latter is the user's value to be compared. In the former case, the value may be adjusted as data is accumulated, or it may be changed by addition or update. Table 2 below summarizes the mental health meanings of the above-described at least one sensor and reference sensor data.

Measure How to measure App notation Measure factor medium Community GPS Total distance traveled in 24 hours in km social activity 10km~50km Affect touch sensor Display the total number of turns on in 24 hours emotional factors 50-90 reps Excercise gyro sensor
& Accelerometer Displays the total number of pedometers in 24 hours Momentum, social distance 3300 steps Mood data usage Total data usage per day social interest and interest 307 to 587 Mb Sleep touch sensor
& Light sensor Number of deep sleep modes sleep time, sleep quality 7-8 hours stable MIC less than 30 dB stable ambient conditions 8-12 reps

If 4 or more points out of 5 scales (corresponding to 4 or more) are higher, it may be determined as a mental abnormality (depressive disorder). The comparator 320 receives at least one sensor data received from the user terminal 100 . Thus, it can be compared with at least one reference sensor data. In this case, the user terminal 100 may upload at least one sensor data measured by at least one sensor in real time or periodically.

In this case, Tables 3 to 7 are tables showing interpretations of data collected from six sensors. Of course, since each numerical value is based on a pilot test result, it may be updated or changed in the future, so it is not limited to the corresponding numerical value.

Red signal result value One low social vitality
decreased social interest 0~3Km 2 2 low social activity 3-10km One 3 Average Step - 10km ~ 60km 0 4 high activity stage 60km to 100km One 5 high level of physical fatigue 100km ~ 2

Table 3 shows the data based on the Co scale, that is, the distance traveled by GPS. As a fixed explanation, analysis of social communication is being carried out, and when the score is high, appropriate social relationships are maintained, but when the score is low, it is perceived as lack of energy, narrow social relationships, lack of communication skills, and lack of intimacy It can be stated that there seems to be a trend. At this time, the explanation for the change according to the numerical value is, 1- Presently in the low social vitality, socially diminished stage, 2- Presently in the low social activity stage, 3- Presently in the average stage- It can be described as, 4- Presently appears to be in the High Activity stage, 5- Presently appears to be in the High Physical Fatigue stage.

Baseline estimation Red signal result value One low emotional vitality 0 to 20 2 2 Adequate emotional control 20 to 50 One 3 average step 50 to 90 0 4 stage of impatience 90 to 120 One 5 Steps to consider smartphone dependence 120 or more 2

Table 4, that is, the af scale is based on the number of times the screen is turned on. As a fixed explanation, it is a measure of emotional state. If the score on this scale is high, you may prefer to do several things at once, or you may have a fast rotation of thoughts and low concentration on one task. On the other hand, if this scale is low, it can be described as can be considered as experiencing a lack of emotional vitality, an explanation of the variation according to the number, 1- presently in a low emotional vitality stage, 2- presently adequate emotional control Appears to be in stage, 3- Current average stage - Appears to be in stage, 4- Presently appears to be in impatient stage, 5- Presently appears to be in Consideration stage for smartphone dependence have.

Baseline estimation Step count/day Red signal result value One optimal exercise time 4,710 ~ 0 2 healthy exercise time 2,355 ~ 4,710 0 3 appropriate exercise time 942 to 2,355 0 4 low exercise time 471 to 942 One 5 low activity less than 471 2

Table 5 is based on the Ex scale, that is, the moving distance. In the fixed description, you can check the basic amount of physical exercise. If the amount of exercise is low, it can be considered that there will be negative emotional experiences such as lethargy and sensitive mood as well as negative effects on physical health. 2- It appears to be in current health exercise time stage, 3- Presently appears to be in moderate exercise time stage, 4- Presently appears to be in low exercise time stage, 5- Present low activity state It can be described as appearing to belong to a stage.

Baseline estimation Data usage per day (MB) Red signal result value One lack of social interest 0 to 34 2 2 Offline Interest Preference 34 to 307 One 3 average step 307 to 887 0 4 Online Interest Preference 887 to 1,024 One 5 Consider your smartphone dependence 1,024 ~ 2

Table 6 is based on data usage to measure the Mo scale. A fixed explanation can be considered when the score is low and the interest in social relationships is limited. On the other hand, when the score is high, it can be described as Considering excessive use of smartphones. Variable explanations by number are: 1- Appears to be currently in Social Interest Lack stage, 2- Presently appears to be in Offline Interest Preference stage, 3- Present Average Stage - Appears to be in stage, 4- Presently online Appears to be in the Interest Preference stage, 5- Can be described as currently in the Consideration stage for smartphone dependence.

Baseline estimation ref reference Red signal result value One Steps to consider insomnia ~6 hours or less 2 2 below average level More than 6 hours to less than 7 hours One 3 proper sleep time More than 7 hours to less than 8 hours 0 4 average award level More than 8 hours to less than 9 hours One 5 tendency to oversleeping ~9 hours or more 2

Table 7 is the Sl scale and is based on the number of deep slips. Fixed explanations, low scores can keep you awake at night or make it hard to fall asleep and wake you up easily even if you fall asleep. You may have difficulty waking up in the morning, and you may experience unresolved fatigue due to lack of sleep, which may lead to discomfort at work, sometimes irritability, and mood swings. On the other hand, if the score is high, it can be described as poor quality of sleep or it is necessary to suspect a sign of disease. can be described as belonging to the level stage, 3- currently appearing in the moderate sleep time stage, 4- currently appearing in the upper average level stage, 5- currently appearing in the hypersomnia tendency stage have.

There is a possibility that he will only live indoors as he has not been able to travel outside due to the recent corona virus. In this case, it is not possible to measure the moving distance because the GPS signal is not well captured. In this case, a method of estimating the distance moved indoors based on deep learning using a smartphone sensor may be further used.

<Estimation of indoor travel distance>

Using GPS, the user walks outdoors in two scenarios, measuring the accelerometer, gyro sensor and GPS every second. The first of the two scenarios may mean a situation in which you hold a cell phone in your hand and look at the screen, and a situation in which you shake the cell phone in your hand. In order to minimize the signal error, the Kalman Filter Algorithm is applied, and the average speed and movement time can be calculated by learning with CNN, and the movement distance can be predicted.

The signals from the smartphone's GPS, 3-axis accelerometer, and 3-axis gyro sensor automatically collected while walking outdoors can be used as learning data. For data collection, the sensor signal can be measured and saved in the form of a log file. If an error occurs due to a message or notification that occurs while collecting data, this can also be reflected in the data. The accelerometer and gyro sensor can measure at a sampling rate of 100Hz, and use GPS to calculate the speed every second. The velocity may be calculated using the displacement between the current position and the previous position, and may be arranged as in Equation 1 below.

Here, d(t) is displacement, v(t) is velocity, and xt and yt of 1ot are GPS latitude and longitude values according to time. In order to minimize the error of the accelerometer signal, it can be corrected by applying a Kalman filter, and after removing the effect of gravity, it can be converted to the earth coordinate system. The error can be minimized by applying the Kalman filter to the gyro sensor and GPS.

The next process is a process of learning through a neural network using data collected outdoors. The dataset used for training can consist of signals from the accelerometer and gyro sensors calibrated at each time t (1 second), and the speed (m/s) obtained using the calibrated GPS can be labeled. Neural network models can use regression to predict speed. In addition, features can be extracted and learned using ID-CNN (Intelligent Detection Using Convolutional Neural Network) and Max Pooling. The optimizer can use regression analysis using RMSprop, in which case the loss function can use mse. It may be desirable not to use the activation function because it limits the range of the output value.

In general, to evaluate the model, data is divided into a training (training) set and a test set. However, when the number of data is insufficient, reliable model evaluation is difficult. In this case, K-fold cross-validation can be used to split the data into K pieces, and a method can also be used to create K models and train them in K-1 partitions to evaluate them in the remaining partitions. .

<Measurement of sleep time>

For sleep time estimation, GPS, sound, and screen state data can be collected at a 10Hz cycle. If it is determined that the GPS data is outdoors, it may not be determined as sleep time even if it is stationary. If a GPS signal is not obtained, it can be assumed that it is indoors. The screen state data can be recorded as 1 when it is ON and 0 when it is OFF by checking whether the screen of the smartphone is in an ON state or an OFF state. If the screen is on, it can be assumed that the user is using the smartphone and not sleeping. Sound data can be recorded by measuring the sound pressure around the smartphone and converting it to dB. The decibel (dB) is a unit representing the ratio of two values, and 2x10 ^-5 [Pa], which is the minimum assumed value of 1 kHz, can be used as a reference value (0 dB).

In order to estimate the sleep time using the collected GPS, screen state, and sound data, a method of scoring each data value may be used. If the GPS accuracy data is outdoors, it is determined that it is active, and the total score is multiplied by 0 to make the score 0.

When the screen is ON, the user determines that the user is active and gives 0 points, and when the screen is OFF, it determines that the user is sleeping and gives 50 points.

Sound data judges that the higher the decibel, the more difficult it is to sleep, and the higher the decibel, the lower the score out of 50.

Equation 5, which is a total of 100 points, can be summarized by synthesizing the three pieces of data.

The range of score change for a short period of time may be large due to the screen turning on due to an alarm or a noise that appears briefly. Accordingly, a Moving Average Filter (MAF) may be used to confirm the trend of the score change. The score distribution of sleep time and activity time is different, and in order to distinguish them, a reference score can be obtained as in Equation (6).

It is the average of the minimum score (IS.T) in sleep time and maximum score (IW.T) in active time. Using the average of the minimum score of sleep time and the maximum score of active time, it is possible to obtain a score that can estimate whether or not sleep has occurred. Since there are only three types of data that can be obtained from a smartphone without using a separate device, it can be assumed that if the smartphone is not used in a quiet environment, it is not in a sleep state. If multiple data are obtained through a smartphone and Ith is obtained through data of multiple days rather than one day, the sleep time can be estimated a little more accurately.

<Time-series behavior recognition>

In an embodiment of the present invention, a method for recognizing human behavior by configuring a deep learning-based deep trust neural network by reflecting the characteristics of a time section using data from a smartphone sensor may be further used. It can be applied to a recognizer based on a Deep Belief Network by using the data from the acceleration sensor and the gyroscope sensor among the sensor information that is widely available in smartphones.

Human behavior is not defined as a snapshot captured in an instant, but as a continuous stream of behavior over time. Therefore, it is possible to reflect the continuous flow of actions using the accumulated data for a specific time. In the flow of time, a deep trust neural network can be hierarchically constructed by reflecting short time intervals and long time intervals, and can be configured to increase stability. Pretraining using a Restricted Boltzmann Machine sets the initial value of the network that avoids the local minimum, and then fine tuning can improve the recognition performance of the network.

To this end, first, accumulated data for the input of the acceleration sensor and the gyro sensor is collected from the user terminal 100, and the limited Boltzmann machine is hierarchically configured with various time intervals for the input as the input. to learn each. They are again constituted by a limited Boltzmann machine to form a hidden layer to learn, and finally, by fine tuning, human walking, upstairs, downstairs, sitting, and standing It is possible to generate recognition results for actions such as (Standing) and lying down (Laying Down).

Smartphones include Light Sensor, Fingerprint Sensor, Image Sensor, Touch Sensor, GPS Sensor, Geo-magnetic Sensor, and Acceleration Sensor. , a gyroscope sensor, and the like exist. In this case, among various sensors, an acceleration sensor and a gyroscope sensor, which are motion sensors, may be used. The acceleration sensor is a sensor that detects and measures the speed change of a moving object or the strength of an impact. It is a sensor that shows the speed increase/decrease ratio in a linear direction within a certain time.

The acceleration sensor consists of three axes, and the X axis is the acceleration acting on the side of the smartphone. The Y axis is the acceleration acting on the upper and lower surfaces, and has positive values above and negative values below. The Z axis is the acceleration acting on the front and back surfaces of the smartphone, with a positive value in front of the display and a negative value behind the display. The gyroscope sensor is a sensor that informs the direction of rotation according to the angular velocity, which is the rotational speed of an object. It is a sensor that numerically indicates the angle of rotation of an object about one axis within a certain period of time. It is calculated by converting the Coriolis force generated when an object rotates into an electrical signal. The rotation around the X axis is called pitch, the rotation around the Y axis is called Roll, and the rotation around the Z axis is called Yaw.

A Deep Belief Network (DBN) consists of a combination of Restricted Boltzmann Machines. A deep trust neural network has each layer, and each layer has a node, the smallest unit in which data exists. The connection between each node is called a weight. In this case, the two adjacent layers are called one limited Boltzmann machine. The input layer consists of 120 nodes of 6*20. As for the data, three accelerometer data and three gyro sensor data are collected at one point in time, and in order to recognize the flow according to time change, it can be configured as one section by grouping n times. The hidden layer is composed of limited Boltzmann machines, and the entire hidden layer is not composed of one limited Boltzmann machine, but may be divided into sections.

The training of deep trust neural networks is divided into pre-learning and fine-tuning stages. In the pre-learning stage, the initial values of all weights of the deep trust neural network are configured, and unsupervised learning is performed on the given input data. . The process of learning the deep trust neural network to fit the target value well after configuring the initial value is called fine tuning.

The Restricted Boltzmann Machine (RBM) is an algorithm that can be used for dimensionality reduction, classification, linear regression analysis, and collaborative filtering, and was proposed by Geoff Hinton. Limited Boltzmann machines can be used on their own, but are also used to help train other neural networks. A restricted Boltzmann machine is a single-layer neural network with one input layer and one hidden layer, also called the visible layer. The limited Boltzmann machine consists of a weight matrix connecting the input node of the input layer and the hidden node of the hidden layer, and bias weights exist in the input layer and the hidden layer. The energy function using this configuration is based on the probability distribution between the hidden node and the input node.

Since the input nodes of the limited Boltzmann machine are independent of each other, whether the input nodes are activated by the activation of the hidden nodes is independent of each other, and likewise, whether the hidden nodes are activated by the activation of the input nodes is independent of each other. Learning of a constrained Boltzmann machine is directed toward maximizing the conditional probability of the training set. For this purpose, a widely used algorithm is Hinton's CDA (Contrastive Divergence) algorithm. Learning uses Gibbs Sampling and Gradient Descent techniques, and weights and biases can be calculated by the CDA algorithm.

Fine tuning is the task of modifying the weights and biases after going through a pre-learning step of setting the initial weights of all limited Boltzmann machines in the network. In this process, supervised learning can be performed by comparing the target value with the results of the neural network by setting the human behavior as the target value for each time point with the Error Back Propagation algorithm. In the error backpropagation algorithm, weights can be updated using stochastic gradient descent.

The extraction unit 330 may extract mental health measurement data previously mapped to at least one reference sensor data including at least one sensor data of the user terminal 100 and stored therein.

The transmitting unit 340 may transmit the extracted mental health measurement data to the user terminal 100 . The user terminal 100 may receive a report on the mental health state measured based on at least one sensor data.

The modeling unit 350 builds big data by accumulating a dataset based on a correlation between at least one sensor data measured from a plurality of user terminals 100 and mental health measurement data, and at least one Modeling can be performed by learning and testing different kinds of artificial intelligence algorithms as datasets. In this case, the user terminal 100 may be a plurality of user terminals 100 .

Hereinafter, an operation process according to the configuration of the mental health measurement service providing server of FIG. 2 will be described in detail with reference to FIG. 3 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3A , it is possible to analyze the user's overall life pattern and evaluate the psychological state using the sensor built into the smartphone, so that it is possible to measure mental health more accurately and conveniently. Referring to FIG. 3B , the mental health state is comprehensively analyzed by checking the amount of physical activity, social relationship, and psychological emotional state of an individual using a GPS sensor, a gyro sensor, an acceleration sensor, an illuminance sensor, and a microphone as a method of measuring the psychological state. can be evaluated Referring to Figure 3c, from the user's point of view, his mental health state is analyzed and shown into three categories: the degree of deviation from the average of the general public, the current state of the individual, and the average score of the individual. It may be designed to confirm and display recognition. In addition, psychological test scores with proven reliability and validity, current physical health status questionnaires, and measured data are stored together so that big data can be utilized.

<Example>

Referring to FIG. 4A, in one embodiment of the present invention, psychiatrists, researchers from the Seoul National University Sports Medicine Research Institute, and mental health center specialists gather together to increase the reliability and validity of the test and collaborate with each other to maximize the value of use. proceeded The Seoul National University Sports Medicine Research Institute established a value for how to define the values of the sensors because the smart phone sensor has to share the user's body contact and activity. The correlations of the papers were identified, and the measurable factors, reliability, and validity were established. Mental health center specialists arranged the pros and cons of tests used to discover actual subjects, and advised on how to use them. A pilot operation was conducted for 30 general people and 15 people with depression, and the current application was updated based on the results.

4B is a diagram showing which smart phone sensor will receive the input value for the diagnostic criterion of the DSM-5. It is a method of measuring the factors necessary for the diagnostic criteria for major depressive disorder of the DSM-5 through a sensor, and through this, a more accurate diagnosis can be made. And, it is shown in the same way as in Fig. 4c. During pilot operation, the internal validity of the relevant expert (advised by 5 psychiatrists) app was high, and the correlation between the sensor and the DSM-5 diagnosis when the actual related data was confirmed appeared significantly. Additionally, when using the app, the app was designed to measure major depressive disorder, depression, and suicide. The contents of FIG. 4d additionally explain the physical and mental health measurement hypothesis.

As shown in Figure 4e, AI algorithm is modeled by accumulating data on the correlation between depression, job stress, fluctuating disease management evaluation, and other mental health/physical health and smartphone sensors, and based on sensor data with proven reliability and validity By publishing a standardization work and interpretation book, it is possible to prepare a foundation that can be used in the local community through collaboration with public institutions in the future, and finally, an educational curriculum can be provided to enhance usability.

<Experimental example>

group of experts number gender age group career note psychiatrist One male 54 29 years Hyundai Hope Medical Foundation (Hospital) 2 male 50 20 years Voice Hope Medical Foundation (Hospital) 3 male 50 25 years Voice Hope Medical Foundation (Hospital) 4 male 40 22 years Voice Hope Medical Foundation (Hospital) 5 male 50 20 years Voice Hope Medical Foundation (Hospital) 6 male 40 12 years Voice Hope Medical Foundation (Hospital) 7 male 50 24 years Hyundai Hope Medical Foundation (Hospital) 8 male 70 40 years Voice Hope Medical Foundation (Hospital) mental health clinical psychologist 9 male 35 8 years Hyundai Hope Medical Foundation (Hospital) 10 female 35 9 years Voice Hope Medical Foundation (Hospital) medical social worker 11 male 50 30 years Voice Hope Medical Foundation (Hospital)

<Content validity analysis>

Content validity shows how well the evaluation index represents the content to be measured. C VR is calculated as the proportion of the panel who answered ‘appropriate’ to each question, and can be calculated by applying the calculation formula developed by Lawshe (1975) as follows. Here, N is the total number of experts, Ne is the frequency of experts who answered ‘relevant’, that is, the number of experts who answered 3 (relevant) or higher on a 4-point Likert scale. When the content validity ratio (CVR) of each index is greater than or equal to the minimum value according to the number of experts suggested by Lawshe (1975) at the 95% confidence level (p<.05), the index is judged to have secured content validity. Since the number of experts in this study is 11, 0.59 corresponds to the minimum content validity ratio.

Number of experts (persons) 5 6 7 8 9 10 11 12 13 14 15 20 CVR Minimum .99 .99 .99 .78 .75 .62 .59 .56 .54 .51 .49 .42

<Agreement & Convergence Analysis>

The degree of agreement between the opinions of the expert panel was examined by the degree of agreement and the degree of convergence. The degree of agreement and the degree of convergence are calculated by applying Equation 8 below, where Mdn is the median value, and Q1 and Q3 are the first and third quartile coefficients, respectively, of 25% and 75% of the cumulative values of the total number of cases. value (Kyung-Yeol Jung, Yoo-Hyeon Choi, 2011). A previous study in which the opinion of the expert panel was interpreted as reaching an agreement when the degree of agreement was greater than .75 and the degree of convergence was less than or equal to .50 (Lee Han-kyu, 2006; Shin Yeon-ju, Kim Seon-tae, Song Ha-deok, 2018; Jeong, Gyeong-yeol) , Yoohyeon Choi, 2011).

<Research Results>

In the evaluation index validity verification, only items with content validity ratio (CVR), agreement, and convergence appropriate to the standard values among the total items were composed, and the results are as follows.

PHQ-9 Questions Questionnaire
(DSM-5 Diagnostic Criteria for Major Depressive Disorder) Utilization sensor agreement Convergence CVR 1. Little interest or fun in doing work GPS movement 0.83 0.25 1.00 number of touches 1.00 0.00 1.00 Data usage 1.00 0.00 0.82 Deep sleep time 0.75 0.25 0.64 number of steps 0.50 0.50 1.00 Decibel noise measurement 0.00 0.50 -0.27 2. Feeling subdued, depressed or hopeless GPS movement 0.67 0.50 0.82 number of touches 0.75 0.25 0.45 Data usage 1.00 0.00 0.82 Deep sleep time 1.00 0.00 0.82 number of steps 0.50 0.50 1.00 Decibel noise measurement 0.75 0.25 0.45 3. Difficulty falling asleep, waking up, or sleeping too much GPS movement 0.75 0.25 0.64 number of touches 0.75 0.25 0.64 Data usage 0.75 0.25 0.45 Deep sleep time 0.83 0.25 1.00 number of steps 1.00 0.00 0.82 Decibel noise measurement 0.75 0.25 1.00 4. Tiredness, low energy GPS movement 0.50 0.50 1.00 number of touches 0.50 0.50 0.27 Data usage 0.75 0.25 0.64 Deep sleep time 0.75 0.25 0.64 number of steps 0.83 0.25 1.00 Decibel noise measurement 0.00 1.00 0.27 5. Loss of appetite or overeating GPS movement 0.50 0.50 0.27 number of touches 1.00 0.00 -0.64 Data usage 0.50 0.25 -0.45 Deep sleep time 0.50 0.50 0.27 number of steps 0.50 0.50 0.09 Decibel noise measurement 1.00 0.00 -0.64 6. Feeling that I am a bad person or that I am a failure or that I or my family are unhappy because of me GPS movement 0.50 0.50 0.09 number of touches 0.00 0.50 -0.09 Data usage 0.00 0.50 -0.09 Deep sleep time 1.00 0.00 0.64 number of steps 1.00 0.00 0.64 Decibel noise measurement 0.00 0.50 -0.27 7. Difficulty concentrating while reading the newspaper or watching TV GPS movement 0.50 0.50 0.27 number of touches 0.83 0.25 0.82 Data usage 0.50 0.50 0.82 Deep sleep time 1.00 0.00 1.00 number of steps 0.50 0.50 0.09 Decibel noise measurement 0.50 0.50 0.27 8. Slow to move or speak enough to be noticed by others, or, conversely, move around and hang around more than usual because of being too impatient and restless GPS movement 0.50 0.50 1.00 number of touches 0.83 0.25 1.00 Data usage 1.00 0.00 0.82 Deep sleep time 0.75 0.25 0.45 number of steps 0.75 0.25 0.82 Decibel noise measurement 0.50 0.50 0.09 9. Thoughts that I would rather die, or thoughts of self-harm in any way GPS movement 0.75 0.25 0.45 number of touches 0.75 0.25 0.45 Data usage 0.75 0.25 0.45 Deep sleep time 0.50 0.50 0.45 number of steps 0.50 0.50 0.27 Decibel noise measurement 0.75 0.25 0.45

PHQ-9 Questions Questionnaire
(DSM-5 Diagnostic Criteria for Major Depressive Disorder) Evaluation Utilization Sensor 1. Little interest or fun in doing work GPS, Touch , Used Data, Deep sleep mode 2. Feeling subdued, depressed or hopeless Used Data, Deep sleep mode 3. Difficulty falling asleep, waking up, or sleeping too much GPS , Touch , Deep sleep , Gyro sensor, Decibel 4. Tiredness, low energy Used Data , Deep sleep mode, Gyro sensor 5. Loss of appetite or overeating - 6. Feeling that I am a bad person or that I am a failure or that I or my family are unhappy because of me Deep sleep mode, Gyro sensor 7. Difficulty concentrating while reading the newspaper or watching TV Touch , deep sleep mode 8. Slow to move or speak enough to be noticed by others, or, conversely, move around and hang around more than usual because of being too impatient and restless Touch, Used Data, Gyro sensor 9. Thoughts that I would rather die, or thoughts of self-harm in any way -

The matters not described with respect to the method of providing a mental health measurement service using a smartphone sensor of FIGS. 2 to 4 are the same as those described for the method of providing a mental health measurement service using a smartphone sensor through FIG. 1 or Since it can be easily inferred from the described content, the following description will be omitted.

5 is a diagram illustrating a process in which data is transmitted/received between components included in the system for providing a mental health measurement service using the smart phone sensor of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each component will be described with reference to FIG. 5, but the present application is not limited to such an embodiment, and the example shown in FIG. 5 according to the various embodiments described above will be described. It is apparent to those skilled in the art that the data transmission/reception process may be changed.

Referring to FIG. 5 , the mental health measurement service providing server maps and stores at least one reference sensor data and at least one mental health measurement data ( S5100 ).

And the mental health measurement service providing server receives at least one sensor data received from the user terminal and compares it with at least one reference sensor data (S5200), and at least one criterion including at least one sensor data of the user terminal The mental health measurement data previously mapped to the sensor data is extracted (S5300). In addition, the mental health measurement service providing server transmits the extracted mental health measurement data to the user terminal (S5400).

The order between the above-described steps (S5100 to S5400) is merely an example, and is not limited thereto. That is, the order between the above-described steps S5100 to S5400 may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters not described with respect to the method of providing a mental health measurement service using a smartphone sensor of FIG. 5 are the same as those described for the method of providing a mental health measurement service using a smartphone sensor through FIGS. 1 to 4 above, or Since it can be easily inferred from the described content, the following description will be omitted.

The method of providing a mental health measurement service using a smart phone sensor according to an embodiment described with reference to FIG. 5 is also in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. can be implemented. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a mental health measurement service using a smart phone sensor according to an embodiment of the present invention described above includes an application basically installed in a terminal (which may include a program included in a platform or an operating system basically installed in the terminal) may be executed by the application store server, an application or an application (ie, a program) directly installed in the master terminal through an application providing server such as a web server related to the application or the corresponding service by the user. In this sense, the method for providing a mental health measurement service using a smart phone sensor according to an embodiment of the present invention described above is implemented as an application (ie, a program) installed by default in a terminal or directly installed by a user, and is provided in the terminal. It may be recorded on a computer-readable recording medium.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

An embodiment of the present invention can be used not only for personal use, but also can be used to immediately check member management and crisis situations of mental health institutions including mental health centers, suicide prevention centers and dementia centers through a server. In addition, although the primary goal is to diagnose major depressive disorder, it can be used for diagnosis and prediction of various psychiatric disorders, including depression, using more than 6 sensors later. Accordingly, an embodiment of the present invention may predict not only major depressive disorder, but also diagnosis of mental disorders including depression.

Claims (6)

a user terminal for uploading at least one sensor data measured by at least one sensor in real time or periodically, and receiving a report on a mental health state measured based on the at least one sensor data; and
A storage unit that maps and stores at least one reference sensor data and at least one mental health measurement data, a comparison unit that receives at least one sensor data received from the user terminal and compares it with the at least one reference sensor data, the An extractor for extracting mental health measurement data previously mapped to at least one reference sensor data including at least one sensor data of the user terminal and extracting the stored mental health measurement data, and a delivery unit for delivering the extracted mental health measurement data to the user terminal Mental health measurement service providing server; including,
the at least one sensor,
a GPS sensor for measuring the moving distance of the user terminal;
a touch sensor for measuring the number of times the screen of the user terminal is turned on;
a gyro sensor and an acceleration sensor for measuring the number of steps of a user having the user terminal;
a data usage meter for measuring the data usage used by the user terminal;
a touch sensor and an illuminance sensor for measuring a sleep mode of a user having the user terminal; and
Including; a microphone for measuring the noise of the environment in which the user terminal is located;
The at least one reference sensor data is
When the GPS sensor expresses the total distance traveled in km for 24 hours, it is 10 km to 50 km,
When counting the total number of turns on for 24 hours in the touch sensor, it is 50 to 90 times,
If the gyro sensor and the acceleration sensor count the total number of pedometers for 24 hours, it shows 3300,
The data usage measured for 24 hours by the data usage meter is 307Mbyte to 587Mbyte,
The time when the illuminance measured by the illuminance sensor is less than the preset illuminance while there is no number of touches of the touch sensor measured by the touch sensor and the illuminance sensor is 7 to 8 hours for 24 hours,
Mental health measurement service providing system using a smart phone sensor, characterized in that the number of decibels (dB) of the noise input into the microphone is less than 30 dB 6 to 12 times in 24 hours.
delete delete The method of claim 1,
At least one mental health measurement data that is mapped and stored with the at least one reference sensor data,
a community corresponding to a social activity based on a movement distance measured through the GPS of the user terminal;
an effect corresponding to an emotional factor based on the number of times the screen is turned on measured using the touch sensor of the user terminal;
an exercise amount corresponding to the number of steps measured by the gyro sensor and the acceleration sensor of the user terminal;
a mood including social interest and interest based on data usage used in the user terminal;
Sleep based on the user's sleep time measured by the touch sensor and the illuminance sensor of the user terminal;
a stable corresponding to the stable environment measured by the user terminal's microphone (Stable);
Mental health measurement service providing system using a smart phone sensor, characterized in that the data including at least one scale including a.
5. The method of claim 4,
At least one scale constituting the at least one mental health measurement data is based on the DSM-5 diagnostic standard (5th edition of Diagnostic and Statistical Manual of Mental Disorder), and 2 or more scales are within the normal range according to symptoms among a total of six scales. Mental health measurement service providing system using a smartphone sensor, characterized in that it is configured to be marked as mental health abnormality if it does not exist in the .
The method of claim 1,
The user terminal is a plurality of user terminals,
The mental health measurement service providing server,
Based on the correlation between the at least one sensor data measured from the plurality of user terminals and the mental health measurement data, a dataset is accumulated to build big data, and at least one kind of artificial intelligence algorithm is used as the data a modeling unit that performs modeling by learning and testing in sets;
Mental health measurement service providing system using a smart phone sensor, characterized in that it further comprises.

Images