KR102012804B1 - Method, apparatus and computer program for recognition of a user activity - Google Patents

Abstract

According to an embodiment of the present disclosure, a method of recognizing a user's behavior using sensing data may include generating sensing data in a user smartphone and a wearable device; In the server, the sensing data is collected and the collected data is applied to a machine learning framework, so that a first model of the smart phone generated data, a second model of the wearable device generated data, and the smart phone generated data Generating a third model integrating the wearable device generation data; And extracting a feature for each of the first model, the second model, and the third model to estimate user behavior for each of the first model, the second model, and the third model.

Description

METHOD, APPARATUS AND COMPUTER PROGRAM FOR RECOGNITION OF A USER ACTIVITY}

The present invention relates to a method of recognizing a user's behavior using sensor data collected from a mobile device such as a smart phone and / or a user device such as wearable diabis such as a smart watch. More specifically, the present invention weights a plurality of models for applying the machine learning framework to more accurately recognize a user's behavior from collected sensor information, and integrates a plurality of weighted models. An invention is directed to a method of determining user behavior from data.

Recently, with the rapid development of the IoT technology, a computing technology capable of more accurately recognizing a situation including recognition of a specific user's behavior from collected IoT data has been developed.

Situational Awareness Technology aims to more accurately grasp user's actions and movements mathematically without human involvement. In general, inertial sensors that attach sensors such as images, accelerations, etc. It can be implemented to collect a large amount of data using a thing sensor using a pressure, a motion sensor and the like and to apply the collected data to the machine learning framework to derive an estimation result.

Among them, in order to collect image data using a camera and sensor data using an object sensor, cameras, pressure sensors, and motion sensor devices should be installed in an arbitrary space. That is, in order to recognize a camera or object sensor-based action, the action may be recognized only in a space where the camera or object sensor is installed in advance. However, because the inertial sensor can be attached to the body of the person, regardless of the location, there is an advantage that can recognize the user's behavior.

Furthermore, when collecting user data based on an inertial sensor, data may be collected to attach a plurality of sensors to a body element part to more accurately recognize a user's behavior and / or situation. However, if this is applied to a real user, a plurality of sensors are attached to the user, which makes the user's movement inconvenient, thereby decreasing user convenience.

Therefore, although a sensor is not directly attached to a user's body, a method of collecting data for behavior recognition may be considered by using a smartphone equipped with various sensors and generally located at a recent distance from the user. Smartphones are equipped with a variety of sensors as well as computing power, has the advantage of high user convenience.

However, in the case of data collection using only a smartphone, there is a limitation in that it is impossible to collect data reflecting each characteristic of a body part that can perform various actions using only a single device, so that a wearable smart device such as a smart watch appeared with a smartphone can be considered. have.

The smart watch may be worn on the user's wrist by adopting a band-shaped design, but this may also be insufficient to collect data reflecting each feature of a body part.

Therefore, the present invention collects the sensing data using both wearable devices such as a smart phone and a smart watch, and the user behavior that can be estimated by integrating and estimating the data collected from each independent device and the data collected from each device. It aims at providing with high reliability.

To this end, according to an embodiment of the present invention, three cognitive models of a single smart phone, a single smart watch, and further, smart phone and smart water integration are generated, and weighting and integration of each model are used to achieve higher reliability. User behavior awareness technology can be provided.

An object of the present invention is to solve the above problems. More specifically, an object of the present invention is to provide a framework capable of estimating a user's behavior with higher reliability by using sensing data collected from a smart phone and / or a smart watch.

According to an embodiment of the present disclosure, a method of recognizing a user's behavior using sensing data may include generating sensing data in a user smartphone and a wearable device; In the server, the sensing data is collected and the collected data is applied to a machine learning framework, so that a first model of the smart phone generated data, a second model of the wearable device generated data, and the smart phone generated data Generating a third model integrating the wearable device generation data; And extracting a feature for each of the first model, the second model, and the third model to estimate user behavior for each of the first model, the second model, and the third model.

Furthermore, after estimating the user behavior, generating a weight table for applying to each of the first model, the second model and the third model; And applying the weight table to user behavior estimated through the first model, the second model, and the third model.

Furthermore, the generating of the weight table may include applying the same first weight to each of the behaviors estimated by the first model, the second model, and the third model; Among the estimated actions, applying a second weight higher than the first weight to an action equal to or less than a predetermined accuracy, and applying a third weight lower than the first weight to an action equal to or greater than a preset accuracy; And setting the weight to 0 for an action estimated through a device unrelated to the action and increasing the weight of the action estimated only through a specific device.

Furthermore, after the applying of the weight table, the method may include comparing priorities for actions having the same weight.

On the other hand, a method for recognizing a user's behavior using the sensing data according to an embodiment of the present invention, generating the sensing data in the user smart phone and wearable device; In the server, the sensing data is collected and the collected data is applied to a machine learning framework, so that a first model of the smart phone generated data, a second model of the wearable device generated data, and the smart phone generated data And generating a third model integrating the wearable device generation data, extracting features for each model, and estimating user behavior for each of the models; Generate a weight table for applying to each of the first model, the second model and the third model, the weight table to the user behavior estimated through the first model, the second model and the third model Applying; And comparing priorities for actions having the same weight.

Furthermore, a system for recognizing user behavior according to an embodiment of the present invention includes: a user smartphone including at least one sensor; A wearable device including at least one sensor and attached to the user; A data processing server for collecting and processing sensing data generated by the smart phone and the wearable device; And applying the data processed by the data processing server to a machine learning framework, to generate a first model of the smartphone generated data, a second model of the wearable device generated data, and the smartphone generated data and the wearable device. A weight table for generating a third model integrating the generated data, extracting features for each model, estimating user behavior for each of the models, and applying the same to each of the first model, the second model, and the third model. And a behavior recognition server for applying the weight table to the user behavior estimated through the first model, the second model, and the third model.

According to the present invention, there is an effect that the computer can more accurately recognize the user's actions without human intervention.

1 is a system configuration of collecting the sensing data from the smart phone and smart watch according to an embodiment of the present invention, and applying it to the machine learning framework
2 is a flowchart illustrating a process of recognizing a user's behavior according to an embodiment of the present invention.
3 is a flowchart illustrating a preprocessing process for collecting and modeling sensor data.

It is apparent that the present invention is not limited to the description of the embodiments described below, and various modifications may be made without departing from the technical gist of the present invention. In describing the embodiments, descriptions of technical contents which are widely known in the technical field to which the present invention belongs and are not directly related to the technical gist of the present invention will be omitted.

On the other hand, the same components in the accompanying drawings are represented by the same reference numerals. In the accompanying drawings, some components may be exaggerated, omitted, or schematically illustrated. This is to clarify the gist of the present invention by omitting unnecessary description that is not related to the gist of the present invention.

1 is a block diagram of a behavior recognition system collecting sensing data from a smart phone and a smart watch and applying the same to a machine learning framework according to an embodiment of the present invention.

In the behavior recognition system according to the embodiment of the present invention, as shown in FIG. 1, the smartphone 10, the wearable device 20, the sensing data collection and processing server 40, and the collected data are estimated. It may be configured to include a server 60 for providing a service using the behavior recognition server 50 and the behavior recognition result.

The smartphone 10 according to an embodiment of the present invention may perform a function of transmitting the collected sensing data to the sensing data collection and processing server 40 using the included sensor. To this end, the smartphone 10 may include a sensor for measuring a physical quantity of specific information, for example, position, motion, brightness, etc., from a measurement target and converting the measured amount into an electrical signal.

Sensors that may be included in a smartphone may exemplify an acceleration sensor, a gravity sensor, a gyro sensor, a temperature sensor, a pressure sensor, an illuminance sensor, a proximity sensor, a GPS sensor, a geomagnetic sensor, a direction sensor, and the like.

More specifically, the gyro sensor measures the rotational speed of the X, Y, and Z axes, thereby generating data that can be used to check information on the degree of tilting or rotation of the smartphone, and the temperature sensor is a temperature outside the smartphone. Data can be measured. In addition, the pressure sensor can generate barometric pressure measurement data, the illuminance sensor can measure ambient illuminance measurement data, and the proximity sensor can measure the presence of the measurement object and the distance to the screen without physical contact. Furthermore, GPS sensors can use GPS satellites to generate data on the time and location of the point in time, geomagnetic sensors measure the azimuth angle using the Earth's magnetic field, and orientation sensors measure the rotation angle, gravity sensors and geomagnetic sensors. Together with the location information may be generated.

Meanwhile, FIG. 1 illustrates a smartphone 10, but this is only an example and the present invention may not be limited thereto. That is, the smartphone 10 according to the embodiment of the present invention is generally portable by the user and should be interpreted to encompass a mobile device including a predetermined sensor. For example, smartphones, tablet personal computers, mobile phones, video phones, e-book readers, desktop personal computers, laptop PCs a personal computer, a netbook computer, a workstation, a server, a personal digital assistant, a portable multimedia player, a MP3 player, a mobile medical device, and a camera. have.

The wearable device 20 illustrated in FIG. 1 may also perform a function of transmitting the collected sensing data to the sensing data collection and processing server 40 using the included sensor. To this end, the wearable device 20 may be configured to include various sensors.

More specifically, the wearable device 20 may be equipped with an acceleration sensor, a gravity sensor, a gyro sensor, a temperature sensor, a pressure sensor, an illuminance sensor, a proximity sensor, a GPS sensor, a geomagnetic sensor, a direction sensor, and the like as the smartphone 10. In addition, a heart rate monitor sensor, an oxygen measurement sensor, a skin conduction sensor, a skin temperature sensor, and the like, which can utilize characteristics attached to a user's body, may be mounted.

The heart rate monitor sensor may sense optical blood flow information, for example, and the oxygen measurement sensor may measure data of oxygen of a user's blood and generate data to determine an accurate pulse rate. Furthermore, the skin conduction sensor can generate data to measure the skin's current response, e.g. how sweaty the user is, which can be used to determine the intensity of physical activity and to calculate calories burned. Can be.

In the example of FIG. 1, the wearable device 20 may be, for example, a smart watch, but the present invention is not limited thereto. Wearable devices according to embodiments of the invention may be accessory, such as watches, rings, bracelets, anklets, necklaces, eyeglasses, contact lenses, or head-mounted-devices (HMDs), textiles, or clothing integral types. (Eg, electronic garments), body-attached, for example, skin pads or tattoos, or bio-implantable, for example, implantable circuits.

In another embodiment, the wearable device 20 may include a variety of medical devices, such as various portable medical measuring devices (such as blood glucose meters, heart rate monitors, blood pressure meters, or body temperature meters), magnetic resonance angiography (MRA), and magnetic resonance (MRI). imaging, computed tomography (CT), imager, or ultrasound.

The sensing data collection and processing server 40 illustrated in FIG. 1 may collect sensing data generated through the smart phone 10 and / or the wearable device 20 and preprocess the sensing data.

In the example of FIG. 1, sensing data collection, processing server 40 and behavior awareness server 50 are shown as separate devices, but this is merely illustrative. In other words, the sensing data collection, processing server 40 and behavior recognition server 50 according to an embodiment of the present invention may be physically one or more hardware devices, or may be virtualized software.

The sensing data collection and processing server 40 may receive the sensing data transmitted by the smart phone 10 and / or the wearable device 20 and store it in a buffer memory. Furthermore, the sensing data can be loaded sequentially from the buffer and preprocessed.

The sensing data collection and processing server 40 may collect sensing data from the smartphone 10 and / or the wearable device 20 through various types of interface protocols. For example, the interface protocol may include File Transfer Protocol (FTP), Hyper Text Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), and the like.

The collected sensing data may be stored in a buffer as needed, and may be sequentially loaded from the buffer, and preprocessing may be performed to generate preprocessed sensing data. The pre-processed sensing data may be stored in the storage or transmitted to the action recognition server 50 for processing a user action recognition result as necessary.

Behavioral awareness server 50 according to an embodiment of the present invention may include a machine learning framework. Furthermore, by applying a machine learning framework to sensing data, a machine learning model may be generated to estimate user's behavior, and the model generated on the collected data may be learned to estimate a user's behavior.

More specifically, the behavior recognition server 50 of the present invention includes a first model for learning data generated from the smartphone 10, a second model for learning data generated from the wearable device 20, a smartphone ( The third model for learning by integrating the data generated from 10 and the data generated from the wearable device 20 may be generated. The behavior recognition server 50 may generate at least one of the first model, the second model, and the third model, respectively.

In the above embodiment, the model applied to the data generated from the smartphone, the model applied to the data generated from the wearable device, and the reason for separately configuring the model applied to the data are separately configured in the user's body. Because the location where the smart watch is attached is different, each sensor has a perceivable behavior and an unrecognizable behavior according to the behavior.

For example, in the case of “eating” behavior, the behavior cannot be recognized unless the sensor is located at the arm. Therefore, according to an exemplary embodiment of the present invention, in order to increase the reliability of behavior recognition, a user model may be recognized by generating an integrated model using data of both devices and models of smartphones and smart watches.

For example, the first model may be generated using data of the acceleration sensor and the gyro sensor of the smartphone, and the second model may be generated using the data of the acceleration sensor and the gyro sensor of the wearable device. Furthermore, the third model may be generated using data of an acceleration sensor and a gyro sensor of a smartphone and a wearable device.

The first model recognizes a behavior that can be perceived by the user's overall movement, the second model recognizes a behavior that can be recognized by the user's arm movement, and the third model is used to increase the final reliability of perception of the overall behavior. Can be used as a criterion for calculating the weight that can be applied to each model.

The first model, the second model, and the third model according to the embodiment of the present invention may be generated according to a behavior recognition technique through conventional data mining. The feature values of each model can be selected from the collected data through Matlab's feature selection function, and the features that are mainly used in the time domain can be used.

For example, the first model and the second model may use 42 features, and the third model may use 49 features in total. More specifically, the feature values include Mean, Max, Min, Min, Standard deviation, Quartile, Absolute mean, Absolute median, and Geometry. Eight feature values, such as the geometric mean, can be used.

For the formula used for each feature, refer to the following formula.

[Equation 1]

? The behavior recognition server 50 of FIG. 1 may generate respective models for integrating and applying the smartphone generation data, the wearable device generation data, and the data generated by the smartphone and the wearable device. Furthermore, the feature may be applied to each model to extract a feature, and the weighted weight may be applied to the estimated user behavior and / or the extracted feature itself using the extracted feature.

More specifically, the behavior recognition server 50 according to the embodiment of the present invention may generate a weight table for applying to each model, respectively. For example, a weight table can be created using a weighted voting-based algorithm that is commonly used in decision-making systems and can compensate for misperceived behavior.

It is important to set weighting criteria because decision-making systems using weighted voting techniques can have different behavioral perceptions depending on how weights are applied.

For example, the behavior recognition server 50 according to an embodiment of the present invention may first assign a weight of two to each behavior. Second, depending on the results of the offline test of each model, a weight of 3 can be assigned to a low-precision behavior (eg, less than 80% accuracy). The third most accurate behavior (e.g. greater than 90% accuracy) can be given a weight of 1.

Furthermore, a fourth weighted value of zero can be assigned to the behavior that is assumed to be unrelated to the behavior. For example, in the case of eating behavior, it cannot be estimated with the data generated by the smartphone. This is because people cannot eat while holding a smartphone in their hands. Therefore, in this case, no weight can be given.

Finally, when a specific behavior is extracted only from a specific model, the weight can be given as 3, and a weight table can be created through this process.

In the above embodiment, the reason why the low weight is given to the high accuracy in the third step is that the behavior having high accuracy may have a very high reliability of behavior recognition regardless of the weight. Furthermore, the reason why high weights are assigned to the low-precision behaviors in the second step is that high-computation behaviors with low-accuracy behaviors can give higher competitiveness than the behavior estimates derived from other models.

Furthermore, the behavior recognition server 50 according to the embodiment of the present invention may apply the created weight table to the behavior recognition result estimated by each model.

For example, when walking in the first model, running in the second model, and walking behavior in the third model are estimated, and the weights of the respective actions are applied to each of the actions through the weight table, α, β, and γ are applied. , gamma walking can be derived. The behavior recognition server may then add the weighted behaviors. In the example above, the result of the act summation would be (α + γ) walking, β running.

Furthermore, the behavior recognition server 50 according to the embodiment of the present invention may perform the behavior priority comparison step. The priority comparison step may be performed when the behavior coefficients are equal and may be omitted when the coefficients are different.

For example, when the behavior recognition server has the same behavior coefficient, the behavior recognition server may increase the behavior coefficient with high accuracy based on the accuracy of each behavior by one. As such, the final action may be determined by integrating the weighted action result.

2 is a flowchart illustrating a process of recognizing a user action according to an embodiment of the present invention.

A user behavior recognition system according to an embodiment of the present invention includes a smart phone and a wearable device, and user behavior data may be collected by processing sensing data through the smart phone and the wearable device. (Step 210)

A more specific example of collecting user behavior data will be described later with reference to FIG. 3.

3 is a flowchart illustrating a preprocessing process for collecting and modeling sensor data.

In operation 310, the smartphone and / or the wearable device according to the embodiment of the present invention may generate sensing data using the included sensor and transmit the sensing data to the data collection and processing server.

To this end, the smartphone or wearable device may include a sensor for measuring a physical quantity of specific information, for example, position, motion, brightness, etc., from a measurement target and converting the measured amount into an electrical signal.

The sensing data collection and processing server may receive the sensing data from the smartphone and / or the wearable device and store the sensing data in the buffer memory. Furthermore, the sensing data can be loaded sequentially from the buffer and preprocessed. (Steps 320, 330)

More specifically, the sensing data collection and processing server may collect the sensing data from various types of interface protocols from the smartphone and / or the wearable device. For example, the interface protocol may include File Transfer Protocol (FTP), Hyper Text Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), and the like.

The collected sensing data may be stored in a buffer as needed, and may be sequentially loaded from the buffer, and preprocessing may be performed to generate preprocessed sensing data. The pre-processed sensing data may be stored in a storage or transmitted to an action recognition server for processing a user action recognition result as necessary. In this case, the sensing data collected and processed in the processing server and the sensing data stored in the behavior recognition server may be synchronized. (Step 340)

Returning to the description of FIG. 2 again, the behavior recognition server that receives the user behavior data from the sensing data collection processing server may generate at least one machine learning model by applying the user behavior data to the machine learning frame week. (Step 220)

More specifically, the behavior recognition server according to the embodiment of the present invention may include a machine learning framework. Further, by applying a machine learning framework to user behavior data, a machine learning model may be generated to estimate user behavior, and the model generated on the collected data may be learned to estimate a user's behavior.

For example, the behavior recognition server may include a first model for learning user behavior data generated from a smartphone, a second model for learning data generated from a wearable device, data generated from a smartphone, and generated from a wearable device. A third model can be created for integrating and learning the data. At least one of the first model, the second model, and the third model may be generated.

In this case, a model applied to data generated from a smartphone, a model applied to data generated from a wearable device, and a model applied integrating to the data are separately configured. Because the attached position is different, each sensor has a perceivable behavior and an unrecognizable behavior according to the behavior.

For example, in the case of “eating” behavior, the behavior cannot be recognized unless the sensor is located at the arm. Therefore, according to an exemplary embodiment of the present invention, in order to increase the reliability of behavior recognition, a user model may be recognized by generating an integrated model using data of both devices and models of smartphones and smart watches.

For example, the first model may be generated using data of the acceleration sensor and the gyro sensor of the smartphone, and the second model may be generated using the data of the acceleration sensor and the gyro sensor of the wearable device. Furthermore, the third model may be generated using data of an acceleration sensor and a gyro sensor of a smartphone and a wearable device.

The first model may be used to recognize a perceivable behavior by the user's overall movement, the second model may be used to recognize a perceivable behavior by the user's arm movement, and the third model may further increase the final reliability of the perception of the overall behavior. It can be used as a criterion for calculating weights that can be applied to each model.

The first model, the second model, and the third model according to an embodiment of the present invention may be generated according to a behavior recognition technique through conventional data mining. The feature values of each model can be selected from the collected data through Matlab's feature selection function, and the features that are mainly used in the time domain can be used.

For example, the first model and the second model may use 42 features, and the third model may use 49 features in total. More specifically, the feature values include Mean, Max, Min, Min, Standard deviation, Quartile, Absolute mean, Absolute median, and Geometry. Eight feature values, such as the geometric mean, can be used.

Furthermore, the behavior recognition server may estimate the user behavior by a plurality of models above a preset reliability. (Step 230)

For example, the behavior recognition server may generate a model for integrating and applying the smartphone generation data, the wearable device generation data, and the data generated by the smartphone and the wearable device, and apply the data to each model. And extract the user behavior using the extracted feature.

The behavior recognition server may then weight the combined behavior and incorporate it. (Step 240)

More specifically, the behavior recognition server according to an embodiment of the present invention may generate a weight table for each model. For example, a weight table can be created using a weighted voting-based algorithm that is commonly used in decision-making systems and can compensate for misperceived behavior.

Decision systems using weighted voting techniques can be important in determining weighting criteria because behavior recognition results may differ depending on how weights are applied.

For example, a weight of 2 is assigned to all estimated behaviors, and a second weight of 3 is assigned to low-accuracy behaviors (e.g., less than 80% accuracy) based on the offline test results of each model. can do. The third most accurate behavior (e.g. greater than 90% accuracy) can be given a weight of 1.

In addition, a weight of 0 may be assigned to an action estimated through a device not related to the action. For example, in the case of eating behavior, it cannot be estimated with the data generated by the smartphone. This is because people cannot eat while holding a smartphone in their hands. Therefore, in this case, no weight can be given.

Finally, when a specific behavior is extracted only from a specific model, the weight can be given as 3, and a weight table can be created through this process.

In the above embodiment, the reason why the low weight is given to the high accuracy in the third step is that the behavior having high accuracy may have a very high reliability of behavior recognition regardless of the weight. Furthermore, the reason why high weights are assigned to the low-precision behaviors in the second step is that high-computation behaviors with low-accuracy behaviors can give higher competitiveness than the behavior estimates derived from other models.

The weight table created in this way will be applied to the behavior recognition result estimated by each model. For example, when walking in the first model, running in the second model, and walking behavior in the third model are estimated, and the weights of the respective actions are applied to each of the actions through the weight table, α, β, and γ are applied. , gamma walking can be derived.

The behavior recognition server may then sum the weighted behaviors, i.e., incorporate the weights. In the example above, the result of the act summation would be (α + γ) walking, β running.

Furthermore, the behavior recognition server may compare the behavior priorities and derive the final behavior by applying the priorities. (Steps 250, 260)

The priority comparison may be performed when the behavior coefficients are the same, and may be omitted when the coefficients are different. For example, when the behavior coefficients are the same, the behavior coefficient showing high accuracy may be increased by 1 based on the accuracy of each behavior. As such, the final action may be determined by integrating the weighted action result.

The embodiments of the present invention disclosed in the specification and the drawings are only specific examples to easily explain the technical contents of the present invention and aid the understanding of the present invention, and are not intended to limit the scope of the present invention. It is apparent to those skilled in the art that other modifications based on the technical idea of the present invention can be carried out in addition to the embodiments disclosed herein.

Claims (8)

In a method of recognizing a user's behavior using the sensing data,
Generating sensing data in a user smartphone and a wearable device;
In the server, the sensing data is collected and the collected data is applied to a machine learning framework, so that a first model of the smart phone generated data, a second model of the wearable device generated data, and the smart phone generated data Generating a third model integrating the wearable device generation data;
Estimating user behavior for each of the first model, the second model, and the third model by extracting a feature for each of the first model, the second model, and the third model;
Applying the same first weight to each of the behaviors estimated through the first model, the second model and the third model;
Among the estimated actions, applying a second weight higher than the first weight to an action equal to or less than a predetermined accuracy in the offline test, and applying a third weight lower than the first weight to an action equal to or greater than a predetermined accuracy. ; And
And setting a weight to 0 for an action estimated through a device unrelated to the action and increasing and setting a weight of the action estimated only through a specific device. The method of claim 1, wherein after estimating the user behavior:
Generating a weight table for applying to each of the first model, the second model and the third model; And
And applying the weight table to user behavior estimated through the first model, the second model, and the third model. delete The method of claim 2, wherein after applying the weight table,
And comparing the priorities with respect to the actions having the same weight. In a method of recognizing a user's behavior using the sensing data,
Generating sensing data in a user smartphone and a wearable device;
In the server, the sensing data is collected and the collected data is applied to a machine learning framework, so that a first model of the smart phone generated data, a second model of the wearable device generated data, and the smart phone generated data And generating a third model integrating the wearable device generation data, extracting features for each model, and estimating user behavior for each of the models;
Generate a weight table for applying to each of the first model, the second model and the third model, the weight table to the user behavior estimated through the first model, the second model and the third model Applying; And
Comparing priorities for actions with the same weight,
Applying to the user behavior,
Applying the same first weight to each of the behaviors estimated through the first model, the second model and the third model;
Among the estimated actions, applying a second weight higher than the first weight to an action equal to or less than a predetermined accuracy in the offline test, and applying a third weight lower than the first weight to an action equal to or greater than a predetermined accuracy. ; And
And setting a weight to 0 for an action estimated through a device unrelated to the action and increasing and setting a weight of the action estimated only through a specific device. In a system that recognizes user behavior,
A user smartphone including at least one sensor;
A wearable device including at least one sensor and attached to the user;
A data processing server for collecting and processing sensing data generated by the smart phone and the wearable device; And
Applying the data processed by the data processing server to a machine learning framework, a first model for the smartphone generation data, a second model for the wearable device generation data, and the smartphone generation data and the wearable device generation Generate a third model incorporating data, extract features for each model, estimate user behavior for each of the models, and apply a weight table to apply to each of the first model, the second model, and the third model. Generating and applying the weight table to a user behavior estimated through the first model, the second model, and the third model;
The act recognition server,
The same first weight is applied to each of the actions estimated through the first model, the second model, and the third model, and among the estimated actions, the first weight is applied to actions below a predetermined accuracy in an offline test. Apply a higher second weight, apply a third weight lower than the first weight for an action greater than or equal to a predetermined accuracy, set the weight to zero for an action estimated through a device independent of the action, and specify a specific device User behavior recognition system, characterized in that the setting by increasing the weight of the behavior estimated only through. delete The method of claim 6, wherein the behavior recognition server,
User behavior recognition system, characterized in that for comparing the priority for the behavior of the same weight.

Images