KR102089002B1 - Method and wearable device for providing feedback on action - Google Patents

Abstract

The server providing feedback on the action includes a sensor data receiving unit receiving a plurality of sensor data measured for a predetermined time by a plurality of sensors built in a wearable device from the wearable device, and at least two or more from the sensor data Singularity detection unit for detecting the singularity, a behavior recognition unit for recognizing the user's behavior based on the similarity between the data for the first feature section including the at least two singularities and a plurality of pre-stored motion pattern data and the recognized behavior And a feedback transmission unit transmitting feedback on at least one of a wearable device or a user terminal.

Description

A wearable device and method for providing feedback on behavior {METHOD AND WEARABLE DEVICE FOR PROVIDING FEEDBACK ON ACTION}

The present invention relates to a wearable device and method for providing feedback on behavior.

A wearable device means a wearable device, and science and technology advances to develop wearable smart devices such as T-shirts, glasses, bracelets, watches, and shoes. A typical wearable device is a smart watch, and the smart watch has an existing clock function that tells the time, and a function that enables you to check notifications, e-mails, and messages in real time without checking your smartphone has been added.

In relation to a wearable device providing such a smart function, Korean Patent Registration No. 10-1689368, which is a prior art, discloses a wearable device.

Recently, wearable devices provide health-related services using data measured through various sensors. However, this is limited to analysis information on the number of steps, movement distance, and exercise time of the user, and thus has a disadvantage in that it is difficult to provide intuitive feedback related to the action and behavior of the user.

The present invention is to solve the above-mentioned problems, a wearable device that provides feedback on an action to monitor when and what actions the user performed based on a plurality of sensor data measured using various sensors built into the wearable device And to provide a method.

It is intended to provide a wearable device and method for providing feedback that induces correct behavior and habits to a user by feeding the user self-digitized information about the monitored user's behavior.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems as described above, and other technical problems may exist.

As a means for achieving the above technical problem, an embodiment of the present invention, a sensor for receiving a plurality of sensor data (Sensor Data) measured for a predetermined time by a plurality of sensors embedded in the wearable device from the wearable device A data receiving unit, a singularity detecting unit detecting at least two singularities from the sensor data, and recognizing a user's behavior based on the similarity between data for a first feature section including the at least two singularities and a plurality of previously stored motion pattern data It is possible to provide a behavior recognition server including a behavior recognition unit and a feedback transmission unit that transmits feedback on the recognized behavior to at least one of a wearable device or a user terminal.

Another embodiment of the present invention, receiving a plurality of sensor data (Sensor Data) measured for a predetermined time by a plurality of sensors embedded in the wearable device from the wearable device, detecting at least two or more singularities from the sensor data Step of recognizing the user's behavior based on the similarity between the data for the first feature section including the at least two singularities and a plurality of pre-stored motion pattern data, and a feedback of the recognized behavior by a wearable device or It may provide a behavior recognition method comprising the step of transmitting to at least one of the user terminal.

Another embodiment of the present invention, collecting a plurality of sensor data (Sensor Data) measured by a plurality of sensors embedded in the wearable device, the collected plurality of sensor data (Sensor Data) as a behavior recognition server Transmitting, receiving feedback on the recognized user's behavior based on the plurality of sensor data from the behavior recognition server, and outputting the received feedback, wherein the user's behavior includes the plurality of A behavior recognition method in which at least two or more singularities are detected from sensor data, and a user's behavior is recognized based on the similarity between data for a first feature section including the at least two singularities and a plurality of previously stored motion pattern data. Can provide

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, the present invention is to solve the above-described problem, based on a plurality of sensor data measured by using a variety of sensors built into the wearable device, the user when, what It is possible to provide a wearable device and method for providing feedback on an action to monitor whether an action has been taken.

It is possible to provide a wearable device and method for providing feedback that induces correct behaviors and habits to a user by feeding the user self-digitized information about the monitored user's behavior.

1 is a block diagram of a system for providing behavioral feedback according to an embodiment of the present invention.
2 is a block diagram of a wearable device according to an embodiment of the present invention.
3 is a flowchart of a method for providing feedback on an action in a wearable device according to an embodiment of the present invention.
4 is a block diagram of a behavior recognition server according to an embodiment of the present invention.
5A to 5D are exemplary views for explaining a process of recognizing a user's motion through data fusion from a plurality of sensor data according to an embodiment of the present invention.
6A and 6B are exemplary diagrams for explaining a process of performing clustering on a user's motion recognized in a behavior recognition server according to an embodiment of the present invention.
7 is a flowchart of a method for providing feedback on a behavior in a behavior recognition server according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another element in between. . Also, when a part is said to “include” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated. However, it should be understood that the existence or addition possibilities of numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

In the present specification, the term “unit” includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be realized by using two or more hardware, and two or more units may be realized by one hardware.

Some of the operations or functions described in this specification as being performed by a terminal or device may be performed instead on a server connected to the corresponding terminal or device. Similarly, some of the operations or functions described as being performed by the server may be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for providing behavioral feedback according to an embodiment of the present invention. Referring to FIG. 1, the behavior feedback providing system 1 may include a wearable device 110, a behavior recognition server 120, and a user terminal 130. The wearable device 110, the behavior recognition server 120, and the user terminal 130 exemplarily illustrate components that can be controlled by the behavior feedback providing system 1.

Each component of the behavior feedback providing system 1 of FIG. 1 is generally connected through a network. For example, as illustrated in FIG. 1, the wearable device 110 or the user terminal 130 may be connected to the behavior recognition server 120 simultaneously or at a time interval.

Network refers to a connection structure capable of exchanging information between nodes such as terminals and servers, and a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World) Wide Web), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The wearable device 110 is installed with an app that provides behavioral feedback, so that the app can be automatically executed without a separate input from the user.

The wearable device 110 may be worn on a user's wrist and collect a plurality of sensor data measured by a plurality of sensors embedded in the wearable device 110.

The wearable device 110 may transmit the collected plurality of sensor data to the behavior recognition server 120.

The wearable device 110 may receive feedback on the recognized user's behavior based on the plurality of sensor data from the behavior recognition server 120. Here, the feedback may include at least one of action information of at least one action included in the user's action history, the user's action history, and meaningful action information included in the at least one action.

The wearable device 110 may output feedback on the user's behavior. For example, feedback may consist of pictures and text, and may be output through a display, and may consist of voice or alarm, and may be output through a speaker, or may be output through vibration. Through this, it is possible to provide visual, audible, and tactile feedback to the user.

This feedback allows users to monitor their behavior. For example, when the user performs an exercise, the user may check the exercise amount measured based on the number of exercise and set calories, exercise time, rest time, and heart rate. In addition, the exercise accuracy can be measured based on the user's exercise posture, and the correct exercise posture can be guided based on the measured exercise accuracy. As another example, based on the type of exercise performed by the user, the user may be able to evenly select various exercises through feedback on an insufficient exercise part to the user. For another example, the user's challenge awareness may be strengthened through feedback including real-time information and ranking information, so that the user is motivated to exercise. As another example, the user may check self-digitization in units of daily, weekly, and monthly by monitoring exercise data. As another example, the user can be guided to the correct posture through feedback on his wrong behavior and habits.

The wearable device 110 may be connected to the user terminal 130 through Bluetooth communication. At this time, the wearable device 110 may synchronize each user's behavior information through an app that provides behavior feedback installed in each of the wearable device 110 and the user terminal 130.

The wearable device 110 may be configured to receive feedback on a user's behavior from the behavior recognition server 120, and the wearable device has pre-stored motion data for a plurality of motion patterns in a memory of the wearable device 110 The 110 itself may also be configured to recognize the user's actions or actions.

The gesture recognition server 120 may store gesture data corresponding to each of a plurality of gesture patterns in a database.

The behavior recognition server 120 may receive a plurality of sensor data measured for a predetermined time by a plurality of sensors built in the wearable device 110 from the wearable device 110. At this time, the behavior recognition server 120 may remove noise from a plurality of sensor data using at least one of a Kalman filter, a low pass filter, a high pass filter, a complementary filter, a moving average filter, and a smoothing filter.

The behavior recognition server 120 may detect at least two or more singularities from sensor data. For example, the behavior recognition server 120 may detect at least two or more singularities using machine learning. For another example, the behavior recognition server 120 may detect at least two singularities through data fusion from sensor data. Specifically, the behavior recognition server 120 may generate at least two singularities by generating a plurality of collected sensor data as sensor fusion data through data fusion. Here, the at least two singularities include a first singularity and a second singularity, and the feature section refers to a section between a time corresponding to the first singularity and a time corresponding to the second singularity, and the at least two singularities are, for example, For example, it can be detected from sensor data based on at least one of peak value, valley, zero crossing, area of the graph, curvature, time, angle, acceleration, distance, and trajectory.

The behavior recognition server 120 may recognize a user's behavior based on the similarity between data for a first feature section including at least two singularities and a plurality of pre-stored motion pattern data.

The behavior recognition server 120 analyzes the similarity between the data for the first feature section and a plurality of pre-stored motion pattern data by using machine learning to recognize the user's behavior, and the first feature section based on the analyzed similarity It is possible to recognize the operation corresponding to the data for. At this time, the behavior recognition server 120 may determine whether the posture of the action performed by the user is the correct posture using the values of the acceleration sensor, the gyro sensor, and the quaternion. To this end, the behavior recognition server 120 may determine whether the user's exercise posture is the correct posture through machine learning such as a Hidden Markov Model (HMM), Convolution Neural Nerwork (CNN), or Deep Neural Network (DNN).

The behavior recognition server 120 clusters the data for the plurality of feature sections into a plurality of similar operation groups by performing the first clustering on the data for the plurality of feature sections including the data for the first feature section. 1 An action corresponding to any one of the similar motion groups including data on the feature section may be recognized as the user's action.

The behavior recognition server 120 may receive location information of the wearable device 110 from the wearable device 110. At this time, the behavior recognition server 120 performs a second clustering on the plurality of similar operation groups classified based on the location information of the wearable device 110 to classify the plurality of similar operation groups into a plurality of location-based behavior groups can do.

The behavior recognition server 120 may store the recognized user's behavior in chronological order, and delete the user's stored behavior based on the association in chronological order. Here, the stored action may include a first action, a second action, and a third action performed within a predetermined time. For example, the behavior recognition server 120 determines the association of each of the first action, the second action, and the third action, and based on each association, one of the first action, the second action, and the third action You can delete an action.

The behavior recognition server 120 may transmit feedback on the recognized behavior to at least one of the wearable device 110 or the user terminal 130. The feedback may include at least one of action information of at least one action included in the user's action history, a user's action history for each time period, and meaningful action information included in the at least one action.

The user terminal 130 is connected to the wearable device 110 through Bluetooth (bluetooth) communication, the user terminal 130 and the wearable device 110 through each of the apps that provide behavioral feedback through the user's behavior information Can be synchronized.

The user terminal 130 may receive feedback on the user's behavior recognized from the behavior recognition server 120. At this time, the feedback may consist of a picture and text, and may be output through a display, may consist of voice or alarm, and may be output through a speaker, or may be output through vibration. Through this, it is possible to provide visual, audible, and tactile feedback to the user.

2 is a block diagram of a wearable device according to an embodiment of the present invention. Referring to FIG. 2, the wearable device 110 may include a sensor data collection unit 210, a transmission unit 220, a feedback reception unit 230, and a feedback output unit 240.

The sensor data collection unit 210 may collect a plurality of sensor data measured by a plurality of sensors embedded in the wearable device 110. The sensor data collection unit 210 may collect a plurality of sensor data measured from a plurality of sensors embedded in the wearable device 110. For example, the sensor data collection unit 210 collects a plurality of sensor data measured by a plurality of sensors built into the wearable device 110 when a user wearing the wearable device 110 performs some action. You can. Here, the plurality of sensors may include an acceleration sensor, a gyro sensor, a heart rate sensor, a gravity sensor, a magnetic sensor, a proximity sensor, an RGB sensor, a brightness sensor, a barometric pressure sensor, and the like. For example, the heart rate sensor (PPG) is a sensor that can detect changes in blood flow in the microvascular tissues. When the LED embedded in the heart rate sensor sends light to the skin, the photodiode is reflected from the skin. By detecting light, it is possible to measure the heart rate.

The transmission unit 220 may transmit the collected plurality of sensor data to the behavior recognition server 120.

The feedback receiver 230 may receive feedback on the recognized user's behavior based on the plurality of sensor data from the behavior recognition server 120.

The feedback output unit 240 may output the received feedback. The feedback may include, for example, at least one of action information of at least one action included in a user's action history, a user's action history, and at least one of meaningful motion information included in at least one action.

For example, the user's action history for each time period is 12: 00 ~ 12: 30 / lunch, 12: 30 ~ 13: 00 / driving, 13: 00 ~ 14: 00 / exercise, 14: 00 ~ 16: 00 / It may include a history of actions performed by the user for one day, such as studying.

The action information of at least one action included in the user's action history includes, for example, the action name, the number of workouts, the time of exercise, and the amount of calories burned in real time when the action performed by the user corresponds to the workout. , Dynamic feedback including at least one of exercise accuracy, and static feedback including at least one of an exercise method, an exercise site, exercise intensity, and ranking. Specifically, when the user has played 'football', the action information includes a run distance, a kick count, a walking time, a rest time, etc., and when the user plays a 'tennis', the run distance, the number of swings, etc. have.

Significant motion information included in the at least one action may include, for example, swing posture, soccer distance, driving time, time sitting in the office, use of the toilet, number of fitness workouts, exercise awareness, distance traveled, and the like. .

The feedback output unit 240 may output a separate attention feedback when the recognized user's behavior includes a negative behavior. For example, if a user has smoked 10 times a day, he or she may output feedback from the week, such as, "You smoked 10 times a day today.

  3 is a flowchart of a method for providing feedback on an action in a wearable device according to an embodiment of the present invention. The method of providing feedback on the behavior in the wearable device 110 illustrated in FIG. 3 includes steps processed in time series by the behavior feedback providing system 1 according to the embodiment illustrated in FIGS. 1 and 3. . Therefore, even if it is omitted below, it is also applied to a method for providing feedback on an action in the wearable device 110 according to the embodiment illustrated in FIGS. 1 and 2.

In operation S310, the wearable device 110 may collect a plurality of sensor data measured by a plurality of sensors embedded in the wearable device 110.

In operation S320, the wearable device 110 may transmit the collected plurality of sensor data to the behavior recognition server 120.

In operation S330, the wearable device 110 may receive feedback on the recognized user's behavior based on the plurality of sensor data from the behavior recognition server 120.

In step S340, the wearable device 110 may output the received feedback.

In the above description, steps S310 to S340 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

4 is a block diagram of a behavior recognition server according to an embodiment of the present invention. 4, the behavior recognition server 120 includes a storage unit 410, a sensor data receiving unit 420, a singularity detection unit 430, a location information receiving unit 440, a behavior recognition unit 450, and a feedback transmission unit ( 460).

The storage unit 410 may store motion data corresponding to each of a plurality of motion patterns in a database.

For example, the storage unit 410 may store the motion motion data corresponding to the motion motion pattern in the database. Specifically, the storage unit 410 stores golf motion data corresponding to a golf motion pattern such as swing, drive, putting, walking, running, walking, running, biking, eating water, curl, shoulder press, front phrase, Stores fitness motion data corresponding to fitness motion patterns such as squats, lunges, cell phone views, watch views, and saves motion data corresponding to soccer motion patterns such as shooting, dribbling, walking, running, raising hands, and backstroke, Motion data corresponding to swimming motion patterns can be stored, such as breaststroke, freestyle, sleep swimming, butterfly, walking, drinking, jumping, arm stretching, full body stretching, leg stretching, cell phone viewing, and watch viewing.

For another example, the storage unit 410 may store business operation data corresponding to the business operation pattern in a database. Specifically, the storage unit 410 is sitting, meeting, eating, typing, drinking, walking, reading a book, moving the mouse, folding arms, opening a drawer, viewing a mobile phone, viewing a clock, etc. Business operation data corresponding to the pattern may be stored.

For another example, the storage unit 410 may store rest motion data corresponding to the rest motion pattern in a database. Specifically, the storage unit 410 may store rest motion data corresponding to a rest motion pattern, such as watching TV, lying down, sleeping, eating, walking, drinking water, and the like.

When the user's action is recognized, the storage unit 410 may store the recognized user's action and action according to a time sequence. For example, the stored action may include a first action (eg, soccer / shoot-running, etc.), a second action (eg, soccer / dribble-raising, etc.), a third action (eg, performed within a predetermined time). For example, soccer / drinking water, walking, and the like). At this time, the storage unit 410 may store the user's actions and actions based on whether the user's actions are general-purpose actions or personal actions.

The sensor data receiving unit 420 may receive a plurality of sensor data measured for a predetermined time by a plurality of sensors embedded in the wearable device 110 from the wearable device 110.

The singularity detection unit 430 includes a noise removal unit (not shown), and the noise removal unit (not shown) uses a Kalman filter, a low-pass filter, a high-pass filter, a complementary filter, a moving average filter, and a smoothing filter. Noise can be removed from a plurality of sensor data. For example, the sensor data may include noise, and a noise removing unit (not shown) may remove noise from the sensor data by applying a filter suitable for the sensor data of each sensor.

Kalman filter (Kalman filter) is a filter that removes noise from the input data that contains noise to enable optimal statistical prediction of the current state, and a low-pass filter (LPF) sets the cutoff frequency to generate a high-frequency noise signal. This filter removes only low-frequency signals. The high-pass filter (HPF) is a filter that uses only high-frequency signals by setting the cut-off frequency to remove low-frequency noise signals. Complementary filters have good response characteristics in the low-frequency region and values measured by the gyro sensor, which has good response characteristics in the high-frequency region. It is a filter that measures the angle by adding the value measured through the acceleration sensor. A moving average filter means a low-pass or flattening filter based on calculating an average value of the surroundings.

The singularity detection unit 430 may detect at least two or more singularities from sensor data. For example, the singularity detection unit 430 may detect at least two singularities based on at least one of a peak value, a valley, a zero crossing, a graph's width, curvature, time, angle, acceleration, distance, and trajectory from sensor data. have.

According to an embodiment, the singularity detection unit 430 may detect at least two or more singularities from sensor data using machine learning. At this time, the singularity extracted or processed through machine learning is data that can be changed by the user.

The behavior recognition unit 450 may recognize a user's behavior based on the similarity between data for a first feature section including at least two singularities and a plurality of pre-stored motion pattern data. Here, the first feature section may be a section between the time corresponding to the first singularity and the time corresponding to the second singularity. At this time, the behavior recognition unit 450 uses, for example, K-Nearest Neighbors Algorithm (KNN), Recurrent Neural Network (RNN), Dynamic Time Warping (DTW), Perceptrons, Correlation clustering, HMM, MEMMs, etc. Can be analyzed.

The behavior recognition unit 450 may recognize an operation corresponding to data for the first feature section based on the analyzed similarity. For example, the behavior recognition unit 450 may recognize an operation corresponding to data for the first feature section as 'backstroke' based on the analyzed similarity.

According to another embodiment, the singularity detection unit 430 may detect at least two or more singularities through data fusion from sensor data. The process of detecting at least two singularities through data fusion from sensor data will be described in detail with reference to FIG. 5.

5A to 5D are exemplary views for explaining a process of recognizing a user's operation through data fusion from a plurality of sensor data according to an embodiment of the present invention.

5A is an exemplary diagram for explaining a process of performing data fusion from sensor data according to a first characteristic section according to an embodiment of the present invention. 5A, it can be seen that the plurality of sensor data 500 is sensor data measured by the plurality of sensors, and each sensor data has a different waveform.

The singularity detection unit 430 may generate the received plurality of sensor data 500 as one sensor fusion data 510 through data fusion. At this time, the sensor fusion data 510 may include at least two singularities 520. Here, the singularity may be represented as the starting point and the ending point of the operation.

The method using data fusion may be composed of sums or squares of each sensor data, and combinations of derivatives and integrals. For example, when expressed as the force of acceleration (3-axis: x, y, z), it can be 'sqrt (x ² + y ² + z ² ), and in the direction of the gyro (roll, pitch, yaw) When expressing, it may be expressed in the direction of the angle. At this time, data fusion may be performed using appropriate formulas according to values and formulas of each sensor on hardware.

The sensor fusion data 510 is one in which a plurality of sensor data is generated as one sensor fusion data 510 through data fusion, and the singularity detection unit 430, for example, uses six axis sensor data as data fusion. Through this, it can be generated as sensor fusion data 510 of one axis. This is, in order to recognize the motion being performed by the user, six or more axes are used, and by generating sensor fusion data from sensor data of six or more axes, it is possible to easily detect singularity. Here, at least two or more singularities detected through the sensor fusion data 500 may be values for finding a flag of the six-axis sensor data.

5B is an exemplary diagram for explaining a process of extracting sensor data corresponding to a first feature section from sensor fusion data generated through data fusion according to an embodiment of the present invention, and FIG. 5C is an embodiment of the present invention. An exemplary diagram for explaining a process of analyzing the similarity between a plurality of sensor data for a feature section and a plurality of pre-stored operation pattern data according to an embodiment. 5B and 5C, the singularity detection unit 430 may detect at least two singularities based on a peak value, a valley, a zero crossing, a graph's area, curvature, time, velocity, acceleration, and distance trajectory. Here, at least two or more singularities include a first singularity 511 and a second singularity 512, and the feature section 513 corresponds to a time corresponding to the first singularity 511 and a second singularity 512 It may be an interval between times.

The singularity extraction unit 430 may extract a plurality of sensor data 540 from the sensor fusion data 530 for the feature section 513. For example, when the feature section 513 is detected based on the first singularity 511 and the second singularity 512 from the sensor fusion data 510, the singularity extracting unit 430 detects the characteristic section 513. A plurality of sensor data 540 for each axis of each sensor may be extracted from the plurality of sensor data 540 included in the fusion data 530.

For example, the behavior recognition unit 450 compares (551) the x-axis sensor data of the extracted acceleration sensor with the x-axis reference data of the previously stored acceleration sensor, and the y-axis sensor data and the pre-stored acceleration of the extracted acceleration sensor. The y-axis reference data of the sensor may be compared (552), and the z-axis sensor data of the extracted acceleration sensor and the z-axis reference data of the previously stored acceleration sensor may be compared (553). Also, the behavior recognition unit 450 compares 561 the roll axis sensor data of the extracted gyro sensor and the roll axis reference data of the pre-stored gyro sensor, and compares the extracted pitch sensor data with the gyro sensor. The pitch axis reference data of the pre-stored gyro sensor may be compared (562), and the yaw axis sensor data of the extracted gyro sensor and the yaw axis reference data of the pre-stored gyro sensor may be compared (563).

Through this process, the behavior recognition unit 450 may calculate an average of accuracy between data for each axis.

5D is an exemplary diagram for explaining a process of analyzing similarity between a plurality of sensor data and a plurality of pre-stored operation pattern data for a feature section according to an embodiment of the present invention. Referring to FIG. 5D, when the average accuracy of the data for each axis of the plurality of sensor data 540 for the feature section 513 is calculated through FIG. 5C, the behavior recognition unit 450 displays the plurality of sensor data 540 Conformity can be calculated by comparing each axis data with each axis of the pre-stored motion pattern data.

For example, when the user is currently backstroke, the behavior recognition unit 450 compares the backstroke and the pre-stored butterfly motion pattern data and calculates the matching degree as 70%, and compares the backstroke and the pre-stored freestyle motion pattern data to compare the match degree to 50. Calculated as%, compares the backstroke and pre-stored backstroke motion pattern data to calculate 95%, compares the backstroke and pre-stored breaststroke motion pattern data to calculate 80%, backstroke and pre-stored walking By comparing the operation pattern data of, it is possible to calculate the concordance as 20%.

Through this process, the behavior recognition unit 450 may recognize that the current user's action is 'backstroke'.

Returning to FIG. 4 again, the behavior recognition unit 450 may recognize a user's behavior by performing clustering on data for a plurality of feature sections including data for the first feature section. The process of recognizing the user's behavior will be described in detail with reference to FIGS. 6A and 6B.

6A is an exemplary diagram illustrating a plurality of operations for a plurality of feature sections according to an embodiment of the present invention.

Referring to (a) of FIG. 6A, the behavior recognition unit 450 may recognize the operation 600 corresponding to data for the first feature section. The operation 600 corresponding to the data for the first feature section may be, for example, 'backstroke'.

Referring to (b) of FIG. 6A, the behavior recognition unit 450 may allow the data for the first characteristic section to be included in any one of the plurality of similar operation groups 610 through the first clustering. . For example, the behavior recognition unit 450 may allow the “backstroke” operation to be included in the swimming operation group 610 among the plurality of similar operation groups through the first clustering. Here, the swimming motion group 610 may be a group of swimming motions such as backstroke, breaststroke, freestyle, sleepstroke, butterfly, walking, stretching.

The behavior recognition unit 450 may recognize an action corresponding to any one of the similar action groups 610 including data on the first characteristic section as a user's action. For example, since the data for the first feature section is included in the swimming action group 610, the action recognition unit 450 may recognize the user's action as 'swimming'.

Referring to FIG. 6A (c), the location information receiving unit 440 may receive location information of the wearable device 110 from the wearable device 110. For example, the location information receiver 410 may receive location information (eg, a school, home, company, fitness center, etc.) of the wearable device 110 measured through GPS of the wearable device 110. .

The behavior recognition unit 450 performs second clustering on a plurality of similar operation groups classified data in a first characteristic section based on the location information of the wearable device 110 to the plurality of location-based action groups 620 Can be classified. For example, when the location information of the wearable device 110 indicates a 'pool', the second clustering is performed to perform the location-based action by reflecting the location information of the swimming action group 610, such as 'pool-swimming-backstroke' It can be classified as a group (620).

6B is an exemplary diagram showing a location-based behavior group of a tree structure according to an embodiment of the present invention. Referring to FIG. 6B, the behavior recognition unit 450 may recognize a user's motion and classify it into small categories.

The sub-classification 630 represents all operations of the user recognized for the first characteristic data, and the sub-classification 630 is, for example, {a1, a2, a3, ..., b1, b2, b3, ..., c1, c2, c3, ..., d1, d2, d3, ..., e1, e2, e3, ..., f1, f2, f3, ..., g1, g2, g3, ..., h1, h2, h3, ..., i1, i2, i3, ..., j1, j2, j3, ..., k1, k2, k3, ..., l1, l2, l3, ...} It can be configured as follows. The sub-classification 630 may include all actions of a user, such as walking, driving, reading a book, viewing a clock, eating, running, butterfly, and the like.

The middle classification 640 represents the user's behavior recognized based on the behavior corresponding to any one of the similar operation groups 610 including data for the first characteristic section, and the middle classification 640 is, for example, {A , B, C, D, E, F, G, H, I, J, K, L, ...}. Here, the set of '{a1, a2, a3, ...}' becomes' A ', the set of' {b1, b2, b3, ...} 'becomes' B', and '{c1, c2, c3, ...} 'can be' C '. For example, if the user's motion includes {butterfly, backstroke, sleep, freestyle, walking, ...}, the user's action may be recognized as 'swimming' (set A) as a set of these. At this time, in case of some actions, the user's actions may be classified only up to the middle class.

The large classification 650 indicates that a plurality of similar operation groups are classified into a plurality of location-based behavior groups by performing second clustering on the plurality of similar operation groups classified based on the location information of the wearable device 110, and the large classification 650 may be configured, for example, {Exercise, Office, House, ...}. For example, if the location information of the wearable device 110 indicates a 'pool', the large category 650 is 'Exercise'-swimming (A)-butterfly / backstroke / submerged / freestyle / walking (a1, a2) , a3, a4).

Returning to FIG. 4 again, the behavior recognition unit 450 analyzes the user's motion and / or behavior in real time through Time Series Analysis, and analyzes whether it is a behavior using Dynamic Time Warping (DTW) during time series analysis. You may. Here, DTW is a speech recognition technology (16,000 Hz or 8,000 Hz) commonly used during time series analysis, and can measure similarity between two sequences having different speeds and times. In general, it is analyzed through endpoint detection, and similarity between data can be measured through warping path.

The action recognition unit 450 may recognize a user's action and / or action by setting a threshold for each action or action. In this case, the user may not recognize the user's motion or behavior for data below the threshold.

In this way, the behavior recognition unit 450 recognizes an operation and determines a user's behavior by a collective behavioral pattern. At this time, the behavior recognition unit 450 may use machine learning to optimize data for each operation. Since machine learning increases the amount of data, it is possible to optimize data by performing data optimization through machine learning, storing each user's behavior and behavior data, and analyzing and comparing them. For example, if the user is performing biceps curl during a fitness workout, User1, User2, User3,? If such data exists, the data of a plurality of users may be analyzed to extract the most optimal data.

The behavior recognition unit 450 may recognize the user's behavior by collecting sensor data through a plurality of wearable devices 110 without using only sensor data collected from one wearable device 110. By using a plurality of wearable devices 110, a user with high accuracy is used by combining sensor data measured from a plurality of wearable devices 110, rather than using sensor data measured from a single wearable device 110. Can recognize the behavior of

If the user's behavior is not recognized, the behavior recognition unit 450 may recognize the user's behavior by analyzing and predicting the user's behavior using various methods.

For example, when a motion corresponding to an unsaved motion pattern is recognized during the user's motion recognition, the feedback transmission unit 460 transmits feedback to the user about the motion to directly input the corresponding feedback from the user. When 'Swimming' is input, the behavior recognition unit 450 may recognize the user's motion and behavior by comparing whether there is motion data similar to 'Swimming'.

For another example, when the user's motion is recognized in the order of 'backstroke-walk-sleep-freestyle-walk-walk-?-Freestyle-walk', the user's machine learning is performed through the user's machine learning. Behavior can be recognized. At this time, if there is no close action or similar action corresponding to '', the action recognition unit 450 may recognize the user's action as 'swimming' by including the corresponding data.

The behavior recognizing unit 450 may store the first behavior, the first behavior, and the first behavior, the second behavior, and the third behavior when the user's behavior in the storage 410 is performed in a predetermined time. The association of each of the second action and the third action may be determined, and one of the first action, the second action, and the third action may be deleted based on each relationship. For example, suppose the user played soccer for a predetermined time. When the user recognizes the first action-football, the second action-tennis, and the third action-football of the user based on the plurality of feature data, the action recognition unit 450 may respectively perform the first action, the second action, and the third action You can judge the association. At this time, the action recognition unit 450 may determine that the user has performed the second action-tennis after the first action-football is finished. However, since the third action-football has been recognized again, the user may determine that he has continued playing football and delete the less relevant tennis.

The feedback transmitting unit 460 may transmit feedback on the recognized behavior to at least one of the wearable device 110 or the user terminal 130. The feedback may include at least one of action information of at least one action included in the user's action history, a user's action history for each time period, and meaningful action information included in the at least one action.

For example, the user's action history for each time period is 12: 00 ~ 12: 30 / lunch, 12: 30 ~ 13: 00 / driving, 13: 00 ~ 14: 00 / exercise, 14: 00 ~ 16: 00 / It may include a history of actions performed by the user for one day, such as studying.

The action information of at least one action included in the user's action history includes, for example, the action name, the number of workouts, the time of exercise, and the amount of calories burned in real time when the action performed by the user corresponds to the workout. , Dynamic feedback including at least one of exercise accuracy, and static feedback including at least one of an exercise method, an exercise site, exercise intensity, and ranking. Specifically, when the user has played 'football', the feedback transmitting unit 460 may transmit the skipped distance, the number of kicks, the walking time, the resting time, etc. as feedback, and when the user has 'tennis', the feedback distance, The number of swings and the like can be transmitted as feedback.

Ranking allows users to determine the level of exercise they are performing (i.e., compare to others) based on their physical conditions (e.g. age, height, weight, country, gender, body fat percentage, climate, race, etc.). You can make it grasp.

Movement accuracy may be calculated based on the user's movement using the similarity of movement patterns. In addition, coaching for the user's posture may be provided by converting the user's posture into 3D coordinates through exercise accuracy. For example, by comparing the coordinate values between reference data for each axis of each sensor and sensor data (real-time data) for a user's characteristic section using exercise accuracy, a user's wrong posture (for example, a wrist bend, a body tilt, etc.) ) In real time.

The calories burned may be calculated according to the displacement amount of movement after the quaternion conversion. At this time, calories can be calculated based on the activity level according to gender, age, weight, heart rate, and time. The activity level can be calculated according to the amount of quaternion displacement and the level of the exercise area. For example, the amount of calories burned may be calculated according to the displacement amount of motion after the sensor data for the feature section is converted into 3D coordinates or quaternions. The calorie consumption can be derived, for example, through Equation 1.

 (Here, T = 1 time, Activity Level = ((quaternion (movement) displacement)) * (exercise level (distinguished motion name))), and exercise area level is related to the Metabolic Equivalent Task (MET) value)

The quaternion displacement amount corresponds to the user's movement, and the displacement amount means the distance the user's wearable device 110 moves. The calories burned may be calculated by measuring the moving distance of the wearable device 110 worn by the user, and using the quaternion displacement as a variable in the existing calorie calculation formula. Here, the amount of calories burned can be calculated by defining the average as '1' in all exercises, and defining a value lower than 1 for a exercise with a small displacement and larger than 1 for a exercise with a large displacement. At this time, the amount of calories burned is changed according to the number of workouts. For example, in the case of arm exercise, the activity level is set low because the movement of the muscle is lower than that of the whole body exercise. You can.

When the user's exercise amount is grasped in units of week, month, and quarter, the exercise intensity may be provided as a feedback according to a ratio of performing the exercise. The exercise guide may calculate the accuracy of the user's exercise posture and provide the guide for the exercise posture to the wearable device 110 or the user terminal 130 as feedback. By analyzing the user's exercise data through machine learning, a user can guide an exercise posture to the user, recommend a suitable exercise to the user, and recommend the exercise intensity of the exercise the user is performing. In addition, each calorie consumption for each exercise can be measured and provided to the user.

Through this, the user who wears the wearable device 110 can intuitively check the feedback on the actions performed by the wearable device 110 through the display of the wearable device 110.

In addition, the feedback transmitting unit 460 may transmit feedback compared with other users having characteristics similar to the user (eg, height, age, occupation, weight, country, gender, body fat percentage, climate, race, region, etc.). . For example, the feedback transmitting unit 460 says, "You are currently spending 8 hours at work, 4 hours at home, 8 hours of sleep, and 4 hours of commute. You're acting with 80% of the time, 250% of the commute time, 60% of the time at home, and 120% of your sleep time, so you need to reduce your commute time. " For another example, the feedback transmission unit 460 may transmit feedback on actions recommended by other users to the wearable device 110. Through this, by recommending the actions recommended by other users to the user, it is possible to induce the user to perform a good action.

Significant motion information included in the at least one action may include, for example, swing posture, soccer distance, driving time, time sitting in the office, use of the toilet, number of fitness workouts, exercise awareness, distance traveled, and the like. .

At this time, if the recognized user's behavior includes negative behavior, a separate attention feedback may be included. For example, if a user has smoked 10 times a day, cautionary feedback may be provided, such as, "You smoked 10 times a day today, and you'd better reduce it for health."

The feedback transmission unit 460 may transmit feedback to the user about the corresponding operation when an unexpected operation is recognized when the user's behavior is recognized by the behavior recognition unit 440. At this time, the feedback can be directly input from the user.

If there is no pre-stored motion pattern data and data for the first characteristic section and pre-stored matching motion pattern data, the register 470 may receive an input for the corresponding motion pattern data from a user. have.

Alternatively, the registration unit 470 may receive a separate operation pattern and operation name from a user, and store the received operation pattern data and operation name in a database to register as operation data.

Through this process, an operation pattern such as an exercise motion may be input from the user, and the user may generate his own motion. In addition, the generated exercise may be shared with other users.

7 is a flowchart of a method for providing feedback on a behavior in a behavior recognition server according to an embodiment of the present invention. The method for providing feedback on the behavior in the behavior recognition server 120 illustrated in FIG. 7 includes steps processed in time series by the behavior feedback providing system 1 according to the embodiment illustrated in FIGS. 1 to 6B. do. Therefore, even if it is omitted below, it is also applied to a method of providing feedback on an action in the action recognition server 120 according to the embodiment shown in FIGS. 1 to 6B.

In step S710, the behavior recognition server 120 may receive a plurality of sensor data measured for a predetermined time by a plurality of sensors embedded in the wearable device 110 from the wearable device.

In step S720, the behavior recognition server 120 may detect at least two singularities from the sensor data.

In step S730, the behavior recognition server 120 may recognize the user's behavior based on the similarity between the data for the first feature section including at least two singularities and the plurality of pre-stored motion pattern data.

In step S740, the behavior recognition server 120 may transmit feedback about the recognized behavior to at least one of the wearable device 110 or the user terminal 130.

In the above description, steps S710 to S740 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order between the steps may be switched.

The method for providing feedback on behavior in the wearable device and the server described with reference to FIGS. 1 to 7 is also implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. Can be. In addition, the method for providing feedback on actions in the wearable device and the server described with reference to FIGS. 1 to 7 may also be implemented in the form of a computer program stored in a medium executed by a computer.

Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration only, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. do.

110: wearable device
120: server
130: user terminal
210: sensor data collection unit
220: transmission unit
230: feedback receiver
240: feedback output
410: storage
420: sensor data receiving unit
430: singularity detection unit
440: behavior recognition unit
450: location information receiving unit
460: feedback transmission unit
470: Register

Claims (19)

In the server that provides feedback on actions,
A sensor data receiver configured to receive a plurality of sensor data measured for a predetermined time by a plurality of sensors embedded in a wearable device from the wearable device;
A singularity detection unit that detects at least two or more singularities from the sensor data;
A behavior recognition unit recognizing a user's behavior based on the similarity between the data for the first feature section including the at least two singularities and a plurality of pre-stored motion pattern data; And
It includes a feedback transmission unit for transmitting the feedback for the recognized behavior to at least one of the wearable device or the user terminal,
The singularity detection unit detects sensor fusion data including the at least two singularities from the collected plurality of sensor data through data fusion, and extracts data for the first feature section from the sensor fusion data. That is, behavior recognition server.
The method of claim 1,
The singularity detection unit detects the at least two singularities based on at least one of a peak value, a valley, a zero crossing, a graph area, a curvature, a time, an angle, an acceleration, a distance, and an orbit from the sensor data. server.
The method of claim 1,
The behavior recognition unit analyzes the similarity between the data for the first feature section and the plurality of pre-stored motion pattern data using machine learning, and corresponds to the data for the first feature section based on the analyzed similarity. A behavior recognition server that recognizes motion.
The method of claim 3,
A storage unit that stores motion data corresponding to each of the plurality of motion patterns in a database
It further includes, behavior recognition server.
The method of claim 3,
The behavior recognition unit,
Clustering the data for the plurality of feature sections into a plurality of similar operation groups by performing first clustering on the data for the plurality of feature sections including the data for the first feature section,
A behavior recognition server that recognizes an action corresponding to any one of the similar operation groups including data for the first feature section as the user's behavior.
The method of claim 4, wherein
Location information receiving unit for receiving the location information of the wearable device from the wearable device
It further includes, behavior recognition server.
The method of claim 5,
The behavior recognition unit classifies the plurality of similar operation groups into a plurality of location-based action groups by performing a second clustering on the classified plurality of similar operation groups based on the location information of the wearable device. server.
The method of claim 4, wherein
The storage unit stores the recognized user's actions in chronological order,
The stored action includes a first action, a second action, and a third action performed within a predetermined time,
The behavior recognition unit determines the association between each of the first behavior, the second behavior, and the third behavior, and deletes any one of the first behavior, the second behavior, and the third behavior based on the association. Is, a behavioral awareness server.
The method of claim 1,
The feedback includes at least one of action information of at least one action included in the action history of the user and the action history of the user, and at least one of meaningful action information included in the at least one action by time zone. Recognition server.
In the method of providing feedback on the actions performed on the server,
Receiving a plurality of sensor data measured for a predetermined time by a plurality of sensors embedded in a wearable device from the wearable device;
Detecting at least two singularities from the sensor data;
Recognizing a user's action based on the similarity between the data for the first feature section including the at least two singularities and a plurality of pre-stored motion pattern data; And
And transmitting feedback on the recognized behavior to at least one of a wearable device or a user terminal,
The step of detecting the singularity,
Detecting sensor fusion data including the at least two singularities from the collected plurality of sensor data through data fusion; And
And extracting data for the first feature section from the sensor fusion data.
The method of claim 10,
The detecting of the singularity may include detecting at least two singularities based on at least one of a peak value, a valley, a zero crossing, an area of a graph, a curvature, a time, an angle, an acceleration, a distance, and an orbit from the sensor data. That includes, behavior recognition method.
The method of claim 10,
The step of recognizing the behavior may analyze the similarity between the data for the first feature section and the plurality of pre-stored motion pattern data using machine learning, and based on the analyzed similarity. And recognizing a motion corresponding to the data.
The method of claim 12,
Storing operation data corresponding to each of the plurality of operation patterns in a database
It further comprises, behavior recognition method.
The method of claim 13,
Recognizing the operation corresponding to the data for the first feature section,
Clustering data for the plurality of feature sections into a plurality of similar operation groups by performing first clustering on data for a plurality of feature sections including data for the first feature section; And
And recognizing an action corresponding to any one of the similar action groups including data for the first feature section as the action of the user.
The method of claim 14,
Receiving location information of the wearable device from the wearable device
It further comprises, behavior recognition method.
The method of claim 15,
The step of recognizing an operation corresponding to the data for the first feature section is performed by performing a second clustering on the classified plurality of similar operation groups based on the location information of the wearable device to perform the plurality of similar operation groups. A method of recognizing behavior, which is classified into a plurality of location-based behavior groups.
The method of claim 16,
The step of storing in the database includes storing the recognized user's actions according to a time sequence,
The stored action includes a first action, a second action, and a third action performed within a predetermined time,
The step of storing the recognized user's actions in chronological order may include determining a correlation between each of the first action, the second action, and the third action; And deleting one of the first action, the second action, and the third action based on each association.
The method of claim 10,
The feedback includes at least one of action information of at least one action included in the action history of the user and the action history of the user, and at least one of meaningful action information included in the at least one action by time zone. Recognition method.
In the method of providing feedback on the behavior in the wearable device,
Collecting a plurality of sensor data measured by a plurality of sensors embedded in the wearable device;
Transmitting the collected plurality of sensor data to a behavior recognition server;
Receiving feedback on the behavior of the recognized user based on the plurality of sensor data from the behavior recognition server and
The user's behavior is recognized based on the similarity between data for a first feature section including at least two singularities and a plurality of pre-stored motion pattern data,
The at least two singularities are detected from the plurality of sensor data,
Sensor fusion data including the at least two singularities is detected from the collected plurality of sensor data through data fusion, and data for the first feature section is extracted from the sensor fusion data. How to recognize behavior.

Images