CN118057369A - Identification method and electronic equipment - Google Patents
Identification method and electronic equipment Download PDFInfo
- Publication number
- CN118057369A CN118057369A CN202211459119.6A CN202211459119A CN118057369A CN 118057369 A CN118057369 A CN 118057369A CN 202211459119 A CN202211459119 A CN 202211459119A CN 118057369 A CN118057369 A CN 118057369A
- Authority
- CN
- China
- Prior art keywords
- data
- electronic device
- behavior
- user
- sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 106
- 230000006399 behavior Effects 0.000 claims description 275
- 230000001133 acceleration Effects 0.000 claims description 89
- 230000008859 change Effects 0.000 claims description 62
- 238000000605 extraction Methods 0.000 claims description 37
- 230000005484 gravity Effects 0.000 claims description 27
- 230000011218 segmentation Effects 0.000 claims description 22
- 238000005070 sampling Methods 0.000 claims description 21
- 239000012634 fragment Substances 0.000 claims description 20
- 230000003542 behavioural effect Effects 0.000 claims description 19
- 238000004458 analytical method Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000000638 solvent extraction Methods 0.000 claims 1
- 230000001976 improved effect Effects 0.000 abstract description 10
- 230000033001 locomotion Effects 0.000 description 106
- 230000009471 action Effects 0.000 description 53
- 230000008569 process Effects 0.000 description 50
- 238000004891 communication Methods 0.000 description 33
- 230000006854 communication Effects 0.000 description 33
- 230000006870 function Effects 0.000 description 32
- 238000012549 training Methods 0.000 description 26
- 238000010586 diagram Methods 0.000 description 25
- 238000012545 processing Methods 0.000 description 24
- 238000003909 pattern recognition Methods 0.000 description 22
- 238000007726 management method Methods 0.000 description 15
- 230000001174 ascending effect Effects 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 13
- 239000000284 extract Substances 0.000 description 13
- 230000005236 sound signal Effects 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 12
- 238000010295 mobile communication Methods 0.000 description 12
- 238000011160 research Methods 0.000 description 11
- 230000003068 static effect Effects 0.000 description 10
- 210000000988 bone and bone Anatomy 0.000 description 9
- 230000002829 reductive effect Effects 0.000 description 7
- 230000000284 resting effect Effects 0.000 description 7
- 238000006073 displacement reaction Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 239000003990 capacitor Substances 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008093 supporting effect Effects 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 239000013078 crystal Substances 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 230000001052 transient effect Effects 0.000 description 3
- 229920001621 AMOLED Polymers 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 2
- 239000000956 alloy Substances 0.000 description 2
- 230000003416 augmentation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000036772 blood pressure Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000006698 induction Effects 0.000 description 2
- 230000001939 inductive effect Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000630 rising effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000002138 osteoinductive effect Effects 0.000 description 1
- 230000010349 pulsation Effects 0.000 description 1
- 239000002096 quantum dot Substances 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Navigation (AREA)
Abstract
The application provides an identification method and electronic equipment. According to the method, the behavior semantic features which can better reflect the behavior pattern of the user are extracted from the data acquired by the sensor of the electronic equipment, the behavior of the user is identified by combining the behavior semantic features, and the accuracy of the identification result of the behavior of the user can be effectively improved.
Description
Technical Field
The present application relates to the field of terminal technologies, and in particular, to an identification method and an electronic device.
Background
According to the related research, people occupy more than 80% of indoor time every day, so that the key to the research of indoor behaviors is that if the behaviors of people in the indoor can be accurately identified, the method can play an important role in many research fields. For example: the method is characterized in that whether the floor switching action of the person is performed is researched, the indoor activities of the person are excavated, the health condition of the old and the children in the house is even concerned, and whether dangerous actions such as falling occur or not. However, nowadays, as the number of sensors integrated on a smart phone is continuously increased along with the updating iteration of the smart phone, how to use the sensors in the smart phone to acquire the current motion state of people is becoming a hot spot of technical research.
At present, the intelligent equipment needs to train a related prediction model by using a large amount of data collected by the sensor, and then predicts the current human behavior state through the prediction model. However, the accuracy of the identification result of the intelligent device on the human body behavior is low, and the identification efficiency is also poor. Thus, there is a need to develop more accurate and efficient methods for identifying human behavior.
Disclosure of Invention
The application aims to provide an identification method and electronic equipment. The electronic equipment can extract the behavior semantic features capable of better reflecting the behavior mode of the user based on the data acquired by the sensor, and identify the behavior of the user by combining the behavior semantic features. By implementing the identification method, the accuracy of the identification result of the electronic equipment on the user behavior can be effectively improved.
The above and other objects are achieved by the features of the independent claims. Further implementations are presented in the dependent claims, the description and the figures.
In a first aspect, the present application provides a method of identification, the method comprising: extracting characteristics of target data to obtain characteristic data, wherein the characteristic data comprise behavior semantic characteristic data, the target data are obtained based on initial data acquired by a sensor in electronic equipment, and the behavior semantic characteristic data comprise first characteristic data representing that the electronic equipment is in a super/weightless state and/or second characteristic data representing that the direction of the electronic equipment is changed; and carrying out feature analysis on the feature data to obtain a behavior recognition result, wherein the behavior recognition result comprises a first recognition result, and the first recognition represents the current behavior mode of the user.
The target data may be obtained based on initial data collected by a sensor in the electronic device, and specific processes related to the foregoing embodiments and related descriptions of subsequent embodiments may be referred to, which are not described herein. In the method, the electronic device may include a plurality of sensors, such as an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, and a WIFI sensor, where the sensors may collect corresponding sensor data at a certain frequency, and the data is the initial data.
Feature extraction refers to the process of converting the target data into feature data that characterizes the behavior of the user. In order to be able to study the behavior patterns of people, five behaviors of resting, walking, going up and down stairs and going up and down stairs of a human body can be identified accurately, in the process of extracting features of the target data, the electronic equipment combines different points of various behavior patterns of users in detail and data, and the behavior semantic feature data is extracted from the target data. The behavioral semantic feature data includes first feature data that characterizes the electronic device in a super/weightless state and/or second feature data that characterizes a change in orientation of the electronic device (user).
Specifically, the feature data may further include statistical feature data having a statistical meaning and time domain feature data having a physical meaning. Compared with the statistical feature data and the time domain feature data, the behavior semantic features better reflect the behavior semantic features of the user behavior pattern. For example, the human body is not overweight or weightless in more than one continuous data frame basically when walking, but the human body is overweight and weightless in a transient and large extent when walking, but the human body is in a static state when in an elevator relative to the elevator, but the human body is in an acceleration motion and deceleration motion when following the elevator, so the human body also presents a small-amplitude overweight (weightless) moving state along with the movement trend of the elevator, namely constant-speed driving and small-amplitude weightless (overweight) moving state along with the movement trend of the elevator, and the overweight state is generally maintained for about 2-4 s. For another example, when a human body enters an adjacent stair from a stair of a certain layer, the human body needs to turn 180 degrees. Therefore, the application can effectively improve the accuracy of the recognition result of the user behavior by extracting the behavior semantic features which can reflect the different points of different behaviors in detail and data
With reference to the first aspect, in one possible implementation manner, the extracting features of the target data includes: performing data segmentation on the initial data by adopting a sliding window with a first length to obtain first data, wherein the first data comprises a plurality of data frames with the first length; data segmentation is carried out on the first data by adopting at least two sample windows to obtain at least two data fragment sets, wherein the length of any one of the at least two sample windows is different from the lengths of other windows in the at least two sample windows; and taking the at least two data fragment sets as the target data.
It should be appreciated that since the action takes place for a certain period of time, the data collected by the sensor at a single moment is not sufficient to characterize the action of the user. Therefore, after the initial data is obtained, the electronic device can select a sliding window with a certain length and a certain window coverage rate to perform data segmentation on the initial data acquired by the sensor, namely, the initial data of a long-time sequence is segmented into smaller data frames (the length of each data frame is the length of the sliding window), so that the first data is obtained, and the data is convenient to extract features.
In addition, the samples extracted from the single data frame cannot well reflect the front-rear semantic features, for example, in the elevator driving process, a constant-speed driving time period exists, and if the features at the moment are extracted by simply using the single data frame, the features at the moment are found to be not greatly different from the plane static. That is, if the feature is extracted at this moment by using only a single data frame at the time of feature extraction, information that can be captured by the data in the window is too small, the recognition result output by the prediction model may be less accurate. The electronic device may then divide the first data using a sample window in units of data frames, and in the feature extraction process, the electronic device performs feature extraction in batches of several data frames. And for different behavior modes, the duration corresponding to the characteristic data with reference meaning corresponding to each behavior mode is also different. For example, when a user rides an elevator to perform a cross-layer exercise, in order to completely extract characteristic data of the user in a weightless state and an overweight state, the length of the sample window needs to be set longer; however, when the user is in the flat-bed static mode or the flat-bed moving mode, the motion state characteristics of the user can be reflected only by a plurality of data frames, and the length of the sample window can be correspondingly set to be shorter. Therefore, in this embodiment, the electronic device may use the at least two sample windows to perform data segmentation on the first data to obtain the at least two sets of data fragment sets, and use the at least two sets of data fragment sets as the target data, so that the feature data obtained later has better referential property, and further improves accuracy of the recognition result.
With reference to the first aspect, in one possible implementation manner, the performing data segmentation on the first data using at least two sample windows to obtain at least two data fragment sets includes: and carrying out data segmentation on the first data by adopting three sample windows to obtain three data fragment sets, wherein the window lengths corresponding to the three sample windows are respectively 5 first lengths, 10 first lengths and 15 first lengths.
In this embodiment, the at least two sample windows may be three sample windows, and the at least two sets of data fragment sets are three sets of data fragment sets. Further, the specific sizes of the three sample windows may be set to 5 data frame lengths, 10 data frame lengths and 15 data frame lengths, respectively. The values of these three window sizes are obtained by observation of a large number of experimental data. When the sampling frequency of each sensor in the electronic device is 100HZ, the data frame length is 2.56s (one data frame length), and the number of sample windows is 3, the accuracy of the prediction model is best when the sample window size is set to 5 data frame lengths, 10 data frame lengths, and 15 data frame lengths, and the accuracy of the output result can reach about 0.98.
With reference to the first aspect, in one possible implementation manner, the behavior recognition result further includes a second recognition result, where the second recognition result characterizes a point of time when the behavior mode of the user is switched.
In this embodiment, the behavior recognition result may further include landmark recognition results of the user, that is, the second recognition result. And landmark identifying the moment point at which the result represents the user to switch the behavior mode. That is, the landmark point identification result may be used to reflect the time when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result may reflect at which time the user finishes the flat layer motion and starts the cross layer motion.
It should be understood that the landmark point identification result is a result obtained by predicting the behavior of the user by the electronic device based on the data collected by the sensor, but there may be a certain error between the time of the result and the actual landmark point of the user. Therefore, in the process of training the prediction model, the electronic device can compare the landmark point identification result with the real landmark point data of the user to determine the delay time (sample number) of the start time and the end time of the predicted landmark point and the real landmark point, the identification accuracy of the predicted landmark point (the total landmark point of the landmark points with the identification error less than or equal to 3 samples) and the identification accuracy of the predicted landmark point (the total landmark point of the landmark points with the identification error less than or equal to 5 samples), and perform a large amount of simulation tests on the error data, so as to continuously optimize the project architecture and the model characteristics, and finally obtain a relatively perfect behavior pattern identification algorithm and a relatively perfect landmark identification algorithm, thereby improving the accuracy of the behavior identification result.
With reference to the first aspect, in a possible implementation manner, the behavior semantic feature data further includes at least one of third feature data, fourth feature data, and fifth feature data, where: the third characteristic data represents the change condition of geomagnetic intensity in the environment detected by the electronic equipment; the fourth characteristic data represents the change condition of the acceleration of the electronic equipment in the gravity direction; and the fifth characteristic data represents the change condition of the number of the WIFI networks detected by the electronic equipment.
It has been found that geomagnetism generally exhibits a generally continuously ascending or descending curve when a user performs a cross-layer movement by means of an escalator (in the present application, the escalator means an escalator); in addition, when the elevator opens or closes the elevator door, severe geomagnetic fluctuations occur in the elevator; in addition, when the human body performs the cross-layer action, acceleration changes in the vertical direction, and the number of the WIFI networks detected by the WIFI sensors in the electronic equipment in the elevator can also be increased or decreased instantaneously at the moment when the elevator door is closed or opened, so that the third data, the fourth data and the fifth data can also help to identify the specific behavior mode of the user. And when the behavior pattern recognition result is a multi-classification result, compared with the process of only extracting the first feature data and the second feature data, the process of further extracting all or part of semantic features in the third data, the fourth data and the fifth data can help the electronic equipment to better acquire the data characteristics corresponding to the current behavior pattern of the user, so that the accuracy of the behavior recognition result is improved.
In this embodiment, the electronic device may extract all or only part of the data features among the first feature data, the second feature data, the third feature data, the fourth feature data, and the fifth feature data included in the behavior semantic feature. For example, the electronic device may extract only the third feature data without extracting the first feature data and the second feature data, and the behavior recognition result obtained by combining the third feature data by the electronic device may be improved in accuracy to some extent compared with extracting only the statistical feature data and the time domain feature data.
With reference to the first aspect, in one possible implementation manner, the first characteristic data includes at least one of a small-amplitude super/weightless ratio, a large-amplitude super/weightless ratio, and a continuous small-amplitude super/weightless ratio, where the small-amplitude super/weightless ratio characterizes a ratio of a number of times that the electronic device is in a super/weightless state in a first data segment and the super/weightless value is smaller than a second threshold value in a total number of times corresponding to the first data segment; the large-amplitude super/weightlessness duty ratio characterizes the duty ratio of the time number of which the electronic equipment is in a super/weightlessness state and the super/weightlessness value is larger than a third threshold value in the first data segment in the corresponding total time number of the first data segment; the continuous small-amplitude super/weightlessness duty ratio characterizes the duty ratio of the time count corresponding to the maximum duration of the electronic equipment continuously in the super/weightlessness state in the first data segment in the total time count corresponding to the first data segment; the first data segment is any data segment in the target data.
As can be seen from the foregoing description, the human body basically does not have overweight or weightlessness exceeding one continuous data frame while walking, but the human body has instantaneous and large overweight and weightlessness while walking, but the human body is in a stationary state relative to the elevator when in the elevator, but the human body has acceleration and deceleration movements when following the elevator, so the human body also has a movement state of small overweight (weightlessness) -constant running-small weightlessness (overweight) along with the movement trend of the elevator, and the overweight state generally maintains about 2-4 s.
In this embodiment, the small-amplitude super/weightlessness duty cycle is different from the continuous small-amplitude super/weightlessness duty cycle in that: the small-amplitude super/weightlessness duty ratio is considered to be whether the super weightlessness amplitude of the electronic equipment at each moment is in a small-amplitude super/weightlessness interval within a period of time, the interval where the last moment and the next moment are positioned is not needed to be considered, and after the number of all the moments in the small-amplitude super/weightlessness interval within a short time is obtained, the ratio obtained by dividing the number by the number of all the moments in the period of time is the small-amplitude super/weightlessness duty ratio; the continuous small-amplitude super/weightlessness duty ratio is concerned with the longest time number continuously in the small-amplitude super/weightlessness range in the period, and then dividing the time number by all time numbers in the period, and the obtained ratio is the continuous small-amplitude super/weightlessness duty ratio.
In the embodiment, the electronic equipment can more accurately identify the behavior mode of the elevator and the behavior mode of flat-bed walking by extracting one or more behavior semantic features of the small-amplitude super/weightlessness duty ratio, the large-amplitude super/weightlessness duty ratio and the continuous small-amplitude super/weightlessness duty ratio.
With reference to the first aspect, in one possible implementation manner, the second feature data includes at least one of a single-sample maximum steering and a double-sample maximum steering, where the single-sample maximum steering characterizes a maximum steering angle of a user in a sampling duration corresponding to one data frame; the double sample maximum steering characterizes the maximum value of the sum of steering angles of any two continuous sampling moments in one data frame by a user.
When a human body enters an adjacent layer of stairs from a certain layer of stairs, the human body needs to turn 180 degrees. When a human body climbs the multi-layer stairs, the human body needs to turn 180 degrees for many times. Furthermore, the time interval between the occurrence of every adjacent two 180 ° turn actions is typically small. Furthermore, in combination with the behavioral habits of the masses, in general, after entering the elevator, the person turns around facing the elevator door and presses the button of the destination floor, after which it is likely to face the elevator door all the way until he leaves the elevator. When the human body reaches the destination floor and goes out of the elevator, the human body generally goes straight, turns left or turns right, but regardless of the direction of travel of the human body after going out of the elevator door, the human body generally does not turn 180 degrees any more, and even if the human body turns 180 degrees again in the subsequent behavior, the time interval between the time of this turning and the time of the last turning may be long.
Therefore, in the embodiment, by extracting one or more behavior semantic features of the small single sample maximum steering and the double sample maximum steering, the electronic device can more accurately perform the behavior pattern of going up and down stairs.
Specifically, the single sample maximum steering satisfies:
MaxAngle＝Max(Angle)；
The double sample maximum steering satisfies:
MaxDoubleAngle＝Max(Angle(t-1)+Anglet)；
Wherein abs represents an absolute value function, degress represents an angle function for converting an radian value into a corresponding radian value, and gyr x, gyr y and gyr z represent an angular velocity of rotation of the gyro sensor about an X axis, an angular velocity of rotation about a Y axis and an angular velocity of rotation about a Z axis, respectively; the acc x, the acc y and the acc z represent components of the acceleration sensor in the X-axis, the Y-axis and the Z-axis, respectively, and the max represents a maximum function.
With reference to the first aspect, in one possible implementation manner, the third feature data includes a geomagnetic fluctuation mean, where the geomagnetic fluctuation mean represents an average value of data change degrees corresponding to geomagnetic data segments with a plurality of second lengths, the data change degree corresponding to any geomagnetic data segment in the geomagnetic data segments with the plurality of second lengths is determined by a maximum magnetic value and a minimum magnetic value in the geomagnetic data segment, the geomagnetic data segments with the plurality of second lengths are all contained in a first data segment, and the first data segment is any data segment in the target data.
It has been found that geomagnetism generally exhibits a generally continuously ascending or descending curve when a user performs a cross-layer movement by means of an escalator (in the present application, the escalator means an escalator); in addition, when the elevator opens or closes the elevator door, severe geomagnetic fluctuations occur in the elevator.
Therefore, in this embodiment, by extracting the geomagnetic fluctuation mean value, the electronic device may more accurately perform a behavior pattern of going up and down stairs.
Specifically, the geomagnetic fluctuation mean value satisfies:
windowSize＝winSize*100；
MagSlice＝Mag[t:t+windowSize]；
The winsize represents the window size corresponding to a first data segment, the first data segment is any one data segment in the target data, which is obtained based on the data acquired by the geomagnetic sensor, and the T represents the total amount of geomagnetic data in the first data segment; the MagSlice represents all data in the time period t to (t+ windowSize) in the first data segment; max and Min respectively represent a maximum value and a minimum value of geomagnetic data acquired in a time period from t to (t+ windowSize) in the first data segment; and MAGDIFFAVG is geomagnetic fluctuation mean value corresponding to the first data segment.
With reference to the first aspect, in one possible implementation manner, the fifth feature data includes a WIFI change proportion, where the WIFI change proportion characterizes a ratio of a first WIFI network number to a second WIFI network number, where the second WIFI network number is a number of WIFI network addresses included in a last data frame of a first data segment, and the first WIFI network number is a number of WIFI network addresses that exist in the last data frame of the first data segment but not in the first data frame of the first data segment.
The WIFI change proportion feature reflects the change condition of the number of the WIFI networks detected by the electronic equipment, and the number of the WIFI networks detected by the WIFI sensor in the electronic equipment in the elevator can be increased or decreased instantaneously at the moment when the elevator door is closed or opened, so that the behavior semantic feature of the WIFI change proportion can be well used for identifying whether a user takes the elevator or not in the embodiment.
In a second aspect, an embodiment of the present application provides an electronic device, including: one or more processors and memory; the memory is coupled with the one or more processors, the memory for storing computer program code comprising computer instructions that the one or more processors invoke the computer instructions to cause the electronic device to perform the method of the first aspect or any of the possible implementations of the first aspect.
In a third aspect, embodiments of the present application provide a chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause the electronic device to perform a method as in any of the possible implementations of the first aspect.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium comprising instructions that, when executed on an electronic device, cause the electronic device to perform a method as in any one of the possible implementations of the first aspect.
Drawings
FIG. 1 is a schematic diagram of a process for identifying a behavior state of a user according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a training/use process of a predictive model according to an embodiment of the application;
FIG. 3 is a schematic view of a data segmentation scenario according to an embodiment of the present application;
FIG. 4 is a schematic diagram of data collected by an acceleration sensor according to an embodiment of the present application;
fig. 5 is a schematic diagram of acceleration variation trend of an electronic device in multiple motion states according to an embodiment of the present application;
fig. 6 is a schematic view of a scene of a person going upstairs according to an embodiment of the present application;
fig. 7 is a schematic view of a scene of a human-body boarding elevator according to an embodiment of the present application;
Fig. 8 is a schematic diagram of a variation trend of geomagnetic values under a plurality of motion states according to an embodiment of the present application;
Fig. 9 is a logic diagram for determining a WIFI change ratio according to an embodiment of the present application;
FIG. 10 is a schematic diagram of a correspondence relationship between a sample window length and model accuracy according to an embodiment of the present application;
Fig. 11 is a schematic view of a scenario in which data is divided from first data according to an embodiment of the present application;
fig. 12 is a schematic diagram of real landmark point data and predicted landmark point data according to an embodiment of the present application;
FIG. 13 is a block diagram of a behavior recognition algorithm according to an embodiment of the present application;
fig. 14 is a schematic diagram of an electronic device according to an embodiment of the present application;
FIG. 15 is a flowchart of an identification method according to an embodiment of the present application;
Fig. 16 is a flowchart of a data processing method according to an embodiment of the present application.
Detailed Description
The terminology used in the following embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates to the contrary. It should also be understood that the term "and/or" as used in this disclosure refers to and encompasses any or all possible combinations of one or more of the listed items.
The embodiment of the application relates to an identification method and electronic equipment, and in order to facilitate understanding, related terms related to the embodiment of the application are introduced.
(1) Behavior recognition and behavior recognition result
The purpose of behavior recognition is a computer technology that analyzes and recognizes the type of human actions and patterns of behavior through a series of observations, and describes them by means of natural language, etc. Due to the complexity and diversity of human behavior, often the identified results are diverse and output with probabilities of behavior types. Methods for human body movement behavior feature recognition can be roughly classified into two main categories, machine vision-based and sensor-based.
The recognition based on machine vision is to describe and recognize human body behaviors in a video sequence frame by taking an image processing technology as a core. The processing process is complex, the operation amount is large, and the method is easily influenced by environmental factors. The identification based on the sensor has the advantages of low cost, small volume, difficult environmental influence and the like. As information technology has evolved, various mobile and wearable devices are growing in acceleration, their performance and embedded sensors have also become diversified, for example: high definition cameras, light sensors, gyroscopic sensors, acceleration sensors, GPS, temperature sensors, and the like. Various sensors record user information at a moment, and the recorded information can be used for not only predicting the position of a user, but also identifying the behavior of the user.
In the application, the behavior recognition result may include a behavior pattern recognition result of the user, and optionally, the behavior recognition result may further include landmark (an up-down elevator, an initial point and an end point of an up-down elevator) recognition result.
The behavior pattern recognition result is the behavior pattern of the user at the moment. Specifically, in the embodiment of the present application, the behavior patterns of the user may be divided into a flat-layer behavior model and a cross-layer behavior model. The flat layer behavior mode can comprise static and walking, and the cross-layer behavior mode can comprise ascending and descending stairs, ascending and descending elevators and ascending and descending elevators. Optionally, when the prediction model 203 predicts the behavior mode of the user, the prediction model may adopt a multi-classification to two-classification mode, firstly determine which of five behavior modes (stationary, walking, going up and down stairs, going up and down elevators, going up and down stairs) the behavior action of the user belongs to, then process the behavior mode according to the requirement, and combine the recognition results, namely, classifying the stationary and walking into a flat-layer behavior mode, and classifying the going up and down stairs, going up and down elevators and going up and down stairs into a cross-layer behavior mode.
The landmark identifying module is used for outputting landmark point identifying results. The landmark point identification result can be used for reflecting the moment when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result can reflect at which time the user finishes the flat layer motion and starts the cross layer motion.
(2) Acceleration sensor
The acceleration sensor is a sensor capable of measuring acceleration and is generally composed of a mass block, a damper, an elastic element, a sensitive element, an adaptive circuit and the like. During acceleration, the sensor obtains an acceleration value by measuring the inertial force borne by the mass block and utilizing Newton's second law. Common acceleration sensors include capacitive, inductive, strain, piezoresistive, piezoelectric, etc., according to the sensor sensing element.
Currently, most acceleration sensors on mobile phones are capacitive. The design principle is as follows: one mass block can move on a certain shaft, one end of a miniature spring is fixed, the other end of the miniature spring is connected with the mass block, the mass block is fixedly connected with one polar plate of a capacitor, and the other polar plate of the capacitor is fixed. When the mobile phone has acceleration in the direction, the mass block compresses or stretches the spring to promote the distance between the two polar plates of the capacitor to be changed, so that the capacitance is changed. If the charge of the capacitor is unchanged, the voltage between the two plates will change along with the change of the distance. Therefore, the acceleration signal in a certain direction of the mobile phone is converted into a voltage signal. Acceleration sensors today generally integrate X, Y, Z sensors in mutually perpendicular directions together, so that the sensors can sense the motion of the mobile phone in three-dimensional space.
(3) Gyroscope sensor
The gyroscope sensor is a simple and easy-to-use control system based on free space movement and gesture positioning, is originally applied to a helicopter model, and is widely applied to mobile portable equipment such as mobile phones and the like. The gyroscope is called an angular velocity sensor, and can well measure the rotation and deflection actions of the mobile phone, so that the mobile phone can be correspondingly operated. The principle of the gyroscope is as follows: the direction of the rotation axis of a rotating object is not changed when it is not affected by external force. According to this theory, one uses it to maintain direction. The direction indicated by the axis is then read in a number of ways and the data signal is automatically transmitted to the control system. Therefore, a gyro sensor is often used for sensing a holding mode of a device and determining a scene such as a direction of a moving object.
(4) Geomagnetic sensor
Geomagnetic sensors are used for determining magnetic field strength by measuring resistance change, and are mostly applied to compass and map navigation. Geomagnetic sensors use Anisotropic Magnetoresistance (AMR) materials to detect the magnitude of magnetic induction in space. The alloy material with the crystal structure is sensitive to an external magnetic field, and the change of the intensity of the magnetic field can lead to the change of the self resistance value of AMR. Therefore, when the magnetic field around the electronic device equipped with the geomagnetic sensor changes, the geomagnetic sensor in the electronic device can sensitively detect the change in the magnetic field strength even if the magnitude of the change in the magnetic field is small.
(5) WIFI sensor
The WIFI sensor may be used to search/scan for WIFI network signals that exist around, and may be used to obtain a physical address for each WIFI signal. It should be understood that the term "WIFI sensor" as used in the application is only used to refer to elements for searching/scanning for the presence of WIFI network signals around. The WIFI sensor may be a processing element that is set up separately, or may be implemented in the same chip as other elements (e.g., other sensors in the foregoing description). In addition, the WIFI sensor may be stored in a memory element of the controller in a form of a program code, and a certain processing element of the processor invokes and executes a function of the WIFI sensor.
(6) Time domain features and statistical features
The action (Movements) is the most basic unit of motion and can be identified without the aid of other actions occurring back and forth, but the motion (Activity) is made up of a continuous sequence of actions.
The time domain characteristics of the behavioral actions characterize changes caused by the accumulation of multiple actions by the user over a period of time, which can be embodied in the displacement and posture of the user, and can be quantified by integrating the data acquired by the various sensors. For example, the acceleration and gyroscope (angular velocity) data are integrated as a double integral and a double integral, respectively, and the obtained result can be roughly understood as the velocity and displacement of the user. According to different behaviors of the human body, such as the human body is at rest, the speed and the displacement are obviously smaller than those of the human body when the human body walks, which can help the model to roughly classify the behaviors of the human body. The statistical characteristics of the behavioural actions reflect the interrelation between the overall data contained in the sequence of actions, taking into account the variation of the data over the time period from the individual's perspective to the overall perspective.
Therefore, when the behavior of the target object is identified, data of an action sequence which can represent the target object is acquired first, and a time sequence relation of actions in the action sequence and a statistical relation of each action are determined based on the data, so that a time domain feature and a statistical feature of the action sequence are obtained, and an identification result of the behavior action is obtained.
(7) Behavior semantic features
Behavior is the largest range of concepts compared to actions and movements, which typically involve interactions and causal relationships with the environment. That is, when a person performs a certain action, especially a floor switching action, the nature of the environment in which it is located may cause certain actions to occur, and the action may also cause certain parameters to change in some form of nature. For example, when a user walks up stairs or down stairs, the user may need to turn around 180 ° when walking to the end of a certain floor ladder to enter the next floor ladder; for another example, when a user enters the elevator, if the elevator is upward, in the movement process of the elevator, the user can be in an overweight state in the acceleration stage of the elevator according to Newton's law because the movement trend of the elevator is that the movement is accelerated firstly, accelerated to the maximum speed, then uniform movement is carried out, then decelerated, and finally decelerated to the speed of 0; accordingly, the user may be in a weightless condition during the deceleration phase of the elevator. And such overweight and weightless conditions typically last for 2-3 seconds. Therefore, the accuracy of the recognition result cannot be ensured by recognizing the behavior and action of the person from the time domain features and the statistical features. Based on the method, the electronic equipment can identify the behavior actions of the user by combining different points, namely semantic features of the behavior actions, of various behaviors in detail and data, so that accuracy of identification results is improved.
According to the related research, people occupy more than 80% of indoor time every day, so that the key to the research of indoor behaviors is that if the behaviors of people in the indoor can be accurately identified, the method can play an important role in many research fields. For example: the study person can judge whether the floor switching action happens or not, and even pay attention to the physical health condition of the old and the children in the house, whether dangerous actions such as falling happen or not.
Today's society, intelligent devices with powerful functions are becoming more and more common and becoming an indispensable necessity in people's daily life. People generally carry intelligent equipment with them wherever they are, which makes intelligent equipment gradually develop into a behavior habit tool capable of identifying people at any time and in any place. In recent years, the number of sensors integrated on intelligent equipment is increased along with the updating iteration of mobile phones, how to acquire the current motion state of people by using the mobile sensors and radio frequency signal information of the equipment is becoming a hot spot of technical research, and behavior recognition is also developing as a branch of technical research.
The current method for researching human body behaviors by using intelligent equipment mainly utilizes the data of various sensors in the intelligent equipment (or the characteristics extracted by utilizing the sensor data) to be put into a machine learning model for training, and then predicts the current human body behavior state by training a complete machine learning model.
Fig. 1 shows a schematic diagram of a process of identifying a behavior state of a user by using data collected by a sensor by using a smart device at present. As shown in fig. 1, most of the sensors in the intelligent device are included in the data acquisition module, and besides the air pressure sensor, the WIFI sensor, the acceleration sensor shown in fig. 1, these sensors may also include a pressure sensor, a gyroscope sensor, and the like. When the behavior state is identified, the intelligent equipment can conduct preprocessing such as filtering and denoising on the original data collected by the sensors, then the obtained data are transmitted to the characteristic engineering module, and the characteristic engineering module analyzes and processes the data, such as integrating and averaging the data, so as to obtain time domain features and statistical features capable of representing the behavior of the user. Furthermore, the intelligent device converts the time domain features and the statistical features into digital features which can be used for machine learning, inputs the digital features into a pre-trained prediction model, and outputs a final recognition result by the prediction model.
However, as can be seen from fig. 1 and the foregoing description, the identification method and the prediction model used by the smart device at present need to use a large amount of sensor information to obtain the current behavior state of the human body, which clearly increases the power consumption of the smart device and reduces the identification efficiency of the smart device. In addition, the currently used prediction model lacks of mining on behavior semantic features in the training and using processes, and the accuracy of the model is reduced when part of sensors fail or are not present. For example, when studying whether a human body is subjected to cross-layer behavior, if the air pressure sensor data is missing, the prediction model is likely to be invalid, and the prediction result is inaccurate.
In order to overcome the defects, the application provides an identification method and electronic equipment, wherein the predictive model provided by the application is deployed in the electronic equipment, and the electronic equipment can identify the behavior and action of a user based on the identification method. In the identification process, the electronic equipment can utilize the behavior characteristics representing the behavior of the user as the input of the prediction model, and can obtain more reference data by using fewer sensors, so that the accuracy of the identification result is improved, and the power consumption of the equipment is reduced.
First, a schematic diagram of a training/application process of a prediction model in an electronic device provided by the application is introduced.
It should be understood that the most different between the training and using processes of the prediction model is the difference of the data, but the training process and the application process of the prediction model are similar to the processing logic of the data, so the prediction model provided by the application is described by taking the application process of the prediction model as an example.
As shown in fig. 2, the data used by the prediction model provided by the present application is mainly acquired by the sensors in the data acquisition module 201 of the electronic device. Specifically, the sensors in the data acquisition module may include an acceleration sensor 201a, a gyroscope sensor 201c, a geomagnetic sensor 201b, and a WIFI sensor 201d. Among them, the acceleration sensor 201a, the gyro sensor 201c, and the geomagnetic sensor 201b may also be referred to as a 3-axis acceleration sensor, a 3-axis gyroscope, and a 3-axis electronic compass (geomagnetic sensor), respectively, and a combination of these three sensors may be referred to as a nine-axis sensor.
It will be appreciated that the structure illustrated by the data acquisition module 201 in this embodiment does not constitute a specific limitation on the data acquisition module 201. In other embodiments of the application, the data acquisition module 201 may include more or less sensors than illustrated, some or all of which may be integrated into one element or may be implemented in hardware, software, or a combination of software and hardware.
It should be noted in advance that, in the training process of the prediction model, the training data set used by the prediction model may be from a plurality of devices. That is, the manufacturer of the electronic device may train the predictive model using a data set collected by a plurality of devices as a training set of the predictive model by a device having a model training function. And then, uniformly deploying the prediction model into the electronic equipment in the production process of the electronic equipment. When the electronic equipment deploying the prediction model identifies the behavior and action of the user (namely, during the use process of the prediction model), each electronic equipment can only use the sensor data acquired by each electronic equipment as the prediction model to input so as to obtain the identification result of the behavior and action of the user.
Specifically, in the process that the electronic device uses the prediction model to identify the behavior action of the user, each sensor in the data acquisition module can acquire data at a certain sampling frequency, and the data directly acquired by each sensor can be called initial data. Alternatively, the sampling frequency of each sensor may be 100HZ, i.e. 100 data acquisitions per second.
After the initial data is collected, the electronic device may select a sliding window with a certain length and a certain window coverage to perform data segmentation on the initial data collected by the sensor after performing preprocessing such as filtering and denoising on the initial data, that is, the initial data of the long-time sequence is segmented into smaller data frames (the length of each data frame is the length of the sliding window, in some embodiments, the length of the data frame may also be referred to as the first length), so that the feature engineering 202 performs feature extraction on the data.
It should be understood that the length of the data frame, i.e. the length of the sliding window described above, not only directly affects the quality and classification performance of feature extraction, but also determines whether the method is suitable for use in a real-time system. For example, if the sliding window is too small, each data frame obtained by segmentation may be only a part of the motion, and the complete behavior state of the motion cannot be reflected, so that the extracted features cannot effectively represent the motion, and further, the recognition performance is reduced, and the electronic device needs to frequently recognize, so that the calculation amount is increased; if the sliding window is too large, multiple actions may be included in one data frame, so that the system cannot effectively identify the current behavior, the real-time performance of the system may be affected, and the accuracy of the identification result may be affected. Therefore, when the above initial data is subjected to data division, the sliding window length will directly affect the accuracy of the recognition result.
In addition, the coverage of the sliding window is also one of important factors affecting the accuracy of the recognition result. The coverage rate of the sliding window is the overlapping rate between two adjacent divided windows in the data dividing process, wherein the coverage rate is 0% which indicates that the adjacent windows are not overlapped, and the coverage rate is 50% which indicates that the current window contains half of the data of the previous window.
The current research shows that the window length is 2.56s, which is a comparatively ideal window length, because the data needs to be subjected to Fourier transform when the frequency domain calculation is carried out, and the N power of 2 is adopted as a time window, thereby ensuring the integrity of the data participating in the frequency domain calculation. In addition, the adoption of the window coverage rate of 50% can effectively reduce disturbance caused by transitional behaviors. Therefore, in the embodiment of the application, the electronic device may use a sliding window with a window length of 2.56s and a window coverage rate of 50% to perform data segmentation on the initial data. For each sensor, the acquired data is divided to obtain a plurality of data segments, wherein one data segment can be called a data frame. It is easy to know that when the electronic device samples at a frequency of 100HZ, and uses a sliding window with a window length of 2.56s and a window coverage rate of 50% to segment the initial data collected by the sensor, each data frame (also referred to as a sample during model training and testing) in the obtained data frames contains 256 times of data.
A scene diagram of data division shown in fig. 3 is taken as an example for explanation. In fig. 3, the initial data 30 may be initial data obtained by sampling one or more sensors (for example, one or more of an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, and a WIFI sensor) in the electronic device at a sampling frequency of 100HZ, and continuously sampling for 2 minutes. Assuming that the electronic device uses a sliding window with a window length of 2.56s and a window coverage of 50% to divide the initial data 30, it is easy to calculate that the initial data 30 may be divided into 93 data frames, where each data frame includes 256 time data. The data frames 301-304 shown in fig. 3 are partially consecutive data frames obtained by dividing the initial data 30. The sampling time corresponding to the data frame 301 is 0s-2.56s, and the sampling time corresponding to the data frame 302 is 1.28s-3.84s. That is, the data collected by each sensor in the electronic device during the period of 1.28s-2.56s is contained in both the data frame 301 and the data frame 302, and the length of the data is (1.28++2.56) ×100+=50% in both the data frame 301 and the data frame 302. Similarly, there is a 50% length data coverage in each of data frames 303 and 302, data frames 304 and 303, and any two subsequent adjacent data frames.
After data acquired by the sensor in the electronic device is segmented, the electronic device further transmits the segmented data to the feature engineering 202 for feature extraction. In the embodiment of the present application and the following embodiments, after the initial data is divided, the data obtained for transmission into the feature engineering 202 may be referred to as first data, and it may be understood that the first data is data in units of data frames.
Feature extraction refers to the process of computing the segmented data frames to generate feature data that characterizes the motion. In the application, in order to be able to study the behavior patterns of people with great importance, five behaviors of resting, walking, going up and down stairs and going up and down stairs of a human body are accurately identified, in the process of extracting the characteristics of data, the electronic equipment not only provides time domain characteristics with physical significance and statistical characteristics with statistical significance from the data obtained by segmentation, but also can comprise behavior semantic characteristics, wherein the behavior semantic characteristics can comprise one or more characteristics of characteristics reflecting the super/weightlessness state of the electronic equipment (user), characteristics reflecting the steering amplitude of the electronic equipment (user), characteristics reflecting the change condition of geomagnetic intensity in the environment where the electronic equipment (user) is located, and characteristics reflecting the change condition of acceleration of the electronic equipment (user) in the gravity direction and the change condition of the number of WIFI networks in the environment where the electronic equipment (user) is located. That is, features obtained by feature extraction of data collected and segmented by the data module by the feature engineering 202 can be roughly classified into three types, namely, statistical features, temporal features, and behavioral semantic features.
When the sensors in the data acquisition module 201 include four sensors, namely, an acceleration sensor 201a, a gyroscope sensor 201c, a geomagnetic sensor 201b, and a WIFI sensor 201d, the electronic device may acquire the following three types of features (statistical features, temporal features, and behavioral semantic features) according to the following table 1:
TABLE 1
As shown in table 1, when the sensors in the data acquisition module 201 include four sensors of an acceleration sensor 201a, a gyroscope sensor 201c, a geomagnetic sensor 201b, and a WIFI sensor 201d, the statistical features extracted by the feature engineering 202 are applied to calculate the statistical results of horizontal acceleration, vertical acceleration, acceleration vector sum, gyroscope vector sum, geomagnetic vector sum, and 5 data in 11 dimensions, respectively, to obtain a total of 55 dimensional features, the 11 dimensions including a mean, a standard deviation, a variance, a median, a minimum, a maximum, a difference between maximum and minimum, a quarter difference, kurtosis, a skewness, and a root mean square. The statistical features are simple to calculate and small in calculation amount, but reflect the mutual connection between overall data, so that the prediction model is beneficial to considering the change condition of the data in the whole time period from the individual angle to the overall angle.
The time domain features extracted by the feature engineering 202 include 4-dimensional features of a double integral of the acceleration vector sum, a double integral of the gyroscope (angular velocity) vector sum, and correlation coefficients of the acceleration vector sum Y-axis and Z-axis. These features may represent the user's movement speed and displacement. For example, the speed and displacement of a human body when stationary is significantly less than the speed and displacement of a human body when walking. For another example, the acceleration Y-Z correlation coefficient refers to the correlation coefficient of the acceleration Y axis and the acceleration Z axis, and in the case that the mobile phone is in a flat end state, since the human body presents different periodicity when walking and going up and down stairs, and the steps of the human body also have different magnitudes, the correlation coefficients of the Y axis and the Z axis are also different. Thus, these temporal features can help the predictive model to coarsely classify human behavior actions.
The behavior semantic features extracted by the feature engineering 202 include one or more behavior semantic features including 11-dimensional behavior semantic features such as a small-amplitude overweight duty cycle, a large-amplitude overweight duty cycle, a small-amplitude weightlessness duty cycle, a large-amplitude weightlessness duty cycle, a continuous small-amplitude overweight duty cycle, a continuous small-amplitude weightlessness duty cycle, a single-sample maximum steering, a double-sample maximum steering, a geomagnetic fluctuation mean value, projection of acceleration in the gravity direction, a WIFI change ratio and the like. The small-amplitude overweight duty ratio, the large-amplitude overweight duty ratio, the small-amplitude weightlessness duty ratio, the large-amplitude weightlessness duty ratio, the continuous small-amplitude overweight duty ratio, the continuous small-amplitude weightlessness duty ratio and the projection of the acceleration in the gravity direction can be obtained based on the initial data acquired by the acceleration sensor 201 a; single sample maximum steering, double sample maximum steering can be obtained based on initial data acquired by the gyro sensor 201 c; the geomagnetic fluctuation mean value can be obtained based on the initial data acquired by the geomagnetic sensor 201 b; and the WIFI change ratio may be obtained based on the initial data collected by WIFI sensor 201 d.
These behavioral semantic features reflect the different points exhibited by various types of behavior in terms of detail as well as in terms of data. In other words, when the user performs different behavior actions, the data values and/or the data change forms corresponding to the behavior semantic features are different; in other words, when a user performs a certain behavioral action, the data value and/or the data change form corresponding to a certain feature in the behavioral semantic features are obviously different from the data value and/or the data change form when the user performs other behavioral actions. The 11-dimensional behavioral semantic features described above are described in detail below in conjunction with fig. 4-9.
1) Small-amplitude overweight duty ratio, large-amplitude overweight duty ratio, small-amplitude weightlessness duty ratio, large-amplitude weightlessness duty ratio, continuous small-amplitude overweight duty ratio, continuous small-amplitude weightlessness duty ratio
The system is characterized by comprising a small-amplitude overweight duty ratio, a large-amplitude overweight duty ratio, a small-amplitude weightlessness duty ratio, a large-amplitude weightlessness duty ratio, a continuous small-amplitude overweight duty ratio and a continuous small-amplitude weightlessness duty ratio, wherein the continuous small-amplitude weightlessness duty ratio is used for reflecting the characteristic of the ultra/weightlessness state of electronic equipment (users) and is mainly used for identifying the walking or the up-down elevator behavior mode of the users.
It is found that the human body is not overweight or weightlessness in more than one continuous data frame basically when walking, but the human body is overweight and weightlessness in a transient and large extent when walking, but the human body is in a static state when being in an elevator in most cases relative to the elevator, but the human body is in an acceleration motion and deceleration motion when following the elevator, so the human body also presents a motion state of small overweight (weightlessness) -constant running-small weightlessness (overweight) along with the motion trend of the elevator, and the overweight state is generally maintained for about 2-4 s.
It is known from the physical knowledge that if the object is only subjected to gravity and supporting force, when the acceleration of the object in the gravity direction is 0, the object is balanced in the gravity direction, and the object is not in an overweight state or a weightless state; when the acceleration of the object in the gravity direction is greater than 0, the gravity borne by the object is greater than the supporting force, and the object is in a weightlessness state; when the acceleration in the gravity direction is smaller than 0, the gravity applied to the object is smaller than the supporting force, and the object is in an overweight state. Fig. 4 shows a schematic representation of the data collected by the acceleration sensor during a user walking into the elevator in a flat floor, and out of the elevator after the elevator has moved to the target location. For ease of understanding, the horizontal axis in the coordinate system shown in fig. 4 represents time, and the vertical axis represents acceleration of the user in the gravity direction (or acceleration of the electronic device in the gravity direction measured by the acceleration sensor), where time t 41-time t42 and time t 45-time t46 are periods of time when the user walks on a flat floor, and time t 42-time t45 are periods of time when the user moves in the elevator along with the elevator. It can be understood that the time t42 is the time when the user enters the elevator, and the time t45 is the time when the user exits the elevator. As can be seen from the images corresponding to the time t 41-t 42 and the time t 45-t 46 in fig. 4, when the user walks on the flat floor, an instantaneous and large-amplitude overweight and weightlessness appears, and the overweight and weightlessness phenomenon is instantaneous and can only be ended at 1 s; however, as can be seen from the image corresponding to the time t 42-t 45 in fig. 4, when the user moves along with the elevator, a small-amplitude overweight and small-amplitude weightlessness state can occur, the amplitude of the overweight at this time is obviously smaller than that of the overweight during walking on a flat floor, and the overweight state generally lasts for a longer time. Specifically, the user is in a weightless state from time t44 to time t45 in fig. 4, and the elevator may be traveling under acceleration; while the user is in an overweight condition at time t 42-time t43 in fig. 4, the elevator may be decelerating downwards.
Fig. 5 shows a schematic diagram of acceleration variation trend of the electronic device in various motion states.
A schematic diagram of the acceleration change trend measured by the electronic device when the user gets on the elevator is shown in fig. 5 (a). For ease of understanding, the image showing the acceleration change trend shown in fig. 5 (a) is an image of an ideal state (i.e., a state in which the electronic device is only subjected to gravity and supporting force), and the image shown in fig. 5 (a) is more clear and intuitive than the image corresponding to the period from time t42 to time t45 in fig. 4. As can be seen from the foregoing description, in (a) of fig. 5, the time t51 and the time t54 are the time when the elevator starts to move and the time when the elevator stops moving, respectively, and the user accelerates downward along with the elevator from the time t51 to the time t52, and is in a weightless state at this time; and the user decelerates downwards with the elevator from time t53 to time t 54. The user is in an overweight state at this time. It will be appreciated that the times from time t51 to time t52 and from time t53 to time t54 are generally long, typically 2-3 seconds, and that the maximum acceleration that a user can achieve in an elevator motion is m (the same direction as gravity) and n (opposite to gravity).
The diagram shown in fig. 5 (B) is a schematic diagram of the acceleration change trend measured by the electronic device when the user walks on the flat floor. In connection with the above description, it is clear that when the user performs this behavior, the fluctuation range of the acceleration of the user (or the electronic device) is large, i.e. the user walks while being overweight and weightless in a transient and large extent, but the user walks substantially without being overweight or weightless for more than one continuous data frame. The maximum acceleration that can be achieved when the user walks on a flat floor is M (same as the direction of gravity) and N (opposite to the direction of gravity), and in general, M is greater than M and N is greater than N.
Fig. 5 (C) is a schematic diagram showing the acceleration change trend measured by the electronic device when the user is stationary on a flat floor. It is easy to understand that when the flat layer is stationary, the user (or the electronic device) is stressed and balanced in the gravity direction, and the object is not in an overweight state or in a weightless state, and then the acceleration of the electronic device is 0.
Therefore, by combining the above description, it can be known that extracting the super-weightlessness duty ratios with different magnitudes can accurately identify whether the person is moving on the elevator or is standing on a flat floor. For example, when the extracted behavioral semantic features reflect that the electronic device is in a super/weightless state for a duration of 2-3 seconds, the electronic device (or predictive model 203) may identify that the user may be riding an elevator for cross-layer movement.
The difference between the small-amplitude overweight (weightlessness) duty ratio and the continuous small-amplitude overweight (weightlessness) duty ratio is: the small-amplitude overweight (weightlessness) duty ratio is considered to be whether the overweight amplitude of the electronic equipment at each moment is in a small-amplitude overweight zone within a period of time or not, the zone where the last moment and the next moment are located is not needed to be considered, and after the number of all the moments in the small-amplitude overweight zone within a short time is obtained, the ratio obtained by dividing the number by the number of all the moments in the period of time is the small-amplitude overweight (weightlessness) duty ratio; the continuous small-amplitude super-weightlessness duty ratio is concerned with the longest time number continuously in the small-amplitude super-weightlessness range in the period, and then dividing the time number by all time numbers in the period, and the obtained ratio is the continuous small-amplitude super-weightlessness duty ratio.
2) Single sample maximum steering and double sample maximum steering
The single-sample maximum steering feature and the double-sample maximum steering feature are features reflecting the direction change amplitude of the electronic equipment (user), and can be mainly used for identifying the ascending and descending stairs behaviors of a human body.
Fig. 6 is a schematic view of a scene of a person going upstairs according to the present application. As shown in fig. 6 (a), the human body is walking upward at this time and reaches the top of a certain stair, and if the destination is not reached at this time, the human body needs to climb down the stair in order to reach a higher floor. In combination with the existing stair structure, most of the stairs have 180-degree rotation angles between two adjacent layers, so that when a human body enters an adjacent layer of stairs from one layer of stairs, the human body needs to turn 180 degrees. As shown in fig. 6 (B), the human body has now turned 180 ° from the direction facing in fig. 6 (a) to enter the next floor. Similarly, when the human body reaches the top of the stairs, if the human body reaches a higher floor, the human body needs to turn 180 degrees again. Further, it should be noted that, assuming that the time when the human body shown in fig. 6 (a) makes a 180-degree turn is t61, and the time when the human body shown in fig. 6 (B) makes a 180-degree turn is t62, the time interval between the time t61 and the time t62 is generally not too long, and is generally 5 to 10 seconds. That is, when a person walks on stairs to perform a cross-layer movement, the time interval between the occurrence of every two adjacent turns to 180 ° is generally small.
Fig. 7 is a schematic view of a scene of a human-body riding elevator provided by the application. In combination with the behavioral habits of the masses, in general, after entering the elevator, a person turns around facing the elevator door and presses the button of the destination floor, after which it is likely to face the elevator door all the way until he leaves the elevator. As shown in fig. 7 (a), the human body just enters the elevator at this time, when the human body is not facing the elevator door, and then the human body turns 180 degrees, as shown in fig. 7 (B), when the human body is facing the elevator door. During the travel of the elevator, the human body will always stand facing the elevator door. Thereafter, as shown in (C) of fig. 7 and (D) of fig. 7, after the human body reaches the target floor and goes out of the elevator, the human body generally goes straight, turns left, or turns right, but regardless of the traveling direction of the human body after going out of the elevator door, the human body generally does not turn 180 degrees any more, even if the human body turns 180 degrees again in the subsequent behavior action, the time interval between the time of this turning and the time of the last turning (i.e., the turning occurs after the user goes into the elevator as shown in (a) of fig. 7) may be long.
Therefore, based on the difference of steering frequency and times among different actions of human bodies, the feature engineering provided by the application can extract a single-sample maximum steering feature and a double-sample maximum steering feature, and the two behavior semantic features can be used as key analysis features when a user walks up/down stairs or goes up/down an elevator. For example, when the extracted behavior semantic features reflect that the electronic device continuously makes a large-scale turning action a plurality of times in a short time, the electronic device can recognize the action pattern of the user to go up/down stairs.
Specifically, the calculation formula of the single sample maximum steering is as follows:
MaxAngle＝Max(Angle)(1-2)
Wherein abs represents a function for obtaining an absolute value, degress represents a function for converting an arc value into a corresponding angle, and gyr x、gyry、gyrz represents an angular velocity of the gyro sensor rotating around the X-axis, an angular velocity of the gyro sensor rotating around the Y-axis, and an angular velocity of the gyro sensor rotating around the Z-axis, respectively; acc x、accy、accz represents the component of the acceleration sensor in the X-axis, the component in the Y-axis, and the component in the Z-axis, respectively.
Furthermore, when the curvature of the stair corner is long or the speed of movement of the user (e.g., an elderly person or a child) is slow, the maximum steering angle that can be achieved in one sample may not be particularly large, in which case the electronic device is very easy to recognize the user's behavior pattern as stationary or flat bed movement. The application can also extract the maximum steering of double samples from the data acquired by the gyroscope sensor, and avoid the electronic equipment from obtaining the wrong recognition result under the conditions that certain stairs with large turning angles and certain users can turn slowly. Specifically, the calculation formula of the double-sample maximum steering is as follows:
MaxDoubleAngle＝max(Angle(t-1)+Anglet) (1-3)
Wherein Angle (t-1)、Anglet represents a single-sample maximum steering value corresponding to two adjacent data frames, and max represents a maximum value of the single-sample maximum steering values corresponding to the two adjacent data frames.
3) Geomagnetic fluctuation mean value
The geomagnetic fluctuation mean value is a characteristic reflecting geomagnetic variation amplitude in an environment where electronic equipment (users) are located, and is mainly used for identifying a behavior pattern of the users riding an elevator or an escalator.
It has been found that geomagnetism generally exhibits a generally continuously ascending or descending curve when a user performs a cross-layer movement by means of an escalator (in the present application, the escalator means an escalator); in addition, when the elevator opens or closes the elevator door, severe geomagnetic fluctuations occur in the elevator.
Fig. 8 shows a schematic diagram of the trend of geomagnetic values measured by the electronic equipment in various motion states.
Fig. 8 (a) shows a schematic diagram of geomagnetic variation trend acquired by the electronic equipment when the user rides on the elevator to perform a cross-layer movement. In fig. 8 (a), time t 81-time t84 is a time period when the user enters the elevator, and time t 1-time t2 is a time period when the user exits the elevator, and in both time periods, the elevator door is opened and then closed. As can be seen from the image shown in fig. 8 (a), in the two periods of time t 81-t 82 and time t 83-t 84, the geomagnetism that is originally stable fluctuates drastically, and in the other periods of time, the geomagnetism values measured by the electronic device show a stable trend.
Fig. 8 (B) shows a schematic diagram of geomagnetic variation trend collected by the electronic device when the user rides on the escalator to perform a cross-layer movement. In fig. 8 (B), time t85 to time t86 are periods when the user approaches the escalator and steps on the escalator, and time t86 to time t87 are periods when the user moves to a higher level along with the escalator. As can be seen from the image shown in (B) of fig. 8, the originally stable geomagnetism fluctuates drastically during the period from time t85 to time t 86; and the value of the geomagnetism appears as a substantially continuously rising curve during the period of time during which the user moves with the escalator, i.e. from time t86 to time t 87.
Fig. 8 (C) shows a schematic diagram of geomagnetic variation trend collected by the electronic apparatus when the user performs a flat bed exercise or goes up and down stairs. In this process, the value of geomagnetism does not fluctuate as a whole.
Therefore, the fluctuation of geomagnetism can reflect the behavior of the human body to a certain extent. For example, when the extracted behavior semantic features reflect that geomagnetic values detected by the electronic device generally show a trend of rising or falling over a period of time, the electronic device may recognize a user's motion pattern as going up/down stairs.
Specifically, the calculation formula of the geomagnetic fluctuation mean value may be as follows:
windowSize＝winSize*100 (1-4)
MagSlice＝Mag[t:t+windowSize] (1-6)
Wherein Winsize represents the window size used for dividing the sample in units of data frames when the feature engineering 202 performs feature extraction, that is, the number of data frames contained in each set of data in the feature extraction process; for example, the specific value of WinSize may be 5, 10 or 15, and specific reference may be made to the following description, which is not repeated here. T represents the total amount of geomagnetic data, and its specific value is equal to (Winsize ×the number of geomagnetic data contained in each data frame); magSlice denotes data in time slices from t to (t+ windowSize) in geomagnetic data. Max and Min represent the maximum value and the minimum value of geomagnetic data in time slices from t to (t+ windowSize) in geomagnetic data, respectively. MAGDIFFAVG is the calculated geomagnetic fluctuation mean value.
4) Projection of acceleration in the direction of gravity
The acceleration of the human body varies in the vertical direction when the cross-layer motion occurs, but the position and posture of the electronic device are uncertain because the human body uses or carries the electronic device. Therefore, the electronic device cannot use the projection in the Z-axis direction in the mobile phone carrier coordinate system when performing feature extraction. However, it is known from the physical knowledge that acceleration is a vector, and acceleration in one direction can be decomposed into a plurality of accelerations in other directions. Thus, in performing feature extraction, the electronic device may calculate a projection of acceleration in the direction of gravity to extract features. Specifically, the electronic device can obtain the gravity acceleration by calculating the average value of the acceleration in one data frame, so that the simple low-pass filtering is performed on the data by calculating the average value of the small window data, and the noise caused by up-and-down swing of the human body during walking can be counteracted. After the electronic equipment acquires the gravitational acceleration, the electronic equipment can acquire the projection of the acceleration in the gravitational direction by utilizing a vector projection mode.
5) WIFI Change ratio
The WIFI change proportion is a characteristic reflecting the change amplitude of the number of WIFI networks in the environment where the electronic equipment is located, and is mainly used for identifying the behavior mode of a user on/off an elevator.
The WIFI change proportion feature reflects the change condition of the number of the WIFI networks detected by the electronic equipment, and the number of the WIFI networks detected by the WIFI sensor in the electronic equipment in the elevator can be increased or reduced instantaneously at the moment when the elevator door is closed or opened, so that the feature of the WIFI change proportion can be well used for identifying whether a user takes the elevator.
In the embodiment of the application, the WIFI change proportion is equal to the ratio of the newly-appearing WIFI network number at the last moment to the total WIFI network number at the last moment in the WIFI data acquired by the WIFI sensor. The number of the newly-appearing WIFI networks at the last moment is obtained by comparing the newly-appearing WIFI networks with the WIFI networks at the first moment. The "first time" and "last time" herein correspond to the first data frame and the last data frame in the sample window, respectively, and the "sample window" refers to a window size used for dividing the first data in units of data frames when the feature engineering 202 performs feature extraction, that is, the number of data frames included in each set of data in the feature extraction process, which is specifically referred to later description, and will not be repeated herein. Specifically, the electronic device may first obtain an intersection of the WIFI hardware address at the last time and the WIFI hardware address at the first time, and then obtain the new WIFI number by using the difference between the WIFI hardware address at the last time and the intersection. Then, the electronic device may divide the number of newly appearing WIFI by the total number of WIFI networks at the last moment, and the obtained ratio is the WIFI change ratio.
An example of this is illustrated in fig. 9. In fig. 9, the address list 901 is the network addresses of all the first data frames collected by the electronic device in a certain sample window, that is, the network addresses mac 1-mac 7 shown in fig. 9; the address list 902 is the mac addresses of all the last data frames collected by the electronic device in the sample window, namely, the network address mac2, the network address mac4, and the network address mac 7-network address mac13 shown in fig. 9; any one of the network addresses in the address list 901 and the address list 902 corresponds to one WIFI network, and the same network address in the two address lists corresponds to the same WIFI network.
As can be seen from the foregoing description, after the electronic device obtains the address list 901 and the address list 902, the electronic device may compare and analyze the network addresses in the two address lists to obtain mac address sets, i.e., the network address mac2, the network address mac4, and the network address mac7, that is, the intersection list 903 in fig. 9, which are both present in the two address lists.
The electronic device then makes a difference with the intersection list using the address list 902 to obtain the difference list 904 in fig. 9. The WIFI network corresponding to the network address in the difference list is the newly-appearing WIFI network. As can be seen from the difference list 904, all newly appeared WIFI networks are WIFI networks corresponding to the network address mac 8-network address mac13, the number is 6, and as can be seen from the address list 902, the number of WIFI networks detected by the electronic device at the last moment is 9. Then, as can be seen from the foregoing description, the WIFI change ratio corresponding to the sample window is: 6/9, i.e. 2/3.
Furthermore, it should be noted that this is because the samples extracted from a single data frame do not reflect the front-rear semantic features well, for example, in elevator driving, there is a constant driving time period, and if the features at this moment are extracted by using a single data frame alone, the features at this moment are found to be not much different from the plane still. Therefore, if the single data frame is simply used to extract the feature at the moment in the feature extraction process, the information captured by the data in the window is too little, and the accuracy of the recognition result output by the prediction model may be lower. Thus, in some embodiments, when the feature engineering 202 performs feature extraction, the electronic device may divide the first data using a sample window with data frames as units, that is, the electronic device may perform feature extraction with several data frames as batches during the feature extraction process.
In addition, for different behavior patterns, the duration corresponding to the feature data with reference meaning corresponding to each behavior pattern is also different. For example, when a user rides an elevator to perform a cross-layer exercise, in order to completely extract characteristic data of the user in a weightless state and an overweight state, the length of the sample window needs to be set longer; however, when the user is in the flat-bed static mode or the flat-bed moving mode, the motion state characteristics of the user can be reflected only by a plurality of data frames, and the length of the sample window can be correspondingly set to be shorter. Thus, in some embodiments, the electronic device may set a plurality of sample windows of different sizes simultaneously to segment the first data.
It will be appreciated that in determining the size of the sample window, it is desirable to have the amount of information that can be captured by the data within the sample window be sufficiently large but the sample window cannot be interspersed with too much data of different behavior. Thus, in some embodiments, during feature extraction, the electronic device may set the sample windows to different sizes (i.e., there are a plurality of sample windows, where the number of data frames included in the sample windows is different), and divide the same piece of the first data using the sample windows with different sizes at the same time to obtain multiple sets of data (in some embodiments, the multiple sets of data may also be referred to as at least two sets of data fragment sets, and any one data fragment in the multiple sets of data may be referred to as a first data fragment). When the characteristics are extracted, the characteristics of the plurality of groups of data are extracted, so that the performance of the prediction model and the accuracy of the identification result can be effectively improved.
In particular, unlike the above-described division of the initial data acquired by each sensor using the sliding window, when the first data is divided, the size of the sample window used is determined by taking the data frame as the minimum unit, and the electronic device can divide the first data by using a plurality of sample windows of different sizes at the same time; in the case of dividing the initial data, the size of the window used is determined in the smallest unit of the number of times, and the coverage between windows may be set to 50%. For example, when the electronic device samples the initial data at a frequency of 100HZ, if the window size determined by the electronic device for the initial data is 2.56s, the number of data in each window is 256 when the electronic device performs data segmentation on the initial data, and the 256 data corresponds to 256 sampling moments, and each data segment obtained by segmenting the initial data is a data frame, and all data frames obtained from the initial data, that is, the first data, may be specifically referred to the related description of fig. 3, which is not repeated herein. When the first data is divided later, the electronic device can determine a sample window by taking the number of data frames as a unit for the first data.
For example, when the first data is cut, the specific size of the sample window may be set to 5, 10, and 15, respectively, that is, the sample window may have a length corresponding to 5 data frames, a length corresponding to 10 data frames, and a length corresponding to 15 data frames. The values of these three window sizes are obtained by observation of a large number of experimental data. As shown in fig. 10, when the number of sample windows is 3, the accuracy of the prediction model is the worst when the sample window sizes are set to 5, 8, and 11, and the accuracy of the output result is only about 0.972; however, the accuracy of the prediction model is best when the sample window size is set to 5, 10, 15, and the accuracy of the output result can reach about 0.98.
Fig. 11 shows a schematic diagram of the electronic device for data segmentation of the first data with sample windows of sizes 5, 10, 15. Here, it is assumed that the sensor in the electronic device continuously collects data for 2 minutes to obtain a piece of initial data, and the electronic device segments the initial data according to a window with a window size of 2.56s and a window coverage rate of 50%, and finally obtains the first data 011 as shown in (a) of fig. 11. As can be seen from simple mathematical calculation, the first data 011 includes about 93 data frames, where each data frame includes data having a sampling duration of 2.56s.
Then, at the time of feature extraction, the electronic device performs data segmentation on the first data with sample windows with sizes of 5, 10 and 15 respectively. As shown in fig. 11 (a), when the first data 011 is divided by a sample window having a size of 5, a first data segment corresponds to the 1 st to 5 th data frames and its corresponding sampling period is 7.68s; when the first data 011 is split by using a sample window with a size of 10, the first data segment corresponds to the 1 st to 10 th data frames, and the sampling duration corresponding to the data segment is 14.08s; when the first data 011 is divided by using a sample window having a size of 15, the first data segment corresponds to 1 data frame to 15 th data frame, and the sampling duration of the data segment is 20.48s.
When the first data is divided by using the sample window with the determined size, a sliding step size of the sample window may be 1. Here, taking the sample window with a size of 5 as an example, as shown in (B) of fig. 11, when the first data 011 is divided by using the sample window with a window size of 5, the data segment 1101 is a first data segment obtained by the electronic device from the first data 011, which corresponds to the 1 st to 5 th data frames in the first data 011, and the data segment 1102 is a second data segment obtained by the electronic device from the first data 011, which corresponds to the 2 nd to 6 th data frames in the first data 011; the data segment 1103 is a third data segment obtained by the electronic device from the first data 011, which corresponds to the 3 rd to 7 th data frames in the first data 011, and so on, until the last data segment (not shown in fig. 11) corresponding to the 89 th to 93 rd data frames in the first data 011 is obtained.
Similarly, the specific process of dividing the first data by using the sample window with other lengths may refer to the description related to (B) in fig. 11, which is not repeated here.
After the feature engineering 202 finishes feature extraction of the target data, the electronic device inputs the time domain features, the statistical features and the behavior semantic features extracted by the feature engineering 202 into the prediction model 203, the prediction model 203 analyzes the features, and finally, the recognition result of the user behavior action is output. In particular, the model used by the predictive model 203 may be the XGBoost model.
In the embodiment of the present application, the prediction model 203 may include two modules, namely, a behavior recognition module and a landmark recognition module shown in fig. 2, and the behavior recognition result output by the prediction model 203 may also include a behavior pattern recognition result of the user and a landmark (an up-down elevator, an initial point and an end point of an up-down elevator) recognition result. Wherein:
The behavior recognition module is configured to output a behavior pattern recognition result of the user, which may be referred to as a first recognition result in some embodiments. The behavior pattern recognition result is the behavior pattern of the user at the moment. Specifically, in the embodiment of the present application, the behavior patterns of the user may be divided into a flat-layer behavior model and a cross-layer behavior model. The flat layer behavior mode can comprise static and walking, and the cross-layer behavior mode can comprise ascending and descending stairs, ascending and descending elevators and ascending and descending elevators. Optionally, when the prediction model 203 predicts the behavior mode of the user, the prediction model may adopt a multi-classification to two-classification mode, firstly determine which of five behavior modes (stationary, walking, going up and down stairs, going up and down elevators, going up and down stairs) the behavior action of the user belongs to, then process the behavior mode according to the requirement, and combine the recognition results, namely, classifying the stationary and walking into a flat-layer behavior mode, and classifying the going up and down stairs, going up and down elevators and going up and down stairs into a cross-layer behavior mode.
In addition, the behavior pattern recognition result output by the prediction model 203 may be represented by a character, and may specifically be represented by a sequence of numbers, where each number represents a duration in which the user acts in the behavior pattern corresponding to the number, and the duration may be a duration corresponding to a data frame. For example, when the behavior pattern recognition result is a multi-classification result (i.e. it is required to specifically recognize which behavior pattern the user is in resting, walking, going up and down stairs), the recognition results output by the electronic device for the five behavior patterns may be respectively represented by 0,1, 2,3,4, and 5; similarly, when the behavior pattern recognition result is a classification result (i.e. only the user needs to be recognized as being in the flat layer motion mode or the cross-layer motion mode), the recognition result output by the electronic device can be used for representing the flat layer motion mode and the cross-layer motion mode by 0 and 1 classification. For example, when the behavior pattern recognition result is a classification result, assuming that the behavior pattern recognition result output by the prediction model 203 is a series of digital sequences such as [0,0,0,0,0,1,1,1,1,1,1,0,0], the digital sequences are a period of time when the user moves in the flat layer motion mode, and the period of time is equal to the period of time corresponding to 5 data frames; then, the user moves another period of time in a cross-layer movement mode, wherein the period of time is equal to the period of time corresponding to 6 data frames; finally, the user moves in the flat layer motion mode for a period of time equal to the period of time corresponding to 2 data frames.
Landmark the identification module is to output landmark point identification, which landmark identification may be referred to as a second identification in some embodiments. The landmark point identification result can be used for reflecting the moment when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result can reflect at which time the user finishes the flat layer motion and starts the cross layer motion.
Optionally, after the prediction model 203 is trained and deployed in the electronic device, when the following electronic device uses the prediction model 203 to act on the behavior of the user, the result output by the prediction model may only include the recognition result of the behavior pattern of the user, and does not need to include the recognition result of the landmark points.
It should be understood that the result of the landmark point identification output by the prediction model 203 is a result obtained by predicting the behavior of the user by the electronic device based on the data collected by the sensor, but there may be a certain error in time between the result and the actual landmark point of the user.
As shown in fig. 12, the sequence 012 is landmark points data representing the real behavior pattern of the user, and the sequence 012' is landmark points data outputted by the prediction model for predicting the behavior pattern of the user, in which the number "0" indicates that the user is in the horizontal movement pattern, and the number "1" indicates that the user is in the cross-layer movement pattern.
As can be seen from the sequence 012, the number of time periods corresponding to the user actually performing the cross-layer motion is 3, and the data periods corresponding to the sequence 012 are an S1-E1 period, an S2-E2 period and an S3-E3 period; similarly, as can be seen from the sequence 012', the number of time periods corresponding to the cross-layer motion in the user behavior identified by the prediction model is also 3, and the data periods corresponding to the sequence 012' are an S1'-E1' period, an S2'-E2' period, and an S3'-E3' period. Wherein, the S1-E1 segment, S2-E2 segment and S3-E3 segment in the sequence 012 correspond to the S1' -E1' segment, S2' -E2' segment and S3' -E3' segment in the sequence 012', respectively. It can be understood that, the data segments corresponding to the sequence 012' are S1' -E1', S2' -E2', and S3' -E3', and the time points corresponding to S1', S2', S3', and E1', E2', E3' are the time points when the behavior pattern of the user detected by the electronic device changes; and the data segments corresponding to the sequence 012 are S1-E1 segments, S2-E2 segments, and S3-E3 segments, and the time points corresponding to S1, S2, S3, and E1, E2, and E3 are the time points at which the behavior pattern of the user actually changes.
As can be seen in conjunction with fig. 12, when the electronic device identifies the behavior of the user, compared with the point landmark of the actual real behavior pattern of the user, the point landmark corresponding to the user identified by the electronic device when performing the cross-layer motion has a certain error. For the S1-E1 segment in the sequence 012, the corresponding data segment in the landmark point data identified by the electronic device is an S1'-E2' segment, and compared with the S1-E1 segment, the start time of the S1'-E1' segment is advanced by 2 sample points (data frames), and the end time is delayed by 2 sample points (data frames); for the S2-E2 segment in the sequence 012, the corresponding data segment in the landmark point data identified by the electronic device is the S2'-E2' segment, and compared with the S2-E2 segment, the start time of the S2'-E2' segment is delayed by 2 sample points (data frames), and the end time is delayed by 3 sample points (data frames); for the S3-E3 segment in the sequence 012, the corresponding data segment in the landmark point data identified by the electronic device is the S3'-E3' segment, and compared with the S3-E3 segment, the starting time of the S3'-E3' segment coincides, but the ending time is delayed by 1 sample point (data frame).
If "+" and "-" are used to represent lead recognition, lag recognition (+representing lead, i.e., prediction before reality, -representing lag, i.e., prediction after reality) of the prediction model, respectively, and numbers are used to represent the number of sample points (data frames) of lead or lag, then the error of the result of the landmark point recognition (i.e., sequence 012 ') output by the prediction model and the landmark point (i.e., sequence 012 ') of the user's actual real behavior pattern in the cross-layer motion pattern can be represented as [ +2, -2,0] (start time) and [ -2, -3, -1] (end time).
Thus, optionally, during the training of the predictive model 203, the dataset used to train the predictive model 203 may also contain user-real landmark point data. In the process of training the prediction model 203, the electronic device can compare the landmark point identification result output by the prediction model 203 with the real landmark point data of the user to determine the delay time (sample number) of the start time and the end time of the predicted landmark point and the real landmark point, the identification accuracy of the predicted landmark point (the ratio of landmark points of less than or equal to 3 samples to total landmark points) and the identification accuracy of the predicted landmark point (the ratio of landmark points of less than or equal to 5 samples to total landmark points), and perform a large amount of simulation tests on error data, so as to continuously optimize the project architecture and model characteristics, and finally obtain a relatively perfect behavior pattern identification algorithm and a relatively perfect landmark identification algorithm.
It should be noted that fig. 2 is merely an exemplary process for identifying a behavior state of a user by an electronic device, and is not meant to limit an embodiment of the present application. In the actual identification process, the electronic device may perform more or fewer steps, for example, after the data acquisition module 201 completes the data acquisition operation, the electronic device may perform data complement and data alignment processing on the initial data, which is not limited in the present application.
In addition, when extracting the behavior semantic features from the initial features, the behavior semantic features extracted by the electronic device (feature engineering module 202) may include one or more of a small-amplitude overweight duty cycle, a large-amplitude overweight duty cycle, a small-amplitude weightlessness duty cycle, a large-amplitude weightlessness duty cycle, a continuous small-amplitude overweight duty cycle, a continuous small-amplitude weightlessness duty cycle, a single-sample maximum steering, a double-sample maximum steering, a geomagnetic fluctuation mean value, projection of acceleration in a gravity direction, and a WIFI change ratio, which is not limited in the present application.
Based on the schematic process diagram of identifying the behavior state of the user by the electronic device provided in fig. 2, the application provides an architecture diagram of a behavior identification algorithm, which can implement the process of executing the identification of the behavior state of the user by the electronic device shown in fig. 2, and refer to fig. 13 specifically.
As shown in fig. 13, the algorithm architecture provided by the present application includes a data layer, a data processing layer, a core algorithm layer, and an output layer, where:
The data layer, which may also be called an input layer, is mainly responsible for achieving the acquisition of data. In particular, in identifying behavioral actions of a user, multiple sensors in an electronic device may collect data in real-time at a frequency, which may be referred to as initial data, as input to a predictive model. Specifically, the sensors related to the data layer may include acceleration sensors, gyroscope sensors, geomagnetic sensors, and WIFI sensors. Among them, the acceleration sensor, the gyro sensor, the geomagnetic sensor may also be referred to as a 3-axis acceleration sensor, a 3-axis gyroscope, and a 3-axis electronic compass (geomagnetic sensor), respectively, and a combination of these three sensors may be referred to as a nine-axis sensor.
The data processing layer may be responsible for feature extraction of the initial data. Specifically, when the feature extraction is performed on the initial data, the extracted data features may include statistical features, time domain features and behavioral semantic features. Optionally, the data processing layer may also be responsible for data alignment and data completion of the initial data.
The core algorithm layer is mainly responsible for training a prediction model or performing feature analysis on the data features extracted by the data processing layer by using the prediction model to obtain a behavior recognition result of the user action behavior. Specifically, the model of the prediction model obtained by training the core algorithm layer and the model for performing feature analysis on the feature data may be XGBoost models.
The output layer is mainly responsible for outputting the recognition result of the prediction model on the action behavior of the user.
Specifically, the behavior recognition result may include a behavior pattern recognition result. The behavior pattern recognition result may be represented by a character, for example, when the behavior pattern recognition result is a multi-classification result (i.e. it is required to specifically recognize which behavior pattern the user is in resting, walking, going up and down stairs, going up and down elevators, going up and down stairs), the recognition results output by the electronic device may be represented by 0,1, 2, 3,4, 5 for the five behavior patterns respectively; similarly, when the behavior pattern recognition result is a classification result (i.e. only the user needs to be recognized as being in the flat layer motion mode or the cross-layer motion mode), the recognition result output by the electronic device can be used for representing the flat layer motion mode and the cross-layer motion mode by 0 and 1 classification.
Further, the behavior recognition result may further include a landmark point recognition result. The landmark point identification result can be used for reflecting the moment when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result can reflect at which time the user finishes the flat layer motion and starts the cross layer motion.
In addition, the specific functions of the data layer, the data processing layer, the core algorithm layer and the output layer, and the content and format of the data related to each layer may refer to the foregoing description of fig. 2-12, which is not repeated herein.
The identification process provided by the embodiment of the application can better reflect the behavior semantic features of the user behavior pattern in the feature extraction stage, further obtain the feature data with different lengths through sliding windows with different scales, fully combine different points of different action patterns in detail and data to analyze the acquired data, effectively improve the accuracy of the user behavior identification result, and achieve the overall accuracy of the identification result of more than 98%.
The electronic device provided by the embodiment of the application is described next.
The electronic device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personaldigital assistant, PDA), or a special camera (e.g., a single-lens reflex camera, a card-type camera), etc., and the application is not limited in any way to the specific type of the electronic device. Specifically, the electronic device may be the electronic device in the foregoing description.
Fig. 14 exemplarily shows a structure of the electronic apparatus.
The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180m, a wifi sensor 180N, and the like.
It should be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural-Network Processor (NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.
The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.
A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.
In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-INTEGRATED CIRCUIT, I2C) interface, an integrated circuit built-in audio (inter-INTEGRATED CIRCUIT SOUND, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.
The I2C interface is a bi-directional synchronous serial bus comprising a serial data line (SERIAL DATA LINE, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc., respectively, through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through an I2C interface, such that the processor 110 communicates with the touch sensor 180K through an I2C bus interface to implement a touch function of the electronic device 100.
The I2S interface may be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface, to implement a function of answering a call through the bluetooth headset.
PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface to implement a function of answering a call through the bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.
The UART interface is a universal serial data bus for asynchronous communications. The bus may be a bi-directional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through a UART interface, to implement a function of playing music through a bluetooth headset.
The MIPI interface may be used to connect the processor 110 to peripheral devices such as a display 194, a camera 193, and the like. The MIPI interfaces include camera serial interfaces (CAMERA SERIAL INTERFACE, CSI), display serial interfaces (DISPLAY SERIAL INTERFACE, DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the photographing functions of electronic device 100. The processor 110 and the display 194 communicate via a DSI interface to implement the display functionality of the electronic device 100.
The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.
The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transfer data between the electronic device 100 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other electronic devices, such as AR devices, etc.
It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present application is only illustrative, and is not meant to limit the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also employ different interfacing manners in the above embodiments, or a combination of multiple interfacing manners.
The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 140 may receive a charging input of a wired charger through the USB interface 130. In some wireless charging embodiments, the charge management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.
The power management module 141 is used for connecting the battery 142, and the charge management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be configured to monitor battery capacity, battery cycle number, battery health (leakage, impedance) and other parameters. In other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charge management module 140 may be disposed in the same device.
The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.
The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.
The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional module, independent of the processor 110.
The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (WIRELESS FIDELITY, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation SATELLITE SYSTEM, GNSS), frequency modulation (frequency modulation, FM), near field communication (NEAR FIELD communication, NFC), infrared (IR), etc., applied to the electronic device 100. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.
In some embodiments, antenna 1 and mobile communication module 150 of electronic device 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, such that electronic device 100 may communicate with a network and other devices through wireless communication techniques. The wireless communication techniques can include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (GENERAL PACKET radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation SATELLITE SYSTEM, GLONASS), a beidou satellite navigation system (beidou navigation SATELLITE SYSTEM, BDS), a quasi zenith satellite system (quasi-zenith SATELLITE SYSTEM, QZSS) and/or a satellite based augmentation system (SATELLITE BASED AUGMENTATION SYSTEMS, SBAS).
The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microLed, micro-oLed, a quantum dot LIGHT EMITTING diode (QLED), or the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.
The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.
The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.
The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.
The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.
Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 100 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.
The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.
The internal memory 121 may be used to store computer executable program code including instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like. The processor 110 performs various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playing, recording, etc.
The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or a portion of the functional modules of the audio module 170 may be disposed in the processor 110.
The speaker 170A, also referred to as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 100 may listen to music, or to hands-free conversations, through the speaker 170A.
A receiver 170B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 100 is answering a telephone call or voice message, voice may be received by placing receiver 170B in close proximity to the human ear.
Microphone 170C, also referred to as a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can sound near the microphone 170C through the mouth, inputting a sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, and may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may also be provided with three, four, or more microphones 170C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.
The earphone interface 170D is used to connect a wired earphone. The headset interface 170D may be a USB interface 130 or a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, a american cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.
The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The pressure sensor 180A is of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a capacitive pressure sensor comprising at least two parallel plates with conductive material. The capacitance between the electrodes changes when a force is applied to the pressure sensor 180A. The electronic device 100 determines the strength of the pressure from the change in capacitance. When a touch operation is applied to the display screen 194, the electronic apparatus 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 may also calculate the location of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations that act on the same touch location, but at different touch operation strengths, may correspond to different operation instructions. For example: and executing an instruction for checking the short message when the touch operation with the touch operation intensity smaller than the first pressure threshold acts on the short message application icon. And executing an instruction for newly creating the short message when the touch operation with the touch operation intensity being greater than or equal to the first pressure threshold acts on the short message application icon.
The gyro sensor 180B may be used to determine a motion gesture of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance to be compensated by the lens module according to the angle, and makes the lens counteract the shake of the electronic device 100 through the reverse motion, so as to realize anti-shake. The gyro sensor 180B may also be used for navigating, somatosensory game scenes.
The air pressure sensor 180C is used to measure air pressure. In some embodiments, electronic device 100 calculates altitude from barometric pressure values measured by barometric pressure sensor 180C, aiding in positioning and navigation.
The geomagnetic sensor 180D is mostly used in compass and map navigation, and can determine the magnetic field strength by measuring resistance change. The geomagnetic sensor adopts an Anisotropic Magnetoresistance (AMR) material to detect the magnetic induction intensity in space, and the alloy material with the crystal structure is sensitive to an external magnetic field, and the change of the intensity of the magnetic field can cause the change of the self resistance value of the AMR. Therefore, when the magnetic field around the electronic device equipped with the geomagnetic sensor changes, the geomagnetic sensor in the electronic device can sensitively detect the change in the magnetic field strength even if the magnitude of the change in the magnetic field is small.
The acceleration sensor 180E is a sensor capable of measuring acceleration, and is generally composed of a mass block, a damper, an elastic element, a sensing element, an adaptive circuit, and the like. The acceleration sensor 180E may obtain an acceleration value by measuring an inertial force applied to the electronic device using newton's second law when the electronic device is in an acceleration motion state. In addition, the acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The electronic equipment gesture recognition method can also be used for recognizing the gesture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.
A distance sensor 180F for measuring a distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, the electronic device 100 may range using the distance sensor 180F to achieve quick focus.
The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 detects infrared reflected light from nearby objects using a photodiode. When sufficient reflected light is detected, it may be determined that there is an object in the vicinity of the electronic device 100. When insufficient reflected light is detected, the electronic device 100 may determine that there is no object in the vicinity of the electronic device 100. The electronic device 100 can detect that the user holds the electronic device 100 close to the ear by using the proximity light sensor 180G, so as to automatically extinguish the screen for the purpose of saving power. The proximity light sensor 180G may also be used in holster mode, pocket mode to automatically unlock and lock the screen.
The ambient light sensor 180L is used to sense ambient light level. The electronic device 100 may adaptively adjust the brightness of the display 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust white balance when taking a photograph. Ambient light sensor 180L may also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent false touches.
The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 may utilize the collected fingerprint feature to unlock the fingerprint, access the application lock, photograph the fingerprint, answer the incoming call, etc.
The temperature sensor 180J is for detecting temperature. In some embodiments, the electronic device 100 performs a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs a reduction in the performance of a processor located in the vicinity of temperature sensor 180J in order to reduce power consumption to implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device 100 heats the battery 142 to avoid the low temperature causing the electronic device 100 to be abnormally shut down. In other embodiments, when the temperature is below a further threshold, the electronic device 100 performs boosting of the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperatures.
The touch sensor 180K, also referred to as a "touch device". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different location than the display 194.
The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, bone conduction sensor 180M may acquire a vibration signal of a human vocal tract vibrating bone pieces. The bone conduction sensor 180M may also contact the pulse of the human body to receive the blood pressure pulsation signal. In some embodiments, bone conduction sensor 180M may also be provided in a headset, in combination with an osteoinductive headset. The audio module 170 may analyze the voice signal based on the vibration signal of the sound portion vibration bone block obtained by the bone conduction sensor 180M, so as to implement a voice function. The application processor may analyze the heart rate information based on the blood pressure beat signal acquired by the bone conduction sensor 180M, so as to implement a heart rate detection function.
The WIFI sensor may be used to search/scan for WIFI network signals that exist around, and may be used to obtain a physical address for each WIFI signal. It should be understood that the term "WIFI sensor" as used in the application is only used to refer to elements for searching/scanning for the presence of WIFI network signals around. The WIFI sensor may be a processing element that is set up separately, or may be implemented in the same chip as other elements (e.g., other sensors in the foregoing description). In addition, the WIFI sensor may be stored in a memory element of the controller in a form of a program code, and a certain processing element of the processor invokes and executes a function of the WIFI sensor.
The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The electronic device 100 may receive key inputs, generating key signal inputs related to user settings and function controls of the electronic device 100.
The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects by touching different areas of the display screen 194. Different application scenarios (such as time reminding, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.
The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.
The SIM card interface 195 is used to connect a SIM card. The SIM card may be inserted into the SIM card interface 195, or removed from the SIM card interface 195 to enable contact and separation with the electronic device 100. The electronic device 100 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 195 may support Nano SIM cards, micro SIM cards, and the like. The same SIM card interface 195 may be used to insert multiple cards simultaneously. The types of the plurality of cards may be the same or different. The SIM card interface 195 may also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as communication and data communication. In some embodiments, the electronic device 100 employs esims, i.e.: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
In some embodiments, the internal memory 121 of the electronic device 100 or the memory in the processor 110 may have disposed therein a predictive model provided by an embodiment of the application, such as predictive model 203 in the foregoing description.
In some embodiments, the acceleration sensor 180E, the gyroscope sensor 180B, the geomagnetic sensor 180D, and the WIFI sensor 180D in the electronic device 100 may collect data in real time, and then the electronic device may perform feature extraction on the data collected by these sensors to obtain time domain feature data, statistical feature data, and behavioral semantic feature data (for details and meanings of the data refer to the foregoing table 1); then, the electronic device may input the time domain feature data, the statistical feature data, and the behavior semantic feature data into the prediction model, and finally output a behavior recognition result for the user behavior action due to a series of complex operations such as feature extraction on the feature data by the prediction model.
In some embodiments, after the initial data acquired by the sensor is acquired, the electronic device may sample a sliding window with a window length of 2.56s and a window coverage rate of 50% to perform data segmentation on the initial data, so as to obtain the first data. Further, in some embodiments, in the process of extracting the features of the data by the electronic device, the electronic device may divide the first data by using sample windows with different sizes at the same time to obtain multiple sets of data with different lengths of the data segments (the multiple sets of data may be referred to as target data); and when the characteristics are extracted, the characteristics of the plurality of groups of data are extracted respectively, so that the performance of the prediction model and the accuracy of the identification result are effectively improved. For example, in determining the size of the sample window for the first data described above, the specific size of the sample window may be set to 5, 10, and 15, respectively.
In some embodiments, the behavior recognition result output by the prediction model may include a behavior pattern recognition result. The behavior pattern recognition result may be represented by a character, for example, when the behavior pattern recognition result is a multi-classification result (i.e. it is required to specifically recognize which behavior pattern the user is in resting, walking, going up and down stairs, going up and down elevators, going up and down stairs), the recognition results output by the electronic device may be represented by 0,1, 2, 3, 4, 5 for the five behavior patterns respectively; similarly, when the behavior pattern recognition result is a classification result (i.e. only the user needs to be recognized as being in the flat layer motion mode or the cross-layer motion mode), the recognition result output by the electronic device can be used for representing the flat layer motion mode and the cross-layer motion mode by 0 and 1 classification. Further, the behavior recognition result may further include a landmark point recognition result. The landmark point identification result can be used for reflecting the moment when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result can reflect at which time the user finishes the flat layer motion and starts the cross layer motion.
Fig. 15 is a flowchart of an identification method according to an embodiment of the present application. According to the method, the behavior semantic features which can better reflect the behavior pattern of the user are extracted from the data acquired by the sensor of the electronic equipment, the behavior of the user is identified by combining the behavior semantic features, and the accuracy of the identification result of the behavior of the user can be effectively improved. As shown in fig. 15, the method provided by the embodiment of the present application may include the following steps:
S101, the electronic equipment performs feature extraction on target data to obtain feature data, wherein the feature data comprises behavior semantic feature data.
The electronic device may be the electronic device 100 described in the foregoing description.
The target data may be obtained based on initial data collected by a sensor in the electronic device, and specific processes related to the foregoing embodiments and related descriptions of subsequent embodiments may be referred to herein, which are not described herein again. In the method, the electronic device may include a plurality of sensors, such as an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, and a WIFI sensor, where the sensors may collect corresponding sensor data at a certain frequency, and these data are the initial data.
Feature extraction refers to the process of converting the target data into feature data capable of representing the behavior of a user. In order to be able to study the behavior patterns of a person, five behaviors of a person, namely resting, walking, going up and down stairs and going up and down stairs, can be accurately identified, in the process of extracting the characteristics of the target data, the electronic equipment combines different points of various behavior patterns of a user, which are shown in detail and data, and extracts the behavior semantic characteristic data from the target data. The behavioral semantic feature data includes first feature data that characterizes the electronic device in a super/weightless state and/or second feature data that characterizes a change in orientation of the electronic device (user).
Further, the behavior semantic features extracted by the electronic device may further include one or more features of a feature reflecting a change condition of geomagnetic intensity in an environment where the electronic device is located, a feature reflecting a change condition of acceleration of the electronic device (user) in a gravity direction, and a feature reflecting a change condition of a WIFI network number in the environment where the electronic device (user) is located.
In addition, besides the behavior semantic features, the electronic device can extract time domain features with physical significance and statistical features with statistical significance based on the target data. That is, the feature data extracted from the target data by the electronic device can be classified into three types, namely, statistical feature data, time domain feature data, and behavior semantic features. The data content and the meaning of each data contained in the three types of feature data can be specifically referred to the related description of table 1, and are not repeated here.
In an alternative embodiment, the electronic device may extract all of the data features or only some of the data features from among the first feature data, the second feature data, the third feature data, the fourth feature data, and the fifth feature data included in the behavior semantic feature, and the present application is not limited thereto. For example, the electronic device may extract only the third feature data without extracting the first feature data and the second feature data, and the behavior recognition result obtained by combining the third feature data by the electronic device may be improved in accuracy to some extent compared with extracting only the statistical feature data and the time domain feature data.
Similarly, in an optional embodiment, the electronic device may extract only one or more of the behavioral semantic features of the 11-dimensional behaviors, such as the first feature data, the second feature data, the third feature data, the fourth feature data, and the fifth feature data, the small-amplitude overweight duty cycle, the large-amplitude overweight duty cycle, the small-amplitude weightlessness duty cycle, the large-amplitude weightlessness duty cycle, the continuous small-amplitude overweight duty cycle, the continuous small-amplitude weightlessness duty cycle, the single-sample maximum steering, the double-sample maximum steering, the geomagnetic fluctuation mean, the projection of acceleration in the gravity direction, and the WIFI change ratio, which may also improve the accuracy of the behavior recognition result to a certain extent.
S102, the electronic equipment performs feature analysis on the feature data to obtain a behavior recognition result.
The behavior recognition result comprises a first recognition result, wherein the first recognition represents the current behavior mode of the user.
Specifically, a prediction model for performing feature analysis on the feature data may be deployed in the electronic device, where the prediction model may be the prediction model 203 in the foregoing description, and in particular, the prediction model may be the XGBoost model. The training data set used for training the prediction model may be sensor data acquired by a plurality of devices via respective sensors (for example, acceleration sensor, gyro sensor, geomagnetic sensor, and WIFI sensor in the foregoing description). The manufacturer of the electronic device may use the data set collected by the plurality of devices as a training set of the predictive model, and train the predictive model by a device having a model training function. And then, uniformly deploying the prediction model into the electronic equipment in the production process of the electronic equipment. Then, when the electronic device deploying the prediction model recognizes the behavior and action of the user (i.e., during the use of the prediction model), the electronic device may use only the sensor data collected by its own sensor (i.e., the initial data) as the input of the prediction model, or use the data obtained by performing a certain preprocessing on the initial data (e.g., the target data) as the input of the prediction model, so as to obtain the recognition result of the behavior and action of the user.
Specifically, the above behavior patterns may be divided into a flat layer behavior model and a cross-layer behavior pattern. The flat layer behavior mode can comprise static and walking, and the cross-layer behavior mode can comprise ascending and descending stairs, ascending and descending elevators and ascending and descending elevators. Optionally, when the prediction model predicts the behavior mode of the user, the prediction model can adopt a multi-classification-to-two-classification mode, firstly judges which of five behavior modes (stationary, walking, going up and down stairs, going up and down elevators and going up and down stairs) the behavior action of the user belongs to, processes the behavior modes according to requirements, combines the recognition results, namely, classifies the stationary and walking into a flat behavior mode, and classifies the going up and down stairs, the going up and down elevators and the going up and down stairs into a cross-layer behavior mode.
In addition, the behavior recognition result output by the prediction model 203 may represent the behavior pattern of the user by a character, which may specifically be represented by a sequence of numbers, where each number represents that the user acts for a duration in the behavior pattern corresponding to the number, and the duration may be a duration corresponding to a data frame. For example, when the behavior pattern recognition result is a multi-classification result (i.e. it is required to specifically recognize which behavior pattern the user is in resting, walking, going up and down stairs), the recognition results output by the electronic device for the five behavior patterns may be respectively represented by 0, 1,2, 3, 4, and 5; similarly, when the behavior pattern recognition result is a classification result (i.e. only the user needs to be recognized as being in the flat layer motion mode or the cross-layer motion mode), the recognition result output by the electronic device can be used for representing the flat layer motion mode and the cross-layer motion mode by 0 and 1 classification.
For example, assuming that after the feature data is subjected to feature analysis by the prediction model, the last output behavior pattern recognition result is a series of digital sequences such as [0,0,0,0,0,1,1,1,1,1,1,0,0], the digital sequences are used for replacing users to move for a period of time in a flat layer motion mode, and the period of time is equal to the period of time corresponding to 5 data frames; then, the user moves another period of time in a cross-layer movement mode, wherein the period of time is equal to the period of time corresponding to 6 data frames; finally, the user moves in the flat layer motion mode for a period of time equal to the period of time corresponding to 2 data frames.
Optionally, in some embodiments, the behavior recognition result may further include a landmark recognition result of the user. The landmark identification characterizes the point in time at which the user switched behavior patterns. In some embodiments, this landmark recognition result may be referred to as a second recognition result. That is, the landmark point identification result may be used to reflect the time when the user behavior is switched in the behavior pattern identification result. For example, if a certain number sequence reflects that the user performs the flat layer motion mode and the cross layer motion mode successively, the landmark point identification result can reflect at which time the user finishes the flat layer motion and starts the cross layer motion. Reference may be made specifically to the foregoing description of fig. 12, and details are not repeated here.
It should be understood that the above-mentioned landmark point identification result is a result obtained by predicting the behavior of the user by the electronic device based on the data collected by the sensor, but there may still be a certain error in time between the result and the actual landmark point of the user. Thus, optionally, during the training process of the prediction model, the dataset for training the prediction model may further include the user's real landmark point data. In the process of training the prediction model, the electronic device can compare the landmark point identification result output by the prediction model with the real landmark point data of the user to determine the delay time (sample number) of the start time and the end time of the predicted landmark point and the real landmark point, the identification accuracy of the predicted landmark point (the ratio of landmark points of less than or equal to 3 samples to total landmark points) and the identification accuracy of the predicted landmark point (the ratio of landmark points of less than or equal to 5 samples to total landmark points), and perform a large amount of simulation tests on error data, so as to continuously optimize the project architecture and model characteristics, and finally obtain a relatively perfect behavior pattern identification algorithm and a relatively perfect landmark identification algorithm.
As can be seen from the foregoing description, in the identification method provided in the application, the initial data collected by the sensor in the electronic device needs to be processed multiple times before the initial data can be finally used for training, and a specific process of the electronic device for obtaining the characteristic data based on the initial data is described next with reference to fig. 16.
Fig. 16 is a flowchart of a data processing method according to the present application. As shown in fig. 16, the method may include:
S201, the electronic equipment collects initial data.
The electronic device may be the electronic device 100 described in the foregoing description.
In the method, the electronic device may include a plurality of sensors, such as an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, and a WIFI sensor, where the sensors may collect corresponding sensor data at a certain frequency, and these data are the initial data.
Specifically, each sensor in the electronic device may collect data at a certain sampling frequency, for example, 100HZ, to obtain the initial data.
S202, the electronic equipment adopts a sliding window with a first length to divide the initial data into data so as to obtain first data.
It should be appreciated that since the action takes place for a certain period of time, the data collected by the sensor at a single moment is not sufficient to characterize the action of the user. Therefore, after the initial data is obtained, the electronic device can select a sliding window with a certain length and a certain window coverage rate to perform data segmentation on the initial data acquired by the sensor, namely, the initial data of a long-time sequence is segmented into smaller data frames (the length of each data frame is the length of the sliding window) to process, so that the data is convenient to perform feature extraction.
It should be appreciated that the length of the data frame, i.e. the length of the sliding window described above, not only directly affects the quality and classification performance of feature extraction, but also determines whether the method is suitable for use in a real-time system. For example, if the sliding window is too small, each data frame obtained by segmentation may be only a part of the motion, and the complete behavior state of the motion cannot be reflected, so that the extracted features cannot effectively represent the motion, and further, the recognition performance is reduced, and the electronic device needs to frequently recognize, so that the calculation amount is increased; if the sliding window is too large, multiple actions may be included in one data frame, so that the system cannot effectively identify the current behavior, the real-time performance of the system may be affected, and the accuracy of the identification result may be affected. Therefore, when the above initial data is subjected to data division, the sliding window length will directly affect the accuracy of the recognition result.
The current research shows that under the condition that the sampling frequency is 100HZ, the window length is 2.56s, which is a comparatively ideal window length, because the data needs to be subjected to Fourier transform when the frequency domain calculation is carried out, the N power of 2 is adopted as a time window, and the integrity of the data participating in the frequency domain calculation is ensured. Thus, alternatively, the first length may be 2.56s. Of course, the first length may be other values, which are not limited by the present application.
In addition, the coverage of the sliding window is also one of important factors affecting the accuracy of the recognition result. The coverage rate of the sliding window is the overlapping rate between two adjacent divided windows in the data dividing process, wherein the coverage rate is 0% which indicates that the adjacent windows are not overlapped, and the coverage rate is 50% which indicates that the current window contains half of the data of the previous window.
In the embodiment of the application, the electronic device can set the window coverage rate of the sliding window to be 50%, and a large amount of experimental data show that the disturbance caused by the transitional behavior can be effectively reduced by setting the window coverage rate to be 50%. Therefore, in the embodiment of the application, the electronic device may use a sliding window with a window length of 2.56s and a window coverage rate of 50% to perform data segmentation on the initial data. For each sensor, the acquired data is divided to obtain a plurality of data segments, wherein one data segment can be called a data frame. It is easy to know that when the electronic device samples at a frequency of 100HZ, and uses a sliding window with a window length of 2.56s and a window coverage rate of 50% to segment the initial data collected by the sensor, each data frame (also referred to as a sample during model training and testing) in the obtained data frames contains 256 times of data. Reference may be made specifically to the foregoing description of fig. 3, and details thereof are omitted herein.
Optionally, before the data splitting of the initial data, the electronic device may further perform a series of preprocessing, such as data alignment, data complementation, denoising, filtering, and so on, on the initial data, and after these preprocessing operations are completed, perform data splitting on the obtained data using the sliding window with the first length.
S203, the electronic equipment adopts at least two sample windows to divide the first data into at least two groups of data fragment sets, and the at least two groups of data fragment sets are used as target data.
As can be seen from the foregoing description, the samples extracted from the single data frame do not reflect the front-rear semantic features well, for example, in the elevator driving, there is a constant-speed driving time period, and if the features at the moment are extracted by using the single data frame alone, the features at the moment are found to be not greatly different from the plane static. That is, if the feature is extracted at this moment by using only a single data frame at the time of feature extraction, information that can be captured by the data in the window is too small, the recognition result output by the prediction model may be less accurate. The electronic device may divide the first data using a sample window in units of data frames, and perform feature extraction in a batch of several data frames in the feature extraction process.
In addition, for different behavior patterns, the duration corresponding to the feature data with reference meaning corresponding to each behavior pattern is also different. For example, when a user rides an elevator to perform a cross-layer exercise, in order to completely extract characteristic data of the user in a weightless state and an overweight state, the length of the sample window needs to be set longer; however, when the user is in the flat-bed static mode or the flat-bed moving mode, the motion state characteristics of the user can be reflected only by a plurality of data frames, and the length of the sample window can be correspondingly set to be shorter. Thus, in some embodiments, the electronic device may set a plurality of sample windows of different sizes simultaneously to segment the first data.
Therefore, in the method, after the first data is obtained, the electronic device may use at least two sample windows to perform data segmentation on the first data, so as to obtain at least two sets of data fragment sets, and use the at least two sets of data fragment sets as target data.
In particular, unlike the above-described division of the initial data acquired by each sensor using the sliding window, when the first data is divided, the size of the sample window used is determined by taking the data frame as the minimum unit, and the electronic device can divide the first data by using a plurality of sample windows of different sizes at the same time; in the case of dividing the initial data, the size of the window used is determined in the smallest unit of the number of times, and the coverage between windows may be set to 50%. For example, when the electronic device samples the initial data at a frequency of 100HZ, if the window size determined by the electronic device for the initial data is 2.56s, the number of data in each window is 256 when the electronic device performs data division on the initial data, and the 256 data corresponds to 256 sampling moments, and each data segment obtained by dividing the initial data is one data frame, and all data frames obtained from the initial data are the first data. When the first data is divided later, the electronic device can determine a sample window by taking the number of data frames as a unit for the first data.
Specifically, the at least two sample windows may be three sample windows, and the at least two sets of data fragment sets are three sets of data fragment sets. Further, the specific sizes of the three sample windows may be set to 5 data frame lengths, 10 data frame lengths and 15 data frame lengths, respectively. The values of these three window sizes are obtained by observation of a large number of experimental data. In the above-described electronic device, when the sampling frequency of each sensor is 100HZ, the data frame length is 2.56s (one data frame length), and the number of sample windows is 3, the accuracy of the prediction model is best when the sample window size is set to 5 data frame lengths, 10 data frame lengths, and 15 data frame lengths, and the accuracy of the output result can reach about 0.98. Reference may be made specifically to the foregoing description of fig. 11, and details are not repeated here.
S204, the electronic equipment performs feature extraction on the target data to obtain feature data.
After the target data is obtained, the electronic device may perform feature extraction on the target data to obtain the feature data.
The feature data extracted from the target data by the electronic device can be classified into three types, namely statistical feature data, time domain feature data and behavior semantic features. The data content and the meaning of each data contained in the three types of feature data can be specifically referred to the related description of table 1, and are not repeated here.
The embodiment of the application also provides electronic equipment, which comprises: one or more processors and memory; wherein a memory is coupled to the one or more processors, the memory for storing computer program code comprising computer instructions that the one or more processors call to cause the electronic device to perform the method shown in the previous embodiments.
As used in the above embodiments, the term "when …" may be interpreted to mean "if …" or "after …" or "in response to determination …" or "in response to detection …" depending on the context. Similarly, the phrase "at the time of determination …" or "if detected (a stated condition or event)" may be interpreted to mean "if determined …" or "in response to determination …" or "at the time of detection (a stated condition or event)" or "in response to detection (a stated condition or event)" depending on the context.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc.
Those of ordinary skill in the art will appreciate that implementing all or part of the above-described method embodiments may be accomplished by a computer program to instruct related hardware, the program may be stored in a computer readable storage medium, and the program may include the above-described method embodiments when executed. And the aforementioned storage medium includes: ROM or random access memory RAM, magnetic or optical disk, etc.
Claims (12)
1. A method of identification, the method comprising:
Extracting characteristics of target data to obtain characteristic data, wherein the characteristic data comprise behavior semantic characteristic data, the target data are obtained based on initial data acquired by a sensor in electronic equipment, and the behavior semantic characteristic data comprise first characteristic data representing that the electronic equipment is in a super/weightless state and/or second characteristic data representing that the direction of the electronic equipment is changed;
And carrying out feature analysis on the feature data to obtain a behavior recognition result, wherein the behavior recognition result comprises a first recognition result, and the first recognition represents the current behavior mode of the user.
2. The method of claim 1, wherein the feature extraction of the target data comprises:
Performing data segmentation on the initial data by adopting a sliding window with a first length to obtain first data, wherein the first data comprises a plurality of data frames with the first length;
data segmentation is carried out on the first data by adopting at least two sample windows to obtain at least two data fragment sets, wherein the length of any one of the at least two sample windows is different from the lengths of other windows in the at least two sample windows;
and taking the at least two data fragment sets as the target data.
3. The method of claim 2, wherein the data partitioning the first data using at least two sample windows to obtain at least two sets of data segments comprises:
And carrying out data segmentation on the first data by adopting three sample windows to obtain three data fragment sets, wherein the window lengths corresponding to the three sample windows are respectively 5 first lengths, 10 first lengths and 15 first lengths.
4. A method according to any of claims 1-3, characterized in that the behavior recognition result further comprises a second recognition result, which second recognition result characterizes the point in time at which the user switches behavior patterns.
5. The method of any of claims 1-4, wherein the behavioral semantic feature data further comprises at least one of third feature data, fourth feature data, and fifth feature data, wherein:
The third characteristic data represents the change condition of geomagnetic intensity in the environment detected by the electronic equipment;
the fourth characteristic data represents the change condition of the acceleration of the electronic equipment in the gravity direction;
and the fifth characteristic data represents the change condition of the number of the WIFI networks detected by the electronic equipment.
6. The method of any one of claims 1 to 5, wherein the first characteristic data includes at least one of a small amplitude super/weightless ratio, a large amplitude super/weightless ratio, a continuous small amplitude super/weightless ratio, the small amplitude super/weightless ratio characterizing a ratio of a number of times in a first data segment that the electronic device is in a super/weightless state and the super/weightless value is less than a second threshold value to a corresponding total number of times in the first data segment; the large-amplitude super/weightlessness duty ratio characterizes the duty ratio of the time number of which the electronic equipment is in a super/weightlessness state and the super/weightlessness value is larger than a third threshold value in the first data segment in the corresponding total time number of the first data segment; the continuous small-amplitude super/weightlessness duty ratio characterizes the duty ratio of the time count corresponding to the maximum duration of the electronic equipment continuously in the super/weightlessness state in the first data segment in the total time count corresponding to the first data segment; the first data segment is any data segment in the target data.
7. The method of any one of claims 1 to 6, wherein the second characteristic data includes at least one of a single sample maximum steer or a double sample maximum steer, the single sample maximum steer characterizing a maximum steer angle of a user in a sampling duration corresponding to one data frame; the double sample maximum steering characterizes the maximum value of the sum of steering angles of any two continuous sampling moments in one data frame by a user.
8. The method of claim 5, wherein the third feature data includes a geomagnetic fluctuation mean, the geomagnetic fluctuation mean represents an average value of data change degrees corresponding to geomagnetic data pieces of a plurality of second lengths, wherein the data change degrees corresponding to any geomagnetic data piece of the plurality of second lengths are determined by a maximum magnetic value and a minimum magnetic value in any geomagnetic data piece, the geomagnetic data pieces of the plurality of second lengths are all contained in a first data piece, and the first data piece is any data piece in the target data.
9. The method of claim 5, wherein the fifth characteristic data comprises a WIFI-change ratio that characterizes a ratio of a first WIFI network number to a second WIFI network number, the second WIFI network number being a number of WIFI network addresses contained in a last data frame of a first data segment, the first WIFI network number being a number of WIFI network addresses that are present in the last data frame of the first data segment but are not present in the first data frame of the first data segment.
10. An electronic device, the electronic device comprising: one or more processors, memory, and a display screen;
The memory is coupled with the one or more processors, the memory for storing computer program code comprising computer instructions that the one or more processors invoke to cause the electronic device to perform the method of any of claims 1-9.
11. A chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause the electronic device to perform the method of any of claims 1-9.
12. A computer readable storage medium comprising instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211459119.6A CN118057369A (en) | 2022-11-21 | 2022-11-21 | Identification method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211459119.6A CN118057369A (en) | 2022-11-21 | 2022-11-21 | Identification method and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118057369A true CN118057369A (en) | 2024-05-21 |
Family
ID=91069278
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211459119.6A Pending CN118057369A (en) | 2022-11-21 | 2022-11-21 | Identification method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118057369A (en) |
-
2022
- 2022-11-21 CN CN202211459119.6A patent/CN118057369A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11871328B2 (en) | Method for identifying specific position on specific route and electronic device | |
| CN115866121B (en) | Application interface interaction method, electronic device and computer readable storage medium | |
| CN110798568B (en) | Display control method of electronic equipment with folding screen and electronic equipment | |
| CN110602309A (en) | Device unlocking method and system and related device | |
| WO2021052139A1 (en) | Gesture input method and electronic device | |
| CN110851067A (en) | Screen display mode switching method and device and electronic equipment | |
| CN112651510B (en) | Model updating method, working node and model updating system | |
| WO2022022319A1 (en) | Image processing method, electronic device, image processing system and chip system | |
| CN113448482B (en) | Sliding response control method and device of touch screen and electronic equipment | |
| CN111147667A (en) | Screen-off control method and electronic equipment | |
| CN110742580A (en) | Sleep state identification method and device | |
| CN114242037A (en) | Virtual character generation method and device | |
| CN111563466A (en) | Face detection method and related product | |
| CN113393856A (en) | Sound pickup method and device and electronic equipment | |
| CN111176465A (en) | Use state identification method and device, storage medium and electronic equipment | |
| CN114822543A (en) | Lip language identification method, sample labeling method, model training method, device, equipment and storage medium | |
| CN111191018B (en) | Response method and device of dialogue system, electronic equipment and intelligent equipment | |
| CN116069139A (en) | Temperature prediction method, device, electronic equipment and medium | |
| CN114064571A (en) | Method, device and terminal for determining file storage position | |
| CN113838478B (en) | Abnormal event detection method and device and electronic equipment | |
| CN113132532B (en) | Ambient light intensity calibration method and device and electronic equipment | |
| CN114356109A (en) | Character input method, electronic device and computer readable storage medium | |
| CN113468929A (en) | Motion state identification method and device, electronic equipment and storage medium | |
| CN111104295A (en) | Method and equipment for testing page loading process | |
| CN114584652B (en) | User graphical interface display method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |